# BettaFish User Guide

BettaFish is a multi-agent **public-opinion (舆情) analysis system**. You ask a question in plain language — "What is the public sentiment around Wuhan University?" — and a team of AI agents fans out across crawled social-media data and live web search, debates findings in a shared "forum," and hands you a polished, chart-rich analysis report in HTML, PDF, and Markdown.

Unlike a dashboard, there are no filters to configure or queries to write. You describe what you want analyzed, and the system does the gathering, cross-checking, and report writing for you. BettaFish combines four specialist AI agents, a dedicated AI web-crawler (MindSpider), and a collection of local sentiment-analysis models into one Python application you run on your own machine.

> [!NOTE]
> BettaFish is research and educational software (GPL-2.0). It analyzes Chinese social platforms (Weibo, Xiaohongshu, Douyin, Bilibili, Kuaishou, Tieba, Zhihu) and global web sources. All agent prompts and on-screen messages are in Chinese; this guide documents the system in English. Exact error messages are quoted verbatim (in Chinese) so you can match them against what you see.

---

## Table of Contents

1. [What BettaFish Does](#1-what-bettafish-does)
2. [Quick Start (Docker)](#2-quick-start-docker)
3. [Installation from Source](#3-installation-from-source)
4. [Configuration: API Keys & Database](#4-configuration-api-keys--database)
5. [Collecting Data with MindSpider](#5-collecting-data-with-mindspider)
6. [Running a Full Analysis (Web App)](#6-running-a-full-analysis-web-app)
7. [Running a Single Agent (Streamlit Apps)](#7-running-a-single-agent-streamlit-apps)
8. [Generating Reports from the Command Line](#8-generating-reports-from-the-command-line)
9. [Using the Sentiment-Analysis Models](#9-using-the-sentiment-analysis-models)
10. [How It Works (Concepts)](#10-how-it-works-concepts)
11. [Common Tasks & Recipes](#11-common-tasks--recipes)
12. [Troubleshooting](#12-troubleshooting)
13. [Frequently Asked Questions](#13-frequently-asked-questions)
14. [Command & Option Reference](#14-command--option-reference)

---

## 1. What BettaFish Does

BettaFish is built from five cooperating pieces. You don't have to understand all of them to get started, but knowing the cast helps you read the output.

| Component | Role | What you see |
|-----------|------|--------------|
| **Query Agent** | Searches domestic + international news and the live web | Real-time information on your topic |
| **Media Agent** | Multimodal search — analyzes video/image content and rich result cards | Coverage from short-video platforms |
| **Insight Agent** | Mines your private/crawled opinion database in depth | Historical patterns, comment-level sentiment |
| **Report Agent** | Assembles everything into a formatted, chart-rich report | The final HTML/PDF/Markdown deliverable |
| **Forum Host** | A moderator LLM that reads all agents and steers the debate | The `[HOST]` summaries in the live forum |

Supporting these are **MindSpider** (the AI crawler that fills your opinion database) and the **SentimentAnalysisModel** collection (local models that score text sentiment without an API call).

A complete run looks like this:

```text
You ask a question
   │
   ▼
Query + Media + Insight agents launch in parallel
   │   each does a broad search, then drills down
   ▼
ForumEngine collects each agent's findings into a shared forum
   │   the Forum Host posts guidance every few rounds; agents adjust
   ▼   (this loop repeats for several rounds)
Report Agent gathers all findings + forum discussion
   │   selects a template, plans chapters, generates content
   ▼
Final report: interactive HTML + PDF + Markdown
```

> [!TIP]
> If you only want to try the system, you do **not** need to run the crawler first. The Query and Media agents work from live web search alone. The Insight agent needs a populated database (see [Section 5](#5-collecting-data-with-mindspider)), but the system runs without it.

---

## 2. Quick Start (Docker)

The fastest path to a running system is Docker Compose. It starts the app and a PostgreSQL database together.

### 2.1 Start the services

From the project root:

```bash
docker compose up -d
```

```text
[+] Running 3/3
 ✔ Network bettafish_default  Created
 ✔ Container bettafish-db-1    Started
 ✔ Container bettafish-app-1   Started
```

> [!NOTE]
> Image pulls can be slow from some regions. The `docker-compose.yml` file ships alternative mirror image addresses as **comments** — open the file and swap them in if the default registry is slow for you.

### 2.2 Database settings (already wired for you)

The Compose stack provisions a PostgreSQL service. Use these values when the app asks for database configuration — they match `docker-compose.yml`:

| Setting | Value | Meaning |
|---------|-------|---------|
| `DB_HOST` | `db` | Database service name inside Compose |
| `DB_PORT` | `5432` | Default PostgreSQL port |
| `DB_USER` | `bettafish` | Database user |
| `DB_PASSWORD` | `bettafish` | Database password |
| `DB_NAME` | `bettafish` | Database name |
| `DB_DIALECT` | `postgresql` | Keep as `postgresql` for the Docker stack |

Leave connection-pool and other advanced settings at their defaults.

### 2.3 Add your LLM keys and open the app

All LLM calls use the OpenAI API standard. After the containers are up, finish the LLM configuration ([Section 4](#4-configuration-api-keys--database)), then open:

```text
http://localhost:5000
```

You now have the full system. Continue to [Section 6](#6-running-a-full-analysis-web-app) to run your first analysis.

---

## 3. Installation from Source

Source installation gives you full control and is the recommended path if you plan to crawl data or customize agents.

### 3.1 System requirements

| Requirement | Detail |
|-------------|--------|
| Operating system | Windows, Linux, or macOS |
| Python | 3.9 or newer (3.11 recommended) |
| Environment manager | Conda (Anaconda/Miniconda) or `uv` |
| Database | PostgreSQL (recommended) or MySQL |
| Memory | 2 GB+ recommended |
| Browser driver | Playwright Chromium (for crawling) |

### 3.2 Clone the repository (with the crawler submodule)

MindSpider's crawler core lives in a **git submodule**. Clone it in one step:

```bash
git clone --recurse-submodules https://github.com/666ghj/BettaFish.git
cd BettaFish
```

If you already cloned without submodules, pull it from the project root:

```bash
git submodule update --init --recursive
```

> [!WARNING]
> If the `MindSpider/DeepSentimentCrawling/MediaCrawler/` directory is empty, crawling will fail with `MediaCrawler子模块未初始化，请先运行: git submodule update --init --recursive`. Always initialize the submodule before crawling.

### 3.3 Create an environment

=== "Conda"

    ```bash
    conda create -n bettafish python=3.11
    conda activate bettafish
    ```

=== "uv"

    ```bash
    uv venv --python 3.11
    # Activate it:
    source .venv/bin/activate      # Linux/macOS
    .venv\Scripts\activate         # Windows
    ```

### 3.4 (Optional) Install PDF-export system dependencies

PDF output uses WeasyPrint, which needs system libraries (GTK/Pango on Windows, etc.). If you want PDF reports, follow the dedicated setup notes in `static/Partial README for PDF Exporting/README-EN.md` **before** installing Python dependencies.

> [!NOTE]
> If you skip this step, WeasyPrint may not install correctly and PDF generation will be unavailable. HTML and Markdown reports still work.

### 3.5 Install Python dependencies

```bash
pip install -r requirements.txt
```

```bash
# Faster alternative with uv:
uv pip install -r requirements.txt
```

> [!TIP]
> Don't want the local sentiment models (they pull in `torch`, `transformers`, etc.)? Comment out the **Machine Learning** section of `requirements.txt` before installing. The system falls back to passing text through un-scored.

### 3.6 Install the Playwright browser driver

```bash
playwright install chromium
```

```text
Downloading Chromium ... done
Chromium installed to /home/you/.cache/ms-playwright/chromium-xxxx
```

You're ready to configure. Continue to [Section 4](#4-configuration-api-keys--database).

---

## 4. Configuration: API Keys & Database

All configuration is unified into a single **`.env` file** in the project root. Every sub-agent inherits from it automatically — there is no per-agent `.env` to maintain.

### 4.1 Create your `.env`

Copy the example file and edit it:

```bash
cp .env.example .env
```

### 4.2 Database configuration

```bash
# Database type: postgresql (recommended) or mysql
DB_DIALECT=postgresql
# Host, e.g. localhost or 127.0.0.1
DB_HOST=your_db_host
# Port: postgresql default 5432, mysql default 3306
DB_PORT=5432
DB_USER=your_db_user
DB_PASSWORD=your_db_password
DB_NAME=your_db_name
# Character set; utf8mb4 keeps emoji working
DB_CHARSET=utf8mb4
```

> [!NOTE]
> You do **not** need to create database tables by hand. The schema is checked and initialized automatically when you run `app.py` (and via MindSpider's `--init-db`).

### 4.3 LLM configuration — seven agent "slots"

BettaFish lets every agent use a **different** LLM, as long as the provider speaks the OpenAI request format (`api_key`, `base_url`, `model`). Each slot has three keys: `*_API_KEY`, `*_BASE_URL`, `*_MODEL_NAME`.

| Slot | Env prefix | Recommended model | Default `BASE_URL` |
|------|-----------|-------------------|--------------------|
| Insight Agent | `INSIGHT_ENGINE_` | `kimi-k2-0711-preview` | `https://api.moonshot.cn/v1` |
| Media Agent | `MEDIA_ENGINE_` | `gemini-2.5-pro` | `https://aihubmix.com/v1` |
| Query Agent | `QUERY_ENGINE_` | `deepseek-chat` | `https://api.deepseek.com` |
| Report Agent | `REPORT_ENGINE_` | `gemini-2.5-pro` | `https://aihubmix.com/v1` |
| MindSpider (topic extraction) | `MINDSPIDER_` | `deepseek-chat` | provider-specific |
| Forum Host (moderator) | `FORUM_HOST_` | `qwen-plus` | provider-specific |
| SQL Keyword Optimizer | `KEYWORD_OPTIMIZER_` | `qwen-plus` | provider-specific |

Example for the Insight Agent:

```bash
INSIGHT_ENGINE_API_KEY=sk-your-moonshot-key
INSIGHT_ENGINE_BASE_URL=https://api.moonshot.cn/v1
INSIGHT_ENGINE_MODEL_NAME=kimi-k2-0711-preview
```

> [!IMPORTANT]
> The maintainers strongly recommend you **start with the recommended models**, get a working run, and only then swap in your own choices. The Report Agent in particular needs a capable model — if your final report has blank charts or malformed paragraphs, switch it to a stronger model first.

> [!TIP]
> You don't need seven separate accounts. Any OpenAI-compatible aggregator (the project sponsors AiHubMix) can serve every slot from one key — just point each `*_BASE_URL` at the aggregator and set each `*_MODEL_NAME` accordingly.

### 4.4 Web-search tools

The Query and Media agents need a web-search provider:

```bash
# Which AI-search backend to use: AnspireAPI (default) or BochaAPI
SEARCH_TOOL_TYPE=AnspireAPI

# Anspire AI Search
ANSPIRE_BASE_URL=https://plugin.anspire.cn/api/ntsearch/search
ANSPIRE_API_KEY=your_anspire_key

# Bocha AI Search (used if SEARCH_TOOL_TYPE=BochaAPI)
BOCHA_BASE_URL=https://api.bocha.cn/v1/ai-search
BOCHA_WEB_SEARCH_API_KEY=your_bocha_key

# Optional: Tavily web search
TAVILY_API_KEY=your_tavily_key
```

### 4.5 Web server (optional)

```bash
HOST=0.0.0.0      # Bind address
PORT=5000         # Web UI port
```

> [!TIP]
> You can also edit all of these later from the web UI's **Configuration** panel — it writes your changes straight back to `.env` and reloads them live, no restart needed.

---

## 5. Collecting Data with MindSpider

The **Insight Agent** analyzes a database of crawled social-media opinion. **MindSpider** is the AI crawler that fills that database. This step is optional for first-time exploration but essential for deep private-data analysis.

MindSpider works in two stages:

1. **Broad Topic Extraction** — collects hot news from 13 platforms, then uses an LLM to extract today's trending topics and keywords.
2. **Deep Sentiment Crawling** — takes those keywords and crawls 7 social platforms for posts and comments.

### 5.1 Move into the crawler directory

```bash
cd MindSpider
```

### 5.2 Check status / initialize

```bash
python main.py --status
```

```text
✓ Configuration valid
✓ Database connected
✓ MediaCrawler submodule present
```

The database initializes automatically; `--init-db` forces it explicitly.

### 5.3 Extract today's hot topics

```bash
python main.py --broad-topic
```

This collects hot news (Weibo hot search, Zhihu, Bilibili, Toutiao, and more), runs LLM topic extraction, and saves topics + keywords to the `daily_topics` table.

### 5.4 Crawl platforms (first run = log in)

> [!IMPORTANT]
> The first time you crawl **each** platform, a QR code appears. Scan it with that platform's mobile app to log in. Your session is saved for subsequent runs.

```bash
# Crawl one platform in test mode (small data) to log in:
python main.py --deep-sentiment --platforms xhs --test
```

Platform codes:

| Code | Platform | Code | Platform |
|------|----------|------|----------|
| `xhs` | Xiaohongshu | `wb` | Weibo |
| `dy` | Douyin | `tieba` | Tieba |
| `ks` | Kuaishou | `zhihu` | Zhihu |
| `bili` | Bilibili | | |

### 5.5 Run the complete pipeline

```bash
# Topic extraction + crawl, small test batch:
python main.py --complete --test
```

```bash
# Full run for a specific date, specific platforms:
python main.py --complete --date 2024-01-20 --platforms xhs dy wb
```

> [!TIP]
> Always start with `--test` (a few keywords, a few posts per keyword) to confirm login and connectivity before launching a full crawl. Tune volume with `--max-keywords` and `--max-notes`.

> [!CAUTION]
> Respect each platform's `robots.txt` and terms of service. BettaFish is for learning and research only; large-scale scraping or data abuse is prohibited and is your responsibility.

See [Section 14](#14-command--option-reference) for the full MindSpider flag list.

---

## 6. Running a Full Analysis (Web App)

The web app is the headline experience: you watch the agents collaborate live and receive a complete report.

### 6.1 Launch

From the project root, with your environment active:

```bash
python app.py
```

```text
INFO  ReportEngine接口已注册
INFO  数据库初始化成功
INFO  query_engine 启动检查: 启动成功
INFO  media_engine 启动检查: 启动成功
INFO  insight_engine 启动检查: 启动成功
INFO  ForumEngine 启动完成
 * Running on http://0.0.0.0:5000
```

Open **http://localhost:5000**.

> [!NOTE]
> `app.py` launches three Streamlit sub-apps (ports 8501–8503), the ForumEngine monitor, and the ReportEngine — all from this one command. You don't start them separately.

### 6.2 Ask your question

In the web interface, enter your analysis request in natural language, for example:

```text
分析武汉大学近期的网络舆情与品牌声誉
(Analyze Wuhan University's recent online public opinion and brand reputation)
```

Press start. The three agents begin working in parallel.

### 6.3 Watch the forum

As agents produce findings, the **forum view** fills with tagged entries:

```text
[INSIGHT] 从历史评论数据看，关注度在 8 月下旬明显上升……
[MEDIA]   短视频平台上相关话题的传播以……为主……
[QUERY]   最新新闻显示……
[HOST]    一、事件梳理与时间线分析……（综合三方观点，指出分歧与趋势）
```

The `[HOST]` entries are the **Forum Host** — a moderator LLM that appears every few rounds to reconcile the agents' findings, flag contradictions, and suggest what to dig into next. The agents read this guidance and adjust their next search.

### 6.4 Receive the report

When the discussion concludes, the **Report Agent** assembles everything into a final report. You can view the interactive HTML in the browser and download PDF and Markdown copies. Generated reports are saved under `final_reports/`.

> [!TIP]
> A real system-generated example ships with the project: open `final_reports/final_report__20250827_131630.html` to see the depth, charts, and structure of a finished report before running your own.

> [!NOTE]
> After you stop a run, a Streamlit sub-app may keep holding its port. If a restart complains the port is busy, find and kill the lingering process. Data crawling is a **separate** operation (Section 5) — it is not triggered from the web app.

---

## 7. Running a Single Agent (Streamlit Apps)

If you want just one perspective — say, only the live web search, or only your private-database analysis — run that agent's standalone Streamlit app. These are also the best way to learn what each agent does.

```bash
# Query Agent (web/news search)
streamlit run SingleEngineApp/query_engine_streamlit_app.py --server.port 8503

# Media Agent (multimodal search)
streamlit run SingleEngineApp/media_engine_streamlit_app.py --server.port 8502

# Insight Agent (private-database mining)
streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501
```

Each opens a simple chat-style page: enter a topic, and the agent runs its full research loop (structure → search → summarize → reflect → format) and writes a Markdown report. Outputs land in the matching `*_streamlit_reports/` directory:

- `query_engine_streamlit_reports/`
- `media_engine_streamlit_reports/`
- `insight_engine_streamlit_reports/`

> [!TIP]
> Single-agent runs are independent of the web app and of each other. They're ideal for quick checks and for producing the per-engine reports that the command-line report tool (Section 8) stitches together.

---

## 8. Generating Reports from the Command Line

`report_engine_only.py` builds a consolidated report **without** the web interface. It skips the three analysis agents entirely, loads their most recent Markdown reports from the `*_streamlit_reports/` directories, and runs only the Report Agent. Use it to quickly re-try a report you weren't happy with, or to debug report generation.

```bash
# Basic — topic auto-extracted from the latest report filenames
python report_engine_only.py
```

The tool shows you which files it picked (with paths and modification times) and waits for confirmation (`y` to proceed, `n` to abort).

```bash
# Specify the report topic explicitly
python report_engine_only.py --query "Civil Engineering Industry Analysis"

# Skip PDF (e.g. if WeasyPrint isn't installed)
python report_engine_only.py --skip-pdf

# Skip the Markdown copy (produced by default after the PDF)
python report_engine_only.py --skip-markdown

# Verbose logging
python report_engine_only.py --verbose
```

Outputs:

| File type | Saved to |
|-----------|----------|
| HTML | `final_reports/` |
| PDF | `final_reports/pdf/` |
| Markdown | `final_reports/md/` |

Naming: `final_report_{topic}_{timestamp}.html/pdf/md`.

> [!NOTE]
> At least one of the three `*_streamlit_reports/` directories must contain a `.md` report, or there is nothing to consolidate. Run a single-agent app (Section 7) or a full web run first.

### Re-rendering the latest output

If you only need to re-render existing chapter data into a different format, three helper scripts skip generation entirely:

```bash
python regenerate_latest_html.py   # re-stitch latest chapters → HTML
python regenerate_latest_md.py     # re-stitch latest chapters → Markdown
python regenerate_latest_pdf.py    # re-export the newest report IR → PDF (vector charts)
```

---

## 9. Using the Sentiment-Analysis Models

BettaFish bundles a collection of standalone sentiment-analysis models under `SentimentAnalysisModel/`. The Insight Agent uses the multilingual model automatically, but you can also run any of them directly from the command line on your own text. Each model folder has its own `predict.py` (and `train.py`).

### 9.1 Multilingual sentiment (used by the Insight Agent)

```bash
cd SentimentAnalysisModel/WeiboMultilingualSentiment
python predict.py --text "This product is amazing!" --lang "en"
```

### 9.2 Traditional machine-learning models

A predict tool wraps Naive Bayes, SVM, LSTM, BERT, and XGBoost classifiers:

```bash
cd SentimentAnalysisModel/WeiboSentiment_MachineLearning
python predict.py --model_type svm --text "服务态度需要改进"
```

```text
预测结果: 负面情感 (置信度: 0.91)
```

```bash
# Interactive mode (default) — type text, get sentiment, repeat
python predict.py --interactive

# Ensemble of all trained models
python predict.py --ensemble --text "今天心情很好"
```

### 9.3 Fine-tuned BERT / GPT-2 (LoRA & Adapter)

```bash
# BERT Chinese LoRA
cd SentimentAnalysisModel/WeiboSentiment_Finetuned/BertChinese-Lora
python predict.py

# GPT-2 LoRA
cd SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora
python predict.py
```

```text
请输入微博内容: 这部电影真是太好看了，我非常喜欢！
预测结果: 正面情感 (置信度: 0.9876)
```

### 9.4 Small-parameter Qwen3

```bash
cd SentimentAnalysisModel/WeiboSentiment_SmallQwen
python predict_universal.py --text "这次活动非常成功"
```

> [!NOTE]
> The first run of a fine-tuned model downloads its base weights (e.g. a ~400 MB BERT model) into a local `model/` folder; later runs load from disk. A trained model file must exist before you can predict — otherwise you'll see `模型 ... 尚未训练，请先调用train方法`. See [Section 14](#14-command--option-reference) for every flag.

---

## 10. How It Works (Concepts)

You don't need this section to use BettaFish, but it explains the output you see.

### The deep-search agent loop

The Query, Media, and Insight agents share the same internal "deep search" loop. Given a topic, an agent:

1. **Plans a structure** — breaks the topic into several paragraphs/sections to research.
2. For each section, **searches** (web search, or database query for the Insight agent), then **summarizes** what it found.
3. **Reflects** — looks at its draft, decides what's missing, searches again, and revises. This repeats a few times per section (default 3 reflection rounds).
4. **Formats** the finished sections into a Markdown report.

The agents differ only in *what they search*: the Query agent searches news/web, the Media agent searches multimodal content, and the Insight agent queries your crawled database (with keyword optimization, result clustering, and automatic sentiment scoring along the way).

### The forum: collaboration by shared notebook

Each agent writes its progress to its own log file. The **ForumEngine** watches those logs, lifts out each agent's latest section summaries, and republishes them into one shared `forum.log` tagged `[INSIGHT]`, `[MEDIA]`, `[QUERY]`. After every five agent contributions, the **Forum Host** (a separate moderator LLM) reads the forum and posts a synthesis: it builds a timeline, compares the agents' views, flags factual conflicts, predicts trends, and proposes questions to pursue. The agents read this `[HOST]` guidance and steer their next round. This "debate moderator" design is what lets multiple models cross-check each other instead of each working in isolation.

### The report: from findings to a finished document

The **Report Agent** turns the raw findings + forum discussion into a publication-quality report:

1. **Template selection** — picks the best-fit template from six built-in report types (brand reputation, market competition, routine monitoring, policy/industry, social hot-event, crisis PR).
2. **Layout & word budget** — designs the title, table of contents, theme, and per-chapter length plan.
3. **Chapter generation** — writes each chapter as structured content blocks (validated against an internal schema).
4. **Rendering** — stitches chapters into an intermediate representation, then renders interactive HTML (with Chart.js charts and MathJax formulas inlined for offline use), plus PDF and Markdown.

---

## 11. Common Tasks & Recipes

**Try the system with zero crawling.** Configure LLM keys + a web-search key only (skip the database/Insight agent). Run a single-agent app: `streamlit run SingleEngineApp/query_engine_streamlit_app.py --server.port 8503`. Ask a question — you get a web-sourced report in minutes.

**Build a private opinion database for one topic area.** Run `python MindSpider/main.py --broad-topic` to get today's keywords, then `python MindSpider/main.py --deep-sentiment --platforms wb xhs --test` (logging in to each platform once). Repeat daily; the Insight agent will have richer history to mine.

**Re-generate a report you didn't like.** Don't re-run the agents. Use `python report_engine_only.py` — it reuses the latest per-agent reports and only re-runs report assembly. Add `--verbose` if it's failing.

**Use one provider for everything.** Point all seven `*_BASE_URL` values at a single OpenAI-compatible aggregator and set each `*_MODEL_NAME`. One key, one bill.

**Repurpose BettaFish for a non-opinion domain.** Because every tool is OpenAI-compatible and prompt-driven, you can adapt the agents to, say, financial-market analysis by editing the agent prompts (`*/prompts/prompts.py`) and the Insight agent's database tools — no orchestration changes required.

**Add a custom report template.** Drop a `.md` (or `.txt`) template into `ReportEngine/report_template/`, or upload one in the web interface. The Report Agent will consider it automatically when selecting a template.

---

## 12. Troubleshooting

The format below is: **error → cause → solution**. Error strings are quoted exactly as they appear (in Chinese).

### Error: `未找到论坛主持人API密钥，请在环境变量文件中设置FORUM_HOST_API_KEY`

- **Cause:** The Forum Host LLM has no API key.
- **Solution:** Set `FORUM_HOST_API_KEY` (and `FORUM_HOST_BASE_URL`, `FORUM_HOST_MODEL_NAME`) in `.env`. Recommended model: `qwen-plus`.

### Error: `Insight Engine INSIGHT_ENGINE_API_KEY is required.` (and the Media/Query/Report equivalents)

- **Cause:** That agent's LLM key is missing.
- **Solution:** Fill in the matching `*_API_KEY` and `*_MODEL_NAME` in `.env`. Each of the four analysis agents needs its own key (or a shared aggregator key).

### Error: `研究过程中发生错误: Error code: 401 - Api key is invalid`

- **Cause:** The API key for that agent is wrong, expired, or for the wrong provider/base URL. (Reported as issues #628, #638, #655.)
- **Solution:** Re-check the key and confirm `*_BASE_URL` matches the provider that issued it. A `403` usually means no access/quota for that model; a `400` means a bad request (often a model name the provider doesn't recognize); a `500` is a provider-side error — retry or switch models.

### Error: `MediaCrawler子模块未初始化，请先运行: git submodule update --init --recursive`

- **Cause:** The crawler submodule wasn't fetched.
- **Solution:** From the project root, run `git submodule update --init --recursive`, then retry the crawl.

### Error: `不支持的平台: {platform}`

- **Cause:** A platform code passed to `--platforms` isn't recognized.
- **Solution:** Use only `xhs`, `dy`, `ks`, `bili`, `wb`, `tieba`, `zhihu`.

### Crawler login fails / QR code doesn't appear

- **Cause:** Headless browser mode, or a platform requiring manual verification.
- **Solution:** Edit `MindSpider/DeepSentimentCrawling/MediaCrawler/config/base_config.py` and set `HEADLESS = False` so you can see and interact with the browser. Re-run the platform's `--test` crawl. To start a login over, delete `MindSpider/DeepSentimentCrawling/MediaCrawler/browser_data/`.

### Error: `WeasyPrint未安装，请运行: pip install weasyprint` or `未找到字体文件，请检查 ... 目录`

- **Cause:** PDF system dependencies or fonts are missing.
- **Solution:** Follow `static/Partial README for PDF Exporting/README-EN.md` to install the system libraries, then reinstall `weasyprint`. Or simply skip PDF with `--skip-pdf` — HTML and Markdown still generate.

### The final report keeps regenerating / never finishes

- **Cause:** The Report Agent's model hit its max-token limit while generating a chapter, causing repeated retries (issue #666). You may also see `章节JSON ... 在 ... 次尝试后仍无法生成`.
- **Solution:** Switch the **Report Agent** to a stronger model with a larger context window. The maintainers explicitly recommend a capable model for `REPORT_ENGINE_*`.

### Crawled data is empty

- **Cause:** Not logged in, no keywords yet, or platform returned nothing.
- **Solution:** Confirm the platform login succeeded, run `--broad-topic` first so keywords exist, and validate with `--test`.

### Database connection / initialization fails

- **Cause:** Wrong DB settings or unreachable database (issues #641, #642).
- **Solution:** Run `python MindSpider/main.py --status` to check connectivity, and verify the `DB_*` values in `.env` (host, port, user, password, dialect).

### Agents don't respond after sending a message

- **Cause:** Usually a misconfigured agent key or base URL, so the agent silently fails to start (issue #672).
- **Solution:** Confirm all four analysis-agent slots are configured and valid. Check the per-engine logs in `logs/`.

---

## 13. Frequently Asked Questions

**Q: Do I have to run the crawler before I can use BettaFish?**
No. The Query and Media agents work from live web search. Only the Insight agent needs a crawled database. You can run a useful analysis with web search alone.

**Q: Which LLM should I use?**
Start with the recommended models per slot (Section 4.3): Kimi for Insight, Gemini 2.5 Pro for Media/Report, DeepSeek for Query/MindSpider, Qwen-Plus for the Forum Host and Keyword Optimizer. Get a working run first, then experiment.

**Q: Can every agent share one API key?**
Yes — point all `*_BASE_URL` values at one OpenAI-compatible aggregator and set the `*_MODEL_NAME` for each. The recommended sponsor aggregator covers all the suggested models.

**Q: What platforms can it crawl?**
Seven for deep crawling: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, Zhihu. Topic extraction pulls hot news from around 13 sources.

**Q: Do I need a GPU?**
No. The local sentiment models default to CPU. A GPU speeds up training/inference of the fine-tuned models but isn't required. You can also skip the local models entirely (Section 3.5).

**Q: Where do my reports go?**
Web-app and `report_engine_only.py` reports go to `final_reports/` (with `pdf/` and `md/` subfolders). Single-agent reports go to the matching `*_streamlit_reports/` directory.

**Q: Can I add my own report template?**
Yes. Put a `.md`/`.txt` template in `ReportEngine/report_template/` or upload one in the web UI; the Report Agent selects the best fit automatically.

**Q: Why are the agents and reports in Chinese?**
BettaFish targets Chinese social platforms, so prompts and output are Chinese by default. You can analyze and report in other languages by editing the agent prompts and using multilingual models.

**Q: PDF generation fails — is my run ruined?**
No. PDF is optional. Use `--skip-pdf`, or read the HTML/Markdown output. Set up WeasyPrint later if you want PDFs.

**Q: Can I use this commercially?**
No. BettaFish is GPL-2.0 and explicitly restricted to educational, academic, and research use. Commercial use and large-scale scraping are prohibited.

---

## 14. Command & Option Reference

### 14.1 Application entry points

| Command | Purpose |
|---------|---------|
| `python app.py` | Launch the full system (web UI on port 5000 + all sub-apps) |
| `streamlit run SingleEngineApp/query_engine_streamlit_app.py --server.port 8503` | Query Agent only |
| `streamlit run SingleEngineApp/media_engine_streamlit_app.py --server.port 8502` | Media Agent only |
| `streamlit run SingleEngineApp/insight_engine_streamlit_app.py --server.port 8501` | Insight Agent only |
| `python report_engine_only.py` | Build a report from the latest agent outputs |
| `python regenerate_latest_html.py` / `_md.py` / `_pdf.py` | Re-render the latest report |
| `docker compose up -d` | Run app + database via Docker |

### 14.2 `report_engine_only.py`

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--query` | str | (auto from filename) | Report topic |
| `--skip-pdf` | flag | off | Skip PDF generation |
| `--skip-markdown` | flag | off | Skip the Markdown copy |
| `--verbose` | flag | off | Verbose logging |

### 14.3 MindSpider — `python MindSpider/main.py`

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--setup` | flag | — | Initialize project (deprecated; init is automatic) |
| `--status` | flag | — | Show project status / check connectivity |
| `--init-db` | flag | — | Initialize the database |
| `--broad-topic` | flag | — | Run topic extraction only |
| `--deep-sentiment` | flag | — | Run the crawler only |
| `--complete` | flag | — | Run the full workflow (topic + crawl) |
| `--date` | str | today | Target date `YYYY-MM-DD` |
| `--platforms` | str… | all | Platforms to crawl (`xhs dy ks bili wb tieba zhihu`) |
| `--keywords-count` | int | 100 | Keywords to extract during topic extraction |
| `--max-keywords` | int | 50 | Max keywords per platform |
| `--max-notes` | int | 50 | Max items crawled per keyword |
| `--test` | flag | — | Test mode (small data volume) |

### 14.4 Topic-extraction sub-module — `MindSpider/BroadTopicExtraction/main.py`

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--sources` | str | all | Specific news-source platforms |
| `--keywords` | int | 100 | Max keyword count |
| `--list-sources` | flag | — | Show supported news sources |
| `--quiet` | flag | — | Simplified output |

### 14.5 Sentiment models — `SentimentAnalysisModel/.../predict.py`

Traditional ML (`WeiboSentiment_MachineLearning/predict.py`):

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--model_type` | str | — | Single model: `bayes`, `svm`, `lstm`, `bert`, `xgboost` |
| `--text` | str | — | Predict one text string directly |
| `--interactive` | flag | on | Interactive prediction loop (default) |
| `--ensemble` | flag | off | Combine all trained models |
| `--model_dir` | str | `./model` | Model directory |

Small Qwen (`WeiboSentiment_SmallQwen/predict_universal.py`) adds `--model_size`, `--load_all`, `--ensemble`. BERT topic detection (`BertTopicDetection_Finetuned/predict.py`) adds `--pretrained_name`, `--gpu`, `--max_length`. Training scripts (`*_train.py`) accept `--train_path`, `--test_path`, `--model_path`, `--epochs`, `--batch_size`, `--learning_rate`, `--eval_only`, and model-specific hyperparameters.

---

_Generated June 2026 from 666ghj/BettaFish at 40327d7. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._