# funNLP — User Guide

## 1. Quick Start

**funNLP** is a curated collection of Chinese Natural Language Processing (NLP) datasets, dictionaries, and corpora. It provides ready-to-use lexical resources for text segmentation, named entity recognition, sentiment analysis, and other NLP tasks — organized by domain and immediately usable in any Chinese NLP pipeline.

**Key resources included:**

- 24+ domain-specific Chinese dictionaries (legal, medical, financial, IT, food, animals, and more)
- Chinese/English/Japanese name corpora with 120W+ entries and gender annotations
- Multiple stop word lists from leading Chinese universities
- Synonym, antonym, and negation word dictionaries
- Classical Chinese poetry organized by 30 themes
- Chinese abbreviation and idiom databases
- Company and organization name corpora (480W+ entries)
- Chinese rumor detection dataset
- Traditional/Simplified Chinese conversion tables
- Character decomposition (拆字) dictionaries
- Chinese word segmentation dictionaries (300K+ entries across multiple formats)

**Prerequisites:** Git (to clone the repository)

```bash
git clone https://github.com/fighting41love/funNLP.git
cd funNLP/data
```

Browse the `data/` directory — each subdirectory contains a themed dictionary or corpus in plain text format, ready for direct use.

> [!TIP]
> Most dictionary files are plain UTF-8 text with one entry per line. You can load them directly into any NLP framework (jieba, HanLP, pkuseg, etc.) as custom dictionaries.

**What to try next:**

- Browse the dataset inventory below to find resources for your task
- See "Loading Dictionaries" in Core Usage for integration examples
- Check the file format reference for each dataset type

## 2. Installation

### System Requirements

| Requirement | Detail |
|---|---|
| Git | Any recent version |
| Disk space | ~500 MB (includes compressed archives) |
| Text encoding | UTF-8 reader required for all text files |
| Archive tools | Needed for `.rar` files in company name corpus |

### Clone the Repository

```bash
git clone https://github.com/fighting41love/funNLP.git
```

```text
Cloning into 'funNLP'...
remote: Enumerating objects: ...
Resolving deltas: 100% ... done.
```

> [!IMPORTANT]
> Some files contain Chinese characters in their filenames. Ensure your filesystem and terminal support UTF-8 encoding. On Windows, use Git Bash or WSL to avoid path encoding issues.

### Verify Installation

```bash
ls data/
```

You should see 24+ subdirectories, each containing domain-specific dictionary files:

```text
IT词库          NLP_BOOK        中文分词词库整理  中文缩写库
中文谣言数据    中英日文名字库  停用词            公司名字词库
动物词库        医学词库        历史名人词库      古诗词库
同义词库、反义词库、否定词库    地名词库          成语词库
拆字词库        汽车品牌、零件词库  法律词库      繁简体转换词库
职业词库        诗词短句词库    财经词库          食物词库
paper
```

### Download Specific Datasets Only

If you only need certain dictionaries, use a sparse checkout:

```bash
git clone --no-checkout https://github.com/fighting41love/funNLP.git
cd funNLP
git sparse-checkout init --cone
git sparse-checkout set data/停用词 data/成语词库
git checkout
```

> [!NOTE]
> The `data/公司名字词库` directory contains `.rar` archives. You need `unrar` or a compatible tool to extract them.

## 3. Core Usage

### Dataset Inventory

| Dictionary | Directory | Format | Scale | Description |
|---|---|---|---|---|
| IT Terms | `data/IT词库/` | TXT | Tsinghua THUOCL | Information technology vocabulary |
| Medical Terms | `data/医学词库/` | TXT | Tsinghua THUOCL | Medical terminology |
| Legal Terms | `data/法律词库/` | TXT | Tsinghua THUOCL | Legal vocabulary |
| Financial Terms | `data/财经词库/` | TXT | Tsinghua THUOCL | Finance and economics terms |
| Food Terms | `data/食物词库/` | TXT | Tsinghua THUOCL | Food-related vocabulary |
| Animal Terms | `data/动物词库/` | TXT | Tsinghua THUOCL | Animal names |
| Place Names | `data/地名词库/` | TXT | Tsinghua THUOCL + Qingdao roads | Geographic names |
| Historical Figures | `data/历史名人词库/` | TXT | Tsinghua THUOCL | Historical person names |
| Poetry Phrases | `data/诗词短句词库/` | TXT | Tsinghua THUOCL | Classical poetry phrases |
| Idioms | `data/成语词库/` | TXT | 5W entries + THUOCL | Chinese four-character idioms |
| Car Brands & Parts | `data/汽车品牌、零件词库/` | TXT | THUOCL + custom | Automotive vocabulary |
| Classical Poetry | `data/古诗词库/` | TXT | 30 themed files | Tang/Song poetry by theme |
| Stop Words | `data/停用词/` | TXT | 4 lists | University-curated stop word lists |
| Synonyms/Antonyms/Negation | `data/同义词库、反义词库、否定词库/` | TXT | 3 files | Lexical relation dictionaries |
| Chinese Names | `data/中英日文名字库/Chinese_Names_Corpus/` | TXT/XLSX | 120W entries | Names with gender, family names, relationships |
| English Names | `data/中英日文名字库/English_Names_Corpus/` | TXT | 48W entries | English names with Chinese translations |
| Japanese Names | `data/中英日文名字库/Japanese_Names_Corpus/` | TXT/XLSX | 18W entries | Japanese names |
| Company Names | `data/公司名字词库/` | RAR/TXT | 480W+ entries | Company and organization names |
| Chinese Abbreviations | `data/中文缩写库/` | TXT | Train/dev/test split | Abbreviation expansion dataset |
| Occupations | `data/职业词库/` | TXT/DOC | National standard | Chinese occupation classification |
| Character Decomposition | `data/拆字词库/` | TXT | Traditional + Simplified | Character radical decomposition |
| Trad./Simp. Conversion | `data/繁简体转换词库/` | TXT | Conversion index | Traditional-Simplified character mapping |
| Word Segmentation | `data/中文分词词库整理/` | TXT | 300K+ entries | Multiple segmentation dictionaries |
| Rumor Data | `data/中文谣言数据/` | JSON | Dated 2017-06-13 | Chinese rumor detection dataset |
| NLP Books | `data/NLP_BOOK/` | PDF | Reference texts | NLP textbooks and notes |
| Research Papers | `data/paper/` | PDF | LLM survey | Academic papers |

### Loading Dictionaries

Most dictionaries are plain text files with one entry per line. Here are common integration patterns:

**Load a dictionary into Python:**

```python
with open("data/停用词/中文停用词库.txt", encoding="utf-8") as f:
    stopwords = set(line.strip() for line in f if line.strip())
```

```text
>>> len(stopwords)
1893
```

**Use as a jieba custom dictionary:**

```python
import jieba

jieba.load_userdict("data/IT词库/THUOCL_it.txt")
words = jieba.lcut("深度学习框架TensorFlow非常流行")
print(words)
```

> [!TIP]
> THUOCL dictionary files use tab-separated format: `word\tfrequency`. Most Chinese NLP tools (jieba, HanLP, pkuseg) accept this format directly as custom dictionaries.

**Load the rumor dataset:**

```python
import json

with open("data/中文谣言数据/rumors_v170613.json", encoding="utf-8") as f:
    rumors = json.load(f)
```

> [!NOTE]
> The rumor dataset is dated June 2017. Treat it as a historical snapshot for model training, not as a current rumor database.

### THUOCL Dictionary Format

Many dictionaries in this collection originate from the Tsinghua University Open Chinese Lexicon (THUOCL). These files follow a consistent format:

```text
深度学习	�1234
机器学习	5678
自然语言处理	2345
```

Each line contains a word and its frequency count, separated by a tab character. Higher frequency indicates more common usage in the Tsinghua corpus.

### Stop Word Lists

Four curated stop word lists are provided, each from a different academic source:

| File | Source |
|---|---|
| `中文停用词库.txt` | General Chinese stop words |
| `哈工大停用词表.txt` | Harbin Institute of Technology |
| `四川大学停用词表.txt` | Sichuan University |
| `百度停用词表.txt` | Baidu |

> [!WARNING]
> These lists have different coverage and aggressiveness. The Harbin Institute list is conservative (fewer words removed), while the general Chinese list is broader. Choose based on your task — aggressive filtering helps topic modeling but harms tasks like sentiment analysis where negation words matter.

### Classical Poetry Collection

The `data/古诗词库/` directory contains classical Chinese poetry organized by theme:

| Theme File | Content |
|---|---|
| `tangshi.txt` | Tang Dynasty poetry anthology |
| `songci.txt` | Song Dynasty ci poetry |
| `patriotic.txt` | Patriotic poetry |
| `love.txt` | Love poetry |
| `farewell.txt` | Farewell poetry |
| `homesick.txt` | Homesickness poetry |
| `landscape.txt` | Landscape/nature poetry |
| `frontier.txt` | Frontier/military poetry |
| `moon.txt` | Moon-themed poetry |
| `rain.txt` / `snow.txt` / `wind.txt` | Weather-themed poetry |
| `flower.txt` / `bird.txt` / `horse.txt` / `willow.txt` | Nature imagery |
| `friendship.txt` | Friendship poetry |
| `encouragement.txt` | Encouragement poetry |
| `war.txt` | War poetry |
| `mourn.txt` | Mourning poetry |
| `miss.txt` / `parting.txt` | Missing/parting poetry |
| `read.txt` | Poetry about reading/study |
| `countryside.txt` | Pastoral/countryside poetry |
| `view.txt` / `wanyue.txt` | Scenic view poetry |
| `haofang.txt` | Bold/heroic style poetry |
| `yongshihuaigu.txt` | Reflecting on history poetry |
| `youguoyoumin.txt` | Concern for country/people poetry |
| `boudoirripinings.txt` | Boudoir poetry |
| `expressemotion.txt` | Emotional expression poetry |

### Name Corpora

The `data/中英日文名字库/` directory provides multilingual name datasets:

**Chinese Names** (`Chinese_Names_Corpus/`):
- `Chinese_Names_Corpus（120W）.txt` — 1.2 million Chinese names
- `Chinese_Names_Corpus_Gender（120W）.txt` — Names with gender labels
- `Ancient_Names_Corpus（25W）.txt` — 250K historical/classical names
- `Chinese_Family_Name（1k）.xlsx` — ~1,000 Chinese family names
- `Chinese_Relationship（4.8k）.xlsx` — 4,800 relationship terms

**English Names** (`English_Names_Corpus/`):
- `English_Names_Corpus（2W）.txt` — 20K English names
- `English_Cn_Name_Corpus（48W）.txt` — 480K English names with Chinese translations
- `English_Cn_Name_Corpus_Gender（48W）.txt` — With gender labels

**Japanese Names** (`Japanese_Names_Corpus/`):
- `Japanese_Names_Corpus（18W）.txt` — 180K Japanese names
- `Japanese_Names_Corpus（1W）.xlsx` — 10K names in spreadsheet format

> [!TIP]
> The gender-annotated name files are particularly useful for training NER (Named Entity Recognition) models and gender prediction classifiers.

### Character Decomposition Dictionary

The `data/拆字词库/` directory contains character-to-radical decomposition tables:

- `chaizi-jt.txt` — Simplified Chinese character decomposition
- `chaizi-ft.txt` — Traditional Chinese character decomposition

These are useful for character-level NLP tasks, Chinese input method development, and educational applications.

## 4. Common Tasks

### Build a Domain-Specific NER System

**Goal:** Create a named entity recognizer using the provided dictionaries.

**Steps:**

1. Load relevant dictionaries as custom word lists:

```python
import jieba

# Load domain dictionaries
jieba.load_userdict("data/医学词库/THUOCL_medical.txt")
jieba.load_userdict("data/地名词库/THUOCL_diming.txt")
jieba.load_userdict("data/历史名人词库/THUOCL_lishimingren.txt")
```

2. Segment text with domain awareness:

```python
text = "张仲景在南阳研究伤寒杂病论"
words = jieba.lcut(text)
print(words)
```

3. Match segmented tokens against dictionary sets for entity classification.

### Create a Chinese Text Preprocessing Pipeline

**Goal:** Build a standard preprocessing pipeline using funNLP resources.

**Steps:**

1. Load stop words:

```python
with open("data/停用词/哈工大停用词表.txt", encoding="utf-8") as f:
    stopwords = set(line.strip() for line in f)
```

2. Load a segmentation dictionary:

```python
import jieba
jieba.load_userdict("data/中文分词词库整理/30wdict_utf8.txt")
```

3. Process text:

```python
def preprocess(text):
    words = jieba.lcut(text)
    return [w for w in words if w.strip() and w not in stopwords]
```

### Build a Sentiment Lexicon

**Goal:** Combine synonym, antonym, and negation dictionaries for sentiment analysis.

**Steps:**

1. Load the three lexical relation files:

```python
def load_dict(path):
    with open(path, encoding="utf-8") as f:
        return [line.strip() for line in f if line.strip()]

synonyms = load_dict("data/同义词库、反义词库、否定词库/同义词库.txt")
antonyms = load_dict("data/同义词库、反义词库、否定词库/反义词库.txt")
negations = load_dict("data/同义词库、反义词库、否定词库/否定词库.txt")
```

2. Use negation words to flip sentiment polarity during analysis.

> [!WARNING]
> The negation dictionary is essential for accurate Chinese sentiment analysis. Removing negation words during stop word filtering will degrade sentiment classification accuracy. Always exclude negation words from your stop word list when building sentiment models.

### Generate Random Chinese Names

**Goal:** Sample realistic Chinese names for test data generation.

**Steps:**

```python
import random

with open("data/中英日文名字库/Chinese_Names_Corpus/Chinese_Names_Corpus（120W）.txt",
          encoding="utf-8") as f:
    names = [line.strip() for line in f if line.strip()]

sample = random.sample(names, 10)
print(sample)
```

### Convert Between Traditional and Simplified Chinese

**Goal:** Use the conversion dictionary for character mapping.

```python
conversion = {}
with open("data/繁简体转换词库/fanjian_suoyin.txt", encoding="utf-8") as f:
    for line in f:
        parts = line.strip().split()
        if len(parts) >= 2:
            conversion[parts[0]] = parts[1]
```

## 5. Troubleshooting

### Error: `UnicodeDecodeError` when reading dictionary files

**Cause:** Files are encoded in UTF-8 but your script uses the system default encoding.

**Solution:**

1. Always specify `encoding="utf-8"` when opening files:

```python
with open("data/停用词/中文停用词库.txt", encoding="utf-8") as f:
    content = f.read()
```

2. On Windows, set the environment variable `PYTHONUTF8=1` for system-wide UTF-8 default.

**Prevention:** Always use explicit `encoding="utf-8"` in every `open()` call when working with this dataset.

### Error: Cannot extract `.rar` archives in company name corpus

**Cause:** The `data/公司名字词库/` directory contains `.rar` compressed files that require a RAR-compatible extractor.

**Solution:**

```bash
# Linux/macOS
sudo apt install unrar  # or: brew install unrar
unrar x "data/公司名字词库/Company-Names-Corpus（480W）.rar"
```

```bash
# Windows
# Use 7-Zip or WinRAR to extract
```

**Prevention:** Consider converting extracted data to `.txt` for easier reuse.

### Error: Chinese characters display as garbled text in filenames

**Cause:** The filesystem or terminal does not support UTF-8 encoded filenames.

**Solution:**

1. On Windows, use Windows Terminal or Git Bash instead of CMD
2. On Linux, ensure locale is set: `export LANG=en_US.UTF-8`
3. Use `ls` with proper locale settings to verify directory names display correctly

**Prevention:** Use a UTF-8 compatible terminal and filesystem from the start.

> [!WARNING]
> Avoid renaming directories to ASCII equivalents — this breaks relative path references and makes it harder to match files back to the dataset inventory.

### Issue: Dictionary file appears empty or has only headers

**Cause:** Some dictionary files may use different line endings (CR/LF vs LF) or BOM markers.

**Solution:**

```python
with open(filepath, encoding="utf-8-sig") as f:  # handles BOM
    lines = [line.strip() for line in f if line.strip()]
```

> [!TIP]
> The `utf-8-sig` encoding in Python automatically strips the BOM (Byte Order Mark) that some Windows-created text files include.

### Issue: `thirtyw.py` fails to run

**Cause:** This script uses Python 2 syntax (`print` without parentheses) and is a one-off data cleaning utility, not a general-purpose tool.

**Solution:** This script is a historical artifact for processing the `30wChinsesSeqDic.txt` file. Its output (`30wdict.txt`) is already included in the repository. You do not need to run it.

## 6. FAQ

**Q: What does "funNLP" stand for?**

The name combines "fun" with "NLP" (Natural Language Processing). The repository is a curated collection of Chinese NLP resources — dictionaries, corpora, datasets, and academic materials — gathered for practical use in NLP tasks.

**Q: What are the THUOCL files?**

THUOCL stands for **THU**inghua University **O**pen **C**hinese **L**exicon. These are domain-specific word frequency lists released by Tsinghua University's NLP lab. Each file contains words and their corpus frequency counts in tab-separated format.

**Q: Are these datasets suitable for commercial use?**

Licensing varies by dataset. The THUOCL dictionaries are released by Tsinghua University for research use. The name corpora and other datasets may have different terms. Check the original source of each dataset before commercial use.

**Q: How current are these datasets?**

The repository's last source commit is dated 2023-08-24. The rumor dataset is from 2017. Most dictionaries are static lexical resources that do not require frequent updates, though domain-specific terminology (IT, finance) may benefit from supplementation with newer terms.

**Q: Can I use these with tools other than jieba?**

Yes. The plain-text, one-word-per-line format is compatible with virtually any Chinese NLP tool including HanLP, pkuseg, LAC, LTP, and custom implementations. The THUOCL tab-separated format (word + frequency) is directly accepted by most segmentation tools.

**Q: Which stop word list should I use?**

It depends on your task:

| List | Best For |
|---|---|
| 哈工大停用词表 | Conservative filtering, preserves more content words |
| 百度停用词表 | Web/search-oriented text processing |
| 四川大学停用词表 | Academic/formal text processing |
| 中文停用词库 | General-purpose, broadest coverage |

For most tasks, start with the general Chinese stop word list and adjust based on results.

**Q: The repository mentions many more resources in the README. Where are they?**

The README serves as a curated directory of external NLP resources across the web. The `data/` directory contains the datasets that are directly included in this repository. External resources linked in the README must be obtained from their respective sources.

> [!NOTE]
> Community issues (#456, #457) indicate the repository is no longer actively updated. The included datasets remain useful as static lexical resources. For the latest NLP tools and models, supplement with actively maintained projects.

**Q: How do I contribute a new dictionary?**

Open a GitHub issue or pull request with your dataset placed in a new subdirectory under `data/`, following the existing naming convention (Chinese category name). Include a brief description of the source and scope.

---

_Generated May 2026 from fighting41love/funNLP at 29f4ac8. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._