funNLP — Administrator & Developer Guide

1. Architecture Overview

funNLP is a static dataset repository — a curated collection of Chinese NLP lexical resources organized into 24+ domain-specific directories under data/. There is no runtime application, no server, no build system, and no API. The repository's architecture is its file organization: a flat hierarchy of themed dictionary directories containing plain-text corpus files.

Repository Architecture

funNLP/
├── README.md              ← Curated directory of external NLP resources (links)
└── data/                  ← All included datasets
    ├── IT词库/            ← THUOCL domain dictionaries (×10 domains)
    ├── 医学词库/
    ├── 法律词库/
    ├── 财经词库/
    ├── 食物词库/
    ├── 动物词库/
    ├── 地名词库/
    ├── 历史名人词库/
    ├── 诗词短句词库/
    ├── 汽车品牌、零件词库/
    ├── 古诗词库/          ← 30 themed classical poetry files
    ├── 停用词/            ← 4 university-curated stop word lists
    ├── 同义词库、反义词库、否定词库/  ← Lexical relation dictionaries
    ├── 成语词库/          ← Idiom corpus (5W entries)
    ├── 中英日文名字库/    ← Multilingual name corpora (120W+ entries)
    │   ├── Chinese_Names_Corpus/
    │   ├── English_Names_Corpus/
    │   └── Japanese_Names_Corpus/
    ├── 公司名字词库/      ← Company/org names (480W+, RAR archives)
    ├── 中文缩写库/        ← Abbreviation expansion (train/dev/test split)
    ├── 职业词库/          ← Occupation classification
    ├── 拆字词库/          ← Character decomposition (trad. + simp.)
    ├── 繁简体转换词库/    ← Traditional↔Simplified conversion index
    ├── 中文分词词库整理/  ← Segmentation dictionaries (300K+ entries)
    ├── 中文谣言数据/      ← Rumor detection dataset (JSON, 2017)
    ├── NLP_BOOK/          ← NLP reference textbooks (PDF)
    └── paper/             ← Research papers (PDF)

Note

This is a data-only repository. There is no application runtime, no deployment pipeline, and no service infrastructure. Administration concerns are limited to storage, data integrity, and contribution management.

Component Inventory

Technology Stack

2. System Requirements

Important

The repository uses Chinese characters in directory and file names extensively. Git on Windows may require core.quotePath=false to display these correctly:

git config --global core.quotePath false

3. Configuration Reference

This repository has no configuration files. There are no settings, environment variables, or config formats to manage.

The only data-processing code is data/中文分词词库整理/thirtyw.py, a self-contained Python 2 script with hardcoded input/output paths:

Note

The script's outputs (30wdict.txt, 30wdict_utf8.txt) are already committed to the repository. There is no need to run this script unless reprocessing the source dictionary.

4. Deployment & Scaling

Distribution

This is a static data repository. "Deployment" means making the data available to consumers:

Direct clone:

git clone https://github.com/fighting41love/funNLP.git

Sparse checkout (selective datasets):

git clone --no-checkout https://github.com/fighting41love/funNLP.git
cd funNLP
git sparse-checkout init --cone
git sparse-checkout set data/停用词 data/成语词库
git checkout

Mirror for internal use:

git clone --mirror https://github.com/fighting41love/funNLP.git funNLP.git

Storage Considerations

Tip

For CI/CD pipelines that only need specific dictionaries, use Git sparse checkout to minimize clone size and build time.

5. Operations

Data Integrity Verification

Since this is a static dataset, verify integrity after cloning:

# Count files per directory
find data -type f | wc -l

Expected: ~100 files

# Verify no corrupted text files (check for valid UTF-8)
find data -name "*.txt" -exec file {} \; | grep -v "UTF-8\|ASCII"

Note

The data/中文分词词库整理/.DS_Store file is a macOS artifact and can be safely ignored or removed.

Tip

Some text files in data/中文分词词库整理/ may use GB2312 or GBK encoding rather than UTF-8. The 30wdict_utf8.txt variant is the UTF-8 conversion of 30wdict.txt.

Archive Extraction

The company name corpus requires extraction:

cd data/公司名字词库
unrar x "Company-Names-Corpus（480W）.rar"
unrar x "Organization-Names-Corpus（110W）.rar"

Extracting from Company-Names-Corpus（480W）.rar
...
All OK

Monitoring Repository Updates

The repository has not been actively updated since 2023 (source commit 29f4ac8, dated 2023-08-24). Community issues #456 and #457 ask about update status.

# Check for upstream changes
git fetch origin
git log HEAD..origin/master --oneline

6. Performance

File I/O Optimization

When loading large dictionary files (name corpora, company names) into memory:

# Efficient: generator for large files
def iter_dict(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            word = line.strip()
            if word:
                yield word

# Use set for O(1) lookup
medical_terms = set(iter_dict("data/医学词库/THUOCL_medical.txt"))

For the largest files (120W+ name entries), consider memory-mapped I/O or database loading rather than holding everything in memory simultaneously.

Tip

For THUOCL files, parse both word and frequency if you need weighted matching:

def load_thuocl(path):
    result = {}
    with open(path, encoding="utf-8") as f:
        for line in f:
            parts = line.strip().split("\t")
            if len(parts) == 2:
                result[parts[0]] = int(parts[1])
    return result

7. Security

Data Provenance

Warning

The company name corpus (480W+ entries) and name corpora (120W+ entries) contain real-world personally identifiable information (PII). If using these in applications subject to data protection regulations (GDPR, China's PIPL), ensure compliance with applicable data handling requirements.

Caution

The rumor dataset (data/中文谣言数据/rumors_v170613.json) contains actual rumor text. Do not surface this content to end users without appropriate labeling. It is intended for training rumor detection models, not as a content source.

Supply Chain Considerations

• No executable dependencies — the repository contains only data files and one inert Python 2 script
• No package manager artifacts — no risk of dependency confusion or supply chain attacks
• The .pyc file (thirtyw.pyc) is a compiled Python 2 bytecode artifact and is harmless but unnecessary

8. Developer Internals

Repository Structure

funNLP/
├── README.md                          ← Main content: curated NLP resource links
├── data/                              ← All included datasets (24+ subdirectories)
│   ├── IT词库/THUOCL_it.txt
│   ├── NLP_BOOK/eisenstein-nlp-notes.pdf
│   ├── paper/LLM_Survey_Chinese_0418.pdf
│   ├── 中文分词词库整理/             ← Largest module: segmentation dicts + utility
│   │   ├── thirtyw.py                ← Python 2 dict cleaner (only code in repo)
│   │   ├── thirtyw.pyc               ← Compiled bytecode artifact
│   │   ├── 30wChinsesSeqDic.txt      ← Source dictionary (input to thirtyw.py)
│   │   ├── 30wdict.txt               ← Cleaned output (words >2 chars)
│   │   ├── 30wdict_utf8.txt          ← UTF-8 conversion
│   │   └── ... (15+ dictionary files)
│   ├── 中文缩写库/                   ← ML-ready train/dev/test split
│   │   ├── train_set.txt
│   │   ├── dev_set.txt
│   │   └── test_set.txt
│   ├── 中文谣言数据/
│   │   ├── README.md                 ← Dataset-specific documentation
│   │   └── rumors_v170613.json       ← Rumor entries (JSON)
│   ├── 中英日文名字库/
│   │   ├── 萌名-中文人名生成器V1.1.xlsx
│   │   ├── Chinese_Names_Corpus/     ← 120W entries, gender labels
│   │   ├── English_Names_Corpus/     ← 48W entries, CN translations
│   │   └── Japanese_Names_Corpus/    ← 18W entries
│   ├── 停用词/                       ← 4 stop word lists
│   ├── 公司名字词库/                 ← RAR archives (480W+ companies, 110W+ orgs)
│   ├── 古诗词库/                     ← 30 themed poetry files
│   ├── 同义词库、反义词库、否定词库/ ← 3 lexical relation files
│   ├── 成语词库/                     ← 5W idioms + THUOCL
│   ├── 拆字词库/                     ← Simplified + Traditional decomposition
│   ├── 繁简体转换词库/              ← Conversion index
│   └── 职业词库/                     ← Occupation classification (TXT + DOC)

Module Reference

data/中文分词词库整理/thirtyw.py — Dictionary Cleaner

The only executable code in the repository. A Python 2 script that:

1. Reads 30wChinsesSeqDic.txt (a raw 300K-entry Chinese dictionary)
2. Parses each line by splitting on space then tab
3. Filters entries to only those where the word is longer than 2 characters
4. Writes matching words to 30wdict.txt (append mode)

# Source: data/中文分词词库整理/thirtyw.py (complete, 15 lines)
f = open('30wChinsesSeqDic.txt')
fout = open('30wdict.txt','a')
count = 0
for line in f:
    temp = line.strip()
    temp_list = temp.split(' ')
    temp_sublist = temp_list[1].split('\t')
    if len(temp_sublist[1]) > 2:
        count = count + 1
        print temp_sublist[1]        # Python 2 print statement
        fout.write(temp_sublist[1] + '\n')
f.close()
fout.close()

Issues:

• Python 2 only (print without parentheses)
• Uses append mode ('a') — running twice doubles the output
• No encoding specification (relies on system default)
• No error handling for malformed lines
• Hardcoded file paths (not parameterized)

Note

This script is a historical artifact. Its output files are already committed. A modern equivalent would take ~5 lines of Python 3 with proper encoding handling.

Data Format Reference

Code Patterns

Pattern: Flat data organization. Each dataset lives in its own descriptively-named Chinese directory. No nesting beyond one level (except name corpora, which split by language). This makes discovery trivial but requires UTF-8 filesystem support.

Pattern: THUOCL as baseline. Ten domain dictionaries follow the identical THUOCL format (word + tab + frequency). This consistency enables generic loaders that work across all domains.

Pattern: No build/test infrastructure. The repository has no tests, no CI, no linting, no packaging. This is appropriate for a static data collection but means data quality depends entirely on manual curation.

Testing Strategy

There are no automated tests. For a data repository, quality assurance would involve:

• Encoding validation: Ensure all .txt files are valid UTF-8
• Format consistency: Verify THUOCL files follow word\tfrequency format
• Duplicate detection: Check for duplicate entries within dictionaries
• Completeness checks: Verify RAR archives extract successfully

Potential Improvements

1. Add a requirements.txt or pyproject.toml — Even if minimal, it signals the project is Python-ecosystem adjacent
2. Convert thirtyw.py to Python 3 — The Python 2 script is non-functional on modern installations
3. Replace RAR archives with open formats — ZIP or gzip would eliminate the unrar dependency
4. Add data validation CI — GitHub Actions workflow to check encoding and format consistency on PR
5. Standardize encoding — Convert all files to UTF-8 (some segmentation dicts use GB2312)
6. Remove .DS_Store and .pyc files — Add a .gitignore to prevent OS/build artifacts
7. Add dataset metadata — A manifest.json describing each dataset's source, license, size, and format
8. Provide unified loader — A Python module that loads any dictionary by category name with proper encoding handling
9. Document licensing per dataset — THUOCL, community-contributed, and scraped data have different terms

Contributing

To add a new dataset:

1. Create a directory under data/ with a descriptive Chinese name
2. Place data files in plain text (UTF-8), one entry per line where applicable
3. Follow the THUOCL format (word\tfrequency) for frequency-annotated dictionaries
4. Add a README.md in the dataset directory if the data requires explanation (see data/中文谣言数据/README.md for an example)
5. Open a pull request with a description of the data source and scope

Important

Do not commit datasets containing sensitive personal data (ID numbers, phone numbers, addresses) or data scraped in violation of terms of service. Name corpora and company name lists are acceptable as they are publicly available information.

9. Integration Points

NLP Framework Compatibility

Data Pipeline Integration

For ETL pipelines that consume these dictionaries:

import os
import glob

def load_all_thuocl(data_dir="data"):
    """Load all THUOCL dictionaries into a unified frequency map."""
    combined = {}
    for path in glob.glob(os.path.join(data_dir, "**/THUOCL_*.txt"), recursive=True):
        domain = os.path.basename(os.path.dirname(path))
        with open(path, encoding="utf-8") as f:
            for line in f:
                parts = line.strip().split("\t")
                if len(parts) == 2:
                    word, freq = parts[0], int(parts[1])
                    combined[word] = combined.get(word, 0) + freq
    return combined

Database Loading

For applications that need indexed search over the dictionaries:

CREATE TABLE dictionary (
    word TEXT NOT NULL,
    frequency INTEGER DEFAULT 0,
    domain TEXT NOT NULL,
    source TEXT DEFAULT 'THUOCL'
);

CREATE INDEX idx_word ON dictionary(word);
CREATE INDEX idx_domain ON dictionary(domain);