face_recognition Administrator & Developer Guide

1. Architecture Overview

face_recognition is a Python library and CLI toolkit that provides face detection, facial landmark identification, face encoding, and face comparison. It is a thin, well-designed Python wrapper around dlib's C++-based machine learning models, exposing them through a simple 7-function public API and two Click-based CLI commands.

The library follows a pipeline architecture: images are loaded as numpy arrays, faces are detected using either a HOG or CNN model, facial landmarks are located using a 68-point or 5-point shape predictor, and 128-dimensional face encodings are generated for comparison via euclidean distance.

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                    User Application                      │
│         (Python script / CLI / Web service)               │
├─────────────┬───────────────────────────────┬────────────┤
│  CLI Layer  │        Python API (api.py)     │  Examples  │
│ face_recog  │  load_image_file()             │  webcam    │
│ face_detect │  face_locations()              │  video     │
│  (Click)    │  batch_face_locations()        │  knn       │
│             │  face_landmarks()              │  svm       │
│             │  face_encodings()              │  flask     │
│             │  compare_faces()               │  rpi       │
│             │  face_distance()               │            │
├─────────────┴───────────────────────────────┴────────────┤
│                  dlib (C++ via Python bindings)            │
│  ┌──────────────┐  ┌───────────────┐  ┌────────────────┐ │
│  │ HOG Detector  │  │ CNN Detector   │  │ Shape Predictor│ │
│  │ (CPU-only)    │  │ (CUDA-capable) │  │ (68pt / 5pt)   │ │
│  └──────────────┘  └───────────────┘  ┌────────────────┐ │
│                                        │ Face Encoder   │ │
│                                        │ (128-dim)      │ │
│                                        └────────────────┘ │
├──────────────────────────────────────────────────────────┤
│              face_recognition_models (data)               │
│  pose_predictor_model  │  cnn_face_detector_model         │
│  pose_predictor_5pt    │  face_recognition_model          │
└──────────────────────────────────────────────────────────┘

Component Inventory

Technology Stack

Note

The entire library has zero runtime configuration — no config files, no environment variables, no registry settings. All behavior is controlled through function parameters at call time. The only external dependency that must be present at import time is the face_recognition_models package, which provides the pre-trained .dat model files.

Data Flow

Image File/Stream
    │
    ▼
load_image_file() ──→ numpy array (H×W×3 uint8 RGB)
    │
    ▼
face_locations() ──→ list of (top, right, bottom, left) tuples
    │                     │
    │                     ▼
    │              face_landmarks() ──→ list of dicts {feature: [(x,y),...]}
    │                     │
    ▼                     ▼
face_encodings() ──→ list of 128-dim numpy arrays
    │
    ▼
compare_faces() ──→ list of True/False
face_distance()  ──→ numpy array of float distances

2. System Requirements

Hardware Requirements

Software Requirements

Docker Images

Tip

The Kubeflow Dockerfile variants (cpu-jupyter-kubeflow, gpu-jupyter-kubeflow) include Jupyter notebook support for interactive face recognition experiments in a Kubeflow pipeline.

3. Configuration Reference

This library has no configuration files, no environment variables, and no persistent settings. All behavior is controlled through function parameters.

API Function Parameters — Complete Reference

load_image_file(file, mode='RGB')

Returns: numpy array of shape (height, width, channels).

face_locations(img, number_of_times_to_upsample=1, model="hog")

Returns: list of (top, right, bottom, left) tuples. Coordinates are clamped to image bounds via _trim_css_to_bounds().

batch_face_locations(images, number_of_times_to_upsample=1, batch_size=128)

Returns: list of lists of (top, right, bottom, left) tuples. Always uses CNN model internally.

Important

batch_face_locations() uses images[0].shape for bounds trimming on all images in the batch. All images should be the same dimensions for correct coordinate clamping.

face_landmarks(face_image, face_locations=None, model="large")

Returns: list of dicts. Large model keys: chin, left_eyebrow, right_eyebrow, nose_bridge, nose_tip, left_eye, right_eye, top_lip, bottom_lip. Small model keys: nose_tip, left_eye, right_eye.

Raises: ValueError if model is not "large" or "small".

face_encodings(face_image, known_face_locations=None, num_jitters=1, model="small")

Returns: list of 128-dimensional numpy arrays.

compare_faces(known_face_encodings, face_encoding_to_check, tolerance=0.6)

Returns: list of True/False values.

face_distance(face_encodings, face_to_compare)

Returns: numpy array of float distances. Uses np.linalg.norm(axis=1) for vectorized euclidean distance. Returns empty array if input list is empty.

CLI Options — Complete Reference

face_recognition

Usage: face_recognition [OPTIONS] KNOWN_PEOPLE_FOLDER IMAGE_TO_CHECK

Output format: filename,person_name[,distance]

Special output values: unknown_person (no match found), no_persons_found (no faces detected).

Images over 1600px are downscaled to 1600x1600 via PIL.Image.thumbnail() with LANCZOS resampling.

face_detection

Usage: face_detection [OPTIONS] IMAGE_TO_CHECK

Output format: filename,top,right,bottom,left

Note

Both CLI tools use forkserver multiprocessing context on macOS to avoid a crash in libdispatch. This is handled automatically.

4. Deployment & Scaling

Docker Deployment

CPU container:

docker build -t face_recognition -f Dockerfile .
docker run -it --rm -v /path/to/images:/images face_recognition

GPU container (requires NVIDIA Container Toolkit):

docker build -t face_recognition:gpu -f Dockerfile.gpu .
docker run -it --rm --runtime=nvidia -v /path/to/images:/images face_recognition:gpu

Docker Compose:

# docker-compose.yml from source
version: '2'
services:
  face_recognition:
    build: .
    volumes:
      - ./:/host

Warning

The GPU Dockerfile requires the NVIDIA Container Toolkit installed on the host. Without it, --runtime=nvidia will fail. Verify with docker run --rm --runtime=nvidia nvidia/cuda nvidia-smi.

pip Installation for Production

# Virtual environment recommended
python -m venv face_env
source face_env/bin/activate

# Install with exact dependency versions
pip install face_recognition==1.4.0
pip install face_recognition_models>=0.3.0

Web Service Deployment

The examples/web_service_example.py demonstrates a minimal Flask-based face recognition API:

pip install flask face_recognition
python web_service_example.py

 * Running on http://0.0.0.0:5001/ (Press CTRL+C to quit)

curl -XPOST -F "file=@photo.jpg" http://127.0.0.1:5001

{
  "face_found_in_image": true,
  "is_picture_of_obama": true
}

For production deployment, use a WSGI server (gunicorn, uWSGI) instead of Flask's built-in development server.

Scaling Considerations

5. Operations

Performance Benchmarking

The library includes a benchmark script at examples/benchmark.py that tests each pipeline stage across different image resolutions:

cd examples
python benchmark.py

Benchmarks (Note: All benchmarks are only using a single CPU core)

Timings at 240p:
 - Face locations: 0.0259s (38.61 fps)
 - Face landmarks: 0.0010s (1028.81 fps)
 - Encode face (inc. landmarks): 0.0384s (26.02 fps)
 - End-to-end: 0.0651s (15.36 fps)

Tip

Face detection is the bottleneck — it gets significantly slower at higher resolutions. For real-time applications, detect faces on a downscaled image and run encoding on the full-resolution crop.

Performance Optimization Techniques

1. Reduce input resolution before face detection:

   small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25)
   

2. Skip frames in video processing:

   process_this_frame = not process_this_frame
   

3. Pre-compute face locations to avoid redundant detection:

   locations = face_recognition.face_locations(image)
   encodings = face_recognition.face_encodings(image, known_face_locations=locations)
   

4. Use the small landmark model when full 68-point precision isn't needed:

   encodings = face_recognition.face_encodings(image, model="small")
   

Note

The num_jitters parameter in face_encodings() trades speed for accuracy. At num_jitters=100, encoding takes 100x longer but produces more stable encodings. The default of 1 is usually sufficient.

Monitoring

The library produces no logs, no metrics, and no health endpoints. For production monitoring:

• Wrap API calls in timing instrumentation
• Track face detection hit rate (faces found / images processed)
• Monitor encoding distance distributions to detect model degradation
• Watch for RuntimeError exceptions from dlib (image format issues, CUDA errors)

6. Performance

Model Comparison

Resolution Impact

Memory Usage

• Each loaded image: height × width × 3 bytes (uint8 RGB)
• Each face encoding: 128 × 8 = 1,024 bytes (float64)
• dlib models loaded at import time: ~100 MB resident

7. Security

Input Validation

The library performs minimal input validation:

• load_image_file() delegates to PIL, which handles format validation
• LOAD_TRUNCATED_IMAGES = True is set at module level, meaning truncated/corrupt images are loaded without error (Pillow fills missing data)
• Face coordinates are clamped to image bounds via _trim_css_to_bounds(), preventing out-of-bounds array access
• The CLI tools filter input files by extension (.jpg, .jpeg, .png only via regex)

Warning

The face_recognition_models import has a bare except Exception handler that calls quit() (which raises SystemExit). This means a missing models package will terminate the entire Python process, not just raise an importable exception. Issue #1663 identifies this as a design concern. In server applications, catch SystemExit or verify the package is installed before importing.

Denial of Service Considerations

• Very large images consume proportional memory (H × W × 3 bytes)
• High number_of_times_to_upsample values multiply processing time exponentially
• High num_jitters values multiply encoding time linearly
• batch_face_locations() with large batch sizes can exhaust GPU memory
• No built-in timeouts or resource limits

Caution

In web-facing applications, validate image dimensions and file size before passing to face_recognition. Untrusted users could submit extremely large images to exhaust server memory.

Dependency Security

Privacy Considerations

• Face encodings are 128-dimensional float vectors that cannot be reversed into face images
• However, encodings can be used to track and identify individuals across images
• The library stores nothing persistently — all state is in-memory
• For GDPR/privacy compliance, treat face encodings as biometric data

8. Developer Internals

Repository Structure

face_recognition/
├── face_recognition/           Core library package
│   ├── __init__.py             Version (1.4.0), public API exports
│   ├── api.py                  7 public functions, 4 internal helpers
│   ├── face_detection_cli.py   face_detection CLI (Click)
│   └── face_recognition_cli.py face_recognition CLI (Click)
├── examples/                   22 example scripts
│   ├── benchmark.py            Performance benchmarking
│   ├── blink_detection.py      Eye blink detection via EAR
│   ├── blur_faces_on_webcam.py Gaussian blur on detected faces
│   ├── digital_makeup.py       Landmark-based digital cosmetics
│   ├── face_distance.py        Distance measurement demo
│   ├── face_recognition_knn.py KNN classifier (scikit-learn)
│   ├── face_recognition_svm.py SVM classifier (scikit-learn)
│   ├── facerec_from_video_file.py Video file processing
│   ├── facerec_from_webcam.py  Basic webcam recognition
│   ├── facerec_from_webcam_faster.py Optimized webcam (1/4 res, skip frames)
│   ├── facerec_from_webcam_multiprocessing.py Multi-process webcam
│   ├── facerec_ipcamera_knn.py IP camera with KNN
│   ├── facerec_on_raspberry_pi.py Raspberry Pi demo
│   ├── find_faces_in_batches.py GPU batch processing
│   ├── find_faces_in_picture.py Basic face detection (HOG)
│   ├── find_faces_in_picture_cnn.py CNN face detection
│   ├── find_facial_features_in_picture.py Landmark visualization
│   ├── identify_and_draw_boxes_on_faces.py Face identification + drawing
│   ├── recognize_faces_in_pictures.py Basic recognition
│   ├── web_service_example.py  Flask web service
│   └── knn_examples/           KNN training/test data
│       ├── train/              Training images (per-person subdirs)
│       └── test/               Test images
├── tests/
│   ├── __init__.py
│   ├── test_face_recognition.py Full test suite
│   └── test_images/            Test fixture images
├── docker/                     Docker configurations
│   ├── cpu/Dockerfile
│   ├── gpu/Dockerfile
│   ├── cpu-jupyter-kubeflow/Dockerfile
│   └── gpu-jupyter-kubeflow/Dockerfile
├── docs/                       Sphinx documentation source
├── Dockerfile                  Root CPU Dockerfile
├── Dockerfile.gpu              Root GPU Dockerfile
├── docker-compose.yml          Docker compose configuration
├── setup.py                    Package build configuration
├── setup.cfg                   Package metadata
├── pyproject.toml              Build system specification
├── requirements.txt            Runtime dependencies
├── requirements_dev.txt        Development dependencies
├── requirements_docs.txt       Documentation build dependencies
├── tox.ini                     Test automation configuration
├── Makefile                    Development convenience targets
├── LICENSE                     MIT license
├── MANIFEST.in                 Source distribution include rules
└── HISTORY.rst                 Changelog

Module Reference

face_recognition/api.py — Core API

This is the entire functional core of the library. It loads four dlib models at import time and exposes seven public functions.

Module-level initialization (runs at import time):

face_detector = dlib.get_frontal_face_detector()                    # HOG detector
pose_predictor_68_point = dlib.shape_predictor(model_68pt_path)     # 68-landmark predictor
pose_predictor_5_point = dlib.shape_predictor(model_5pt_path)       # 5-landmark predictor
cnn_face_detector = dlib.cnn_face_detection_model_v1(cnn_model_path)  # CNN detector
face_encoder = dlib.face_recognition_model_v1(face_rec_model_path) # 128-dim encoder

All four models are loaded eagerly. This adds ~1-2 seconds to first import but avoids lazy-loading latency during processing.

Internal helpers:

Public API:

face_recognition/face_recognition_cli.py — Recognition CLI

Key behavior: images larger than 1600px are thumbnailed to 1600×1600. Unknown faces print unknown_person. Images with no detectable faces print no_persons_found.

face_recognition/face_detection_cli.py — Detection CLI

Code Patterns and Conventions

Coordinate system: All face locations use CSS-style (top, right, bottom, left) ordering — not the more common (x, y, width, height) or (top, left, bottom, right). This is consistent throughout the API and CLI.

Image format convention: All functions expect numpy arrays in RGB color order with uint8 dtype. OpenCV users must convert BGR → RGB before calling any face_recognition function.

Error handling: The library uses minimal error handling:

• Module import: bare except Exception with quit() for missing models
• API functions: no explicit error handling (dlib exceptions propagate)
• face_landmarks(): explicit ValueError for invalid model names
• CLI: no try/except (crashes propagate to the user)

Multiprocessing: Both CLI tools use forkserver start method on macOS to work around a libdispatch crash bug. The detection is:

if "forkserver" in multiprocessing.get_all_start_methods():
    context = multiprocessing.get_context("forkserver")

Truncated images: ImageFile.LOAD_TRUNCATED_IMAGES = True is set globally at module level, meaning corrupt/truncated images are silently loaded with missing data filled in by Pillow.

Testing

The test suite is in tests/test_face_recognition.py using Python's unittest framework.

Test categories:

Running tests:

# Direct
python -m pytest tests/

# Via tox
tox

# Via make
make test

Note

Tests use fixed test images in tests/test_images/ with known face positions. The CNN tests use assertAlmostEqual with delta=25 because CNN detection coordinates vary slightly across dlib versions.

Test dependencies: tox, flake8, click.testing.CliRunner

Potential Improvements

1. Replace quit() with proper exception in api.py models import — quit() calls SystemExit, which is inappropriate in library code (issue #1663)
2. Add input validation to public API functions — currently no type checking on img parameter
3. Thread-safe model loading — dlib models are loaded as module-level globals; concurrent imports in multi-threaded servers could race
4. Configurable image size limit for the CLI's 1600px thumbnail threshold
5. Structured output format (JSON) option for CLI tools instead of CSV-only
6. Logging — the library uses print() for warnings instead of Python's logging module
7. Type hints — no type annotations on any public API function
8. batch_face_locations coordinate bug — uses images[0].shape for bounds trimming on all images in the batch, which is incorrect if images have different dimensions
9. LOAD_TRUNCATED_IMAGES global side effect — setting this Pillow global affects all PIL usage in the process, not just face_recognition

Contributing

Development setup:

git clone https://github.com/ageitgey/face_recognition.git
cd face_recognition
pip install -r requirements_dev.txt
pip install -e .

Development dependencies (requirements_dev.txt): tox, flake8

Code style: No enforced formatter. The codebase uses PEP 8-adjacent style with 4-space indentation. flake8 is configured via tox.ini.

Test workflow:

# Lint
flake8 face_recognition tests
# Test
python -m pytest tests/
# Or use tox for full matrix
tox

Build configuration:

The project uses setup.py with setuptools:

setup(
    name='face_recognition',
    version='1.4.0',
    packages=['face_recognition'],
    package_data={'face_recognition': ['models/*.dat']},
    entry_points={
        'console_scripts': [
            'face_recognition=face_recognition.face_recognition_cli:main',
            'face_detection=face_recognition.face_detection_cli:main'
        ]
    },
    install_requires=[
        'face_recognition_models>=0.3.0',
        'Click>=6.0',
        'dlib>=19.7',
        'numpy',
        'Pillow'
    ],
)

9. Integration Points

Python Library Integration

Import and use the public API directly:

import face_recognition

# All 7 public functions available at package level
face_recognition.load_image_file()
face_recognition.face_locations()
face_recognition.batch_face_locations()
face_recognition.face_landmarks()
face_recognition.face_encodings()
face_recognition.compare_faces()
face_recognition.face_distance()

CLI Integration

Both commands accept files or directories and output CSV to stdout, making them suitable for shell pipelines:

# Pipe to grep for specific people
face_recognition ./known/ ./photos/ | grep "John"

# Count faces in a directory
face_detection ./photos/ | wc -l

# Process and store results
face_recognition --show-distance 1 ./known/ ./photos/ > results.csv

Flask/Web Integration

The examples/web_service_example.py pattern works with any WSGI framework. Key integration points:

• load_image_file() accepts file-like objects (e.g., request.files['file'])
• All functions are stateless — safe for concurrent requests
• Face encodings are JSON-serializable as lists of floats

scikit-learn Integration

The KNN and SVM examples demonstrate classifier-based recognition:

• face_encodings() output is directly usable as feature vectors for any scikit-learn classifier
• Models can be serialized with pickle for persistent storage
• The image_files_in_folder() utility from the CLI module is importable for building training sets

OpenCV Integration

OpenCV provides video capture; face_recognition provides face analysis:

import cv2
import face_recognition

cap = cv2.VideoCapture(0)
ret, frame = cap.read()

# OpenCV BGR → face_recognition RGB
rgb_frame = frame[:, :, ::-1]

# Now use any face_recognition function
locations = face_recognition.face_locations(rgb_frame)

Important

OpenCV uses BGR color order. Always convert to RGB before passing frames to face_recognition functions. Forgetting this conversion is the most common integration mistake and results in degraded accuracy or no faces found.

For basic usage, installation, and getting started, see the User Guide.