Unit 42 Timely Threat Intel Administrator & Developer Guide

This guide is for the engineers and operators who put PaloAltoNetworks/Unit42-timely-threat-intel to work inside a security program — the people running the cron jobs, the SOAR pipelines, the SIEM ingestors, and the threat-intel platforms that consume the dataset. It covers what the repository actually is at a system level, how to host and scale a mirror, how to integrate cleanly with existing detection stacks, and the security and operational considerations that come with handling a dataset full of live malicious indicators.

The User Guide explains how to navigate the repo and write one-off recipes. This document covers the long-running, multi-team operational concerns that surround it.

1. Architecture Overview

What the system is

Unit42-timely-threat-intel is a flat, append-only dataset of plaintext threat-intelligence reports authored by Palo Alto Networks' Unit 42 team. It contains 370 entries at the root — 367 dated reports, plus README.md, LICENSE (GPL-3.0), and SUPPORT.md. There is no service, no daemon, no API, no schema validator, and no build step.

The repository is the product: a versioned, GitHub-hosted document store whose interface is git, whose schema is "filename starts with YYYY-MM-DD", and whose payloads are human-readable text files with a soft-but-stable internal layout.

Note

The architectural decision to use a flat directory rather than a folder hierarchy or a structured database is load-bearing. Filenames sort chronologically out of the box; classification by actor, malware family, or technique would require either deep folder trees that age poorly (today's group names get retired or merged) or duplicate placement (multi-actor campaigns), and either approach would couple consumers to a taxonomy maintained by Unit 42 alone. The current design pushes classification to consumers, who can index the dataset along whatever axes their detection programs care about.

Component inventory

High-level data flow

Unit 42 researcher
   │
   │ commits new YYYY-MM-DD-*.txt or .md
   ▼
github.com/PaloAltoNetworks/Unit42-timely-threat-intel  (single main branch)
   │
   ├── git clone / git pull         ─►  Analyst workstation
   ├── raw.githubusercontent.com    ─►  Curl / SOAR fetcher
   ├── api.github.com/.../contents  ─►  Listing automations
   └── github.com/.../commits.atom  ─►  Change-notification subscribers

Consumer side (you build this):
   Fetch ─► Diff (since last seen commit) ─► Extract (regex per IOC type)
        ─► Refang ─► Normalize ─► Push to TIP / SIEM / blocklist

There is one upstream branch (main); commits are append-only in practice. Edits to existing files do happen (typo fixes — see issue #2), so consumers must handle "file changed" as well as "file added".

Technology stack

Note

There is no parser, no validator, and no test suite shipped with the repo. The "schema" is enforced by author convention. Plan for variance — section heading capitalization, label phrasing, and bullet markers all drift across the 5+ years of history (2020-08 through 2026-04).

Confirmed integration points

Direct, supported by the upstream:

• Git over HTTPS — clone or pull https://github.com/PaloAltoNetworks/Unit42-timely-threat-intel.git.
• Raw file URLs — https://raw.githubusercontent.com/PaloAltoNetworks/Unit42-timely-threat-intel/main/<filename>.
• GitHub Contents API — https://api.github.com/repos/PaloAltoNetworks/Unit42-timely-threat-intel/contents/.
• GitHub commit Atom feed — https://github.com/PaloAltoNetworks/Unit42-timely-threat-intel/commits/main.atom.

Indirect, build-it-yourself: any feed format your downstream needs — STIX, MISP, CSV, JSON, TAXII — must be produced by a transform you write. See section 9.

2. System Requirements

Per-consumer minimums

Software dependencies

Network egress

If your environment is air-gapped, run a mirror on the perimeter (see section 4) and pull from the mirror inside the airgap.

3. Configuration Reference

Files of record (upstream)

There are no config files inside the repo. There is no pyproject.toml, no package.json, no Makefile, no Dockerfile, no GitHub Actions workflow under .github/workflows/, and no CONTRIBUTING.md. Treat absence as a constraint your tooling must respect.

Filename grammar (the soft schema)

Approximate ABNF for the filename convention seen across the corpus:

filename     = date "-" descriptor "." ext
date         = 4DIGIT "-" 2DIGIT "-" 2DIGIT          ; YYYY-MM-DD
descriptor   = 1*( ALPHA / DIGIT / "-" / "_" / " " )  ; kebab-case typical
                                                      ; mixed case occurs
                                                      ; spaces occur (rare)
ext          = "txt" / "md"

Edge cases observed in the file tree:

• 2026-02-20- AI-Accelerated Malicious Chrome Extension Campaigns.txt — leading space after the date hyphen, internal spaces in the descriptor.
• 2026-02-27-IOCs-for-Alloy-Taurus-infrastructure — no extension at all on this entry. Consumers must handle both extensioned and bare-name files.
• All-caps descriptors (THE-GHOST-IN-CAMPAIGN, KIMWOLF-V7-IoT, LORIKAZZ-ANDROID-IOT).

Important

The filename grammar is descriptive, not enforced. Build robust automation that does not crash on a missing extension, an embedded space, or a deviated descriptor. A practical strategy: parse ^(\d{4}-\d{2}-\d{2}) from the basename and let the rest be opaque.

Body section grammar (per-report)

The body of a .txt report is informally organized as:

<DATE> (<WEEKDAY>): <TITLE-IN-CAPS>

AUTHOR(S):
- <name>[, <name>, ...]

REFERENCES:
- https://www.linkedin.com/posts/...
- https://twitter.com/Unit42_Intel/status/...

NOTES:
- <bullet 1>
- <bullet 2>
...

[<INDICATOR-CATEGORY-HEADING>:]
- <indicator>
- <indicator>
...

Markdown reports use H1 for the title, H2 for Authors/References/Notes/Indicators of Compromise (IoCs), and code fences for embedded code or HTML samples. Section names vary in case (AUTHOR: vs Authors, INDICATORS: vs Indicators of Compromise (IoCs)). Consumers should match section labels case-insensitively.

Indicator-category headings observed

These exact strings (or close variants) appear as section headings across reports:

Refanging substitutions (the only "config" you must hard-code)

There are no other consistent obfuscations. There is no zero-width-character defang, no hex-encoded host defang, no unicode confusable substitution applied programmatically by Unit 42 — the four substitutions above cover the corpus.

Consumer-side environment variables (recommended pattern)

Names below are conventions your ingestion service should adopt; they are not read by the upstream repo.

Important

Ship U42_AV_EXCLUDED=false as the default and refuse to run the pull/extract loop unless the operator has explicitly set true. This forces conscious sign-off that the AV exclusion is in place — preventing the most common incident: an AV agent on the cron host quarantines the working tree mid-pull and corrupts state.

4. Deployment & Scaling

Topology options

Option A — Per-analyst workstation

Each analyst clones to their own machine, runs extraction locally, and pushes nothing further upstream. Suitable for small teams (1-5 analysts) with mature individual workstation hygiene.

Option B — Central mirror + ingestion pipeline (recommended)

A single internal mirror (a self-hosted GitLab project, an internal Gitea instance, or a read-only path on a NAS) is updated by a service account that pulls from upstream on a schedule. Downstream services (SIEM ingestor, TIP feeder, blocklist generator) read from the mirror.

Internet
   │
   ▼
[Mirror puller]   cron: git pull --ff-only
   │
   ▼
Internal mirror (read-only to analysts, write only by puller service)
   │
   ├──► Extraction service  (regex per indicator type, refang at boundary)
   │       │
   │       ├──► TIP / TAXII server
   │       ├──► SIEM indicator API
   │       └──► Blocklist generator (DNS sinkhole, proxy denylist)
   │
   └──► Analyst workstations  (read-only for browsing, no individual pulls)

Option C — Air-gapped / offline mirror

For classified or regulated environments, run Option B's puller in a DMZ host, then sync the mirror across the air gap (one-way diode, removable media, or scheduled SFTP). Inside the gap, treat the synced mirror as read-only.

Note

The repository ships no signed commits, no SBOM, and no attestation. If your environment requires cryptographic provenance, your mirror layer should sign each downstream artifact you derive (signed STIX bundle, signed CSV, signed git tag) using your own keys.

Scaling characteristics

The dataset is small in absolute terms — ~370 files, low-tens-of-MB compressed — but its access pattern matters more than its size:

Tip

Cache the last-seen commit SHA in your pipeline state and use git diff --name-only --diff-filter=AM <last-seen>..HEAD to enumerate only files that have been added or modified. This converts "process every file every hour" into "process the 0-3 files that actually changed", and makes incremental ingestion idempotent.

Backup & restore

The upstream is itself a Git repository — your local clone IS a backup. For a true cold backup:

git clone --mirror https://github.com/PaloAltoNetworks/Unit42-timely-threat-intel.git \
  /backup/u42.git

A mirror clone preserves all refs, including any tags or branches that may appear later. To restore, simply git clone /backup/u42.git.

If the upstream repository is ever taken down, a git clone --mirror from any healthy consumer machine recovers the full history.

Mirror puller — minimal sketch

#!/usr/bin/env bash
# /opt/u42/puller.sh — run from cron under a dedicated service account
set -euo pipefail
REPO_DIR="/srv/mirror/Unit42-timely-threat-intel"
LOG="/var/log/u42-puller.log"

cd "$REPO_DIR"
LAST="$(git rev-parse HEAD)"
git pull --ff-only --quiet
NEW="$(git rev-parse HEAD)"

if [[ "$LAST" != "$NEW" ]]; then
  echo "$(date -u +%FT%TZ) pulled $LAST..$NEW" >> "$LOG"
  /opt/u42/extract.sh "$LAST" "$NEW"
fi

The corresponding crontab entry, run hourly at a non-zero minute to avoid cron-thundering on shared hosts:

17 * * * * /opt/u42/puller.sh >> /var/log/u42-puller.cron.log 2>&1

5. Operations

Day-2 runbook overview

Note

The puller's "last successful commit SHA" is the single most valuable observability signal you have. Surface it to your monitoring stack and alert when it has not advanced in N × pull-interval (where N is your tolerance for upstream-quiet days; 7 days is a reasonable starting point given the historical cadence).

Health checks

Three signals tell you the pipeline is healthy:

1. Reachability — git ls-remote https://github.com/PaloAltoNetworks/Unit42-timely-threat-intel.git HEAD returns a SHA without prompting.
2. Freshness — git -C $REPO_DIR log -1 --format=%cI is within your expected upstream cadence (commits within last 14 days during normal operations; longer dry spells do happen).
3. Extractor sanity — total IOC counts trend up monotonically over the long run. A sudden drop across a pull cycle indicates a parser bug, not an upstream change.

Common operational alerts

Rolling out an extractor change

Because the dataset is append-only in practice, a regression in your extractor is silent — you only notice when downstream rules stop firing. Mitigations:

• Snapshot test fixtures — keep ~10 fixed report files (across 2020, 2022, 2024, 2026) and a hand-curated expected-IOC set. Run extractor changes against the fixtures in CI.
• Backfill on extractor deploy — when shipping a new extractor, run it across the full corpus (not just the latest pull) and diff against the previous extractor's output before promoting.
• Canary one report type first — if you change SHA256 extraction, push to staging; verify against TIP indicator count; promote.

Tip

Pin extractor changes to a specific upstream commit when running differential tests. This isolates "extractor changed" from "upstream report layout drifted" — both can produce a count delta, and you cannot tell them apart without freezing one variable.

6. Performance

The dataset is too small to have a "performance problem" in the classical sense, but pipeline performance still matters because consumers run on a clock.

Baseline numbers (reference)

Bottlenecks worth knowing about

Optimization patterns

• Extract on diff, not on world. Process only the files in git diff --name-only HEAD~1 HEAD after each pull.
• Memoize refang. A single function, called per-string, with no global state. Cheap; should be inlined by your interpreter.
• Parallelize across files, not within a file. Files are independent; use a thread pool sized to your CPU count.

7. Security

Caution

This dataset is, by design, full of live malicious indicators — real malware SHA256 hashes, currently or recently active C2 domains, malicious URLs, and decoded payload fragments. Treat the working tree as you would a malware analysis sandbox: assume your endpoint security software will react to it, and make conscious decisions about exposure on every host that touches it.

Threat model

The repo itself is not malware — it ships no executables, no compiled artifacts, no installers. The risks are:

1. AV/EDR false positives triggering on string-form hashes, malicious URLs, or fragments of malicious code embedded in writeups, leading to:
   • quarantined working tree mid-pull
   • alerts in your endpoint console that have to be triaged
   • clone aborted with partial state
2. Operator error during refanging — a refanged URL pasted into chat/email/ticket triggers link unfurlers to contact live attacker infrastructure, leaking the operator's IP and timing information.
3. Supply-chain risk — although low for a public PAN-owned repo, a hypothetical compromise of the upstream could push tampered indicators that, if blindly ingested, would cause downstream blocklist or detection actions against benign assets.
4. Information disclosure — IOCs and the timing of when you ingest them can themselves be operationally sensitive material; they reveal what your security team is monitoring.

Secure handling — handling-grade requirements

Warning

When you check out the working tree on a host with EDR enabled, expect alerts. Some EDR products will:

• Quarantine .txt files containing a high density of known-bad hash strings.
• Block file writes to a path containing a malicious URL string.
• Generate a "malware staged" detection because the filename literally contains words like "malware", "stealer", "ransomware".

All of these are false positives in the strict sense, but each one is a tier-one ticket if you do not pre-empt it. Coordinate the AV/EDR exclusion before the first clone.

Secrets & credentials

The repo contains no secrets — no API keys, no tokens, no passwords. All references are to public LinkedIn and X/Twitter posts. Your consumer-side pipeline may carry secrets (TIP API keys, SIEM auth, signing keys) — store them in your usual secret manager, not in the repo or its mirror.

Vulnerability handling

There is no software product to CVE. Any "vulnerabilities" here would be content errors — a wrong indicator, a misattributed campaign, a wrong filename date (issue #2 was an example of the latter, since closed). The community channel for these is the GitHub Issues tracker on the upstream repo.

Important

If you spot a misattribution that could lead to operational harm (for example, a hash for a benign file being labeled as malicious, which would cause downstream takedowns of legitimate infrastructure if blindly ingested), do not open a public issue with the corrected indicator unless you have confirmed it through your own analysis. Contact Palo Alto Networks through your account team so the correction can be coordinated with affected parties.

Data classification

For most security teams, the working tree should be classified at internal-use at minimum: the data is public, but the timing and shape of your consumption (which campaigns you mirror, how fast, into which detection products) is itself sensitive. Treat consumer logs and downstream feeds accordingly.

8. Developer Internals

Repository structure (annotated)

Unit42-timely-threat-intel/
├── README.md                       2-line statement of intent
├── SUPPORT.md                      "Released as-is", no normal PAN support
├── LICENSE                         GPL-3.0 full text
├── 2020-08-20-IOCs-for-Emotet-...  earliest report
├── ...
├── 2022-05-15-...for-Deadbolt-...md   first Markdown-format report
├── ...
├── 2026-04-21-LummaStealer-...     latest report at this snapshot
└── (370 entries total at the root; no subdirectories)

There is no .github/ directory shipped with workflows or templates that the briefing surfaced; there is no tests/, no docs/, no tools/. Anything you build — extractors, fixture sets, CI — lives in your repo, not this one.

How a new report appears

Operationally, the lifecycle looks like:

1. A Unit 42 researcher publishes a LinkedIn or X/Twitter post.
2. They commit a new YYYY-MM-DD-<descriptor>.<txt|md> to main.
3. The new file appears in the next git pull your puller runs.

There is no PR review surfaced publicly, no draft branch, and no staging. The first time consumers see a report is when it is published.

Author conventions to know

• Author lines are real names of Unit 42 staff — Daniel Bunce, Dominik Reichel, Beliz Kaleli, Shehroze Farooqi, Alex Starov, Richard Emerson, Keerthiraj Nagaraj, Vishwa Thothathri, Nabeel Mohamed, Billy Melicher, Lee Wei Yeong, Jonny Westfall, Alex Payne, and others. Useful for cross-referencing related research from the same author cluster.
• References point to LinkedIn and X/Twitter (formerly twitter.com), in that order of preference for newer reports. Older reports lean on twitter.com/Unit42_Intel/status/...; newer reports lean on linkedin.com/posts/unit42_....
• Threat actor names follow Unit 42's "celestial" taxonomy. Names confirmed in the 370-file file tree include: Stately Taurus, Bitter Scorpius, Iron Taurus, Alloy Taurus, Boggy Serpens, Dark Scorpius, White Lynx, Agent Serpens, Squeamish Libra, Playful Taurus, Mirai (a botnet, not an actor in the celestial sense). Track these as a controlled vocabulary for your extractor.

Module-level patterns

There are no modules — the whole repo is data. The closest analog to "code patterns" is the report layout pattern repeated 367 times. Document and version your understanding of that pattern in your repo, not theirs.

A reasonable internal abstraction:

from dataclasses import dataclass, field
from datetime import date
from typing import Optional

@dataclass(frozen=True)
class Indicator:
    kind: str           # 'sha256', 'domain', 'url', 'ipv4', 'filepath'
    value: str          # refanged form
    role: Optional[str] = None  # 'sideloaded DLL', 'C2', 'dropper', etc.

@dataclass(frozen=True)
class Report:
    publication_date: date
    title: str
    authors: tuple[str, ...]
    references: tuple[str, ...]
    notes: str
    indicators: tuple[Indicator, ...] = field(default_factory=tuple)
    source_filename: str = ''
    source_commit: str = ''

Pinning source_commit on every Report lets your downstream artifacts be reproduced from a specific upstream snapshot — important for incident timelines and audit defense.

Testing strategy (consumer-side)

You cannot test the upstream repo, but you must test your interpretation of it.

Potential improvements (consumer-facing)

These are improvements you can build atop the dataset; not requests of the upstream:

• A normalized JSON or STIX 2.1 export of the entire corpus, refanged and tagged with ATT&CK IDs derived from prose keyword mapping.
• A web index (your own, internal) keyed by malware family, threat actor, and ATT&CK technique — the axes the upstream deliberately does not provide.
• A diff-summary email that fires on each new commit, listing the new indicators by type and linking back to the source file at a pinned SHA.
• A historical backfill script that walks the entire git history and produces "first-seen" timestamps for every unique indicator across the corpus.

Contributing upstream

Contributions to the upstream are not solicited via PR in any documented way. Per SUPPORT.md: "released as-is", "community supported". The practical channel for corrections is GitHub Issues — file a clear, narrow issue (see issue #2 as the canonical example).

9. Integration Points

Consumers (who reads from this repo)

Producers (what feeds in)

The repo has exactly one producer: the Unit 42 team, committing directly to main. There is no automation surface upstream that consumers can wire into.

Authentication

If you authenticate, use a token scoped to public-read only with no organization membership and no repository write access. Rotate quarterly.

Output contracts (what you owe downstream)

When your extraction pipeline pushes indicators to a SIEM/TIP/EDR, the downstream contract should include:

Important

Carry source_commit and source_filename end-to-end, not just at extraction. When a SOC analyst asks "where did this domain block come from?", the answer must be a one-click trip back to the canonical report, not a guess. Without those fields, indicator provenance is lost and the dataset's value drops sharply.

Compatibility with adjacent feeds

This dataset complements but does not duplicate:

• The Unit 42 blog (unit42.paloaltonetworks.com) — long-form research; some articles cross-link to specific reports here, but most don't.
• AlienVault OTX, MISP communities, Abuse.ch (URLhaus, MalwareBazaar, ThreatFox) — broader IOC sharing; this repo is narrower and Unit 42-attributed.
• Vendor takedown advisories (CISA, Microsoft, Mandiant) — sometimes overlapping subjects, with different indicator coverage. Use all of them; deduplicate at the TIP layer.

Tip

If you already run a TIP that ingests OTX, MISP, and abuse.ch, add this repo as a separate source-of-record rather than merging it into an existing feed. Per-source provenance lets you measure each source's hit rate, false-positive rate, and uniqueness — and decide if it's worth keeping.