# pancloud-nodejs User Guide

`pancloud-nodejs` is the Palo Alto Networks Application Framework SDK for
Node.js. It is a TypeScript library that wraps the three Cortex Data Lake
services that PAN exposes to third-party tooling — **Logging Service**,
**Event Service**, and **Directory Sync Service** — and adds the OAuth2
plumbing, automatic access-token refresh, and retry logic you would
otherwise have to write yourself.

This guide is for the developer building a Node.js application that needs
to read PAN-OS firewall logs, subscribe to log events, or query Active
Directory data out of a Cortex Data Lake instance. It is organized as a
linear path from the first `npm install` through credential setup,
day-to-day querying patterns, and the failure modes you will hit in
production.

## 1. Quick Start

The fastest way to talk to a Cortex Data Lake from Node.js: install the
package, set three environment variables, and call `autoCredentials()`.
The SDK figures out the credential type from the variables you set, hands
you back a refresh-aware `Credentials` object, and you are ready to query.

### Install

```bash
npm install pancloud-nodejs
```

> [!TIP]
> The package is published as `pancloud` on its `package.json` `name` field
> but is consumed under the GitHub repository name. The TypeScript test
> suite imports it as `from 'pancloud-nodejs'` and that is the import path
> shown throughout this guide.

### A 5-line query

Save this as `quickstart.ts`:

```typescript
import { autoCredentials, LoggingService } from 'pancloud-nodejs'

async function main(): Promise<void> {
    const cred = await autoCredentials()
    const ls = LoggingService.factory(cred)
    const job = await ls.query({
        query: "SELECT * FROM panw.traffic LIMIT 10",
        startTime: Math.floor(Date.now() / 1000) - 3600,
        endTime: Math.floor(Date.now() / 1000)
    })
    console.log(`Query ${job.queryId}: ${job.queryStatus}`)
}

main().catch(e => console.error(e))
```

Provide credentials via environment variables (the simplest path):

```bash
export PAN_CLIENT_ID="<your-oauth2-client-id>"
export PAN_CLIENT_SECRET="<your-oauth2-client-secret>"
export PAN_REFRESH_TOKEN="<your-refresh-token>"
export PAN_ENTRYPOINT="https://api.us.paloaltonetworks.com"
node --loader ts-node/esm quickstart.ts
```

Expected output:

```text
PANCLOUD: {"source":"AutoCredentials","message":"Using memory based credentials provider"}
PANCLOUD: {"source":"defaultCredentialsFactory","message":"Got 'client_id'"}
PANCLOUD: {"source":"defaultCredentialsFactory","message":"Got 'client_secret'"}
PANCLOUD: {"source":"LoggingService","message":"Creating new LoggingService object for entryPoint https://api.us.paloaltonetworks.com"}
Query 1234abcd-...: RUNNING
```

> [!NOTE]
> The query returns a `JobResult` that may be in `RUNNING`, `FINISHED`,
> `JOB_FINISHED`, `JOB_FAILED`, or `CANCELLED` state. For the synchronous
> result, see `LoggingServiceClient` (the streaming wrapper) or the
> auto-poll `callBack` pattern shown later in this guide.

### What you get

- **OAuth2 management** — the SDK refreshes the access token automatically
  when it is within 5 minutes of expiry (`accTokenGuardTime` default).
- **Three service clients** — `LoggingService`, `EventService`,
  `DirectorySyncService`, all reached via a `factory(cred)` static method.
- **Retry-with-backoff** — every HTTP call is wrapped in a `retrier`
  (default 3 attempts, 100 ms apart).
- **Async log streaming** — `LoggingServiceClient` is a Node `Readable`
  stream that yields log records as they become available.
- **Multi-datalake / multi-tenant** — `CortexHubHelper` and
  `CortexCredentialProvider` cover the OAuth2 authorization-code flow plus
  per-tenant credential storage.

### What to try next

- Section 2 walks through the supported install methods.
- Section 3 covers the four credential strategies the SDK ships with.
- Section 4 has the full surface area — every public class and its main
  methods.
- Section 5 has copy-paste recipes for the eight workflows most apps need.

## 2. Installation

### Requirements

| Component | Minimum | Notes |
|---|---|---|
| Node.js | 8.x or later | TypeScript compile target is `es2017` (`tsconfig.json`); native `async`/`await` and `Symbol.iterator` are required. |
| TypeScript | 2.7+ | Optional. The package ships JavaScript built into `dist/` and TypeScript declarations alongside. |
| OS | Linux, macOS, Windows | Pure JavaScript with `https`, `crypto`, `fs`, `stream`, `events`, `url`, `querystring` from the Node standard library. No native modules. |
| Network | Outbound HTTPS to `*.paloaltonetworks.com` | The SDK calls `https://api.us.paloaltonetworks.com` (Americas) or `https://api.eu.paloaltonetworks.com` (Europe), plus `https://api.paloaltonetworks.com` for IDP. |

> [!IMPORTANT]
> You need OAuth2 credentials issued by Palo Alto Networks before any of
> this works. That means a registered application in the Cortex hub plus a
> `client_id`, a `client_secret`, and either a `refresh_token` (for
> long-lived headless usage) or a developer token (for quick testing). The
> SDK does not issue credentials; it only consumes them.

### Install methods

=== "npm"
    ```bash
    npm install pancloud-nodejs
    ```

=== "yarn"
    ```bash
    yarn add pancloud-nodejs
    ```

=== "pnpm"
    ```bash
    pnpm add pancloud-nodejs
    ```

=== "From source"
    ```bash
    git clone https://github.com/PaloAltoNetworks/pancloud-nodejs.git
    cd pancloud-nodejs
    npm run build:library
    ```

The `build:library` script empties `dist/`, runs `tsc -p tsconfig.json`,
and emits CommonJS modules with declaration files alongside.

### Verify the install

Once installed, this one-liner confirms the import path works:

```bash
node -e "console.log(Object.keys(require('pancloud-nodejs')).slice(0, 6))"
```

Expected output:

```text
[
  'defaultCredentialsFactory',
  'DevTokenCredentials',
  'autoCredentials',
  'LoggingService',
  'LoggingServiceClient',
  'EventService'
]
```

> [!NOTE]
> The package's `package.json` declares `"main": "dist/index.js"` and has
> an empty `dependencies` block — every external API is satisfied by the
> Node standard library. Your bundle gets no transitive runtime
> dependencies from this SDK.

### TypeScript projects

For TypeScript consumers, no `@types/*` package is required — type
declarations ship in the package. The `tsconfig.json` recommended for an
SDK consumer matches the test suite shipped with this repo:

```json
{
    "compilerOptions": {
        "noImplicitAny": true,
        "strictNullChecks": true,
        "target": "es2017",
        "module": "commonjs"
    }
}
```

> [!WARNING]
> The `Symbol.iterator` pattern that `EventService` exposes (`for (let p
> of es) { ... }`) requires an iteration target of `es2015` or later. If
> you set `target: "es5"` you must also enable
> `"downlevelIteration": true` to get the same behaviour.

## 3. Configuration

The only thing the SDK has to be told is **how to get an access token**.
Everything else (retry counts, timeouts, log level, channel ID) has a
sensible default.

### The four credential strategies

The SDK ships four ways to produce a `Credentials` object. Pick the one
that matches your deployment shape:

| Strategy | Factory | Use when |
|---|---|---|
| **Auto** | `autoCredentials()` | Hands-off setup. SDK reads env vars and picks the best path. |
| **Memory provider** (refresh token) | `defaultCredentialsProviderFactory()` | Long-lived headless service, refresh token in memory only. |
| **Filesystem provider** | `fsCredentialsFactory()` | Headless service with persistent encrypted credential cache for multiple data lakes. |
| **Developer token** | `new DevTokenCredentials()` | Quick scripts, demos, the `apiexplorer.rocks` developer token flow. |
| **Static** | `defaultCredentialsFactory()` | Tests where you have a still-valid `access_token` in hand. No refresh. |

### Environment variables

`autoCredentials()` and the two factory functions read from `process.env`.
The names are fixed:

| Variable | Read by | Purpose |
|---|---|---|
| `PAN_CLIENT_ID` | memory + filesystem providers, `autoCredentials` | OAuth2 `client_id` |
| `PAN_CLIENT_SECRET` | memory + filesystem providers, `autoCredentials` | OAuth2 `client_secret` |
| `PAN_REFRESH_TOKEN` | memory provider, `autoCredentials` | OAuth2 `refresh_token` |
| `PAN_ACCESS_TOKEN` | `autoCredentials` (static fallback) | Pre-fetched JWT access token |
| `PAN_ENTRYPOINT` | all credential paths | Regional API endpoint (`https://api.us.paloaltonetworks.com` or `https://api.eu.paloaltonetworks.com`) |
| `PAN_DEVELOPER_TOKEN` | `DevTokenCredentials` | Token issued by the developer-token provider |
| `PAN_DEVELOPER_TOKEN_PROVIDER` | `DevTokenCredentials` | Override the default `https://app.apiexplorer.rocks/request_token` URL |

> [!IMPORTANT]
> Treat `PAN_CLIENT_SECRET` and `PAN_REFRESH_TOKEN` as long-lived secrets.
> A leaked refresh token means full access to your Cortex Data Lake until
> someone manually revokes it. Store them in a vault, inject at runtime,
> never commit to source control.

### The two regional entry points

The SDK exports a string-literal type `EntryPoint` with exactly two
allowed values:

```typescript
export type EntryPoint =
    'https://api.eu.paloaltonetworks.com' |
    'https://api.us.paloaltonetworks.com'
```

A convenience map `region2EntryPoint` translates `'americas'` and
`'europe'` to the corresponding URLs.

> [!TIP]
> If you set `PAN_ENTRYPOINT` to anything else, the SDK still accepts it
> (it casts `string` to `EntryPoint`) but you have left the supported
> matrix. Stick to one of the two literal values unless PAN issues a new
> regional API endpoint.

### Common knobs (all optional)

Both `EventService.factory()` and `LoggingService.factory()` accept the
same `EmitterOptions` configuration object, which extends `CoreOptions`:

| Option | Type | Default | Description |
|---|---|---|---|
| `level` | `LogLevel` | `LogLevel.INFO` | Minimum severity that the SDK's internal logger prints. |
| `retrierCount` | `number` | `3` | Attempts per HTTP call before failing. |
| `retrierDelay` | `number` (ms) | `100` | Delay between retry attempts. |
| `fetchTimeout` | `number` (ms) | unset | Per-request timeout passed to the underlying `https` request. |
| `allowDup` | `boolean` | `false` | Allow registering the same event listener multiple times. |
| `l2Corr` | `object` | unset | Enable the L2/L3 correlation engine (Event Service only). |
| `channelId` | `string` | `"EventFilter"` | Event Service channel ID (`EventService` only). |
| `autoPollSleep` | `number` (ms) | `200` | Delay between auto-poll iterations. |

For everything else, see the Configuration Reference in the Admin Guide.

## 4. Core Usage

This section walks through the four pieces every application touches:
credentials, the three service clients, and the iteration patterns the
clients support.

### 4.1 Credentials

Every service factory takes a `Credentials` instance. `Credentials` is
abstract — the SDK ships four concrete subclasses plus a factory for
each.

#### `autoCredentials()` — let the SDK pick

The simplest entry point. It inspects environment variables and chooses:

1. If `PAN_CLIENT_ID` + `PAN_CLIENT_SECRET` + `PAN_REFRESH_TOKEN` are all
   set, it uses the **memory provider** (refresh-token flow).
2. Else if `PAN_ACCESS_TOKEN` is set, it builds a **static credentials**
   object (no refresh).
3. Else it falls back to **developer token** mode and reads
   `PAN_DEVELOPER_TOKEN`.

```typescript
import { autoCredentials } from 'pancloud-nodejs'

const cred = await autoCredentials()
// cred is now a Credentials instance ready to hand to any factory
```

You can override individual fields without setting environment variables:

```typescript
const cred = await autoCredentials({
    accessToken: '<jwt>',
    entryPoint: 'https://api.eu.paloaltonetworks.com'
})
```

#### `defaultCredentialsProviderFactory()` — explicit memory provider

When you want the refresh-token flow but do not want it picked from env:

```typescript
import { defaultCredentialsProviderFactory } from 'pancloud-nodejs'

const cred = await defaultCredentialsProviderFactory({
    clientId:     '<oauth2-client-id>',
    clientSecret: '<oauth2-client-secret>',
    refreshToken: '<oauth2-refresh-token>',
    entryPoint:   'https://api.us.paloaltonetworks.com'
})
```

Internally this returns a `Credentials` bound to a one-datalake
`DefaultCredentialsProvider` instance with the fixed datalake ID
`'DEFAULT'`.

#### `fsCredentialsFactory()` — encrypted file cache

For headless services that want refresh tokens persisted across restarts.
Tokens are stored in `${PAN_PREFIX}_PANCLOUD_CONFIG.json` (default
`PAN_PANCLOUD_CONFIG.json`) encrypted with `aes128` derived from the
secret you provide:

```typescript
import { fsCredentialsFactory } from 'pancloud-nodejs'

const provider = await fsCredentialsFactory({
    clientId:     process.env.PAN_CLIENT_ID!,
    clientSecret: process.env.PAN_CLIENT_SECRET!,
    secret:       process.env.PAN_FS_SECRET!  // encryption key (any string)
})

const cred = await provider.registerManualDatalake(
    'datalake-1',
    'https://api.us.paloaltonetworks.com',
    '<refresh-token>'
)
```

#### `DevTokenCredentials` — quick demos

Useful with the `apiexplorer.rocks` flow if you have a developer token:

```typescript
import { DevTokenCredentials } from 'pancloud-nodejs'

const cred = new DevTokenCredentials({
    developerToken: process.env.PAN_DEVELOPER_TOKEN,
    entryPoint:     'https://api.us.paloaltonetworks.com'
})
await cred.retrieveAccessToken()
```

#### `defaultCredentialsFactory()` — static, no refresh

For tests where you already have a fresh JWT in hand:

```typescript
import { defaultCredentialsFactory } from 'pancloud-nodejs'

const cred = defaultCredentialsFactory(
    'https://api.us.paloaltonetworks.com',
    '<eyJ...JWT...>'
)
```

> [!WARNING]
> `defaultCredentialsFactory()` returns a `StaticCredentials` instance.
> Its `retrieveAccessToken()` is a no-op — when the JWT expires, queries
> will start failing with 401 errors. Use it for fixtures only.

#### Automatic refresh

Every concrete `Credentials` subclass implements `autoRefresh()`. The
service clients call it before every HTTP request via
`CoreClass.checkAutoRefresh()`. If the access token is within
`guardTime` (5 minutes by default) of expiry, the SDK calls
`retrieveAccessToken()` and updates the `Authorization` header
transparently.

```typescript
// You almost never call these directly; the service clients do.
await cred.getAccessToken()      // Returns the current access token
await cred.getExpiration()       // Returns the unix epoch validity
await cred.autoRefresh()         // Refreshes if needed, returns validUntil
cred.getEntryPoint()             // Returns the regional entry point
```

### 4.2 LoggingService — the main query API

`LoggingService` runs SQL-style queries against the Cortex Data Lake's log
indexes (`panw.traffic`, `panw.threat`, `panw.system`, etc.) and returns
results in batches.

#### Synchronous query (poll-yourself)

```typescript
import { autoCredentials, LoggingService } from 'pancloud-nodejs'

const cred = await autoCredentials()
const ls = LoggingService.factory(cred)

const job = await ls.query({
    query:     "SELECT * FROM panw.threat WHERE risk-of-app > 3",
    startTime: 1577836800,
    endTime:   1580515200,
    maxWaitTime: 30000
})

console.log(`queryId: ${job.queryId}, status: ${job.queryStatus}`)

// If the result didn't arrive on the first call, poll until done:
let next = job
while (next.queryStatus !== 'JOB_FINISHED' && next.queryStatus !== 'JOB_FAILED') {
    next = await ls.poll(next.queryId, next.sequenceNo + 1, 30000)
}
```

#### Asynchronous query with callback (the recommended pattern)

When you provide a `callBack.event` handler, the SDK starts an
auto-poller in the background. Every time a sequence of results is ready,
your handler is called with an `EmitterInterface<any[]>` containing the
records.

```typescript
import { autoCredentials, LoggingService, EmitterInterface } from 'pancloud-nodejs'

const cred = await autoCredentials()
const ls = LoggingService.factory(cred)

const completed = await ls.query({
    query:     "SELECT * FROM panw.traffic",
    startTime: Math.floor(Date.now() / 1000) - 3600,
    endTime:   Math.floor(Date.now() / 1000),
    logType:   'panw.traffic',
    callBack: {
        event: (e: EmitterInterface<any[]>) => {
            if (e.message) {
                console.log(`Got ${e.message.length} ${e.logType} records`)
            }
        }
    }
})
console.log(`Job ${completed.queryId} done, ${ls.getLsStats().records} records total`)
```

#### Streaming query with `LoggingServiceClient`

`LoggingServiceClient` extends Node's `Readable` stream interface. It
lazy-starts the query on the first `_read()` and pushes each batch as a
single chunk in object mode:

```typescript
import { autoCredentials, LoggingService, LoggingServiceClient } from 'pancloud-nodejs'

const cred = await autoCredentials()
const ls = LoggingService.factory(cred)

const stream = new LoggingServiceClient(ls, {
    query:     "SELECT * FROM panw.traffic",
    startTime: Math.floor(Date.now() / 1000) - 3600,
    endTime:   Math.floor(Date.now() / 1000),
    retries:   20,        // attempts before giving up on RUNNING state
    delay:     200        // ms between RUNNING-state polls
})

stream.on('data', (records: any[]) => {
    console.log(`stream chunk: ${records.length} records`)
})
stream.on('end', () => console.log('stream finished'))
stream.on('error', e => console.error('stream error:', e))
```

#### Job statuses

Every result carries a `queryStatus` field:

| Status | Meaning |
|---|---|
| `RUNNING` | Job dispatched, no results yet. Keep polling. |
| `FINISHED` | This sequence delivered; more sequences are coming. Increment `sequenceNo` and poll again. |
| `JOB_FINISHED` | All sequences delivered; you are done. |
| `JOB_FAILED` | Server-side failure. Record the `queryId` and stop polling. |
| `CANCELLED` | Caller invoked `cancelPoll()` or `deleteQuery()`. |

#### Other methods

| Method | What it does |
|---|---|
| `ls.poll(qid, seqNo, maxWaitTime?)` | Fetch the next sequence of a running query. |
| `ls.deleteQuery(queryId)` | Cancel a query on the server side. |
| `ls.cancelPoll(queryId, err?)` | Remove a query from the auto-poll queue and call `deleteQuery`. |
| `ls.write(vendorName, logType, data[])` | Push log records into the service. Requires registered vendor + log type. |
| `ls.getLsStats()` | Return per-instance counters: queries, polls, deletes, records, writes, plus inherited Emitter stats. |

### 4.3 EventService — the streaming subscription API

Where `LoggingService` runs ad-hoc SQL, `EventService` opens a long-lived
**channel** (default name `EventFilter`) into which the Application
Framework publishes matching events as they happen. Your code applies a
**filter** to that channel — a list of tables and optional WHERE clauses
— and then **polls** for events.

#### Set a filter and poll synchronously

```typescript
import { autoCredentials, EventService, EsFilterBuilderCfg } from 'pancloud-nodejs'

const builderCfg: EsFilterBuilderCfg = {
    filter: [
        { table: "panw.traffic", timeout: 1000, batchSize: 8000 },
        { table: "panw.dpi",     timeout: 1000, batchSize: 8000 },
        { table: "panw.threat",  where: 'where risk-of-app > 3' }
    ],
    flush: false,
    filterOptions: {}
}

const cred = await autoCredentials()
const es = EventService.factory(cred)
await es.filterBuilder(builderCfg)

const events = await es.poll()
events.forEach(e => {
    console.log(`Event Type: ${e.logType}, Record Count: ${e.event.length}`)
})
```

#### Set a filter and stream asynchronously

When you provide a `callBack` in `filterOptions`, the SDK auto-polls and
pushes each event batch through the handler:

```typescript
const builderCfg: EsFilterBuilderCfg = {
    filter: [
        { table: "panw.traffic", timeout: 1000 },
        { table: "panw.threat",  where: 'where risk-of-app > 3' }
    ],
    filterOptions: {
        callBack: {
            event: e => {
                if (e.message) console.log(`got ${e.message.length} ${e.logType} events`)
            }
        },
        poolOptions: { ack: true, pollTimeout: 1000 }
    }
}

const es = EventService.factory(cred)
await es.filterBuilder(builderCfg)
// Auto-poll starts in the background; let it run for 60 seconds:
await new Promise<void>(resolve => setTimeout(resolve, 60000))
es.pause()
await es.clearFilter(true)
console.log(JSON.stringify(es.getEsStats(), undefined, ' '))
```

#### Iterating without a callback

`EventService` implements `Iterable<Promise<EsEvent[]>>`. You can drive
the polling loop directly with `for...of`:

```typescript
const es = EventService.factory(cred)
await es.filterBuilder(builderCfg)
let iterations = 10
for (const promise of es) {
    if (iterations-- === 0) break
    const response = await promise
    response.forEach(e => console.log(`${e.event.length} ${e.logType} events`))
}
await es.clearFilter()
```

#### Channel control methods

| Method | Description |
|---|---|
| `es.filterBuilder(cfg)` | Build and apply a filter from a friendly config. |
| `es.setFilters(cfg)` | Lower-level: apply an `EsFilterCfg` directly. |
| `es.getFilters()` | Return the current filter set. |
| `es.clearFilter(flush?)` | Replace the filter with an empty one (and optionally flush). |
| `es.poll()` | Fetch one batch of events. |
| `es.ack()` | Acknowledge the current batch (mark as consumed). |
| `es.nack()` | Negative acknowledge — leave the batch on the channel. |
| `es.flush()` | Discard everything currently buffered server-side. |
| `es.pause()` | Stop the auto-poll loop without changing the filter. |
| `es.resume()` | Restart auto-poll after `pause()`. |
| `es.getEsStats()` | Per-instance counters: polls, acks, nacks, flushes, filtersets, filtergets. |

> [!WARNING]
> The Event Service rejects the `tms.analytics`, `tms.config`,
> `tms.system`, and `tms.threat` log types — only `tms.traps` is allowed
> from the `tms.*` namespace. `filterBuilder()` validates this before the
> HTTP call and throws `PanCloudError(CONFIG)` if you pass an invalid
> table.

### 4.4 DirectorySyncService — Active Directory data

`DirectorySyncService` exposes the directory inventory (computers,
containers, groups, organizational units, users) that the
PAN-OS-integrated Directory Sync agent has uploaded.

```typescript
import { autoCredentials, DirectorySyncService } from 'pancloud-nodejs'

const cred = await autoCredentials()
const dss = await DirectorySyncService.factory(cred)

const domains = await dss.domains()
console.log(`${domains.length} domains synced`)

const userCount = await dss.count(domains[0].dns, 'users')
console.log(`${userCount} users in ${domains[0].dns}`)

const userPage = await dss.query('users', {
    domain: domains[0].dns,
    pageNumber: 1,
    pageSize: 100
})
console.log(`page 1 has ${userPage.result.length} entries`)
```

The five object classes are exposed as the type `DssObjClass`:

```typescript
type DssObjClass = "containers" | "computers" | "ous" | "groups" | "users"
```

| Method | Endpoint | Returns |
|---|---|---|
| `dss.attributes()` | `GET /attributes` | The schema for each object class. |
| `dss.domains()` | `GET /domains` | Synced domains with status timestamps. |
| `dss.count(domain, objClass)` | `GET /{objClass}/count?domain=...` | Entry count for that class. |
| `dss.query(objClass, filter?)` | `POST /{objClass}` | Paginated entries matching filter. |
| `dss.getDssStats()` | (local) | Counters: queryCalls, domainCalls, attributeCalls, countCalls. |

`DssQueryFilter` supports name-based matching with operators
`startWith`, `contain`, `equal` against attributes
`Common-Name`, `Distinguished Name`, `Name`, `SAM Account Name`,
`SID`, `User Principal Name`, plus pagination via `pageNumber` and
`pageSize` (max 1000 per page).

### 4.5 Logging the SDK itself

Every operation logs through a single `commonLogger` instance:

```typescript
import { setLogLevel, setLogger, LogLevel } from 'pancloud-nodejs'

setLogLevel(LogLevel.DEBUG)   // Print everything

// Or replace the logger entirely:
setLogger({
    level: LogLevel.INFO,
    error: (e) => myMonitoring.report(e),
    alert: (src, msg, name) => console.warn(`[${src.className}] ${msg}`),
    info:  (src, msg, name) => {},
    debug: (src, msg, name, payload) => {}
})
```

`LogLevel` values: `DEBUG = 0`, `INFO = 1`, `ALERT = 2`, `ERROR = 3`. The
default is `INFO`.

> [!TIP]
> Output lines are prefixed with `PANCLOUD: ` and contain a JSON object
> of `{source, message, ...}`. Pipe stdout through `jq` or your log
> aggregator's JSON parser to slice them up.

## 5. Common Tasks

This section is a copy-paste cookbook for the eight workflows that show
up in nearly every integration.

### 5.1 Run a one-shot logs query and print the records

**Goal:** retrieve traffic logs from the last hour, log how many you got,
exit.

**Prerequisites:** `PAN_CLIENT_ID`, `PAN_CLIENT_SECRET`,
`PAN_REFRESH_TOKEN`, `PAN_ENTRYPOINT` set in environment.

**Steps:**

```typescript
import { autoCredentials, LoggingService, EmitterInterface } from 'pancloud-nodejs'

async function main(): Promise<void> {
    const cred = await autoCredentials()
    const ls = LoggingService.factory(cred)

    let totalRecords = 0
    await ls.query({
        query:     "SELECT * FROM panw.traffic LIMIT 1000",
        startTime: Math.floor(Date.now() / 1000) - 3600,
        endTime:   Math.floor(Date.now() / 1000),
        logType:   'panw.traffic',
        callBack: {
            event: (e: EmitterInterface<any[]>) => {
                if (e.message) {
                    totalRecords += e.message.length
                    console.log(`batch of ${e.message.length} (running total ${totalRecords})`)
                }
            }
        }
    })
    console.log(`done, ${totalRecords} records`)
}

main().catch(e => console.error(e))
```

**Verify:** the script prints one or more `batch of N` lines, then
`done, N records`. The `LoggingService` stats reflect the same totals via
`ls.getLsStats()`.

**If it fails:** start by raising the log level (`setLogLevel(LogLevel.DEBUG)`).
A 401 means the refresh token is bad — re-issue from the Cortex hub. A
`PARSER` error means the endpoint returned non-JSON, usually a transient
upstream issue; the SDK already retried 3 times.

### 5.2 Stream logs to a file as a Node Readable

**Goal:** pipe a query result straight to disk without buffering it in
memory.

**Steps:**

```typescript
import { createWriteStream } from 'fs'
import { autoCredentials, LoggingService, LoggingServiceClient } from 'pancloud-nodejs'

const cred = await autoCredentials()
const ls = LoggingService.factory(cred)

const stream = new LoggingServiceClient(ls, {
    query:     "SELECT * FROM panw.threat",
    startTime: Math.floor(Date.now() / 1000) - 86400,
    endTime:   Math.floor(Date.now() / 1000)
})

const out = createWriteStream('threats.ndjson', { encoding: 'utf8' })
stream.on('data', (records: any[]) => {
    for (const r of records) out.write(JSON.stringify(r) + '\n')
})
stream.on('end', () => out.end())
stream.on('error', e => { out.end(); console.error(e) })
```

**Verify:** `wc -l threats.ndjson` matches `ls.getLsStats().records`.

**If it fails:** `LoggingServiceClient` throws if the query stays in
`RUNNING` for more than `retries` polls (default 20, 200 ms apart =
4 seconds). Pass a higher `retries` for very large queries.

### 5.3 Subscribe to live traffic events and ack each batch

**Goal:** keep a long-running process consuming `panw.traffic` events,
acknowledging each batch so the channel does not back up.

**Steps:**

```typescript
import { autoCredentials, EventService, EsFilterBuilderCfg } from 'pancloud-nodejs'

const cred = await autoCredentials()
const es = EventService.factory(cred)

const cfg: EsFilterBuilderCfg = {
    filter: [{ table: "panw.traffic", timeout: 1000, batchSize: 8000 }],
    filterOptions: {
        callBack: {
            event: e => console.log(`${e.message?.length ?? 0} events at ${new Date().toISOString()}`)
        },
        poolOptions: { ack: true, pollTimeout: 1000 }
    }
}
await es.filterBuilder(cfg)

process.on('SIGINT', async () => {
    es.pause()
    await es.clearFilter(true)
    process.exit(0)
})
```

**Verify:** the process prints a line every poll iteration. `es.getEsStats()`
shows `polls` climbing and `acks` matching it.

**If it fails:** if you see no output, the filter probably has no
matching traffic. Drop the WHERE clause and try `panw.system`, which
is always populated.

### 5.4 Page through Active Directory users

**Goal:** enumerate every user in a synced domain, 500 at a time.

**Steps:**

```typescript
import { autoCredentials, DirectorySyncService } from 'pancloud-nodejs'

const cred = await autoCredentials()
const dss = await DirectorySyncService.factory(cred)
const domains = await dss.domains()
const dom = domains[0].dns

const total = await dss.count(dom, 'users')
console.log(`paging ${total} users in ${dom}`)

let page = 1
while (true) {
    const resp = await dss.query('users', { domain: dom, pageNumber: page, pageSize: 500 })
    if (resp.result.length === 0) break
    console.log(`page ${page}: ${resp.result.length} entries (count=${resp.count})`)
    if (resp.unreadResults === undefined || resp.unreadResults === 0) break
    page++
}
```

**Verify:** the sum of `result.length` across pages equals the original
`count`.

**If it fails:** a 4xx response often means the directory hasn't been
synced yet — `dss.domains()[0].status.statusUpdateTime` will be 0.

### 5.5 Spin the SDK up for a quick test with a static token

**Goal:** validate connectivity in a 30-second test without provisioning
a refresh-token flow.

**Steps:**

```typescript
import { defaultCredentialsFactory, LoggingService } from 'pancloud-nodejs'

const cred = defaultCredentialsFactory(
    'https://api.us.paloaltonetworks.com',
    process.env.PAN_ACCESS_TOKEN!
)

const ls = LoggingService.factory(cred)
const job = await ls.query({
    query:     "SELECT * FROM panw.traffic LIMIT 1",
    startTime: Math.floor(Date.now() / 1000) - 600,
    endTime:   Math.floor(Date.now() / 1000)
})
console.log(`status: ${job.queryStatus}, queryId: ${job.queryId}`)
```

**Verify:** `queryStatus` is one of `RUNNING`, `FINISHED`, `JOB_FINISHED`.

**If it fails:** if the SDK throws `CONFIG: not a valid JWT access
token`, the value in `PAN_ACCESS_TOKEN` is missing the three
dot-separated segments. Re-fetch a fresh token.

### 5.6 Enable the L2/L3 correlation engine

**Goal:** receive synthesized correlation events that join L2 (DPI / MAC)
and L3 (traffic / threat) flows by `sessionid`.

**Prerequisites:** filter must include both DPI events for L2 and
`panw.traffic` or `panw.threat` for L3.

**Steps:**

```typescript
import { autoCredentials, EventService, EsFilterBuilderCfg, L2correlation, EmitterInterface } from 'pancloud-nodejs'

const cred = await autoCredentials()
const es = EventService.factory(cred, {
    l2Corr: { timeWindow: 120, absoluteTime: false, gcMultiplier: 0 }
})

const cfg: EsFilterBuilderCfg = {
    filter: [
        { table: "panw.traffic", timeout: 1000 },
        { table: "panw.dpi",     timeout: 1000 }
    ],
    filterOptions: {
        callBack: {
            corr: (e: EmitterInterface<L2correlation[]>) => {
                if (e.message) {
                    console.log(`${e.message.length} correlation events`)
                    e.message.forEach(x => console.log(`  ${x.src} <-> ${x.dst} mac=${x["extended-traffic-log-mac"]}`))
                }
            }
        },
        poolOptions: { ack: true, pollTimeout: 1000 }
    }
}
await es.filterBuilder(cfg)

setTimeout(async () => {
    es.pause()
    await es.clearFilter(true)
    console.log(JSON.stringify(es.getEsStats(), undefined, ' '))
}, 120000)
```

**Verify:** `es.getEsStats().correlationStats` shows `dbInserts > 0` and
`dbWaterMark > 0`.

**If it fails:** without `panw.dpi` events the engine has no L2 data and
will never produce correlations. Make sure the firewall sending data is
configured to forward extended-application logs.

### 5.7 Decode PCAP payloads from threat events

**Goal:** save every PCAP payload embedded in a `panw.threat` event to a
separate `.pcap` file.

**Steps:**

```typescript
import { writeFileSync } from 'fs'
import { autoCredentials, EventService, EsFilterBuilderCfg, EmitterInterface } from 'pancloud-nodejs'

const cred = await autoCredentials()
const es = EventService.factory(cred)

let n = 0
const cfg: EsFilterBuilderCfg = {
    filter: [{ table: "panw.threat", timeout: 1000 }],
    filterOptions: {
        callBack: {
            pcap: (p: EmitterInterface<Buffer>) => {
                if (p.message) {
                    const name = `pcap${('00' + n++).slice(-3)}.pcap`
                    writeFileSync(name, p.message)
                    console.log(`wrote ${name} (${p.message.length} bytes)`)
                }
            }
        },
        poolOptions: { ack: true, pollTimeout: 1000 }
    },
    flush: true
}
await es.filterBuilder(cfg)
```

**Verify:** files appear on disk; each is a valid libpcap file.

**If it fails:** `Util.pcaptize()` returns `null` if the event has no
`pcap` property, so no file is written. Switch to a higher-volume table
like `panw.dpi` to confirm the pipeline before debugging the threat
filter.

### 5.8 Turn on verbose tracing

**Goal:** see every HTTP request the SDK makes, with the JSON payload.

**Steps:**

```typescript
import { setLogLevel, LogLevel, autoCredentials, EventService } from 'pancloud-nodejs'

setLogLevel(LogLevel.DEBUG)

const cred = await autoCredentials()
const es = EventService.factory(cred)
const f = await es.getFilters()
console.log(JSON.stringify(f, undefined, ' '))
```

**Verify:** stdout now shows `PANCLOUD: ...` debug lines for header
updates, fetches, and JSON payloads (truncated to 300 bytes).

**If it fails:** if you replaced the logger via `setLogger()`, your
custom `debug` implementation may be swallowing messages. Confirm by
reverting to the default `commonLogger` first.

## 6. Troubleshooting

### Diagnostic decision tree

1. **Does `autoCredentials()` succeed?** If not, check env vars (`PAN_CLIENT_ID` etc.) and `PAN_ENTRYPOINT`.
2. **Does the first HTTP call succeed?** If you get `IDENTITY` or 401, the refresh token is invalid or the entry point is wrong.
3. **Is the query returning records?** Confirm with `setLogLevel(LogLevel.DEBUG)` and look at the JSON payload echoed in the logs.

### Error: `Environment variable PAN_CLIENT_ID not found or empty value`

**Cause:** `defaultCredentialsProviderFactory()` (and therefore
`autoCredentials()` in memory mode) requires `PAN_CLIENT_ID`,
`PAN_CLIENT_SECRET`, `PAN_REFRESH_TOKEN`, and `PAN_ENTRYPOINT` to all be
present.

**Solution:**
1. `printenv | grep PAN_` to see what your shell has.
2. If you are inside Docker, confirm the variables made it through `docker run -e PAN_...` or `env_file:`.
3. As a fallback, pass the values directly: `defaultCredentialsProviderFactory({clientId, clientSecret, refreshToken, entryPoint})`.

**Prevention:** validate all four variables at process start and exit
non-zero with a clear message before constructing the SDK.

### Error: `not a valid JWT access token`

**Cause:** `defaultCredentialsFactory()` and `StaticCredentials` decode
the JWT to extract its `exp` claim. The token must have three
dot-separated segments and a base64-encoded payload with a numeric `exp`.

**Solution:**
1. Verify the token has three segments: `echo $TOKEN | tr '.' '\n' | wc -l` should print `3`.
2. Decode the middle segment: `echo "<segment>" | base64 -d | jq .` and confirm `exp` is a number.

**Prevention:** do not log or copy access tokens through systems that
strip or wrap whitespace; even a stray newline corrupts the JWT.

### Error: `PanCloudError() only "tms.traps" is accepted in the EventService`

**Cause:** `EventService.filterBuilder()` rejects `tms.analytics`,
`tms.config`, `tms.system`, and `tms.threat` because the channel does
not deliver them.

**Solution:** drop those tables from your filter. If you need
`tms.threat`, query it through `LoggingService` instead.

**Prevention:** use the `LogType` TypeScript type for filter tables —
the compiler will not stop you from passing invalid values today, but
keeping the table list typed makes future SDK upgrades surface the
mismatch.

### Error: `Property 'accTokenGuardTime' must be, at max 3300 seconds`

**Cause:** every credentials object refuses a guard time greater than
55 minutes because Cortex access tokens are typically 1-hour bearers.

**Solution:** set `accTokenGuardTime` (or the `Credentials` constructor's
`guardTime`) to a value below `3300`. The default of `300` (5 minutes)
is appropriate for almost all deployments.

**Prevention:** if you find yourself wanting longer, you are likely
attempting to keep tokens alive across a long blocking call. Refactor to
release the credential object during long sleeps and let the next call
re-fetch.

### Error: `Still in RUNNING state after 20 attempts`

**Cause:** `LoggingServiceClient` polls a query 20 times at 200 ms
spacing. If the query is still computing, it gives up and deletes the
query.

**Solution:**
1. For very large queries, raise `retries` and `delay` in the
   `LoggingServiceClient` constructor.
2. Switch to the auto-poll callback pattern; it has no built-in retry
   ceiling and runs as long as the query lives on the server.

**Prevention:** for queries that span months of data, narrow the
`startTime`/`endTime` window or add `LIMIT` to the SQL.

### Error: HTTP 401 mid-stream

**Cause:** the access token expired and the refresh attempt failed
(e.g. the refresh token was revoked).

**Solution:**
1. Re-issue a refresh token from the Cortex hub.
2. Restart the process so `autoCredentials()` picks up the new value.
3. For long-running services, watch for repeated `IDENTITY` errors and
   surface them to your alerting layer; do not silently retry forever.

**Prevention:** rotate refresh tokens deliberately on a known cadence,
not by waiting for failures. Use `fsCredentialsFactory()` so the new
token persists across restarts without code changes.

### Issue #30: Community Health Assessment

The PaloAltoNetworks/pancloud-nodejs repository tracks community-health
items under [issue #30](https://github.com/PaloAltoNetworks/pancloud-nodejs/issues/30).
If you encounter behaviour that contradicts what this guide describes,
check that issue (and the project's broader issue tracker) before filing
a new ticket — the active maintainer cadence and contribution channels
are documented there.

> [!CAUTION]
> The repository's last source commit captured here is `e0d9e6b` from
> 2019-10-29. The Cortex Data Lake API surface and OAuth2 flow may
> have changed since then. Always cross-check endpoint
> behaviour against the current Cortex Data Lake reference at
> [docs.paloaltonetworks.com](https://docs.paloaltonetworks.com/) before
> debugging unexpected responses.

## 7. FAQ

**Q: Why is the package called `pancloud` in `package.json` but I import it as `pancloud-nodejs`?**

The `package.json` `name` field is `pancloud`, version `1.0.0`. The
GitHub repository is `pancloud-nodejs`, and that is the npm install path.
The TypeScript test suite imports from `'pancloud-nodejs'` and that is
the recommended import path for consumers.

**Q: Do I need a separate `@types/pancloud-nodejs` package?**

No. TypeScript declarations are emitted alongside the JavaScript by
`build:library` (the `tsc -p tsconfig.json` step). `npm install` brings
both.

**Q: How does the SDK refresh access tokens?**

Every `Credentials` subclass has a `guardTime` (default 300 seconds = 5
minutes). Before each HTTP request, the service client calls
`Credentials.autoRefresh()`, which checks whether the cached token
expires within `guardTime`. If yes, it calls `retrieveAccessToken()`,
which posts the refresh token to
`https://api.paloaltonetworks.com/api/oauth2/RequestToken` and stores
the new access token. Static credentials skip this — they are no-ops.

**Q: What is the difference between `LoggingService` and `EventService`?**

`LoggingService` runs ad-hoc SQL queries against historical log indexes
and returns a finite result set in batches. `EventService` opens a
long-lived channel that emits new events as they happen, with optional
filters. Use `LoggingService` for "show me everything between Tuesday
and Thursday"; use `EventService` for "stream me new threat events
forever".

**Q: Can I run multiple service clients off the same `Credentials`?**

Yes. The `Credentials` object is the auth-token cache. You can pass it
into `LoggingService.factory()`, `EventService.factory()`, and
`DirectorySyncService.factory()` simultaneously. Each client manages its
own per-instance state (channel ID, query queue, stats).

**Q: How do I cancel an in-flight query?**

Call `ls.cancelPoll(queryId)` to remove it from the auto-poll queue and
issue a server-side delete, or call `ls.deleteQuery(queryId)` directly
for the same effect on the server side without touching the local queue.

**Q: What does `flush: true` do in an Event Service filter?**

When you set a filter with `flush: true`, the channel discards every
event currently buffered server-side. Use it when you have changed the
filter so significantly that old events are no longer relevant. The
default is `flush: false` (keep buffered events).

**Q: Can I use this SDK in a browser?**

No. The SDK depends on Node's `https`, `crypto`, `fs`, and `events`
modules. There is no browser bundle and no plan to ship one — Cortex
Data Lake credentials should not live in a browser anyway.

**Q: Where is the published TypeScript source?**

The `lib/` directory at the repository root contains the canonical
TypeScript. `tsc` compiles it into `dist/`, which is what `package.json`
points `main` at. There is no separate `src/` versus `dist/` split in
the repo at commit `e0d9e6b`.

**Q: Does the SDK validate JSON responses?**

Yes — every response goes through a hand-written type guard
(`isJobResult`, `isEsFilter`, `isDssDomain`, etc.) before it is exposed
to your code. A response that does not match the expected schema raises
`PanCloudError(PARSER, ...)`.

**Q: Is `tms.traps` the only `tms.*` table I can subscribe to?**

For Event Service, yes. `EventService.filterBuilder()` validates against
the `invalidTables` list (`tms.analytics`, `tms.config`, `tms.system`,
`tms.threat`). All `panw.*` tables are accepted. `LoggingService` has
no such restriction — you can query any indexed table.

**Q: How do I test code that uses this SDK without a real Cortex tenant?**

Use `defaultCredentialsFactory(entryPoint, fakeJwt)` to build a static
credentials object, then mock the underlying `fetch` module. The SDK has
a single internal `fetch` export (`lib/fetch.ts`) that wraps Node's
`https.request`; intercepting it gives you full control over responses.

**Q: How do I support multiple Cortex Data Lake tenants in one process?**

Use `fsCredentialsFactory()` (or implement your own subclass of
`CortexCredentialProvider`) and call `registerManualDatalake()` once per
tenant. Each call returns its own `Credentials` object bound to its own
datalake ID. The provider stores all of them in a single encrypted
`PANCLOUD_CONFIG.json` and refreshes each independently.

**Q: What is the L2/L3 correlation engine?**

It is an optional in-memory join that pairs L3 (IP) flows with L2 (MAC)
flows by `sessionid`. When you pass `l2Corr: { timeWindow, absoluteTime,
gcMultiplier }` to `EventService.factory()`, the SDK buffers L2 and L3
events and emits a synthesized "correlated" event whenever a session
appears in both — see Section 5.6 for the full pattern.

---

_Generated April 2026 from PaloAltoNetworks/pancloud-nodejs at e0d9e6b. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial, AI-generated guide for a Palo Alto Networks open-source repository; not reviewed by Palo Alto Networks._