Jesse A. Green / notebook

Live demo: the DocCompiler guide library as a static, read-only snapshot (October 2026). No backend, no accounts, nothing to submit.

Forge — Administrator & Developer Guide

Generated May 2026 from cisco-open/forge at 1a8b9a7

AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it.

1. Architecture Overview

Forge is Cisco's open-source Infrastructure-as-Code (IaC) framework for deploying and operating multi-tenant, self-hosted GitHub Actions runners on AWS. It provides a modular Terraform/Terragrunt architecture that provisions both EC2-based and Kubernetes-based (ARC) runner backends, along with comprehensive observability, security, and lifecycle management infrastructure.

The framework is designed around three architectural layers:

  • Core — GitHub Actions Runner Controller (ARC) on EKS, including scale set management and Helm-based deployment
  • Infrastructure — Foundational AWS resources: EKS clusters, ECR registries, S3 storage, AMI lifecycle policies, CloudFormation stacks, and cross-account subscription roles
  • Platform — Runner orchestration: EC2 and ARC deployment, GitHub App management, webhook relay, job log archival, trust validation, and dead-letter queue redriving
  • Integrations — Third-party services: Splunk Cloud (data ingestion, dashboards, props/transforms), Splunk Observability (SignalFx, OTel), Teleport (secure access), and GitHub webhook relay (EventBridge cross-account forwarding with Webex notification support)

Architecture Diagram

┌──────────────────────────────────────────────────────────────────────┐
│                        GitHub (GHES / github.com)                    │
│  ┌─────────┐   ┌──────────┐   ┌───────────────┐                    │
│  │ Webhooks │   │ Runner   │   │ GitHub App    │                    │
│  │          │   │ Groups   │   │ (JWT + Install│                    │
│  └────┬─────┘   └─────┬────┘   │  Token Auth)  │                    │
│       │               │        └───────┬───────┘                    │
└───────┼───────────────┼────────────────┼────────────────────────────┘
        │               │                │
        ▼               │                ▼
┌───────────────────────┼──────────────────────────────────────────────┐
│  AWS Account          │                                              │
│                       │                                              │
│  ┌──────────────┐     │     ┌─────────────────────────────────────┐  │
│  │ API Gateway  │─────┼────▶│ Webhook Relay Lambda               │  │
│  │ (webhook     │     │     │ (validate_signature.py)            │  │
│  │  ingress)    │     │     └──────────────┬──────────────────────┘  │
│  └──────────────┘     │                    │                         │
│                       │                    ▼                         │
│                       │     ┌─────────────────────────────────────┐  │
│                       │     │ EventBridge                         │  │
│                       │     │ ┌─────────┐  ┌──────────────────┐  │  │
│                       │     │ │ Source   │─▶│ Destination Bus  │  │  │
│                       │     │ │ Rules    │  │ (cross-account)  │  │  │
│                       │     │ └─────────┘  └──────┬───────────┘  │  │
│                       │     └─────────────────────┼──────────────┘  │
│                       │                           │                  │
│           ┌───────────┴───────────┐               ▼                  │
│           │                       │    ┌──────────────────┐          │
│  ┌────────▼─────┐  ┌─────────────▼┐   │ Webex Relay      │          │
│  │ EC2 Runners  │  │ ARC Runners  │   │ Lambda           │          │
│  │ (ASG/Fleet)  │  │ (EKS Pods)   │   └──────────────────┘          │
│  │ ┌──────────┐ │  │ ┌──────────┐ │                                  │
│  │ │ Linux    │ │  │ │ DinD     │ │   ┌──────────────────────────┐  │
│  │ │ macOS    │ │  │ │ K8s Mode │ │   │ Support Lambdas          │  │
│  │ │ Windows  │ │  │ └──────────┘ │   │ ┌──────────────────────┐ │  │
│  │ └──────────┘ │  └──────────────┘   │ │ trust_validator      │ │  │
│  └──────────────┘                     │ │ ssm_ami_updater      │ │  │
│                                       │ │ runner_tag_updater   │ │  │
│  ┌────────────────────────────────┐   │ │ global_lock_cleaner  │ │  │
│  │ EKS Cluster                   │   │ │ app_runner_group     │ │  │
│  │ ┌────────┐ ┌────────────────┐ │   │ │ job_log_archiver     │ │  │
│  │ │Karpenter│ │ARC Controller │ │   │ │ job_log_dispatcher   │ │  │
│  │ │        │ │(Helm)         │ │   │ │ redrive_deadletter   │ │  │
│  │ └────────┘ └────────────────┘ │   │ └──────────────────────┘ │  │
│  └────────────────────────────────┘   └──────────────────────────┘  │
│                                                                      │
│  ┌───────────────────────────────────────────────────────────────┐   │
│  │ Observability (Splunk)                                        │   │
│  │ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌────────────────────┐│   │
│  │ │Data Mgr  │ │O11y/SFx  │ │OTel EKS │ │Billing (CUR→HEC)  ││   │
│  │ │(CloudFmt)│ │(Dashbds) │ │(Helm)   │ │(S3→Lambda→Splunk) ││   │
│  │ └──────────┘ └──────────┘ └─────────┘ └────────────────────┘│   │
│  └───────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐               │
│  │ ECR      │ │ S3       │ │ SSM      │ │ Secrets  │               │
│  │ (images) │ │ (logs,   │ │ (params) │ │ Manager  │               │
│  │          │ │  billing)│ │          │ │          │               │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘               │
└──────────────────────────────────────────────────────────────────────┘

Component Inventory

Component Path Role
ARC Core modules/core/arc/ Scale set controller, scale sets, Karpenter, storage classes
ARC Deployment modules/platform/arc_deployment/ ARC runner deployment wrapper
EC2 Deployment modules/platform/ec2_deployment/ EC2 runner instances, AMI management, security groups, user data
Forge Runners modules/platform/forge_runners/ Main orchestration: composes EC2 + ARC, GitHub App, webhooks, logs
EKS modules/infra/eks/ EKS cluster, addons, Karpenter, Calico, node groups
ECR modules/infra/ecr/ Container registry for runner images
Storage modules/infra/storage/ S3 buckets for artifacts and logs
AMI Policy modules/infra/ami_policy/ DLM lifecycle policies, EBS encryption
AMI Sharing modules/infra/ami_sharing/ Cross-account AMI sharing
Cloud Custodian modules/infra/cloud_custodian/ Cost governance policies
CloudFormation modules/infra/cloud_formation/ CloudFormation stack management
Forge Subscription modules/infra/forge_subscription/ Cross-account IAM roles and ECR policies for subscribers
Opt-in Regions modules/infra/opt_in_regions/ AWS opt-in region enablement
Service Linked Roles modules/infra/service_linked_roles/ AWS service-linked role provisioning
Splunk Cloud Conf modules/integrations/splunk_cloud_conf_shared/ Splunk props, transforms, dashboards
Splunk Data Manager modules/integrations/splunk_cloud_data_manager/ CloudWatch + custom data ingestion to Splunk
Splunk Data Manager Common modules/integrations/splunk_cloud_data_manager_common/ Shared Splunk DM IAM roles and config
Splunk AWS Billing modules/integrations/splunk_aws_billing/ CUR billing data pipeline to Splunk
Splunk O11y Integration modules/integrations/splunk_o11y_aws_integration/ CloudFormation-based AWS integration
Splunk O11y Common modules/integrations/splunk_o11y_aws_integration_common/ SignalFx API integration + IAM
Splunk O11y Dashboards modules/integrations/splunk_o11y_conf_shared/ SignalFx dashboard group + 7 dashboard types
Splunk OTel EKS modules/integrations/splunk_otel_eks/ Splunk OTel collector Helm deployment
Splunk S3 Runner Logs modules/integrations/splunk_cloud_s3_runner_logs/ Runner log shipping via Kinesis Firehose
Splunk Secrets modules/integrations/splunk_secrets/ Secrets Manager entries for Splunk credentials
Teleport modules/integrations/teleport/ Teleport agent for secure cluster access
Teleport Tenant modules/integrations/teleport/tenant/ Per-tenant Teleport RBAC
Webhook Relay Source modules/integrations/github_webhook_relay_source/ API Gateway + Lambda webhook ingress
Webhook Relay Destination modules/integrations/github_webhook_relay_destination/ EventBridge destination bus + rules
Webhook Relay Receivers modules/integrations/github_webhook_relay_destination_receivers/ Lambda receivers (e.g., Webex relay)

Technology Stack

Technology Version Purpose
Terraform / OpenTofu ~> 1.11 Infrastructure-as-Code engine
Terragrunt Latest Configuration hierarchy, dependency management
AWS Provider >= 6.25 AWS resource provisioning
Helm Provider >= 3.0.0 Kubernetes Helm chart management
Kubernetes Provider >= 3.0 Kubernetes resource management
Splunk Provider >= 1.4.30 Splunk Cloud configuration
SignalFx Provider < 10.0.0 Splunk Observability dashboards
Python 3.12 Lambda functions, pre-commit hooks
terraform-aws-modules/eks 21.15.1 EKS cluster module
terraform-aws-modules/lambda 8.7.0 Lambda function module
terraform-aws-modules/iam 6.4.0 IAM IRSA roles

2. System Requirements

AWS Account Prerequisites

Resource Requirement
VPC With private subnets for runners, NAT gateway for internet access
IAM Permissions to create roles, policies, Lambda functions, EKS clusters
Secrets Manager For Splunk credentials, GitHub App secrets
SSM Parameter Store For GitHub App configuration
S3 For Terraform state, runner logs, billing data, CloudFormation templates
EKS For ARC runners (if using Kubernetes backend)
ECR For custom runner container images
EventBridge For webhook relay and scheduled Lambda triggers
CloudWatch For Lambda logging and metrics
KMS For encryption of logs, SQS queues, and S3 buckets

GitHub App Requirements

The GitHub App must have these permissions:

Permission Level Purpose
Actions Read Job log archival
Metadata Read Repository metadata access
Self-hosted runners Read & Write Runner registration/deregistration
Organization self-hosted runners Read & Write Runner group management

Important

The GitHub App must be installed on the target organization. The installation ID, app ID, client ID, and base64-encoded private key must be available for Terraform to store in SSM.

Toolchain

Tool Version Required For
Terraform / OpenTofu ~> 1.11 All modules
Terragrunt Latest Deployment orchestration
AWS CLI v2 Latest Provider authentication, scripts
kubectl EKS-version compatible EKS management, Karpenter, ARC
Helm >= 3.0.0 ARC controller + scale sets, OTel
Python 3.12 Pre-commit hooks, Lambda development
pre-commit Latest Code quality enforcement
shellcheck, shfmt Latest Shell script linting
tflint Latest Terraform linting
hadolint Latest Dockerfile linting
ansible-lint v26.3.0 Ansible linting
Gitleaks v8.30.0 Secret detection
commitizen v4.13.9 Commit message validation

3. Configuration Reference

Global Settings (_global_settings/)

_global.yaml — shared values inherited by all environments:

# Example from templates/tenant/_global_settings/_global.yaml
# Contains: aws_profile, aws_region, default_tags, product metadata

_global.hcl — shared HCL configuration for provider setup, backend, and default inputs.

Per-module .hcl files — each module has a dedicated HCL file (e.g., tenant.hcl, eks.hcl, storage.hcl) that specifies the module source path and version:

# Example: _global_settings/eks.hcl
locals {
  source = "${get_repo_root()}/modules/infra/eks"
}

Environment Settings (_environment_wide_settings/)

_environment.yaml — environment-specific overrides (e.g., dev vs. prod instance sizes, scaling limits).

_environment.hcl — environment-level HCL (e.g., different AWS profiles per environment).

Region Settings (_region_wide_settings/)

_region.hcl — region-specific configuration (subnet IDs, VPC IDs, availability zones).

VPC Settings (_vpc_wide_settings/)

_vpc.yaml / _vpc.hcl — VPC-level configuration for multi-VPC deployments, including subnet assignments and security group references.

Forge Runners Module — Key Variables

The forge_runners module (modules/platform/forge_runners/) is the main orchestration module. Key input variables:

deployment_config

variable "deployment_config" {
  type = object({
    deployment_prefix = string
    secret_suffix     = string
    env               = string
    github_app = object({
      id              = string
      client_id       = string
      installation_id = string
      name            = string
    })
    github = object({
      ghes_org             = string
      ghes_url             = string
      repository_selection = string  # "all" or "selected"
      runner_group_name    = string
    })
    tenant = object({
      name                         = string
      iam_roles_to_assume          = optional(list(string), [])
      ecr_registries               = optional(list(string), [])
      github_logs_reader_role_arns = optional(list(string), [])
    })
  })
}

Important

The deployment_prefix is used to namespace all SSM parameters, log groups, and resource names. It must be unique per tenant deployment within an AWS account.

ec2_deployment_specs

variable "ec2_deployment_specs" {
  type = object({
    lambda_subnet_ids = list(string)
    subnet_ids        = list(string)
    lambda_vpc_id     = string
    vpc_id            = string
    scale_errors      = optional(list(string), [])
    runner_specs = map(object({
      ami_filter                    = object({ name = list(string), state = list(string) })
      ami_kms_key_arn               = string
      ami_owners                    = list(string)
      runner_labels                 = list(string)
      runner_os                     = string    # "linux", "osx", "windows"
      runner_architecture           = string    # "x86_64", "arm64"
      extra_labels                  = list(string)
      max_instances                 = number
      min_run_time                  = number
      instance_types                = list(string)
      pool_config                   = list(object({
        size                         = number
        schedule_expression          = string
        schedule_expression_timezone = string
      }))
      runner_user                   = string
      enable_userdata               = bool
      instance_target_capacity_type = string  # "spot" or "on-demand"
      block_device_mappings         = list(object({...}))
    }))
  })
}

arc_deployment_specs

variable "arc_deployment_specs" {
  type = object({
    cluster_name    = string
    migrate_cluster = optional(bool, false)
    runner_specs = map(object({
      runner_size = object({
        max_runners = number
        min_runners = number
      })
      scale_set_name               = string
      scale_set_type               = string  # "k8s" or "dind"
      container_actions_runner     = string
      container_limits_cpu         = string
      container_limits_memory      = string
      container_requests_cpu       = string
      container_requests_memory    = string
      volume_requests_storage_size = string
      volume_requests_storage_type = string
    }))
  })
}

github_webhook_relay

variable "github_webhook_relay" {
  type = object({
    enabled                     = bool
    destination_account_id      = optional(string)
    destination_event_bus_name  = optional(string)
    destination_region          = optional(string)
    destination_reader_role_arn = optional(string)
  })
  default = { enabled = false }
}

EKS Module — Key Variables

Variable Type Default Description
cluster_name string required EKS cluster name
cluster_version string required Kubernetes version
cluster_size object required Instance type, min/max/desired node count
cluster_volume object required EBS size, IOPS, throughput, type
cluster_endpoint_public_access bool false Whether the API endpoint is public
external_access_cidr_blocks list(string) [] CIDRs allowed to access the K8s API
cluster_admin_role_arn string "" IAM role for cluster admin access
subnet_ids list(string) required Private subnets for worker nodes
vpc_id string required VPC ID

Common Variables (shared across modules)

Variable Type Description
aws_profile string AWS CLI profile name
aws_region string Target AWS region
default_tags map(string) Default tags applied to all resources
tags map(string) Additional resource-specific tags
log_level string Lambda log level (INFO, DEBUG, WARN, ERROR)
logging_retention_in_days number CloudWatch log retention (default varies: 3 or 30 days)

Splunk Configuration Variables

Module Key Variable Description
splunk_aws_billing splunk_aws_billing_config.splunk_hec_url Splunk HEC endpoint URL
splunk_aws_billing splunk_aws_billing_config.splunk_index Target Splunk index
splunk_aws_billing splunk_aws_billing_config.splunk_metrics_url Splunk metrics endpoint
splunk_cloud_conf_shared splunk_conf.splunk_cloud Splunk Cloud endpoint
splunk_cloud_conf_shared splunk_conf.index Splunk index name
splunk_cloud_conf_shared splunk_conf.tenant_names List of tenant names
splunk_cloud_data_manager splunk_cloud Splunk Cloud endpoint
splunk_o11y_aws_integration_common splunk_api_url SignalFx API URL
splunk_o11y_aws_integration_common splunk_organization_id SignalFx org ID
splunk_o11y_aws_integration_common integration_regions AWS regions to monitor
splunk_otel_eks splunk_otel_collector OTel collector config object
splunk_cloud_s3_runner_logs splunk_hec_endpoint Splunk HEC URL for runner logs

Note

All Splunk secrets (HEC tokens, API tokens, Splunk Cloud credentials) are stored in AWS Secrets Manager and referenced by the modules via aws_secretsmanager_secret data sources. Never hardcode credentials in Terraform variables.

4. Deployment & Scaling

Deployment Order

Forge deployments follow a strict dependency order:

  1. infra-forge — Foundation: EKS, ECR, storage, AMI policies, service-linked roles, opt-in regions
  2. forge-extras — Optional: Cloud Custodian, additional ECR repos, Forge subscription (cross-account)
  3. forge-eks — EKS cluster with Karpenter and Calico (if using ARC runners)
  4. forge-integrations — Splunk Cloud configuration (props, transforms, dashboards)
  5. splunk-deployment — Splunk O11y, OTel, Data Manager, billing pipeline
  6. forge-tenant — Per-tenant runner deployment (last, depends on all above)

Deploying a Full Stack

# 1. Foundation
cd examples/infra-forge/terragrunt/environments/dev
terragrunt run-all apply

# 2. EKS cluster (if using ARC)
cd examples/deployments/forge-eks/terragrunt/environments/prod
terragrunt run-all apply

# 3. Integrations
cd examples/deployments/forge-integrations/terragrunt/environments/prod
terragrunt run-all apply

# 4. Splunk
cd examples/deployments/splunk-deployment/terragrunt/environments/prod
terragrunt run-all apply

# 5. Tenants
cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all apply

Scaling EC2 Runners

EC2 runner pools scale via two mechanisms:

  1. Reactive scaling — The upstream terraform-aws-github-runner module responds to GitHub Actions webhook events to launch/terminate instances based on demand.

  2. Scheduled scaling — The pool_config variable defines baseline capacity on a cron schedule:

pool_config = [
  {
    size                         = 2
    schedule_expression          = "cron(0 8 ? * MON-FRI *)"
    schedule_expression_timezone = "America/New_York"
  },
  {
    size                         = 0
    schedule_expression          = "cron(0 20 ? * MON-FRI *)"
    schedule_expression_timezone = "America/New_York"
  }
]

Key scaling parameters per runner spec:

Parameter Description
max_instances Hard ceiling on concurrent runners
min_run_time Minimum job duration (minutes) before scale-down eligibility
instance_target_capacity_type spot or on-demand
instance_types List of allowed EC2 instance types

Scaling ARC Runners

ARC runner pools scale via the GitHub Actions Runner Controller:

Parameter Description
max_runners Maximum concurrent runners in the scale set
min_runners Minimum warm runners (always running)
container_limits_cpu / memory Pod resource limits
container_requests_cpu / memory Pod resource requests (baseline)

Karpenter manages the underlying EKS node capacity, provisioning nodes as pods demand increases.

Tip

Set min_runners to at least 1 for latency-sensitive workflows. Cold starts (launching a new runner pod + node) can add 2-5 minutes to job startup.

ARC Cluster Migration

Set migrate_arc_cluster = true when performing a blue/green cluster cutover. This flag is propagated to the scale set controller and scale set modules to coordinate the migration.

5. Operations

Lambda Functions

Forge deploys several operational Lambda functions:

Lambda Module Trigger Purpose
forge_trust_validator forge_runners/forge_trust_validator EventBridge schedule Validates IAM trust relationships for runner roles
ec2_update_runner_ssm_ami ec2_deployment/ec2_update_runner_ssm_ami EventBridge schedule Updates SSM with latest AMI matching filters
ec2_update_runner_tags ec2_deployment/ec2_update_runner_tags EventBridge (event bus) Tags EC2 runner instances on state changes
github_app_runner_group forge_runners/github_app_runner_group EventBridge schedule Creates/updates GitHub runner groups, attaches repos
github_clean_global_lock forge_runners/github_global_lock EventBridge schedule Cleans stale GitHub global locks
redrive_deadletter forge_runners/redrive_deadletter EventBridge schedule Redrives messages from DLQ back to main queue
job_log_dispatcher forge_runners/github_actions_job_logs EventBridge (webhook) Dispatches completed job events to SQS
job_log_archiver forge_runners/github_actions_job_logs SQS Downloads and archives GitHub Actions job logs to S3
validate_signature github_webhook_relay_source API Gateway Validates GitHub webhook HMAC signatures
webex_webhook_relay github_webhook_relay_destination_receivers EventBridge Forwards failure notifications to Webex rooms
splunk_s3_runner_logs splunk_cloud_s3_runner_logs SQS (S3 notifications) Ships runner logs to Splunk via Kinesis Firehose
cur_per_resource splunk_aws_billing S3 notification Processes per-resource CUR billing data
cur_per_resource_process splunk_aws_billing S3 notification Aggregates and sends per-resource costs to Splunk HEC
cur_per_service splunk_aws_billing S3 notification Processes per-service CUR billing data to Splunk
sec_meta_ec2_tags splunk_cloud_data_manager/sec_meta_ec2_tags (data manager) EC2 tag metadata for Splunk security enrichment

Note

All Lambda functions use a configurable log_level variable (default INFO) and logging_retention_in_days for CloudWatch log group retention. Common values: 3 days for integration Lambdas, 30 days for platform Lambdas.

GitHub Actions Job Log Archival

The job log archiver supports two modes:

Queue Pipeline (default, recommended):

GitHub Webhook → EventBridge → Dispatcher Lambda → SQS → Archiver Lambda → S3
  • Better isolation, retry semantics, and backpressure handling
  • SQS dead-letter queue for failed downloads
  • Configurable batch size (downloader_batch_size) and concurrency (downloader_max_concurrency)

Direct Mode (fallback):

GitHub Webhook → EventBridge → Archiver Lambda → S3
  • Simpler, fewer moving parts
  • Set enable_queue_pipeline = false to use

Logs are stored at: s3://<bucket>/<workflow_name>/<run_attempt>/<job_id>.zip

Monitoring

Forge provides built-in monitoring through Splunk Observability (SignalFx) dashboards:

Dashboard Module Path Metrics
Runner EC2 dashboards/runner_ec2/ CPU, memory, disk, network utilization; active hosts by AZ and instance type
Runner K8s dashboards/runner_k8s/ Active pods, memory usage, network bytes, pod phases
Billing dashboards/billing/ Cost per service, cost per tenant, net costs
Lambda dashboards/lambda/ Invocations, errors, duration, throttles, concurrency
SQS dashboards/sqs/ Queue depth, message age, processing trends
EBS dashboards/ebs/ IOPS, throughput, latency, queue length, utilization
DynamoDB dashboards/dynamodb/ Request latency, throttles, capacity utilization

Each dashboard supports dynamic variables for filtering by tenant via the dashboard_variables input.

Tip

The Splunk Cloud Conf module (splunk_cloud_conf_shared) provisions three Splunk dashboards (CI Jobs, Tenant, Trust Relationship Validation) as saved searches using splunk_data_ui_views resources.

Secrets Management

Secret Type Storage Modules
GitHub App private key SSM Parameter Store (SecureString) forge_runners
GitHub App webhook secret SSM Parameter Store forge_runners (auto-rotated every 30 days)
Splunk Cloud credentials Secrets Manager All Splunk integration modules
Splunk HEC tokens Secrets Manager splunk_aws_billing, splunk_cloud_s3_runner_logs
SignalFx API token Secrets Manager splunk_o11y_aws_integration_common, splunk_o11y_conf_shared
Webex bot token Secrets Manager webex_webhook_relay

Warning

The Webex webhook relay Lambda expects a Secrets Manager secret at /cicd/common/webex_webhook_relay_bot_token containing JSON with token and room_id keys. Both are required. The Lambda auto-prepends Bearer to the token if not present.

Operational Scripts

Script Purpose
scripts/migrate-tenant.sh Migrate a single tenant to new configuration
scripts/migrate-all-tenants.sh Batch tenant migration
scripts/reinstall-eks-with-deps.sh Reinstall EKS cluster with all dependencies
scripts/update-github-app-secrets.sh Update GitHub App credentials in SSM
scripts/update-terraform-docs.sh Regenerate terraform-docs for all modules
scripts/generate_and_patch_github_app.sh Generate and patch GitHub App webhook configuration
scripts/download_lambdas.sh Download Lambda deployment packages

6. Performance

EKS Cluster Sizing

The EKS module exposes cluster_size for the self-managed node group:

cluster_size = {
  instance_type = "m5.large"
  min_size      = 2
  max_size      = 10
  desired_size  = 3
}

And cluster_volume for EBS configuration:

cluster_volume = {
  size       = 100
  iops       = 3000
  throughput = 125
  type       = "gp3"
}

Karpenter handles additional node provisioning beyond the base node group, using NodePool and EC2NodeClass custom resources applied via null_resource provisioners.

ARC Scale Set Resource Tuning

Each ARC runner pod's resources are individually configurable:

Setting Example Impact
container_requests_cpu "500m" Baseline CPU reservation
container_requests_memory "1Gi" Baseline memory reservation
container_limits_cpu "2000m" CPU burst ceiling
container_limits_memory "4Gi" Memory ceiling (OOM-kill above)
volume_requests_storage_size "20Gi" Persistent volume for workspace
volume_requests_storage_type "gp3" Storage class

EC2 Runner Instance Selection

Configure instance types per runner pool for cost/performance optimization:

instance_types = ["m5.xlarge", "m5a.xlarge", "m5d.xlarge"]
instance_target_capacity_type = "spot"  # or "on-demand"

Tip

Use multiple instance types in the instance_types list when using spot capacity to improve availability. The runner module selects from the list based on current spot pricing and capacity.

7. Security

IAM Architecture

Forge implements a layered IAM model:

  • Runner IAM roles — Each tenant's runners assume a dedicated IAM role (role_for_forge_runners) with scoped permissions for S3, Secrets Manager, ECR, and Packer operations
  • Forge trust validator — A scheduled Lambda that validates IAM trust policies on runner roles, ensuring no unauthorized principals can assume them
  • ECR access policies — Per-tenant ECR repository policies granting pull access only to authorized runner roles
  • Cross-account roles — The forge_subscription module creates IAM roles and ECR policies for subscriber accounts

Trust Validation

The forge_trust_validator Lambda (modules/platform/forge_runners/forge_trust_validator/lambda/forge_trust_validator.py) runs on a schedule and:

  1. Reads the trust policy of each configured Forge IAM role
  2. Validates that only expected principals appear in the trust relationship
  3. Reports violations (the Splunk Cloud Conf module provides a Trust Relationship Validation dashboard)

Webhook Signature Validation

The GitHub webhook relay source Lambda (modules/integrations/github_webhook_relay_source/lambda/validate_signature.py) validates incoming webhook payloads using HMAC-SHA256 with the GitHub App webhook secret before forwarding events to EventBridge.

Encryption

Resource Encryption
EBS volumes Default encryption enabled via aws_ebs_encryption_by_default (in ami_policy module)
S3 buckets Server-side encryption (SSE-S3 or SSE-KMS depending on module)
SQS queues KMS encryption for job log queues
Kinesis streams KMS encryption for runner log streaming
SSM parameters SecureString type for sensitive values

Caution

The ami_policy module enables EBS encryption by default at the account level via aws_ebs_encryption_by_default. This affects ALL EBS volumes in the region, not just Forge resources. Coordinate with other teams before applying.

Network Security

  • EC2 runners are deployed in private subnets with security groups managed by the ec2_deployment module
  • EKS clusters default to cluster_endpoint_public_access = false (private API endpoint)
  • external_access_cidr_blocks must be explicitly configured for any external access to the K8s API
  • Calico network policies are applied via the EKS module for pod-level network segmentation

Pre-Commit Security Hooks

Hook Purpose
detect-private-key Prevents committing private keys
gitleaks (v8.30.0) Scans for secrets and credentials
check-added-large-files Prevents accidentally committing large binaries

8. Developer Internals

Repository Structure

cisco-open/forge/
├── modules/
│   ├── core/
│   │   └── arc/                    # ARC scale set controller + scale sets
│   │       ├── scale_set/          # Individual scale set (Helm + K8s + IAM)
│   │       ├── scale_set_controller/  # ARC controller (Helm + namespace + secret)
│   │       └── templates/          # Karpenter NodePool template
│   ├── infra/
│   │   ├── ami_policy/             # DLM lifecycle, EBS encryption
│   │   ├── ami_sharing/            # Cross-account AMI sharing
│   │   ├── cloud_custodian/        # Cost governance
│   │   ├── cloud_formation/        # CF stack management
│   │   ├── ecr/                    # Container registry
│   │   ├── eks/                    # EKS cluster + addons + Karpenter
│   │   ├── forge_subscription/     # Cross-account subscription
│   │   ├── opt_in_regions/         # AWS region opt-in
│   │   ├── service_linked_roles/   # SLR provisioning
│   │   └── storage/                # S3 buckets
│   ├── integrations/
│   │   ├── github_webhook_relay_source/        # Webhook ingress (API GW + Lambda)
│   │   ├── github_webhook_relay_destination/   # EventBridge destination bus
│   │   ├── github_webhook_relay_destination_receivers/  # Lambda receivers
│   │   ├── splunk_aws_billing/                 # CUR billing pipeline
│   │   ├── splunk_cloud_conf_shared/           # Splunk props/transforms/dashboards
│   │   ├── splunk_cloud_data_manager/          # CloudWatch data ingestion
│   │   ├── splunk_cloud_data_manager_common/   # Shared DM config
│   │   ├── splunk_cloud_s3_runner_logs/        # Runner log shipping
│   │   ├── splunk_o11y_aws_integration/        # AWS integration (CloudFormation)
│   │   ├── splunk_o11y_aws_integration_common/ # SignalFx API integration
│   │   ├── splunk_o11y_conf_shared/            # SignalFx dashboards (7 types)
│   │   ├── splunk_otel_eks/                    # OTel collector Helm
│   │   ├── splunk_secrets/                     # Secrets Manager entries
│   │   └── teleport/                           # Teleport agent + tenant RBAC
│   └── platform/
│       ├── arc_deployment/         # ARC deployment wrapper
│       ├── ec2_deployment/         # EC2 runners (AMI, SG, user data, Lambdas)
│       └── forge_runners/          # Main orchestration module
│           ├── forge_trust_validator/     # IAM trust validation Lambda
│           ├── github_actions_job_logs/   # Job log archiver (dispatcher + archiver)
│           ├── github_app_runner_group/   # Runner group management Lambda
│           ├── github_global_lock/        # Global lock cleanup Lambda
│           ├── github_webhook_relay/      # Webhook relay orchestration
│           ├── redrive_deadletter/        # DLQ redrive Lambda
│           └── scripts/                   # GitHub App patching script
├── examples/
│   ├── deployments/                # Full deployment examples per stack
│   │   ├── forge-eks/
│   │   ├── forge-extras/
│   │   ├── forge-integrations/
│   │   ├── forge-tenant/
│   │   └── splunk-deployment/
│   ├── infra-forge/                # Foundation infrastructure deployment
│   └── templates/                  # Configuration templates per stack type
├── docs/                           # Internal documentation
│   ├── configurations/             # Deployment configuration guides
│   └── tenant-usage/               # Tenant onboarding and Renovate
├── scripts/                        # Operational scripts
├── .pre-commit-config.yaml         # 30+ pre-commit hooks
├── .terraform-docs.yml             # terraform-docs configuration
├── renovate.json                   # Renovate dependency management
├── .ansible-lint                   # Ansible linting config
└── .yamllint                       # YAML linting config

Module Composition Pattern

The forge_runners module demonstrates the primary composition pattern:

# modules/platform/forge_runners/
module "ec2_runners"              { source = "../ec2_deployment" }
module "arc_runners"              { source = "../arc_deployment" }
module "forge_trust_validator"    { source = "./forge_trust_validator" }
module "github_actions_job_logs"  { source = "./github_actions_job_logs" }
module "github_app_runner_group"  { source = "./github_app_runner_group" }
module "github_global_lock"       { source = "./github_global_lock" }
module "github_webhook_relay"     { source = "./github_webhook_relay" }
module "redrive_deadletter"       { source = "./redrive_deadletter" }

This single module composes all runner-related infrastructure, creating a cohesive deployment unit per tenant.

Code Patterns and Conventions

State management: Each module uses a backend.tf with S3 backend configuration. State paths follow the Terragrunt hierarchy.

Provider configuration: Providers are declared in providers.tf per module, with versions pinned in versions.tf.

Tagging: Every module accepts default_tags and tags variables. The AWS provider default_tags feature applies organization-wide tags, while tags adds module-specific tags.

Naming: Resources use a prefix or deployment_prefix variable for namespace isolation. SSM parameters follow /forge/<prefix>/<parameter_name> convention.

Lambda packaging: Lambda functions are Python 3.12, packaged in-module under lambda/ directories. The terraform-aws-modules/lambda/aws module (v8.7.0) handles packaging and deployment.

Template files: Terraform template files use .tftpl extension and live in template_files/ directories within their respective modules.

External data sources: Several modules use external_external data sources to call shell scripts (e.g., fetch_secret_value.sh, list_buckets.sh) for operations that lack native Terraform providers.

Webhook Relay Architecture

The webhook relay spans three modules:

  1. Source (github_webhook_relay_source/) — API Gateway endpoint receives GitHub webhooks → Lambda validates HMAC signature → forwards valid events to EventBridge source bus

  2. Destination (github_webhook_relay_destination/) — EventBridge destination bus in another account receives cross-account events → rules route to Lambda targets based on event patterns

  3. Receivers (github_webhook_relay_destination_receivers/) — Lambda functions that process forwarded events. The included webex_webhook_relay receiver:

    • Filters for main branch failures (cancelled, failure, timed_out, timeout, startup_failure)
    • Builds a Slack-format card, converts to Webex Adaptive Card format
    • Sends to Webex via the Messages API using a bot token from Secrets Manager

Splunk Billing Pipeline

The billing integration (splunk_aws_billing/) implements a three-Lambda pipeline:

  1. cur_per_resource — Triggered by S3 notifications when new CUR (Cost and Usage Report) parquet files arrive. Splits data by date and re-uploads to S3 for processing.

  2. cur_per_resource_process — Groups billing data by resource, service, and application. Sends events to Splunk HEC in batches (respecting MAX_BATCH_COUNT and MAX_BATCH_SIZE_BYTES). Also sends metrics to SignalFx (forge.per_resource.cost_usd, forge.per_resource.net_cost_usd).

  3. cur_per_service — Similar to per-resource but aggregated at the service level. Sends forge.per_service.cost_usd and forge.per_service.net_cost_usd metrics.

All three Lambdas use a shared common.py module for Splunk HEC/SignalFx communication and data preprocessing.

Testing Strategy

Forge relies on:

  • Pre-commit validation — terraform validate, terraform fmt, tflint, tofu-validate run on every commit
  • Terraform plan — terragrunt run-all plan serves as the primary integration test
  • Safety hooks — safety_hook.sh provides environment-aware guardrails before apply

Code Quality Toolchain

The .pre-commit-config.yaml configures 30+ hooks across categories:

Category Hooks
General YAML/JSON/TOML/XML syntax, merge conflict markers, large file guard, line endings
Python autopep8, isort, autoflake, flake8 (E501 ignored), pyupgrade, validate-pyproject
Shell shellcheck, shfmt (4-space indent), bashate, script-must-have-extension
Terraform/IaC terragrunt-hcl-fmt, tofu-fmt, tofu-validate, terraform_fmt, terraform_tflint, terraform_validate, packer-validate
Security gitleaks, detect-private-key
Docker hadolint
Ansible ansible-lint (v26.3.0, Python 3.12)
Markdown mdformat (with GFM, ruff, frontmatter plugins, excludes modules/)
JSON Schema GitHub workflows, GitHub Actions, Docker Compose, Dependabot, Renovate
Commit commitizen with cz-conventional-gitmoji

Dependency Management

Renovate (renovate.json) manages all dependency updates:

Rule Behavior
Security patches Immediate processing, auto-merge, security + critical labels
Patch/pin/digest Auto-merge with simple-review label
Minor updates simple-review label, no auto-merge
Major updates 7-day stability period, major-update + breaking-change labels
AWS providers/modules Grouped as "AWS providers and modules"
GitHub Actions Pinned to commit SHA, grouped
Pre-commit hooks Grouped as "Pre-commit hooks"
Docker images Digest pinning
Terraform providers Resolved from OpenTofu registry

Custom regex managers extract versions from:

  • .tf files (required_version)
  • download_lambdas.sh references (GitHub runner version)
  • Dockerfiles (with renovate comments)

Contributing

Forge follows conventional commits with gitmoji (enforced by commitizen pre-commit hook). Key practices:

  • All Terraform code must pass terraform validate, terraform fmt, and tflint
  • Shell scripts must pass shellcheck and shfmt
  • Python code must pass flake8 (E501 ignored), autopep8, isort, and autoflake
  • Module documentation is auto-generated — update source code, then run scripts/update-terraform-docs.sh
  • YAML files are formatted with yamlfmt (2-space mapping, 4-space sequence, 2-space offset, 180 char width)

Potential Improvements

Based on open GitHub issues:

  • Lifecycle field refactoring (#236) — Replace count conditionals with the lifecycle.enabled field for cleaner resource toggling
  • Grafana/PagerDuty integration (#125, #124) — Add alerting pipeline from Grafana and Splunk Cloud to PagerDuty
  • AWS IAM Roles Anywhere (#122) — Enable multi-cloud ephemeral runners via certificate-based authentication
  • Open-source monitoring (#121) — Add Loki, Prometheus, and Grafana as alternatives to Splunk
  • Cost safeguards (#296) — Default cost guardrails and hardening for runner environments

9. Integration Points

AWS Service Integration

AWS Service Forge Usage
EC2 Runner instances (Linux, macOS, Windows)
EKS Kubernetes clusters for ARC runners
ECR Container image registry for runner images
S3 Terraform state, runner logs, billing data, CloudFormation templates, Firehose backup
SSM Parameter Store GitHub App configuration, AMI IDs
Secrets Manager Splunk credentials, Webex bot tokens
EventBridge Webhook relay, Lambda scheduling, EC2 state change events
API Gateway GitHub webhook ingress endpoint
Lambda 14+ operational functions
CloudWatch Lambda logs, metrics, event rules
KMS Encryption for S3, SQS, Kinesis, EBS
IAM Runner roles, service roles, cross-account access
CloudFormation Splunk O11y AWS integration
DLM EBS snapshot lifecycle policies
Kinesis Data Streams Runner log streaming
Kinesis Firehose Splunk HEC delivery
SQS Job log dispatch queue, dead-letter queues
Service Catalog AppRegistry for Forge application tracking
BCM Data Exports Cost and Usage Report (CUR) exports
Calico EKS network policies

Third-Party Integrations

Service Integration Module Purpose
Splunk Cloud splunk_cloud_conf_shared, splunk_cloud_data_manager Log ingestion, props/transforms, dashboards
Splunk Observability (SignalFx) splunk_o11y_aws_integration_common, splunk_o11y_conf_shared Metrics, dashboards
Splunk OTel splunk_otel_eks Kubernetes observability via Helm
Teleport teleport/, teleport/tenant/ Secure cluster access with per-tenant RBAC
Webex webex_webhook_relay CI failure notifications
GitHub (API) forge_runners, github_app_runner_group Runner management, App webhook configuration

Terraform Module Dependencies

External Module Version Used By
terraform-aws-modules/eks/aws 21.15.1 modules/infra/eks/
terraform-aws-modules/eks/aws//modules/karpenter 21.15.1 modules/infra/eks/
terraform-aws-modules/eks/aws//modules/self-managed-node-group 21.15.1 modules/infra/eks/
terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts 6.4.0 modules/infra/eks/
terraform-aws-modules/lambda/aws 8.7.0 Multiple Lambda-based modules