1. Architecture Overview
Forge is Cisco's open-source Infrastructure-as-Code (IaC) framework for deploying and operating multi-tenant, self-hosted GitHub Actions runners on AWS. It provides a modular Terraform/Terragrunt architecture that provisions both EC2-based and Kubernetes-based (ARC) runner backends, along with comprehensive observability, security, and lifecycle management infrastructure.
The framework is designed around three architectural layers:
- Core — GitHub Actions Runner Controller (ARC) on EKS, including scale set management and Helm-based deployment
- Infrastructure — Foundational AWS resources: EKS clusters, ECR registries, S3 storage, AMI lifecycle policies, CloudFormation stacks, and cross-account subscription roles
- Platform — Runner orchestration: EC2 and ARC deployment, GitHub App management, webhook relay, job log archival, trust validation, and dead-letter queue redriving
- Integrations — Third-party services: Splunk Cloud (data ingestion, dashboards, props/transforms), Splunk Observability (SignalFx, OTel), Teleport (secure access), and GitHub webhook relay (EventBridge cross-account forwarding with Webex notification support)
Architecture Diagram
┌──────────────────────────────────────────────────────────────────────┐
│ GitHub (GHES / github.com) │
│ ┌─────────┐ ┌──────────┐ ┌───────────────┐ │
│ │ Webhooks │ │ Runner │ │ GitHub App │ │
│ │ │ │ Groups │ │ (JWT + Install│ │
│ └────┬─────┘ └─────┬────┘ │ Token Auth) │ │
│ │ │ └───────┬───────┘ │
└───────┼───────────────┼────────────────┼────────────────────────────┘
│ │ │
▼ │ ▼
┌───────────────────────┼──────────────────────────────────────────────┐
│ AWS Account │ │
│ │ │
│ ┌──────────────┐ │ ┌─────────────────────────────────────┐ │
│ │ API Gateway │─────┼────▶│ Webhook Relay Lambda │ │
│ │ (webhook │ │ │ (validate_signature.py) │ │
│ │ ingress) │ │ └──────────────┬──────────────────────┘ │
│ └──────────────┘ │ │ │
│ │ ▼ │
│ │ ┌─────────────────────────────────────┐ │
│ │ │ EventBridge │ │
│ │ │ ┌─────────┐ ┌──────────────────┐ │ │
│ │ │ │ Source │─▶│ Destination Bus │ │ │
│ │ │ │ Rules │ │ (cross-account) │ │ │
│ │ │ └─────────┘ └──────┬───────────┘ │ │
│ │ └─────────────────────┼──────────────┘ │
│ │ │ │
│ ┌───────────┴───────────┐ ▼ │
│ │ │ ┌──────────────────┐ │
│ ┌────────▼─────┐ ┌─────────────▼┐ │ Webex Relay │ │
│ │ EC2 Runners │ │ ARC Runners │ │ Lambda │ │
│ │ (ASG/Fleet) │ │ (EKS Pods) │ └──────────────────┘ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ Linux │ │ │ │ DinD │ │ ┌──────────────────────────┐ │
│ │ │ macOS │ │ │ │ K8s Mode │ │ │ Support Lambdas │ │
│ │ │ Windows │ │ │ └──────────┘ │ │ ┌──────────────────────┐ │ │
│ │ └──────────┘ │ └──────────────┘ │ │ trust_validator │ │ │
│ └──────────────┘ │ │ ssm_ami_updater │ │ │
│ │ │ runner_tag_updater │ │ │
│ ┌────────────────────────────────┐ │ │ global_lock_cleaner │ │ │
│ │ EKS Cluster │ │ │ app_runner_group │ │ │
│ │ ┌────────┐ ┌────────────────┐ │ │ │ job_log_archiver │ │ │
│ │ │Karpenter│ │ARC Controller │ │ │ │ job_log_dispatcher │ │ │
│ │ │ │ │(Helm) │ │ │ │ redrive_deadletter │ │ │
│ │ └────────┘ └────────────────┘ │ │ └──────────────────────┘ │ │
│ └────────────────────────────────┘ └──────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ Observability (Splunk) │ │
│ │ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌────────────────────┐│ │
│ │ │Data Mgr │ │O11y/SFx │ │OTel EKS │ │Billing (CUR→HEC) ││ │
│ │ │(CloudFmt)│ │(Dashbds) │ │(Helm) │ │(S3→Lambda→Splunk) ││ │
│ │ └──────────┘ └──────────┘ └─────────┘ └────────────────────┘│ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ ECR │ │ S3 │ │ SSM │ │ Secrets │ │
│ │ (images) │ │ (logs, │ │ (params) │ │ Manager │ │
│ │ │ │ billing)│ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────────────────────────────────────────────┘
Component Inventory
| Component | Path | Role |
|---|---|---|
| ARC Core | modules/core/arc/ |
Scale set controller, scale sets, Karpenter, storage classes |
| ARC Deployment | modules/platform/arc_deployment/ |
ARC runner deployment wrapper |
| EC2 Deployment | modules/platform/ec2_deployment/ |
EC2 runner instances, AMI management, security groups, user data |
| Forge Runners | modules/platform/forge_runners/ |
Main orchestration: composes EC2 + ARC, GitHub App, webhooks, logs |
| EKS | modules/infra/eks/ |
EKS cluster, addons, Karpenter, Calico, node groups |
| ECR | modules/infra/ecr/ |
Container registry for runner images |
| Storage | modules/infra/storage/ |
S3 buckets for artifacts and logs |
| AMI Policy | modules/infra/ami_policy/ |
DLM lifecycle policies, EBS encryption |
| AMI Sharing | modules/infra/ami_sharing/ |
Cross-account AMI sharing |
| Cloud Custodian | modules/infra/cloud_custodian/ |
Cost governance policies |
| CloudFormation | modules/infra/cloud_formation/ |
CloudFormation stack management |
| Forge Subscription | modules/infra/forge_subscription/ |
Cross-account IAM roles and ECR policies for subscribers |
| Opt-in Regions | modules/infra/opt_in_regions/ |
AWS opt-in region enablement |
| Service Linked Roles | modules/infra/service_linked_roles/ |
AWS service-linked role provisioning |
| Splunk Cloud Conf | modules/integrations/splunk_cloud_conf_shared/ |
Splunk props, transforms, dashboards |
| Splunk Data Manager | modules/integrations/splunk_cloud_data_manager/ |
CloudWatch + custom data ingestion to Splunk |
| Splunk Data Manager Common | modules/integrations/splunk_cloud_data_manager_common/ |
Shared Splunk DM IAM roles and config |
| Splunk AWS Billing | modules/integrations/splunk_aws_billing/ |
CUR billing data pipeline to Splunk |
| Splunk O11y Integration | modules/integrations/splunk_o11y_aws_integration/ |
CloudFormation-based AWS integration |
| Splunk O11y Common | modules/integrations/splunk_o11y_aws_integration_common/ |
SignalFx API integration + IAM |
| Splunk O11y Dashboards | modules/integrations/splunk_o11y_conf_shared/ |
SignalFx dashboard group + 7 dashboard types |
| Splunk OTel EKS | modules/integrations/splunk_otel_eks/ |
Splunk OTel collector Helm deployment |
| Splunk S3 Runner Logs | modules/integrations/splunk_cloud_s3_runner_logs/ |
Runner log shipping via Kinesis Firehose |
| Splunk Secrets | modules/integrations/splunk_secrets/ |
Secrets Manager entries for Splunk credentials |
| Teleport | modules/integrations/teleport/ |
Teleport agent for secure cluster access |
| Teleport Tenant | modules/integrations/teleport/tenant/ |
Per-tenant Teleport RBAC |
| Webhook Relay Source | modules/integrations/github_webhook_relay_source/ |
API Gateway + Lambda webhook ingress |
| Webhook Relay Destination | modules/integrations/github_webhook_relay_destination/ |
EventBridge destination bus + rules |
| Webhook Relay Receivers | modules/integrations/github_webhook_relay_destination_receivers/ |
Lambda receivers (e.g., Webex relay) |
Technology Stack
| Technology | Version | Purpose |
|---|---|---|
| Terraform / OpenTofu | ~> 1.11 | Infrastructure-as-Code engine |
| Terragrunt | Latest | Configuration hierarchy, dependency management |
| AWS Provider | >= 6.25 | AWS resource provisioning |
| Helm Provider | >= 3.0.0 | Kubernetes Helm chart management |
| Kubernetes Provider | >= 3.0 | Kubernetes resource management |
| Splunk Provider | >= 1.4.30 | Splunk Cloud configuration |
| SignalFx Provider | < 10.0.0 | Splunk Observability dashboards |
| Python | 3.12 | Lambda functions, pre-commit hooks |
| terraform-aws-modules/eks | 21.15.1 | EKS cluster module |
| terraform-aws-modules/lambda | 8.7.0 | Lambda function module |
| terraform-aws-modules/iam | 6.4.0 | IAM IRSA roles |
2. System Requirements
AWS Account Prerequisites
| Resource | Requirement |
|---|---|
| VPC | With private subnets for runners, NAT gateway for internet access |
| IAM | Permissions to create roles, policies, Lambda functions, EKS clusters |
| Secrets Manager | For Splunk credentials, GitHub App secrets |
| SSM Parameter Store | For GitHub App configuration |
| S3 | For Terraform state, runner logs, billing data, CloudFormation templates |
| EKS | For ARC runners (if using Kubernetes backend) |
| ECR | For custom runner container images |
| EventBridge | For webhook relay and scheduled Lambda triggers |
| CloudWatch | For Lambda logging and metrics |
| KMS | For encryption of logs, SQS queues, and S3 buckets |
GitHub App Requirements
The GitHub App must have these permissions:
| Permission | Level | Purpose |
|---|---|---|
| Actions | Read | Job log archival |
| Metadata | Read | Repository metadata access |
| Self-hosted runners | Read & Write | Runner registration/deregistration |
| Organization self-hosted runners | Read & Write | Runner group management |
Important
The GitHub App must be installed on the target organization. The installation ID, app ID, client ID, and base64-encoded private key must be available for Terraform to store in SSM.
Toolchain
| Tool | Version | Required For |
|---|---|---|
| Terraform / OpenTofu | ~> 1.11 | All modules |
| Terragrunt | Latest | Deployment orchestration |
| AWS CLI v2 | Latest | Provider authentication, scripts |
| kubectl | EKS-version compatible | EKS management, Karpenter, ARC |
| Helm | >= 3.0.0 | ARC controller + scale sets, OTel |
| Python | 3.12 | Pre-commit hooks, Lambda development |
| pre-commit | Latest | Code quality enforcement |
| shellcheck, shfmt | Latest | Shell script linting |
| tflint | Latest | Terraform linting |
| hadolint | Latest | Dockerfile linting |
| ansible-lint | v26.3.0 | Ansible linting |
| Gitleaks | v8.30.0 | Secret detection |
| commitizen | v4.13.9 | Commit message validation |
3. Configuration Reference
Global Settings (_global_settings/)
_global.yaml — shared values inherited by all environments:
# Example from templates/tenant/_global_settings/_global.yaml
# Contains: aws_profile, aws_region, default_tags, product metadata
_global.hcl — shared HCL configuration for provider setup, backend, and default inputs.
Per-module .hcl files — each module has a dedicated HCL file (e.g., tenant.hcl, eks.hcl, storage.hcl) that specifies the module source path and version:
# Example: _global_settings/eks.hcl
locals {
source = "${get_repo_root()}/modules/infra/eks"
}
Environment Settings (_environment_wide_settings/)
_environment.yaml — environment-specific overrides (e.g., dev vs. prod instance sizes, scaling limits).
_environment.hcl — environment-level HCL (e.g., different AWS profiles per environment).
Region Settings (_region_wide_settings/)
_region.hcl — region-specific configuration (subnet IDs, VPC IDs, availability zones).
VPC Settings (_vpc_wide_settings/)
_vpc.yaml / _vpc.hcl — VPC-level configuration for multi-VPC deployments, including subnet assignments and security group references.
Forge Runners Module — Key Variables
The forge_runners module (modules/platform/forge_runners/) is the main orchestration module. Key input variables:
deployment_config
variable "deployment_config" {
type = object({
deployment_prefix = string
secret_suffix = string
env = string
github_app = object({
id = string
client_id = string
installation_id = string
name = string
})
github = object({
ghes_org = string
ghes_url = string
repository_selection = string # "all" or "selected"
runner_group_name = string
})
tenant = object({
name = string
iam_roles_to_assume = optional(list(string), [])
ecr_registries = optional(list(string), [])
github_logs_reader_role_arns = optional(list(string), [])
})
})
}
Important
The
deployment_prefixis used to namespace all SSM parameters, log groups, and resource names. It must be unique per tenant deployment within an AWS account.
ec2_deployment_specs
variable "ec2_deployment_specs" {
type = object({
lambda_subnet_ids = list(string)
subnet_ids = list(string)
lambda_vpc_id = string
vpc_id = string
scale_errors = optional(list(string), [])
runner_specs = map(object({
ami_filter = object({ name = list(string), state = list(string) })
ami_kms_key_arn = string
ami_owners = list(string)
runner_labels = list(string)
runner_os = string # "linux", "osx", "windows"
runner_architecture = string # "x86_64", "arm64"
extra_labels = list(string)
max_instances = number
min_run_time = number
instance_types = list(string)
pool_config = list(object({
size = number
schedule_expression = string
schedule_expression_timezone = string
}))
runner_user = string
enable_userdata = bool
instance_target_capacity_type = string # "spot" or "on-demand"
block_device_mappings = list(object({...}))
}))
})
}
arc_deployment_specs
variable "arc_deployment_specs" {
type = object({
cluster_name = string
migrate_cluster = optional(bool, false)
runner_specs = map(object({
runner_size = object({
max_runners = number
min_runners = number
})
scale_set_name = string
scale_set_type = string # "k8s" or "dind"
container_actions_runner = string
container_limits_cpu = string
container_limits_memory = string
container_requests_cpu = string
container_requests_memory = string
volume_requests_storage_size = string
volume_requests_storage_type = string
}))
})
}
github_webhook_relay
variable "github_webhook_relay" {
type = object({
enabled = bool
destination_account_id = optional(string)
destination_event_bus_name = optional(string)
destination_region = optional(string)
destination_reader_role_arn = optional(string)
})
default = { enabled = false }
}
EKS Module — Key Variables
| Variable | Type | Default | Description |
|---|---|---|---|
cluster_name |
string |
required | EKS cluster name |
cluster_version |
string |
required | Kubernetes version |
cluster_size |
object |
required | Instance type, min/max/desired node count |
cluster_volume |
object |
required | EBS size, IOPS, throughput, type |
cluster_endpoint_public_access |
bool |
false |
Whether the API endpoint is public |
external_access_cidr_blocks |
list(string) |
[] |
CIDRs allowed to access the K8s API |
cluster_admin_role_arn |
string |
"" |
IAM role for cluster admin access |
subnet_ids |
list(string) |
required | Private subnets for worker nodes |
vpc_id |
string |
required | VPC ID |
Common Variables (shared across modules)
| Variable | Type | Description |
|---|---|---|
aws_profile |
string |
AWS CLI profile name |
aws_region |
string |
Target AWS region |
default_tags |
map(string) |
Default tags applied to all resources |
tags |
map(string) |
Additional resource-specific tags |
log_level |
string |
Lambda log level (INFO, DEBUG, WARN, ERROR) |
logging_retention_in_days |
number |
CloudWatch log retention (default varies: 3 or 30 days) |
Splunk Configuration Variables
| Module | Key Variable | Description |
|---|---|---|
splunk_aws_billing |
splunk_aws_billing_config.splunk_hec_url |
Splunk HEC endpoint URL |
splunk_aws_billing |
splunk_aws_billing_config.splunk_index |
Target Splunk index |
splunk_aws_billing |
splunk_aws_billing_config.splunk_metrics_url |
Splunk metrics endpoint |
splunk_cloud_conf_shared |
splunk_conf.splunk_cloud |
Splunk Cloud endpoint |
splunk_cloud_conf_shared |
splunk_conf.index |
Splunk index name |
splunk_cloud_conf_shared |
splunk_conf.tenant_names |
List of tenant names |
splunk_cloud_data_manager |
splunk_cloud |
Splunk Cloud endpoint |
splunk_o11y_aws_integration_common |
splunk_api_url |
SignalFx API URL |
splunk_o11y_aws_integration_common |
splunk_organization_id |
SignalFx org ID |
splunk_o11y_aws_integration_common |
integration_regions |
AWS regions to monitor |
splunk_otel_eks |
splunk_otel_collector |
OTel collector config object |
splunk_cloud_s3_runner_logs |
splunk_hec_endpoint |
Splunk HEC URL for runner logs |
Note
All Splunk secrets (HEC tokens, API tokens, Splunk Cloud credentials) are stored in AWS Secrets Manager and referenced by the modules via
aws_secretsmanager_secretdata sources. Never hardcode credentials in Terraform variables.
4. Deployment & Scaling
Deployment Order
Forge deployments follow a strict dependency order:
infra-forge— Foundation: EKS, ECR, storage, AMI policies, service-linked roles, opt-in regionsforge-extras— Optional: Cloud Custodian, additional ECR repos, Forge subscription (cross-account)forge-eks— EKS cluster with Karpenter and Calico (if using ARC runners)forge-integrations— Splunk Cloud configuration (props, transforms, dashboards)splunk-deployment— Splunk O11y, OTel, Data Manager, billing pipelineforge-tenant— Per-tenant runner deployment (last, depends on all above)
Deploying a Full Stack
# 1. Foundation
cd examples/infra-forge/terragrunt/environments/dev
terragrunt run-all apply
# 2. EKS cluster (if using ARC)
cd examples/deployments/forge-eks/terragrunt/environments/prod
terragrunt run-all apply
# 3. Integrations
cd examples/deployments/forge-integrations/terragrunt/environments/prod
terragrunt run-all apply
# 4. Splunk
cd examples/deployments/splunk-deployment/terragrunt/environments/prod
terragrunt run-all apply
# 5. Tenants
cd examples/deployments/forge-tenant/terragrunt/environments/prod
terragrunt run-all apply
Scaling EC2 Runners
EC2 runner pools scale via two mechanisms:
-
Reactive scaling — The upstream
terraform-aws-github-runnermodule responds to GitHub Actions webhook events to launch/terminate instances based on demand. -
Scheduled scaling — The
pool_configvariable defines baseline capacity on a cron schedule:
pool_config = [
{
size = 2
schedule_expression = "cron(0 8 ? * MON-FRI *)"
schedule_expression_timezone = "America/New_York"
},
{
size = 0
schedule_expression = "cron(0 20 ? * MON-FRI *)"
schedule_expression_timezone = "America/New_York"
}
]
Key scaling parameters per runner spec:
| Parameter | Description |
|---|---|
max_instances |
Hard ceiling on concurrent runners |
min_run_time |
Minimum job duration (minutes) before scale-down eligibility |
instance_target_capacity_type |
spot or on-demand |
instance_types |
List of allowed EC2 instance types |
Scaling ARC Runners
ARC runner pools scale via the GitHub Actions Runner Controller:
| Parameter | Description |
|---|---|
max_runners |
Maximum concurrent runners in the scale set |
min_runners |
Minimum warm runners (always running) |
container_limits_cpu / memory |
Pod resource limits |
container_requests_cpu / memory |
Pod resource requests (baseline) |
Karpenter manages the underlying EKS node capacity, provisioning nodes as pods demand increases.
Tip
Set
min_runnersto at least 1 for latency-sensitive workflows. Cold starts (launching a new runner pod + node) can add 2-5 minutes to job startup.
ARC Cluster Migration
Set migrate_arc_cluster = true when performing a blue/green cluster cutover. This flag is propagated to the scale set controller and scale set modules to coordinate the migration.
5. Operations
Lambda Functions
Forge deploys several operational Lambda functions:
| Lambda | Module | Trigger | Purpose |
|---|---|---|---|
forge_trust_validator |
forge_runners/forge_trust_validator |
EventBridge schedule | Validates IAM trust relationships for runner roles |
ec2_update_runner_ssm_ami |
ec2_deployment/ec2_update_runner_ssm_ami |
EventBridge schedule | Updates SSM with latest AMI matching filters |
ec2_update_runner_tags |
ec2_deployment/ec2_update_runner_tags |
EventBridge (event bus) | Tags EC2 runner instances on state changes |
github_app_runner_group |
forge_runners/github_app_runner_group |
EventBridge schedule | Creates/updates GitHub runner groups, attaches repos |
github_clean_global_lock |
forge_runners/github_global_lock |
EventBridge schedule | Cleans stale GitHub global locks |
redrive_deadletter |
forge_runners/redrive_deadletter |
EventBridge schedule | Redrives messages from DLQ back to main queue |
job_log_dispatcher |
forge_runners/github_actions_job_logs |
EventBridge (webhook) | Dispatches completed job events to SQS |
job_log_archiver |
forge_runners/github_actions_job_logs |
SQS | Downloads and archives GitHub Actions job logs to S3 |
validate_signature |
github_webhook_relay_source |
API Gateway | Validates GitHub webhook HMAC signatures |
webex_webhook_relay |
github_webhook_relay_destination_receivers |
EventBridge | Forwards failure notifications to Webex rooms |
splunk_s3_runner_logs |
splunk_cloud_s3_runner_logs |
SQS (S3 notifications) | Ships runner logs to Splunk via Kinesis Firehose |
cur_per_resource |
splunk_aws_billing |
S3 notification | Processes per-resource CUR billing data |
cur_per_resource_process |
splunk_aws_billing |
S3 notification | Aggregates and sends per-resource costs to Splunk HEC |
cur_per_service |
splunk_aws_billing |
S3 notification | Processes per-service CUR billing data to Splunk |
sec_meta_ec2_tags |
splunk_cloud_data_manager/sec_meta_ec2_tags |
(data manager) | EC2 tag metadata for Splunk security enrichment |
Note
All Lambda functions use a configurable
log_levelvariable (defaultINFO) andlogging_retention_in_daysfor CloudWatch log group retention. Common values: 3 days for integration Lambdas, 30 days for platform Lambdas.
GitHub Actions Job Log Archival
The job log archiver supports two modes:
Queue Pipeline (default, recommended):
GitHub Webhook → EventBridge → Dispatcher Lambda → SQS → Archiver Lambda → S3
- Better isolation, retry semantics, and backpressure handling
- SQS dead-letter queue for failed downloads
- Configurable batch size (
downloader_batch_size) and concurrency (downloader_max_concurrency)
Direct Mode (fallback):
GitHub Webhook → EventBridge → Archiver Lambda → S3
- Simpler, fewer moving parts
- Set
enable_queue_pipeline = falseto use
Logs are stored at: s3://<bucket>/<workflow_name>/<run_attempt>/<job_id>.zip
Monitoring
Forge provides built-in monitoring through Splunk Observability (SignalFx) dashboards:
| Dashboard | Module Path | Metrics |
|---|---|---|
| Runner EC2 | dashboards/runner_ec2/ |
CPU, memory, disk, network utilization; active hosts by AZ and instance type |
| Runner K8s | dashboards/runner_k8s/ |
Active pods, memory usage, network bytes, pod phases |
| Billing | dashboards/billing/ |
Cost per service, cost per tenant, net costs |
| Lambda | dashboards/lambda/ |
Invocations, errors, duration, throttles, concurrency |
| SQS | dashboards/sqs/ |
Queue depth, message age, processing trends |
| EBS | dashboards/ebs/ |
IOPS, throughput, latency, queue length, utilization |
| DynamoDB | dashboards/dynamodb/ |
Request latency, throttles, capacity utilization |
Each dashboard supports dynamic variables for filtering by tenant via the dashboard_variables input.
Tip
The Splunk Cloud Conf module (
splunk_cloud_conf_shared) provisions three Splunk dashboards (CI Jobs, Tenant, Trust Relationship Validation) as saved searches usingsplunk_data_ui_viewsresources.
Secrets Management
| Secret Type | Storage | Modules |
|---|---|---|
| GitHub App private key | SSM Parameter Store (SecureString) | forge_runners |
| GitHub App webhook secret | SSM Parameter Store | forge_runners (auto-rotated every 30 days) |
| Splunk Cloud credentials | Secrets Manager | All Splunk integration modules |
| Splunk HEC tokens | Secrets Manager | splunk_aws_billing, splunk_cloud_s3_runner_logs |
| SignalFx API token | Secrets Manager | splunk_o11y_aws_integration_common, splunk_o11y_conf_shared |
| Webex bot token | Secrets Manager | webex_webhook_relay |
Warning
The Webex webhook relay Lambda expects a Secrets Manager secret at
/cicd/common/webex_webhook_relay_bot_tokencontaining JSON withtokenandroom_idkeys. Both are required. The Lambda auto-prependsBearerto the token if not present.
Operational Scripts
| Script | Purpose |
|---|---|
scripts/migrate-tenant.sh |
Migrate a single tenant to new configuration |
scripts/migrate-all-tenants.sh |
Batch tenant migration |
scripts/reinstall-eks-with-deps.sh |
Reinstall EKS cluster with all dependencies |
scripts/update-github-app-secrets.sh |
Update GitHub App credentials in SSM |
scripts/update-terraform-docs.sh |
Regenerate terraform-docs for all modules |
scripts/generate_and_patch_github_app.sh |
Generate and patch GitHub App webhook configuration |
scripts/download_lambdas.sh |
Download Lambda deployment packages |
6. Performance
EKS Cluster Sizing
The EKS module exposes cluster_size for the self-managed node group:
cluster_size = {
instance_type = "m5.large"
min_size = 2
max_size = 10
desired_size = 3
}
And cluster_volume for EBS configuration:
cluster_volume = {
size = 100
iops = 3000
throughput = 125
type = "gp3"
}
Karpenter handles additional node provisioning beyond the base node group, using NodePool and EC2NodeClass custom resources applied via null_resource provisioners.
ARC Scale Set Resource Tuning
Each ARC runner pod's resources are individually configurable:
| Setting | Example | Impact |
|---|---|---|
container_requests_cpu |
"500m" |
Baseline CPU reservation |
container_requests_memory |
"1Gi" |
Baseline memory reservation |
container_limits_cpu |
"2000m" |
CPU burst ceiling |
container_limits_memory |
"4Gi" |
Memory ceiling (OOM-kill above) |
volume_requests_storage_size |
"20Gi" |
Persistent volume for workspace |
volume_requests_storage_type |
"gp3" |
Storage class |
EC2 Runner Instance Selection
Configure instance types per runner pool for cost/performance optimization:
instance_types = ["m5.xlarge", "m5a.xlarge", "m5d.xlarge"]
instance_target_capacity_type = "spot" # or "on-demand"
Tip
Use multiple instance types in the
instance_typeslist when using spot capacity to improve availability. The runner module selects from the list based on current spot pricing and capacity.
7. Security
IAM Architecture
Forge implements a layered IAM model:
- Runner IAM roles — Each tenant's runners assume a dedicated IAM role (
role_for_forge_runners) with scoped permissions for S3, Secrets Manager, ECR, and Packer operations - Forge trust validator — A scheduled Lambda that validates IAM trust policies on runner roles, ensuring no unauthorized principals can assume them
- ECR access policies — Per-tenant ECR repository policies granting pull access only to authorized runner roles
- Cross-account roles — The
forge_subscriptionmodule creates IAM roles and ECR policies for subscriber accounts
Trust Validation
The forge_trust_validator Lambda (modules/platform/forge_runners/forge_trust_validator/lambda/forge_trust_validator.py) runs on a schedule and:
- Reads the trust policy of each configured Forge IAM role
- Validates that only expected principals appear in the trust relationship
- Reports violations (the Splunk Cloud Conf module provides a Trust Relationship Validation dashboard)
Webhook Signature Validation
The GitHub webhook relay source Lambda (modules/integrations/github_webhook_relay_source/lambda/validate_signature.py) validates incoming webhook payloads using HMAC-SHA256 with the GitHub App webhook secret before forwarding events to EventBridge.
Encryption
| Resource | Encryption |
|---|---|
| EBS volumes | Default encryption enabled via aws_ebs_encryption_by_default (in ami_policy module) |
| S3 buckets | Server-side encryption (SSE-S3 or SSE-KMS depending on module) |
| SQS queues | KMS encryption for job log queues |
| Kinesis streams | KMS encryption for runner log streaming |
| SSM parameters | SecureString type for sensitive values |
Caution
The
ami_policymodule enables EBS encryption by default at the account level viaaws_ebs_encryption_by_default. This affects ALL EBS volumes in the region, not just Forge resources. Coordinate with other teams before applying.
Network Security
- EC2 runners are deployed in private subnets with security groups managed by the
ec2_deploymentmodule - EKS clusters default to
cluster_endpoint_public_access = false(private API endpoint) external_access_cidr_blocksmust be explicitly configured for any external access to the K8s API- Calico network policies are applied via the EKS module for pod-level network segmentation
Pre-Commit Security Hooks
| Hook | Purpose |
|---|---|
detect-private-key |
Prevents committing private keys |
gitleaks (v8.30.0) |
Scans for secrets and credentials |
check-added-large-files |
Prevents accidentally committing large binaries |
8. Developer Internals
Repository Structure
cisco-open/forge/
├── modules/
│ ├── core/
│ │ └── arc/ # ARC scale set controller + scale sets
│ │ ├── scale_set/ # Individual scale set (Helm + K8s + IAM)
│ │ ├── scale_set_controller/ # ARC controller (Helm + namespace + secret)
│ │ └── templates/ # Karpenter NodePool template
│ ├── infra/
│ │ ├── ami_policy/ # DLM lifecycle, EBS encryption
│ │ ├── ami_sharing/ # Cross-account AMI sharing
│ │ ├── cloud_custodian/ # Cost governance
│ │ ├── cloud_formation/ # CF stack management
│ │ ├── ecr/ # Container registry
│ │ ├── eks/ # EKS cluster + addons + Karpenter
│ │ ├── forge_subscription/ # Cross-account subscription
│ │ ├── opt_in_regions/ # AWS region opt-in
│ │ ├── service_linked_roles/ # SLR provisioning
│ │ └── storage/ # S3 buckets
│ ├── integrations/
│ │ ├── github_webhook_relay_source/ # Webhook ingress (API GW + Lambda)
│ │ ├── github_webhook_relay_destination/ # EventBridge destination bus
│ │ ├── github_webhook_relay_destination_receivers/ # Lambda receivers
│ │ ├── splunk_aws_billing/ # CUR billing pipeline
│ │ ├── splunk_cloud_conf_shared/ # Splunk props/transforms/dashboards
│ │ ├── splunk_cloud_data_manager/ # CloudWatch data ingestion
│ │ ├── splunk_cloud_data_manager_common/ # Shared DM config
│ │ ├── splunk_cloud_s3_runner_logs/ # Runner log shipping
│ │ ├── splunk_o11y_aws_integration/ # AWS integration (CloudFormation)
│ │ ├── splunk_o11y_aws_integration_common/ # SignalFx API integration
│ │ ├── splunk_o11y_conf_shared/ # SignalFx dashboards (7 types)
│ │ ├── splunk_otel_eks/ # OTel collector Helm
│ │ ├── splunk_secrets/ # Secrets Manager entries
│ │ └── teleport/ # Teleport agent + tenant RBAC
│ └── platform/
│ ├── arc_deployment/ # ARC deployment wrapper
│ ├── ec2_deployment/ # EC2 runners (AMI, SG, user data, Lambdas)
│ └── forge_runners/ # Main orchestration module
│ ├── forge_trust_validator/ # IAM trust validation Lambda
│ ├── github_actions_job_logs/ # Job log archiver (dispatcher + archiver)
│ ├── github_app_runner_group/ # Runner group management Lambda
│ ├── github_global_lock/ # Global lock cleanup Lambda
│ ├── github_webhook_relay/ # Webhook relay orchestration
│ ├── redrive_deadletter/ # DLQ redrive Lambda
│ └── scripts/ # GitHub App patching script
├── examples/
│ ├── deployments/ # Full deployment examples per stack
│ │ ├── forge-eks/
│ │ ├── forge-extras/
│ │ ├── forge-integrations/
│ │ ├── forge-tenant/
│ │ └── splunk-deployment/
│ ├── infra-forge/ # Foundation infrastructure deployment
│ └── templates/ # Configuration templates per stack type
├── docs/ # Internal documentation
│ ├── configurations/ # Deployment configuration guides
│ └── tenant-usage/ # Tenant onboarding and Renovate
├── scripts/ # Operational scripts
├── .pre-commit-config.yaml # 30+ pre-commit hooks
├── .terraform-docs.yml # terraform-docs configuration
├── renovate.json # Renovate dependency management
├── .ansible-lint # Ansible linting config
└── .yamllint # YAML linting config
Module Composition Pattern
The forge_runners module demonstrates the primary composition pattern:
# modules/platform/forge_runners/
module "ec2_runners" { source = "../ec2_deployment" }
module "arc_runners" { source = "../arc_deployment" }
module "forge_trust_validator" { source = "./forge_trust_validator" }
module "github_actions_job_logs" { source = "./github_actions_job_logs" }
module "github_app_runner_group" { source = "./github_app_runner_group" }
module "github_global_lock" { source = "./github_global_lock" }
module "github_webhook_relay" { source = "./github_webhook_relay" }
module "redrive_deadletter" { source = "./redrive_deadletter" }
This single module composes all runner-related infrastructure, creating a cohesive deployment unit per tenant.
Code Patterns and Conventions
State management: Each module uses a backend.tf with S3 backend configuration. State paths follow the Terragrunt hierarchy.
Provider configuration: Providers are declared in providers.tf per module, with versions pinned in versions.tf.
Tagging: Every module accepts default_tags and tags variables. The AWS provider default_tags feature applies organization-wide tags, while tags adds module-specific tags.
Naming: Resources use a prefix or deployment_prefix variable for namespace isolation. SSM parameters follow /forge/<prefix>/<parameter_name> convention.
Lambda packaging: Lambda functions are Python 3.12, packaged in-module under lambda/ directories. The terraform-aws-modules/lambda/aws module (v8.7.0) handles packaging and deployment.
Template files: Terraform template files use .tftpl extension and live in template_files/ directories within their respective modules.
External data sources: Several modules use external_external data sources to call shell scripts (e.g., fetch_secret_value.sh, list_buckets.sh) for operations that lack native Terraform providers.
Webhook Relay Architecture
The webhook relay spans three modules:
-
Source (
github_webhook_relay_source/) — API Gateway endpoint receives GitHub webhooks → Lambda validates HMAC signature → forwards valid events to EventBridge source bus -
Destination (
github_webhook_relay_destination/) — EventBridge destination bus in another account receives cross-account events → rules route to Lambda targets based on event patterns -
Receivers (
github_webhook_relay_destination_receivers/) — Lambda functions that process forwarded events. The includedwebex_webhook_relayreceiver:- Filters for
mainbranch failures (cancelled,failure,timed_out,timeout,startup_failure) - Builds a Slack-format card, converts to Webex Adaptive Card format
- Sends to Webex via the Messages API using a bot token from Secrets Manager
- Filters for
Splunk Billing Pipeline
The billing integration (splunk_aws_billing/) implements a three-Lambda pipeline:
-
cur_per_resource— Triggered by S3 notifications when new CUR (Cost and Usage Report) parquet files arrive. Splits data by date and re-uploads to S3 for processing. -
cur_per_resource_process— Groups billing data by resource, service, and application. Sends events to Splunk HEC in batches (respectingMAX_BATCH_COUNTandMAX_BATCH_SIZE_BYTES). Also sends metrics to SignalFx (forge.per_resource.cost_usd,forge.per_resource.net_cost_usd). -
cur_per_service— Similar to per-resource but aggregated at the service level. Sendsforge.per_service.cost_usdandforge.per_service.net_cost_usdmetrics.
All three Lambdas use a shared common.py module for Splunk HEC/SignalFx communication and data preprocessing.
Testing Strategy
Forge relies on:
- Pre-commit validation —
terraform validate,terraform fmt,tflint,tofu-validaterun on every commit - Terraform plan —
terragrunt run-all planserves as the primary integration test - Safety hooks —
safety_hook.shprovides environment-aware guardrails before apply
Code Quality Toolchain
The .pre-commit-config.yaml configures 30+ hooks across categories:
| Category | Hooks |
|---|---|
| General | YAML/JSON/TOML/XML syntax, merge conflict markers, large file guard, line endings |
| Python | autopep8, isort, autoflake, flake8 (E501 ignored), pyupgrade, validate-pyproject |
| Shell | shellcheck, shfmt (4-space indent), bashate, script-must-have-extension |
| Terraform/IaC | terragrunt-hcl-fmt, tofu-fmt, tofu-validate, terraform_fmt, terraform_tflint, terraform_validate, packer-validate |
| Security | gitleaks, detect-private-key |
| Docker | hadolint |
| Ansible | ansible-lint (v26.3.0, Python 3.12) |
| Markdown | mdformat (with GFM, ruff, frontmatter plugins, excludes modules/) |
| JSON Schema | GitHub workflows, GitHub Actions, Docker Compose, Dependabot, Renovate |
| Commit | commitizen with cz-conventional-gitmoji |
Dependency Management
Renovate (renovate.json) manages all dependency updates:
| Rule | Behavior |
|---|---|
| Security patches | Immediate processing, auto-merge, security + critical labels |
| Patch/pin/digest | Auto-merge with simple-review label |
| Minor updates | simple-review label, no auto-merge |
| Major updates | 7-day stability period, major-update + breaking-change labels |
| AWS providers/modules | Grouped as "AWS providers and modules" |
| GitHub Actions | Pinned to commit SHA, grouped |
| Pre-commit hooks | Grouped as "Pre-commit hooks" |
| Docker images | Digest pinning |
| Terraform providers | Resolved from OpenTofu registry |
Custom regex managers extract versions from:
.tffiles (required_version)download_lambdas.shreferences (GitHub runner version)- Dockerfiles (with renovate comments)
Contributing
Forge follows conventional commits with gitmoji (enforced by commitizen pre-commit hook). Key practices:
- All Terraform code must pass
terraform validate,terraform fmt, andtflint - Shell scripts must pass
shellcheckandshfmt - Python code must pass
flake8(E501 ignored),autopep8,isort, andautoflake - Module documentation is auto-generated — update source code, then run
scripts/update-terraform-docs.sh - YAML files are formatted with
yamlfmt(2-space mapping, 4-space sequence, 2-space offset, 180 char width)
Potential Improvements
Based on open GitHub issues:
- Lifecycle field refactoring (#236) — Replace
countconditionals with thelifecycle.enabledfield for cleaner resource toggling - Grafana/PagerDuty integration (#125, #124) — Add alerting pipeline from Grafana and Splunk Cloud to PagerDuty
- AWS IAM Roles Anywhere (#122) — Enable multi-cloud ephemeral runners via certificate-based authentication
- Open-source monitoring (#121) — Add Loki, Prometheus, and Grafana as alternatives to Splunk
- Cost safeguards (#296) — Default cost guardrails and hardening for runner environments
9. Integration Points
AWS Service Integration
| AWS Service | Forge Usage |
|---|---|
| EC2 | Runner instances (Linux, macOS, Windows) |
| EKS | Kubernetes clusters for ARC runners |
| ECR | Container image registry for runner images |
| S3 | Terraform state, runner logs, billing data, CloudFormation templates, Firehose backup |
| SSM Parameter Store | GitHub App configuration, AMI IDs |
| Secrets Manager | Splunk credentials, Webex bot tokens |
| EventBridge | Webhook relay, Lambda scheduling, EC2 state change events |
| API Gateway | GitHub webhook ingress endpoint |
| Lambda | 14+ operational functions |
| CloudWatch | Lambda logs, metrics, event rules |
| KMS | Encryption for S3, SQS, Kinesis, EBS |
| IAM | Runner roles, service roles, cross-account access |
| CloudFormation | Splunk O11y AWS integration |
| DLM | EBS snapshot lifecycle policies |
| Kinesis Data Streams | Runner log streaming |
| Kinesis Firehose | Splunk HEC delivery |
| SQS | Job log dispatch queue, dead-letter queues |
| Service Catalog | AppRegistry for Forge application tracking |
| BCM Data Exports | Cost and Usage Report (CUR) exports |
| Calico | EKS network policies |
Third-Party Integrations
| Service | Integration Module | Purpose |
|---|---|---|
| Splunk Cloud | splunk_cloud_conf_shared, splunk_cloud_data_manager |
Log ingestion, props/transforms, dashboards |
| Splunk Observability (SignalFx) | splunk_o11y_aws_integration_common, splunk_o11y_conf_shared |
Metrics, dashboards |
| Splunk OTel | splunk_otel_eks |
Kubernetes observability via Helm |
| Teleport | teleport/, teleport/tenant/ |
Secure cluster access with per-tenant RBAC |
| Webex | webex_webhook_relay |
CI failure notifications |
| GitHub (API) | forge_runners, github_app_runner_group |
Runner management, App webhook configuration |
Terraform Module Dependencies
| External Module | Version | Used By |
|---|---|---|
terraform-aws-modules/eks/aws |
21.15.1 | modules/infra/eks/ |
terraform-aws-modules/eks/aws//modules/karpenter |
21.15.1 | modules/infra/eks/ |
terraform-aws-modules/eks/aws//modules/self-managed-node-group |
21.15.1 | modules/infra/eks/ |
terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts |
6.4.0 | modules/infra/eks/ |
terraform-aws-modules/lambda/aws |
8.7.0 | Multiple Lambda-based modules |