AWS CLI Administrator & Developer Guide

1. Architecture Overview

AWS CLI v1 (version 1.44.85) is a universal command-line interface for interacting with Amazon Web Services. Built in Python, it wraps the botocore SDK with a full-featured CLI layer including argument parsing, output formatting, help generation, and a plugin/event system for extensibility.

Note

This document covers the v1 branch of AWS CLI. AWS CLI v2 is a separate codebase with different architecture decisions (bundled Python, auto-prompt, SSO integration). V1 remains actively maintained.

High-Level Component Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         User Shell                                   │
│   $ aws <service> <operation> [--params]                            │
└────────────────────────────┬────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│  bin/aws  →  awscli/__main__.py  →  awscli.clidriver.main()        │
└────────────────────────────┬────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      CLIDriver                                       │
│  ┌──────────────┐  ┌───────────────┐  ┌────────────────────────┐   │
│  │ MainArgParser│  │ Command Table │  │ botocore Session       │   │
│  │              │  │ (from models) │  │  - credentials         │   │
│  └──────┬───────┘  └───────┬───────┘  │  - config             │   │
│         │                   │          │  - event system        │   │
│         ▼                   ▼          │  - service models      │   │
│  ┌──────────────────────────────┐     └────────────┬───────────┘   │
│  │     ServiceCommand           │                  │               │
│  │       └─ ServiceOperation    │◄─────────────────┘               │
│  │            └─ botocore call  │                                   │
│  └──────────────────────────────┘                                   │
└─────────────────────────────────────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│                   Output Pipeline                                    │
│  ┌──────────┐  ┌──────────┐  ┌────────────┐  ┌──────────────────┐ │
│  │ JSON     │  │ Text     │  │ Table      │  │ YAML (optional)  │ │
│  │Formatter │  │Formatter │  │Formatter   │  │ Formatter        │ │
│  └──────────┘  └──────────┘  └────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Boot Sequence

1. bin/aws invokes awscli/__main__.py
2. main() calls create_clidriver()
3. create_clidriver() instantiates a botocore Session
4. Plugins are loaded from ~/.aws/config [plugins] section
5. CLIDriver is constructed with the session
6. CLIDriver.main() parses top-level args, locates the service command
7. ServiceCommand.__call__() parses the operation, delegates to ServiceOperation
8. ServiceOperation.__call__() invokes the botocore API call
9. Response passes through the formatter and is written to stdout

Event-Driven Extensibility

The entire CLI is wired through botocore's event system. Events fire at every stage:

Customizations in awscli/customizations/ register handlers for these events to modify behavior, inject arguments, rewrite parameters, or add entirely new commands.

2. System Requirements

Runtime Requirements

Python Dependencies

Important

The botocore version is pinned exactly (==1.42.95). Upgrading botocore independently will break API compatibility. Always upgrade aws-cli and botocore together.

Build/Development Requirements

3. Configuration Reference

Configuration Files

Environment Variables

Config File Options (~/.aws/config)

[default]
region = us-east-1
output = json
cli_pager = less

[profile dev]
region = us-west-2
output = table
role_arn = arn:aws:iam::123456789012:role/dev-role
source_profile = default
credential_source = Environment

[plugins]
myplugin = awscli_plugin_module

Tip

Use aws configure to interactively set up profiles. Use aws configure set for scripted configuration. Use aws configure list to see which config values are active and where they came from.

Alias Configuration (~/.aws/cli/alias)

[toplevel]
whoami = sts get-caller-identity
running-instances = ec2 describe-instances --filters Name=instance-state-name,Values=running
myip = !curl -s https://checkip.amazonaws.com

Two alias types exist:

Service aliases are parsed and executed within the CLI process. External aliases (prefixed with !) spawn a subprocess.

4. Deployment & Scaling

Installation Methods

pip (recommended for v1):

pip install awscli

From source:

git clone https://github.com/aws/aws-cli.git
cd aws-cli
pip install -e .

System package managers:

# Debian/Ubuntu
sudo apt install awscli

# RHEL/CentOS
sudo yum install awscli

# macOS (Homebrew)
brew install awscli@1

Shell Completion Setup

Bash:

# Add to ~/.bashrc
complete -C aws_completer aws
# Or source the helper:
source /path/to/bin/aws_bash_completer

Zsh:

# Add to ~/.zshrc
source /path/to/bin/aws_zsh_completer.sh

The completer binary (bin/aws_completer) implements tab completion by introspecting botocore service models dynamically.

Multi-Account / Multi-Region Deployment

Note

AWS CLI v1 supports multiple named profiles natively. Each profile can target a different account, region, and credential source.

For fleet deployments:

1. Shared credentials via instance profiles - Use credential_source = Ec2InstanceMetadata or EcsContainer to avoid distributing static keys.
2. Role chaining - Use role_arn + source_profile to centralize credential management.
3. Config distribution - Distribute ~/.aws/config via configuration management (Ansible, Chef, Puppet). Keep ~/.aws/credentials empty when using instance roles.

Containerized Usage

FROM python:3.11-slim
RUN pip install awscli==1.44.85
# Credentials injected via environment or instance role
ENV AWS_DEFAULT_REGION=us-east-1
ENTRYPOINT ["aws"]

5. Operations

Credential Resolution Order

The CLI resolves credentials in this fixed order (first match wins):

1. Command-line options (--profile)
2. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
3. Credentials file (~/.aws/credentials)
4. Config file (~/.aws/config)
5. Assume-role (if role_arn configured, using source credentials)
6. Container credentials (ECS task role via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)
7. Instance metadata (EC2 instance profile via IMDS)

Credential Caching

When using role_arn (assume-role), the CLI caches temporary credentials in ~/.aws/cli/cache/ as JSON files. The cache is implemented by JSONFileCache and keyed by a hash of the role parameters. Cached credentials are reused until they expire.

Output Formats

All formatters support --query (JMESPath) for server-side result filtering.

Pagination

The CLI auto-paginates by default. Control with:

Retry Behavior

Retries are handled by botocore with exponential backoff. The default retry mode retries up to 4 times for throttling and transient errors. Configure via:

[default]
retry_mode = standard  # or adaptive, legacy
max_attempts = 5

Warning

Setting max_attempts too high with adaptive mode can cause long delays under sustained throttling. Monitor CloudWatch API call metrics to tune appropriately.

Logging and Debugging

# Enable debug output (sent to stderr)
aws --debug s3 ls

# Log botocore wire-level details
aws --debug ec2 describe-instances 2> debug.log

Debug output includes: credential resolution, endpoint calculation, request signing, wire-format request/response, retry attempts, and event handler invocations.

History (Command Logging)

The aws history customization (awscli/customizations/history/) records CLI invocations:

aws history show       # Show recent commands
aws history list       # List recorded command IDs

6. Performance

S3 Transfer Configuration

The aws s3 commands (cp, sync, mv) use s3transfer for high-performance multipart operations:

Configure in ~/.aws/config:

[default]
s3 =
  max_concurrent_requests = 20
  multipart_threshold = 64MB
  multipart_chunksize = 16MB
  max_bandwidth = 50MB/s

S3 Benchmarking

The scripts/performance/benchmark_utils.py module provides utilities for measuring S3 transfer performance. It supports:

• Configurable file sizes and concurrency levels
• Throughput measurement (MB/s)
• Comparison across different s3transfer configurations

CLI Startup Time

CLI startup involves loading botocore, parsing config, loading plugins, and building the command table. Strategies to minimize cold-start latency:

• Avoid unnecessary plugins in [plugins]
• Use AWS_DATA_PATH only when custom models are needed
• On Lambda/containers, pre-warm by running aws --version in init

Tip

If you only use a few services, startup time is not significantly affected because command table construction is lazy - individual service models load only when that service is invoked.

7. Security

Credential Security Best Practices

File Permissions

chmod 600 ~/.aws/credentials
chmod 600 ~/.aws/config
chmod 700 ~/.aws/cli/cache/

Caution

The credentials file and cached role credentials contain sensitive material. Ensure they are readable only by the owning user. The CLI does NOT enforce file permissions automatically.

Request Signing

All API requests are signed using AWS Signature Version 4 (handled by botocore). The signing process:

1. Creates a canonical request (method, path, query, headers, payload hash)
2. Creates a string to sign (algorithm, timestamp, credential scope, canonical request hash)
3. Derives a signing key from the secret access key + date + region + service
4. Computes HMAC-SHA256 signature
5. Adds Authorization header with credential, signed headers, and signature

SSL/TLS

• All requests use HTTPS by default
• Custom CA bundles: --ca-bundle flag or AWS_CA_BUNDLE env var or ca_bundle config
• Certificate verification can be disabled with --no-verify-ssl (NOT recommended for production)

Caution

Disabling SSL verification (--no-verify-ssl) exposes credentials and data to man-in-the-middle attacks. Only use for debugging with self-signed certificates in development environments.

CloudFront URL Signing

The awscli/customizations/cloudfront.py module uses the rsa dependency to generate signed URLs and signed cookies for CloudFront distributions. Private keys must be PEM-encoded RSA keys associated with a CloudFront key pair.

Paramfile Security

The CLI supports loading parameter values from files (file://path) and URLs (https://url). This is implemented in awscli/paramfile.py:

• file:// - reads local file contents as the parameter value
• fileb:// - reads local file as binary (base64-encoded for JSON params)
• https:// - fetches URL contents (use with caution - validates SSL by default)

Warning

The https:// paramfile scheme fetches content from arbitrary URLs at request time. Ensure URLs are trusted and served over verified TLS to prevent parameter injection.

8. Developer Internals

Module Reference

Command Dispatch Architecture

CLIDriver.main(args)
  │
  ├── parse top-level args (--region, --output, --profile, --debug)
  │
  ├── look up service in command_table
  │        command_table built from botocore.session.get_available_services()
  │        + customizations injected via 'building-command-table.main' event
  │
  ├── ServiceCommand.__call__(remaining_args, parsed_globals)
  │     │
  │     ├── build operation sub-command table from service model
  │     │     + customizations via 'building-command-table.<service>' event
  │     │
  │     └── ServiceOperation.__call__(remaining_args, parsed_globals)
  │           │
  │           ├── build argument table from operation model
  │           │     + customizations via 'building-argument-table' event
  │           │
  │           ├── parse operation-specific args
  │           │
  │           ├── fire 'process-cli-arg' for each argument
  │           │
  │           ├── invoke botocore API call
  │           │
  │           └── format + display response
  │
  └── return exit code (0=success, 1=client error, 2=server error, 255=other)

Argument Type System

The argument hierarchy enables both model-driven and custom arguments:

BaseCLIArgument (abstract)
  ├── CLIArgument         ← from botocore operation model
  ├── CustomArgument      ← manually defined (customizations)
  │     ├── BooleanArgument
  │     └── ListArgument
  └── (per-customization subclasses)

Each argument knows how to:

• Add itself to an argparse parser (add_to_parser())
• Process its parsed value into API-compatible format (add_to_params())

Shorthand Syntax Parser

The shorthand parser (awscli/shorthand.py) allows compact input without JSON:

# Instead of:
aws ec2 run-instances --tag-specifications '[{"ResourceType":"instance","Tags":[{"Key":"Name","Value":"test"}]}]'

# Shorthand:
aws ec2 run-instances --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=test}]

Parser implementation uses a recursive descent approach. Key error types:

• ShorthandParseSyntaxError - malformed shorthand expression
• DuplicateKeyInObjectError - repeated keys in shorthand object
• ParamUnknownKeyError - unrecognized key name

Customization Registration Pattern

All customizations are registered centrally in awscli/handlers.py:

# Pattern: register handler for a specific event
def register_custom_commands(event_handlers):
    event_handlers.register('building-command-table.main', add_custom_commands)

# Handler modifies the command table in-place
def add_custom_commands(command_table, session, **kwargs):
    command_table['my-command'] = MyCustomCommand(session)

Major customizations include:

Help System Internals

Help generation uses the bcdoc subpackage:

1. CLIDocumentEventHandler (and variants: Provider, Service, Operation) listens for doc events
2. Generates reStructuredText from botocore model descriptions + awscli/data/ overlays
3. On Unix: pipes reST through rst2man.py (docutils) then groff/mandoc to the pager
4. On Windows: renders plain text directly to the pager
5. awscli/data/ contains per-service overlays adding CLI-specific examples to help pages

Document event flow:

doc-title → doc-description → doc-synopsis → doc-options → doc-examples → doc-output

Each event handler class (ProviderDocumentEventHandler, ServiceDocumentEventHandler, OperationDocumentEventHandler) generates appropriate content for its scope level.

Plugin System

Plugins are loaded from the [plugins] section of ~/.aws/config:

[plugins]
my_plugin = my_plugin_module

The plugin loading process (awscli/plugin.py):

1. Import the module specified as the value
2. Call module.awscli_initialize(event_hooks) passing the botocore event emitter
3. The plugin registers handlers for whatever events it needs

Plugins can:

• Add new top-level commands
• Modify argument tables
• Intercept and transform requests/responses
• Add new output formats
• Inject custom credential providers

Data-Driven Model Architecture

The CLI is fundamentally data-driven via botocore service models:

• Service models (JSON): define every API operation, its inputs, outputs, and errors
• CLI overlays (awscli/data/): add CLI-specific metadata (examples, extra descriptions)
• Custom data path (AWS_DATA_PATH): allows injecting additional/override models

This means adding a new AWS service to the CLI requires zero code changes - only a new service model JSON file from botocore.

Error Handling

The codebase defines 86 error types across modules:

Exit codes: 0 = success, 1 = client error (4xx or argument error), 2 = server error (5xx), 255 = unhandled exception.

Testing Strategy

Note

The test suite uses pytest and is organized into three tiers with increasing scope and cost.

Key test infrastructure:

• CLIRunner - test helper that invokes CLI commands programmatically
• SessionStubber - patches botocore HTTP layer to return canned responses
• Pre-commit hooks: check-yaml, end-of-file-fixer, trailing-whitespace, isort, ruff

Running tests:

# Unit tests
pytest tests/unit/

# Functional tests
pytest tests/functional/

# Specific test file
pytest tests/unit/test_clidriver.py

# With coverage
pytest --cov=awscli tests/unit/

Contributing Patterns

When adding a new customization:

1. Create module in awscli/customizations/
2. Define command class extending BasicCommand (for new commands) or register event handlers (for modifications)
3. Register in awscli/handlers.py via the appropriate event
4. Add CLI overlay data in awscli/data/ if needed (examples, descriptions)
5. Write unit tests in tests/unit/customizations/
6. Write functional tests in tests/functional/

9. Integration Points

Botocore Integration

The CLI is a thin layer over botocore. Key integration points:

S3Transfer Integration

High-level S3 commands (aws s3 cp/sync/mv) use the s3transfer library for:

• Multipart uploads (parallel part uploading)
• Multipart downloads (parallel part downloading)
• Transfer progress callbacks
• Bandwidth throttling
• Automatic content-type detection

Shell Integration

Important

When piping CLI output, the pager is automatically disabled. The CLI detects non-TTY stdout and writes directly to the pipe.

CI/CD Integration

# Non-interactive usage pattern
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=$CI_AWS_KEY
export AWS_SECRET_ACCESS_KEY=$CI_AWS_SECRET

# Disable pager for scripted use
export AWS_PAGER=""

# Use --output text for easy parsing
INSTANCE_ID=$(aws ec2 run-instances \
  --image-id ami-12345 \
  --instance-type t3.micro \
  --query 'Instances[0].InstanceId' \
  --output text)

Docker / Container Integration

When running in containers:

• ECS tasks: credentials auto-injected via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
• EC2 instances: IMDS provides credentials automatically
• Kubernetes: use IAM Roles for Service Accounts (IRSA) which sets AWS_WEB_IDENTITY_TOKEN_FILE

Tip

In ephemeral containers, avoid writing to ~/.aws/ entirely. Use environment variables for credentials and pass --region explicitly. This eliminates filesystem state and simplifies container image management.

Custom Data Path

The AWS_DATA_PATH environment variable allows injecting additional service models:

export AWS_DATA_PATH=/opt/custom-models

# Structure:
# /opt/custom-models/
#   myservice/
#     2024-01-01/
#       service-2.json    (service model)
#       paginators-1.json (pagination config)
#       waiters-2.json    (waiter definitions)

This enables using AWS CLI with custom/internal services that follow the botocore model format, or overriding behavior of existing services for testing.

Quick Reference

Common Administrative Commands

# Show active configuration and source
aws configure list

# Show caller identity (verify credentials)
aws sts get-caller-identity

# List all configured profiles
aws configure list-profiles

# Set a config value
aws configure set region us-west-2 --profile prod

# Get a config value
aws configure get region --profile prod

# Show CLI version
aws --version

Troubleshooting Checklist