Practical AWS scripts built for task automation, debugging, and cloud resource management.

• Python 3.6+
• AWS credentials configured
• AWS Cost Explorer API enabled

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Configure AWS credentials:

aws configure

Required IAM permissions:

{
    "Effect": "Allow",
    "Action": [
        "ce:GetCostAndUsage",
        "lambda:ListFunctions",
        "lambda:GetFunction",
        "lambda:ListTags",
        "cloudwatch:GetMetricStatistics"
    ],
    "Resource": "*"
}

Note: Enable Cost Explorer in AWS Console if not already active.

Analyzes ECS costs by service and usage type. Groups costs by aws:ecs:serviceName tag and displays daily totals.

# Last 7 days (default)
python ecs_cost_analyzer.py

# Last 30 days
python ecs_cost_analyzer.py --days 30

# Filter by cluster
python ecs_cost_analyzer.py --cluster my-cluster

# Combine options
python ecs_cost_analyzer.py --days 14 --cluster production

• --days: Number of days to analyze (default: 7)
• --cluster: Filter by ECS cluster name (requires aws:ecs:clusterName tag)

Analyzes EC2 costs for "Amazon Elastic Compute Cloud - Compute" and "EC2 - Other" services. Groups costs by a specified tag (default: Name) and displays daily totals.

# Last 7 days (default) - grouped by Name tag
python ec2_cost_analyzer.py

# Last 30 days
python ec2_cost_analyzer.py --days 30

# Group by a different tag (e.g., Environment)
python ec2_cost_analyzer.py --tag Environment

• --days: Number of days to analyze (default: 7)
• --tag: Cost Allocation Tag to group by (default: 'Name')

Detailed table showing:

• Date
• Tag Value (e.g., Instance Name)
• Usage Type (e.g., BoxUsage:t3.medium)
• Cost

Analyzes AWS Lambda costs in two ways: actual costs from Cost Explorer and estimated compute costs per function from CloudWatch metrics.

# Last 7 days (default)
python lambda_cost_analyzer.py

# Last 30 days
python lambda_cost_analyzer.py --days 30

# Specific region
python lambda_cost_analyzer.py --region us-east-1

• --days: Number of days to analyze (default: 7)
• --region: AWS region (default: uses default region)

PART 1: Actual Costs (Cost Explorer)

• Real billing data grouped by usage type (compute, requests, etc.)
• Daily breakdown of actual Lambda costs
• This is the real cost from AWS billing

PART 2: Per-Function Estimate (CloudWatch) ⚠️ ESTIMATE ONLY

• Estimated compute costs for each Lambda function
• Based on CloudWatch metrics: invocations, duration, memory
• Shows real metrics: invocation count, duration (seconds), memory
• Note: This is an ESTIMATE - does not include:
  • Provisioned Concurrency costs
  • Data Transfer costs
  • Other AWS charges

1. Part 1 fetches actual costs from Cost Explorer API (real billing data)
2. Part 2 calculates estimated costs per function using:
   • CloudWatch Invocations metric (real invocation count)
   • CloudWatch Duration metric (real execution time in milliseconds)
   • Function memory configuration
   • Pricing: $0.0000166667 per GB-second (x86) or $0.0000133334 (ARM)
   • Request cost: $0.20 per 1M requests

• Part 2 costs are ESTIMATES based on CloudWatch metrics
• Actual billing may differ due to Provisioned Concurrency, data transfer, etc.
• The difference between Part 1 and Part 2 shows unaccounted costs
• Metrics show real usage data (invocations, duration) even if cost is estimated

Automatically scales ECS tasks based on RabbitMQ queue size. Designed to be integrated into your own system (Celery, cron, etc.).

# Run once (standalone)
python ecs_task_autoscaler.py

# Dry run mode (no actual scaling)
python ecs_task_autoscaler.py --dry-run

You can integrate this into your own system by calling the autoscale_ecs_workers() function:

from ecs_task_autoscaler import autoscale_ecs_workers

# Call in your scheduler (Celery Beat, cron, etc.)
result = autoscale_ecs_workers()
print(result)

• MIN_WORKERS: Minimum number of workers (default: 1)
• MAX_WORKERS: Maximum number of workers (default: 20)
• TASKS_PER_WORKER: Tasks per worker for scaling calculation (default: 200)
• SCALE_DOWN_DELAY: Delay before scaling down in seconds (default: 900 = 15 min)
• RABBITMQ_HOST: RabbitMQ host (default: rabbitmq)
• RABBITMQ_MANAGEMENT_PORT: RabbitMQ management port (default: 80)
• RABBITMQ_DEFAULT_USER: RabbitMQ username (default: guest)
• RABBITMQ_DEFAULT_PASS: RabbitMQ password (default: guest)
• RABBITMQ_QUEUE_NAME: Queue name to monitor (default: celery)
• RABBITMQ_VHOST: Virtual host, URL encoded (default: %2F which is /)
• ECS_CLUSTER_NAME: ECS cluster name (default: my-ecs-cluster) REQUIRED
• ECS_WORKER_SERVICE: ECS service name to scale (default: my-worker-service) REQUIRED
• AWS_REGION: AWS region (default: us-east-1)
• REDIS_HOST: Redis host for scale-down delay tracking (default: localhost)
• REDIS_PORT: Redis port (default: 6379)
• REDIS_DB: Redis database number (default: 1)
• REDIS_PASSWORD: Redis password (optional)
• DRY_RUN: Set to 'true' for dry run mode (default: false)

1. Checks RabbitMQ queue size (messages_ready)
2. Calculates needed workers: ceil(queue_size / TASKS_PER_WORKER)
3. Scale Up: Immediate scaling when queue size increases
4. Scale Down: Delays scaling down by SCALE_DOWN_DELAY seconds (prevents flapping)
5. Uses Redis to track scale-down timer state
6. Updates ECS service desiredCount

• Calculation: workers = ceil(queue_size / TASKS_PER_WORKER)
• Special case: If queue >= 100 tasks, scale to at least 2 workers
• Scale up: Immediate (no delay)
• Scale down: Requires delay period (default 15 minutes) to prevent rapid scaling

Based on default values (MIN_WORKERS=1, MAX_WORKERS=20, TASKS_PER_WORKER=200):

Examples:

• Queue size 150 → 2 workers (special case)
• Queue size 350 → 2 workers (ceil(350/200) = 2)
• Queue size 450 → 3 workers (ceil(450/200) = 3)
• Queue size 5000 → 20 workers (MAX_WORKERS limit)

Returns a dictionary with action taken:

{
    'action': 'scale_up' | 'scale_down' | 'no_change' | 'scale_down_delayed' | 'scale_down_waiting',
    'workers': int,  # Current worker count
    'queue': int,    # Queue size
    # ... other fields depending on action
}

• "Error fetching data": Check AWS credentials and IAM permissions
• "No data found": Verify tags are active in Cost Explorer preferences
• Tags may take a few hours to appear after enabling in Cost Explorer

• "No Name Tag": The script looks for the 'Name' tag by default. Ensure your instances are tagged and 'Name' is enabled as a Cost Allocation Tag in billing console.
• "No data found": Check permissions or credentials. Note that this script filters for EC2 services specifically.

• "No Lambda functions found": Check region setting or AWS credentials
• CloudWatch metrics not available: Ensure functions have been invoked in the selected period
• Zero costs shown: Functions might not have been used, or costs are below $0.0001 threshold
• Region errors: Specify --region if functions are in multiple regions

• "RabbitMQ API error": Check RabbitMQ host, port, and credentials
• "ECS API error": Verify AWS credentials, region, cluster, and service names
• "Redis connection error": Redis is optional (only for scale-down delay), check connection if needed
• No scaling happening: Check MIN_WORKERS and MAX_WORKERS limits
• Scale-down not working: Verify Redis connection and SCALE_DOWN_DELAY setting