Slack App - AWS DevOps Agent Bi-directional Integration

Introduction

I got tired of jumping between Slack, the AWS console, and DevOps Agent every time something broke. So I built a Slack app that bridges the gap: type a message in Slack, DevOps Agent investigates, and the results come back in the same thread. You can ask follow-up questions right there — no context switching required.

---

What It Does

You post a message in your Slack channel. DevOps Agent responds in a thread. If it kicks off an investigation, the results come back in that same thread when it's done. You can keep chatting — asking follow-up questions, requesting deeper analysis — all within one conversation.

Key advantage over the console: Multiple engineers can collaborate with DevOps Agent on the same incident thread simultaneously. In the console, a chat session is single-user. In Slack, your entire on-call team can see the investigation in real time, ask their own follow-up questions, and build on each other's context — all in one shared thread.

#incidents:

🧑 "The payments API is throwing 503s in us-east-1. Started ~10 min ago.
     arn:aws:ecs:us-east-1:123456789012:service/prod/payments-api"

  └── 🤖 "Looking into it. I can see the ECS service health degraded
           at 14:02 UTC. Checking recent deployments and CloudWatch..."

  └── 🤖 "✅ Investigation Complete
           Root cause: deploy-7f3a reduced the connection pool from 100
           to 10. All tasks are hitting connection exhaustion.
           Recommendation: rollback to previous task definition."

  └── 🧑 "Are other regions affected?"

  └── 🤖 "Checked us-west-2 and eu-west-1 — both showing normal
           error rates. This was isolated to us-east-1."

No forms, no special commands, no clicking through UIs. Just type like you're talking to a colleague.

---

How It Works (The Short Version)

You type in Slack
    → Slack sends the message to your API Gateway
    → Lambda creates a DevOps Agent chat session and relays your message
    → Agent responds (streamed) → Lambda posts the response in your thread
    → If Agent starts an investigation, Lambda notes the task ID + thread
    → Investigation completes → EventBridge fires → second Lambda posts results in the same thread

---

Prerequisites

Before we start, make sure you've got:

• An AWS account with an AWS DevOps Agent space already set up and working
• A Slack workspace where you can create apps (a sandbox workspace is perfect for testing)
• AWS CLI installed and configured with permissions to create Lambda, API Gateway, S3, Secrets Manager, EventBridge, and IAM resources

---

Step 1: Create the Slack App

Go to api.slack.com/apps and click Create New App → From scratch.

• App Name: Something like devops-agent or incident-helper
• Workspace: Pick your workspace (sandbox if you're testing)

OAuth & Permissions

Go to OAuth & Permissions in the left sidebar.

Under Bot Token Scopes, add:

chat:write

That's the only scope you need. The app posts messages — it doesn't need to read history or manage channels.

Click Install to Workspace → Allow. Copy the Bot User OAuth Token (xoxb-...). You'll need this.

Event Subscriptions

Go to Event Subscriptions in the left sidebar.

1. Toggle Enable Events to ON
2. Request URL — leave this blank for now (we'll come back after deploying the infrastructure)
3. Under Subscribe to bot events, add:
   • message.channels (public channels)
   • message.groups (private channels)
4. Click Save Changes

Socket Mode — TURN IT OFF

This one is easy to miss. Go to Socket Mode in the left sidebar.

Make sure it's OFF. If Socket Mode is on, Slack sends events via WebSocket instead of HTTP — your API Gateway endpoint will never receive anything. This is a common source of silent failures.

Signing Secret

Go to Basic Information → App Credentials. Copy the Signing Secret. You'll need this for verifying incoming requests.

Invite the App's Bot User

In your Slack channel, type:

/invite @your-app-name

The app's bot user must be a member of the channel to receive events from it.

---

Step 2: Deploy the Infrastructure

Here's the CloudFormation template. It creates everything you need — the API Gateway, both Lambda functions with their IAM roles, the S3 bucket, Secrets Manager secrets, and the EventBridge rule.

You'll pass your Slack credentials as parameters — CloudFormation stores them securely in Secrets Manager for you.

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  DevOps Agent Slack Integration (Chat-Only) - Chat with DevOps Agent
  directly in Slack. Investigation results posted as thread replies.

Parameters:
  AgentSpaceId:
    Type: String
    Description: DevOps Agent Space ID (UUID format)
    AllowedPattern: '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'

  SlackBotToken:
    Type: String
    Description: Slack Bot User OAuth Token (xoxb-...)
    NoEcho: true

  SlackChannelId:
    Type: String
    Description: Slack Channel ID where the bot operates (e.g. C********)

  SlackSigningSecret:
    Type: String
    Description: Slack App Signing Secret (for verifying incoming requests)
    NoEcho: true

  SlackWebhookUrl:
    Type: String
    Description: Slack Incoming Webhook URL (https://hooks.slack.com/services/...)
    NoEcho: true

Resources:

  SlackBotTokenSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub '${AWS::StackName}-bot-token'
      Description: Slack bot token, channel ID, and signing secret
      SecretString: !Sub |
        {"bot-token":"${SlackBotToken}","channel-id":"${SlackChannelId}","signing-secret":"${SlackSigningSecret}"}

  SlackWebhookSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub '${AWS::StackName}-webhook'
      Description: Slack incoming webhook URL for notifications
      SecretString: !Sub '{"webhookUrl":"${SlackWebhookUrl}"}'

  StateBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub '${AWS::StackName}-state-${AWS::AccountId}'
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      LifecycleConfiguration:
        Rules:
          - Id: expire-thread-maps
            Status: Enabled
            Prefix: thread-map-by-task/
            ExpirationInDays: 1
          - Id: expire-chat-sessions
            Status: Enabled
            Prefix: chat-sessions/
            ExpirationInDays: 7

  HttpApi:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: !Sub '${AWS::StackName}-api'
      ProtocolType: HTTP

  HttpApiStage:
    Type: AWS::ApiGatewayV2::Stage
    Properties:
      ApiId: !Ref HttpApi
      StageName: $default
      AutoDeploy: true

  EventsIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref HttpApi
      IntegrationType: AWS_PROXY
      IntegrationUri: !GetAtt EventsLambda.Arn
      PayloadFormatVersion: '2.0'

  EventsRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref HttpApi
      RouteKey: POST /slack-events
      Target: !Sub 'integrations/${EventsIntegration}'

  EventsLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${AWS::StackName}-events-role'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: SecretsAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: secretsmanager:GetSecretValue
                Resource: !Ref SlackBotTokenSecret
        - PolicyName: S3Access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                Resource:
                  - !Sub '${StateBucket.Arn}/chat-sessions/*'
                  - !Sub '${StateBucket.Arn}/thread-map-by-task/*'
        - PolicyName: DevOpsAgentAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - aidevops:CreateChat
                  - aidevops:SendMessage
                Resource: !Sub 'arn:aws:aidevops:${AWS::Region}:${AWS::AccountId}:agentspace/${AgentSpaceId}'

  EventsLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-events'
      Runtime: python3.12
      Handler: index.handler
      Role: !GetAtt EventsLambdaRole.Arn
      Timeout: 90
      MemorySize: 128
      Environment:
        Variables:
          AGENT_SPACE_ID: !Ref AgentSpaceId
          BOT_TOKEN_SECRET: !Sub '${AWS::StackName}-bot-token'
          NONCE_BUCKET: !Ref StateBucket
      Code:
        ZipFile: |
          # Placeholder - handles Slack URL verification challenge.
          # Deploy actual code via:
          # cd lambda-events && zip -j /tmp/events.zip index.py
          # aws lambda update-function-code --function-name <stack-name>-events --zip-file fileb:///tmp/events.zip
          import json
          def handler(event, context):
              body = json.loads(event.get('body', '{}'))
              if body.get('type') == 'url_verification':
                  return {
                      'statusCode': 200,
                      'headers': {'Content-Type': 'application/json'},
                      'body': json.dumps({'challenge': body.get('challenge', '')})
                  }
              return {'statusCode': 200, 'body': 'ok'}

  EventsLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref EventsLambda
      Action: lambda:InvokeFunction
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub 'arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${HttpApi}/*/*'

  NotifyLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${AWS::StackName}-notify-role'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: SecretsAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: secretsmanager:GetSecretValue
                Resource: !Ref SlackWebhookSecret
        - PolicyName: S3Access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:DeleteObject
                  - s3:PutObject
                Resource:
                  - !Sub '${StateBucket.Arn}/thread-map-by-task/*'
                  - !Sub '${StateBucket.Arn}/chat-sessions/*'
        - PolicyName: DevOpsAgentAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - aidevops:ListJournalRecords
                  - aidevops:GetBacklogTask
                  - aidevops:ListBacklogTasks
                Resource: !Sub 'arn:aws:aidevops:${AWS::Region}:${AWS::AccountId}:agentspace/${AgentSpaceId}'

  NotifyLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub '${AWS::StackName}-notify'
      Runtime: python3.12
      Handler: index.lambda_handler
      Role: !GetAtt NotifyLambdaRole.Arn
      Timeout: 60
      MemorySize: 128
      Environment:
        Variables:
          SLACK_SECRET_ID: !Sub '${AWS::StackName}-webhook'
          NONCE_BUCKET: !Ref StateBucket
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          # Deploy actual code via: aws lambda update-function-code
          def lambda_handler(event, context):
              return {'statusCode': 200}

  NotifyLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref NotifyLambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt EventBridgeRule.Arn

  EventBridgeRule:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub '${AWS::StackName}-notify-rule'
      Description: Routes DevOps Agent investigation events to Slack notification Lambda
      State: ENABLED
      EventPattern:
        source:
          - aws.aidevops
        detail-type:
          - Investigation Completed
          - Investigation Linked
          - Investigation Failed
          - Investigation Timed Out
          - Investigation Cancelled
          - Investigation Skipped
        resources:
          - !Sub 'arn:aws:aidevops:${AWS::Region}:${AWS::AccountId}:agentspace/${AgentSpaceId}'
      Targets:
        - Id: NotifyLambda
          Arn: !GetAtt NotifyLambda.Arn

Outputs:
  SlackEventsUrl:
    Description: Set this as your Slack app's Request URL (Event Subscriptions)
    Value: !Sub 'https://${HttpApi}.execute-api.${AWS::Region}.amazonaws.com/slack-events'

  StateBucketName:
    Description: S3 bucket for thread maps and chat sessions
    Value: !Ref StateBucket

Deploy it:

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name devops-agent-slack \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
    AgentSpaceId=your-agent-space-id \
    SlackBotToken=xoxb-your-bot-token \
    SlackChannelId=C0******** \
    SlackSigningSecret=your-signing-secret \
    SlackWebhookUrl=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Where to find these values:

• AgentSpaceId — your DevOps Agent space UUID
• SlackBotToken — from Step 1: OAuth & Permissions → Bot User OAuth Token
• SlackChannelId — right-click your channel → View channel details → scroll to bottom
• SlackSigningSecret — from Step 1: Basic Information → App Credentials → Signing Secret
• SlackWebhookUrl — from your Slack app: Incoming Webhooks → Add New Webhook to Workspace → pick your channel

---

Step 3: Connect Slack to Your Endpoint

After the stack deploys, grab the SlackEventsUrl from the outputs:

aws cloudformation describe-stacks \
  --stack-name devops-agent-slack \
  --query 'Stacks[0].Outputs[?OutputKey==`SlackEventsUrl`].OutputValue' \
  --output text

Go back to your Slack app → Event Subscriptions → paste the URL into Request URL.

Slack will send a challenge request. The Lambda handles it automatically and you'll see a green "Verified" checkmark. If it doesn't verify, double-check:

• The stack deployed successfully
• The Lambda has API Gateway permissions
• You're pasting the full URL including /slack-events

Click Save Changes.

---

Step 4: The Lambda Code

Here's the actual code for both Lambda functions. Save each one as index.py in its respective directory.

Events Lambda (lambda-events/index.py)

This is the heart of the system — it receives Slack messages, relays them to DevOps Agent, and posts responses back as thread replies.

"""
Lambda: devops-agent-slack-events
Handles Slack Events API:
- URL verification challenge
- Thread replies → forwards to DevOps Agent chat, posts response back

Flow:
1. User replies in an investigation thread
2. Lambda verifies Slack signature
3. Looks up chat session from S3 (chat-sessions/<thread_ts>)
4. If no session exists, creates one via CreateChat
5. Sends user message via SendMessage
6. Collects streamed response
7. Posts response back in thread
"""

import json
import hashlib
import hmac
import time
import os
import boto3
import urllib.request

AGENT_SPACE_ID = os.environ['AGENT_SPACE_ID']
NONCE_BUCKET = os.environ['NONCE_BUCKET']
BOT_TOKEN_SECRET = os.environ['BOT_TOKEN_SECRET']
REGION = os.environ.get('AWS_REGION', 'us-east-1')

secrets_client = boto3.client('secretsmanager', region_name=REGION)
s3_client = boto3.client('s3', region_name=REGION)
devops_client = boto3.client('devops-agent', region_name=REGION)

_bot_config = None


def get_bot_config():
    global _bot_config
    if _bot_config is None:
        resp = secrets_client.get_secret_value(SecretId=BOT_TOKEN_SECRET)
        _bot_config = json.loads(resp['SecretString'])
    return _bot_config


def verify_slack_signature(event):
    """Verify the request is from Slack using the signing secret."""
    config = get_bot_config()
    signing_secret = config['signing-secret']

    headers = event.get('headers', {})
    timestamp = headers.get('x-slack-request-timestamp', '')
    signature = headers.get('x-slack-signature', '')
    body = event.get('body', '')

    if not timestamp or not signature:
        return False

    if abs(time.time() - int(timestamp)) > 300:
        return False

    sig_basestring = f"v0:{timestamp}:{body}"
    expected = 'v0=' + hmac.new(
        signing_secret.encode(),
        sig_basestring.encode(),
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(expected, signature)


def get_or_create_chat_session(thread_ts):
    """Get existing chat session or create a new one with task context for this thread."""
    key = f"chat-sessions/{thread_ts}"

    try:
        resp = s3_client.get_object(Bucket=NONCE_BUCKET, Key=key)
        data = json.loads(resp['Body'].read())
        if data.get('chat_execution_id'):
            return data['chat_execution_id']

        task_id = data.get('task_id', '')
        title = data.get('title', '')
        summary = data.get('summary', '')
        agent_space_id = data.get('agent_space_id', AGENT_SPACE_ID)

        chat_resp = devops_client.create_chat(agentSpaceId=agent_space_id)
        execution_id = chat_resp['executionId']

        context_message = f"""You are continuing a conversation about a DevOps investigation.

Task ID: {task_id}
Title: {title}
Agent Space: {agent_space_id}

Investigation Summary:
{summary}

The user will now ask follow-up questions about this specific investigation."""

        try:
            context_resp = devops_client.send_message(
                agentSpaceId=agent_space_id,
                executionId=execution_id,
                content=context_message
            )
            for _ in context_resp.get('events', []):
                pass
        except Exception as e:
            print(f"Warning: context message failed: {e}")

        data['chat_execution_id'] = execution_id
        s3_client.put_object(
            Bucket=NONCE_BUCKET,
            Key=key,
            Body=json.dumps(data).encode(),
            ContentType='application/json'
        )
        return execution_id

    except Exception as e:
        if 'NoSuchKey' in str(e) or '404' in str(e) or '403' in str(e):
            chat_resp = devops_client.create_chat(agentSpaceId=AGENT_SPACE_ID)
            execution_id = chat_resp['executionId']
            s3_client.put_object(
                Bucket=NONCE_BUCKET,
                Key=key,
                Body=json.dumps({
                    'chat_execution_id': execution_id,
                    'created_at': time.time()
                }).encode(),
                ContentType='application/json'
            )
            return execution_id
        raise


def send_message_to_agent(execution_id, message, thread_ts=None, channel=None):
    """Send a message to DevOps Agent, collect response, and detect investigation task IDs."""
    import re

    resp = devops_client.send_message(
        agentSpaceId=AGENT_SPACE_ID,
        executionId=execution_id,
        content=message
    )

    full_text = []
    events = resp.get('events', [])

    for event in events:
        if 'contentBlockDelta' in event:
            delta = event['contentBlockDelta'].get('delta', {})
            if isinstance(delta, dict):
                text_delta = delta.get('textDelta', {})
                if isinstance(text_delta, dict):
                    text = text_delta.get('text', '')
                    if text:
                        full_text.append(text)
        elif 'contentBlockStop' in event:
            text = event['contentBlockStop'].get('text', '')
            if text:
                full_text.append(text)
        elif 'summary' in event:
            content = event['summary'].get('content', '')
            if content:
                full_text.append(content)
        elif 'responseFailed' in event:
            error_msg = event['responseFailed'].get('errorMessage', 'Unknown error')
            return f":warning: DevOps Agent error: {error_msg}"

    response_text = ''.join(full_text) if full_text else "No response from DevOps Agent."

    # Parse response for investigation task IDs and store thread mapping
    if thread_ts and channel:
        task_ids = re.findall(r'\[\[investigation:([0-9a-f-]{36}):', response_text)
        if not task_ids:
            task_ids = re.findall(
                r'investigation[^0-9a-f]*([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})',
                response_text, re.IGNORECASE
            )

        for task_id in set(task_ids):
            try:
                s3_client.put_object(
                    Bucket=NONCE_BUCKET,
                    Key=f'thread-map-by-task/{task_id}',
                    Body=json.dumps({'thread_ts': thread_ts, 'channel': channel}).encode(),
                    ContentType='application/json'
                )
                print(f"Mapped task {task_id} -> thread {thread_ts}")
            except Exception as e:
                print(f"Error storing task mapping: {e}")

    return response_text


def post_to_slack_thread(channel, thread_ts, text):
    """Post a message as a thread reply."""
    config = get_bot_config()
    bot_token = config['bot-token']

    MAX_LEN = 3900
    if len(text) <= MAX_LEN:
        chunks = [text]
    else:
        chunks = []
        while text:
            if len(text) <= MAX_LEN:
                chunks.append(text)
                break
            split_at = text.rfind('\n', 0, MAX_LEN)
            if split_at == -1 or split_at < MAX_LEN // 2:
                split_at = MAX_LEN
            chunks.append(text[:split_at])
            text = text[split_at:].lstrip('\n')

    for chunk in chunks:
        payload = json.dumps({
            'channel': channel,
            'thread_ts': thread_ts,
            'text': chunk,
            'mrkdwn': True
        }).encode()

        req = urllib.request.Request(
            'https://slack.com/api/chat.postMessage',
            data=payload,
            headers={
                'Content-Type': 'application/json',
                'Authorization': f'Bearer {bot_token}'
            }
        )

        with urllib.request.urlopen(req) as resp:
            result = json.loads(resp.read())
            if not result.get('ok'):
                print(f"Slack error: {result.get('error')}")


def handler(event, context):
    """Handle Slack Events API requests."""

    raw_body = event.get('body', '{}')
    body = json.loads(raw_body) if isinstance(raw_body, str) else raw_body

    # Handle Slack URL verification challenge
    if body.get('type') == 'url_verification':
        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'challenge': body.get('challenge', '')})
        }

    # Verify Slack signature
    if not verify_slack_signature(event):
        print("ERROR: Slack signature verification failed - rejecting request")
        return {'statusCode': 401, 'body': 'Unauthorized'}

    # Handle event callbacks
    if body.get('type') == 'event_callback':
        headers = event.get('headers', {})
        if headers.get('x-slack-retry-num'):
            return {'statusCode': 200, 'body': 'ok'}

        slack_event = body.get('event', {})

        if slack_event.get('type') != 'message':
            return {'statusCode': 200, 'body': 'ok'}

        # Skip bot messages (avoid infinite loop)
        if slack_event.get('bot_id') or slack_event.get('subtype'):
            return {'statusCode': 200, 'body': 'ok'}

        # Only handle messages in our channel
        config = get_bot_config()
        if slack_event.get('channel') != config['channel-id']:
            return {'statusCode': 200, 'body': 'ok'}

        user_message = slack_event.get('text', '')
        channel = slack_event.get('channel')
        thread_ts = slack_event.get('thread_ts')
        message_ts = slack_event.get('ts')

        if not user_message:
            return {'statusCode': 200, 'body': 'ok'}

        try:
            if thread_ts:
                # Thread reply - continue existing conversation
                execution_id = get_or_create_chat_session(thread_ts)
                response_text = send_message_to_agent(
                    execution_id, user_message, thread_ts=thread_ts, channel=channel
                )
                post_to_slack_thread(channel, thread_ts, response_text)
            else:
                # Top-level message - fresh chat session
                chat_resp = devops_client.create_chat(agentSpaceId=AGENT_SPACE_ID)
                execution_id = chat_resp['executionId']

                s3_client.put_object(
                    Bucket=NONCE_BUCKET,
                    Key=f'chat-sessions/{message_ts}',
                    Body=json.dumps({
                        'chat_execution_id': execution_id,
                        'created_at': time.time()
                    }).encode(),
                    ContentType='application/json'
                )

                response_text = send_message_to_agent(
                    execution_id, user_message, thread_ts=message_ts, channel=channel
                )
                post_to_slack_thread(channel, message_ts, response_text)

        except Exception as e:
            print(f"Error handling message: {e}")
            reply_ts = thread_ts or message_ts
            post_to_slack_thread(channel, reply_ts, f":warning: Error: {str(e)}")

    return {'statusCode': 200, 'body': 'ok'}

Notification Lambda (lambda-notify/index.py)

This Lambda listens for EventBridge events when investigations complete and posts the results back to the correct Slack thread.

import json
import os
import boto3
from urllib import request, error

REGION = os.environ.get('AWS_REGION', os.environ.get('REGION', 'us-east-1'))
NONCE_BUCKET = os.environ['NONCE_BUCKET']
SLACK_SECRET_ID = os.environ['SLACK_SECRET_ID']

secrets_client = boto3.client('secretsmanager', region_name=REGION)
devops_client = boto3.client('devops-agent', region_name=REGION)
s3_client = boto3.client('s3', region_name=REGION)

_slack_webhook_url = None


def get_slack_webhook_url():
    global _slack_webhook_url
    if _slack_webhook_url is None:
        resp = secrets_client.get_secret_value(SecretId=SLACK_SECRET_ID)
        secret = json.loads(resp['SecretString'])
        _slack_webhook_url = secret['webhookUrl']
    return _slack_webhook_url


def get_summary(agent_space_id, execution_id):
    """Fetch investigation summary from journal records."""
    for record_type in ['investigation_summary_md', 'investigation_result']:
        try:
            resp = devops_client.list_journal_records(
                agentSpaceId=agent_space_id,
                executionId=execution_id,
                recordType=record_type
            )
            records = resp.get('records', [])
            if records:
                content = records[0].get('content', '')
                try:
                    parsed = json.loads(content)
                    return parsed.get('text', content)
                except (json.JSONDecodeError, TypeError):
                    return content
        except Exception as e:
            print(f"Error fetching {record_type}: {e}")
    return None


def get_linked_summary(agent_space_id, task_id):
    """For a linked investigation, fetch the primary investigation's summary."""
    try:
        resp = devops_client.get_backlog_task(agentSpaceId=agent_space_id, taskId=task_id)
        task = resp.get('task', {})
        primary_task_id = task.get('primaryTaskId')

        if primary_task_id:
            primary_resp = devops_client.get_backlog_task(
                agentSpaceId=agent_space_id, taskId=primary_task_id
            )
            primary_task = primary_resp.get('task', {})
            primary_execution_id = primary_task.get('executionId', '')

            if primary_execution_id:
                summary = get_summary(agent_space_id, primary_execution_id)
                return summary, primary_task_id
            else:
                summary = get_summary(agent_space_id, primary_task_id)
                return summary, primary_task_id

        return None, None
    except Exception as e:
        print(f"Error getting linked summary: {e}")
        return None, None


def post_to_slack(payload):
    """Post a message to the Slack webhook."""
    webhook_url = get_slack_webhook_url()
    data = json.dumps(payload).encode('utf-8')
    req = request.Request(webhook_url, data=data, headers={'Content-Type': 'application/json'})
    try:
        with request.urlopen(req, timeout=15) as resp:
            return resp.status
    except error.URLError as e:
        print(f"Slack POST failed: {e}")
        raise


def build_slack_message(event_type, detail, summary=None, primary_task_id=None,
                        status_reason=None, title=None):
    """Build a Slack Block Kit message."""
    metadata = detail.get('metadata', {})
    data = detail.get('data', {})

    task_id = metadata.get('task_id', 'unknown')
    agent_space_id = metadata.get('agent_space_id', 'unknown')
    priority = data.get('priority', 'N/A')
    created_at = data.get('created_at', '')
    updated_at = data.get('updated_at', '')

    event_config = {
        'Investigation Completed': ('✅', 'Investigation Completed'),
        'Investigation Linked': ('🔗', 'Investigation Linked to Existing'),
        'Investigation Failed': ('❌', 'Investigation Failed'),
        'Investigation Timed Out': ('⏰', 'Investigation Timed Out'),
        'Investigation Cancelled': ('🚫', 'Investigation Cancelled'),
        'Investigation Skipped': ('⏭️', 'Investigation Skipped'),
    }
    emoji, status_text = event_config.get(event_type, ('ℹ️', event_type))

    header_text = f"{emoji} {status_text}"
    if title:
        header_text = f"{emoji} {status_text}: {title}"

    blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": header_text[:150], "emoji": True}
        },
        {
            "type": "section",
            "fields": [
                {"type": "mrkdwn", "text": f"*Priority:* {priority}"},
                {"type": "mrkdwn", "text": f"*Task ID:* `{task_id}`"},
            ]
        },
    ]

    if event_type == 'Investigation Linked' and primary_task_id:
        blocks.append({"type": "divider"})
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn",
                     "text": f"🔗 *Linked to existing investigation:* `{primary_task_id}`"}
        })

    if status_reason:
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn", "text": f"*Reason:* {status_reason}"}
        })

    if summary:
        blocks.append({"type": "divider"})
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn", "text": "*Investigation Findings:*"}
        })
        CHUNK_SIZE = 2900
        chunks = [summary[i:i + CHUNK_SIZE] for i in range(0, len(summary), CHUNK_SIZE)]
        for chunk in chunks[:40]:
            blocks.append({
                "type": "section",
                "text": {"type": "mrkdwn", "text": f"```\n{chunk}\n```"}
            })
        if len(chunks) > 40:
            blocks.append({
                "type": "section",
                "text": {"type": "mrkdwn",
                         "text": "_⚠️ Output truncated. Check the console for full results._"}
            })
    elif event_type in ['Investigation Completed', 'Investigation Linked']:
        blocks.append({"type": "divider"})
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn",
                     "text": "_No summary available for this investigation._"}
        })

    blocks.append({"type": "divider"})
    blocks.append({
        "type": "context",
        "elements": [{"type": "mrkdwn",
                      "text": f"📅 {updated_at or created_at} | Space: `{agent_space_id}` | AWS DevOps Agent"}]
    })

    return {
        "blocks": blocks,
        "text": f"{emoji} {status_text} — Task: {task_id} (Priority: {priority})"
    }


def lambda_handler(event, context):
    print(f"Received: {json.dumps(event)}")

    event_type = event.get('detail-type', 'Unknown')
    detail = event.get('detail', {})
    metadata = detail.get('metadata', {})
    task_id = metadata.get('task_id', '')
    execution_id = metadata.get('execution_id', '')
    agent_space_id = metadata.get('agent_space_id', '')

    summary = None
    primary_task_id = None
    status_reason = None
    title = None
    slack_thread_ts = None

    # Get task details
    try:
        resp = devops_client.get_backlog_task(agentSpaceId=agent_space_id, taskId=task_id)
        task = resp.get('task', {})
        title = task.get('title', '') or task.get('name', '')
        status_reason = task.get('statusReason', '')
        if not status_reason:
            meta = task.get('metadata', {})
            status_reason = (meta.get('canceledReason', '') or
                           meta.get('failedReason', '') or
                           meta.get('skippedReason', ''))
    except Exception as e:
        print(f"Error getting task details: {e}")

    # Look up thread_ts by task_id
    try:
        task_resp = s3_client.get_object(
            Bucket=NONCE_BUCKET, Key=f'thread-map-by-task/{task_id}'
        )
        task_data = json.loads(task_resp['Body'].read())
        slack_thread_ts = task_data.get('thread_ts')
        s3_client.delete_object(Bucket=NONCE_BUCKET, Key=f'thread-map-by-task/{task_id}')
    except Exception:
        pass

    # Fetch summary based on event type
    if event_type == 'Investigation Completed':
        summary = get_summary(agent_space_id, execution_id)
    elif event_type == 'Investigation Linked':
        summary, primary_task_id = get_linked_summary(agent_space_id, task_id)

    # Build and post Slack message
    slack_payload = build_slack_message(
        event_type, detail, summary, primary_task_id, status_reason, title
    )
    if slack_thread_ts:
        slack_payload["thread_ts"] = slack_thread_ts

        # Store chat-session so users can ask follow-ups
        try:
            s3_client.put_object(
                Bucket=NONCE_BUCKET,
                Key=f'chat-sessions/{slack_thread_ts}',
                Body=json.dumps({
                    'task_id': task_id,
                    'execution_id': execution_id,
                    'title': title or '',
                    'agent_space_id': agent_space_id,
                    'summary': (summary or '')[:2000],
                    'created_at': detail.get('data', {}).get('created_at', '')
                }).encode(),
                ContentType='application/json'
            )
        except Exception as e:
            print(f"Failed to store chat-session: {e}")

    post_to_slack(slack_payload)
    return {'statusCode': 200, 'task_id': task_id, 'summary_found': summary is not None}

---

Step 5: Deploy the Lambda Code

The CloudFormation template deploys placeholder (stub) Lambda functions — just enough to pass Slack's URL verification in Step 3. Now it's time to deploy the actual handler code.

Why this extra step? CloudFormation can't upload local files to Lambda directly. You need to zip your code and push it separately. It's two commands per function — takes about 10 seconds.

Package the code into zip files:

# From the project root directory:

# 1. Zip the Events Lambda code
zip -j lambda-events.zip lambda-events/index.py

# 2. Zip the Notification Lambda code
zip -j lambda-notify.zip lambda-notify/index.py

Note: The -j flag strips the directory path so index.py sits at the root of the zip — Lambda requires the handler file at the top level.

Deploy the zip files to your Lambda functions:

# Deploy Events Lambda (handles chat)
aws lambda update-function-code \
  --function-name devops-agent-slack-events \
  --zip-file fileb://lambda-events.zip \
  --region us-east-1

# Deploy Notification Lambda (posts investigation results)
aws lambda update-function-code \
  --function-name devops-agent-slack-notify \
  --zip-file fileb://lambda-notify.zip \
  --region us-east-1

If you used a different stack name, replace devops-agent-slack with your stack name in the function names above (e.g. my-stack-name-events and my-stack-name-notify).

You can re-run these commands anytime you update the Lambda code.

---

Step 6: Test It

Everything's deployed — time to verify it works. Post something in your channel:

Hey, can you check the health of my ECS services?

You should see a threaded response within a few seconds. If the channel stays silent, here's where to look:

1. Is the app in the channel? Type /invite @your-app if you haven't already
2. Is Socket Mode OFF? This is the #1 silent killer (see Step 1)
3. Check the Lambda logs: aws logs tail /aws/lambda/devops-agent-slack-events --since 5m

---

The Architecture (In More Detail)

Now that it's running, let me explain what's actually happening under the hood.

The Events Lambda

This is the heart of the system. It receives every message posted in channels where the bot is a member. Here's its decision tree:

Incoming Slack event
    │
    ├── type: url_verification → return challenge (Slack handshake)
    │
    ├── Retry header present → skip (prevents duplicate processing)
    │
    ├── Bot message or subtype → skip (avoids infinite loops)
    │
    ├── Has thread_ts (it's a reply):
    │   → Load chat-sessions/<thread_ts> for context
    │   → CreateChat with investigation context injected
    │   → SendMessage with user's question
    │   → Parse response for new investigation task IDs
    │   → Post response in same thread
    │
    └── No thread_ts (top-level message):
        → CreateChat (fresh session)
        → SendMessage with user's text
        → Store chat-sessions/<message_ts>
        → Parse response for investigation task IDs
        → Store thread-map-by-task/<task_id> if found
        → Post response as thread reply

The Streaming Response

DevOps Agent's SendMessage API returns a stream of events. The actual text is buried in contentBlockDelta events:

full_text = []
for event in response["events"]:
    if "contentBlockDelta" in event:
        delta = event["contentBlockDelta"]["delta"]
        text = delta.get("textDelta", {}).get("text", "")
        if text:
            full_text.append(text)

return "".join(full_text)

The Threading Concept

When DevOps Agent decides to kick off a background investigation, it includes a structured marker in its response:

[[investigation:fcd497d6-3b9d-45fc-af6f-a2356c41b46a:High CPU on payments-api]]

The Lambda parses this with a regex, extracts the task ID, and stores it in S3 mapped to the current thread's ts. Later, when EventBridge fires the completion event, the Notification Lambda looks up that task ID and knows exactly which thread to reply to.

It's simple pattern matching, but it reliably links asynchronous investigation results back to the right conversation.

The Notification Lambda

Listens on EventBridge for investigation status events (aws.aidevops). When one fires:

1. Gets the task details (title, priority) via get_backlog_task
2. Gets the findings via list_journal_records
3. Looks up thread-map-by-task/<task_id> in S3
4. If found → posts as thread reply. If not → posts as top-level message.

Investigations created outside Slack (via the console, for example) still get reported in the channel — they just post as top-level messages since there's no thread to link to.

---

What It Costs for this Integration Only

Assuming a team doing 100 chat conversations and 10 investigations per day (3,000+ messages/month):

Service	Monthly
Lambda (6,300 invocations, ~563 GB-s)	$0.00 (free tier)
API Gateway HTTP (6,000 requests)	$0.01
S3 (~6,900 requests, auto-cleaned)	$0.02
EventBridge (300 events)	$0.00
Secrets Manager (2 secrets)	$0.80
Total	~$0.83

Above is high level cost, do estimate your own cost based on your own use case.

---

Common Pitfalls and Lessons Learned

Socket Mode breaks everything silently. If it's on, Slack sends events via WebSocket and your HTTP endpoint gets nothing. No errors, no logs, just silence. Turn it off.

Slack retries aggressively. If your Lambda takes more than 3 seconds to respond, Slack assumes it failed and retries. The fix: check for the x-slack-retry-num header and skip retries.

The app sees its own messages. Without a bot_id filter, your Lambda processes its own responses, creating an infinite loop. Always skip events where bot_id is present.

S3 returns 403 (not 404) for missing keys if you don't have ListBucket permission. Either add ListBucket or handle both 403 and 404 as "not found."

DevOps Agent's streaming format nests text in event["contentBlockDelta"]["delta"]["textDelta"]["text"]. The contentBlockStop event doesn't contain the text — only the deltas do.

---

The Complete Flow

User types in Slack
    → Slack Events API delivers to API Gateway
    → Events Lambda verifies signature, skips bots/retries
    → Creates or reuses a DevOps Agent chat session
    → Sends message via SendMessage (streaming)
    → Collects response text from contentBlockDelta events
    → Parses [[investigation:<task_id>:...]] patterns
    → Stores task_id → thread_ts mapping in S3
    → Posts response as thread reply via chat.postMessage

Investigation completes (minutes later)
    → EventBridge event fires (aws.aidevops)
    → Notify Lambda gets task_id from event
    → Looks up thread-map-by-task/<task_id> in S3
    → Gets investigation findings via list_journal_records
    → Posts results as thread reply in the original conversation

---

Wrapping Up

If you've already got a DevOps Agent space running, you can realistically have this working in 30 minutes. Most of that time is clicking through Slack's app configuration UI. The actual AWS infrastructure deploys in under two minutes.

The Integration uses two Lambdas, One API Gateway, One bucket, One channel.