codepipeline.putJobSuccessResult() Hanging when Invoking a Lambda from VPC

0

After much trial and tribulation, I have determined that the code below (nodejs8.1 runtime), which is a basic CodePipeline setup with no additional code, works fine when one invokes a Lambda normally, but if one tries to invoke the Lambda from within a VPC, the codepipeline.putJobSuccessResult() hangs and the Lambda times out. Any code that comes before the call runs fine, but it just won't run codepipeline.putJobSuccessResult() and give a proper return value back to CodePipeline despite both codepipeline and the Lambda having a role that has all kinds of policies and trust relationships and the VPC has lots of endpoints as well as a NAT Gateway and Internet Gateway. This results in CodePipeline continually retrying until the timeout period (~15 minutes).

Note also that prior to adding the Lambda to CodePipeline and adding an endpoint that I was running the Lambda manually and successfully utilizing a static IP through a NAT => Internet Gateway (https://medium.com/@matthewleak/aws-lambda-functions-with-a-static-ip-89a3ada0b471) and again, even within the CodePipeline, the Lambda runs fine until it utilizes the AWS SDK aws.CodePipeline.putJobSuccessResult()/aws.CodePipeline.putJobFailureResult() functions; all other code is successfully executed.

In theory, to reproduce, one need only take the code below and create a Lambda, set up a VPC as described in the article above, set up a basic CodePipeline and invoke the Lambda as part of the pipeline. The first run through should work fine. Then assign the Lambda to the VPC and subnets, then run the pipeline again and see if it doesn't hang when it tries to putJobSuccessResult.

The hanging behavior implies it is a networking issue, but if CodePipeline has an endpoint to the VPC and successfully is able to invoke the Lambda, why would the Lambda not be able to talk back to CodePipeline to putJobSuccessResult/putJobFailureResult? My guess is that either I'm missing something in terms of the VPC or CodePipeline isn't working correctly and/or utilizing its endpoint correctly -- but I'd like to figure it out.

It's looking more and more to me like an endpoint issue. I found this thread while digging: https://forums.aws.amazon.com/thread.jspa?threadID=293780 – If I enabled the endpoint in the VPC, have the internet gateway, the route table points to the internet gateway, and the subnet uses the internet routable route table, and there's no security group or ACL denying traffic it is supposed to work.

// Working Base response code for CodePipeline  
  
'use strict';  
const aws = require('aws-sdk');  
const codepipeline = new aws.CodePipeline();  
  
let environment = 'dev';  
let callback;  
let context = {  
    invokeid: ''  
}  
  
exports.handler = async (event, context, callback) => {  
    context = context;  
    callback = callback;  
    console.log('Inside deploy-website Lambda');  
    if (!('CodePipeline.job' in event)) {  
        return Promise.resolve();  
    }  
    // Retrieve the Job ID from the Lambda action  
    let jobId;  
    if (event\["CodePipeline.job"]) {  
        jobId = event\["CodePipeline.job"].id;  
  
        // Retrieve the value of UserParameters from the Lambda action configuration in AWS CodePipeline, in this case the environment  
        // to deploy to from this function  
        environment = event\["CodePipeline.job"].data.actionConfiguration.configuration.UserParameters || environment;  
    }  
  
    console.log(`Envrionment: ${environment}`);  
  
  
    console.log('Copy Successful');  
    console.log('Entering Results');  
    return await putJobSuccess('Copy Successful', jobId);  
}  
  
// Notify AWS CodePipeline of a successful job  
async function putJobSuccess(message, jobId) {  
    console.log(`Post Job Success For JobID: ${jobId}`);  
    const params = {  
        jobId: jobId  
    };  
    console.log(`Job Success Params: ${JSON.stringify(params)}`);  
    await codepipeline.putJobSuccessResult(params).promise();  
    console.log('Job Success: Successfully reported hook results');  
    return callback(null, 'Job Success: Successfully reported hook results');  
}
maplion
asked 5 years ago988 views
1 Answer
0

It turns out it was indeed a networking issue. It seems that the VPC route tables are what got me. When you create a route table, it has you choose a name and a VPC to associate it to. What I forgot to do was go to the subnets and associate them to the proper route table under the Route Table tab and/or I didn't choose the right one on one of them because when you choose a route table to associate it to, it doesn't show the logical name, just the route table ID, which makes it more prone to error. So, while it was definitely a "Newb" mistake, I think there is something left to be desired in terms of user experience in associating route tables.

maplion
answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions