SSM agent - supposedly race conditions with websocket connections

0

I'm trying to open a websocket connection with an SSM agent running on my EC2 to fetch container logs

locally it works great,I open the ws connection and get the logs as I wanted.

my issue is when I deploy my server on another EC2, then I don't get any "new" message from the ws connection, I assume there is a race condition (?) between the init phase and the communication part

I do get the a response after the init part in the "open" handler :

ws listeners :

const command = `sudo docker logs $(sudo docker ps | grep -i '${taskFamilyName}' | awk '{print $1}' | head -1) -f --tail 0\n`;

ws.on("open", () => {
  init(ws, {
    token: TokenValue,
    termOptions: termOptions,
  });
 //  I assume this is the 'problematic' part
  sendText(ws, textEncoder.encode(command));
});

ws.on("message", (event) => {
  let agentMessage = this.decode(event);
  this.sendACK(ws, agentMessage);
   ... 
});

examples of the agent responses (decoded) :

  headerLength: 116,
  messageType: 'output_stream_data',
  schemaVersion: 1,
  createdDate: 1681738240767,
  sequenceNumber: 0,
  flags: 1,
  messageId: 'xxxxx,
  payloadDigest: 'xxxxx,
  payloadType: 1,
  payloadLength: 16,
  payload: Uint8Array(16) [
    27,  91,  63,  49, 48, 51,
    52, 104, 115, 104, 45, 52,
    46,  50,  36,  32
  ]
}

-> which is 'sh-4.2$'

and an acknowledge after I send my command (i assume ?) :

{
  headerLength: 116,
  messageType: 'acknowledge',
  schemaVersion: 1,
  createdDate: 1681738242743,
  sequenceNumber: 0,
  flags: 3,
  messageId: 'xxxxx',
  payloadDigest: 'xxxxx,
  payloadType: 0,
  payloadLength: 175,
  payload: Uint8Array(175) [
    123,  34,  65,  99, 107, 110, 111, 119, 108, 101, 100, 103,
    101, 100,  77, 101, 115, 115,  97, 103, 101,  84, 121, 112,
    101,  34,  58,  34, 105, 110, 112, 117, 116,  95, 115, 116,
    114, 101,  97, 109,  95, 100,  97, 116,  97,  34,  44,  34,
     65,  99, 107, 110, 111, 119, 108, 101, 100, 103, 101, 100,
     77, 101, 115, 115,  97, 103, 101,  73, 100,  34,  58,  34,
     53,  49,  53, 100,  99, 101,  49,  56,  45,  97,  54,  55,
     56,  45,  52,  51,  57,  53,  45,  57,  50, 100, 102,  45,
     53,  55,  52,  57,
    ... 75 more items
  ]
}
Eyal
asked a year ago347 views
1 Answer
0

It sounds like you cannot connect to the websocket. This could be caused by having a security group that don't allow expect port to expect destination.

The easiest way to test if the route is open is to use the Reachability Analyzer. If the destination is reachable you may want to use VPC Flow logs to identify the traffic to see if you can see what is happening.

profile picture
answered a year ago
  • Hi Robert, thanks for the answer.

    It's worth mentioning, I tried installing the aws cli and the ssm plugin to verify the connectivity between both, which worked perfectly. This lead me to thinking the issue might be in the ssm agent itself ?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions