Debugging FAILED status in Fleet

0

Greetings!

I am having difficulty determining the cause of the FAILED status in my fleet.

I am currently attempting to deploy a new fleet via a build. After activating, the fleet moves to a FLEET_STATE_ERROR status - specifically:

  1. FLEET_STATE_VALIDATING (1m 34s)
  2. FLEET_STATE_BUILDING (1s)
  3. FLEET_STATE_ACTIVATING (59m 53s)
  4. FLEET_STATE_ERROR

I suspect that the problem could be either the amount of time the ACTIVATING part takes (it seems long to me), or that I'm using AWS-SDK within my build.

However, the Events Log only tells me the status, and does not show me any log files that I can view.

I also puttied into the fleet to see what's there. I can see the game scripts, but I could not find any indication of logs files (they may be there - I am not a Linux expert so I may have missed them).

Is there a way to view the logs on the building of the fleet so I can determine the cause of the failure?

Thanks.

asked 4 years ago502 views
12 Answers
0

ACTIVATING -> ERROR means your server didn't complete the InitSDK, ProcessReady hooks

So to debug:

  • Firstly, look at the Fleet events and check the log from your install.sh file
  • Now look to see if there are other fleet events, that may indicate that your server process couldn't be started/crashed/failed to health check etc

Your build is run out of /local/game so your logs should be there thought it could be that your server build is trying to log to a location outside of this area and so won't produce logs.

Potential Next steps:

  • Ensure your server produces logs when hosted on Gamelift
  • Try manually running your server on the GameLift instance to see if it starts up as expected (a lot of fleet activation issues are to do with missing dependencies such as dlls, bad executable launch paths etc)
  • If you are building locally on Linux ensure you are correctly testing your packaged builds as closely as they are run on GameLift
  • Double check your GameLift integration with GameLift local which covers some of this stuff.
answered 4 years ago
0

If you are a really stuck and this is urgent, you can open an issue with GameLift support and they can probably look at your fleet. Otherwise, I would try getting logs working so you can see where in the process your fleet may be failing.

answered 4 years ago
0

Thanks for the reply. Luckily, it's not urgent, and I'm still trying to learn the ins and outs of this Build system.

The short answer is that there is not an install.sh file, as the entire package was downloaded. Earlier, I had tried with an install.sh file that had "#!/bin/sh \ npm install" (2 lines), and the Events log returned a 127 error. I took that to mean that "npm" wasn't found. So the second try I attempted to load the entire package, node_modules and all. I'd rather not do that, but I'm testing different options.

The package currently works on my local Ubuntu server.

Previously, I created my fleets from uploaded scripts (simple scripts to test communications) with great success. However, now that I'm using aws-sdk so I can access my Dynamodb servers, I needed to add aws-sdk to my package. It's too large to upload as-is via a script (limit of 5m), so the Build process is how I'm attempting to do now.

answered 4 years ago
0

Sorry should have asked if you were using GameLift with your own servers or GameLift with realtime scripts.

For realtime scripts I would start with a working version and work from there.

Adding the AWS SDKs can be tricky and obviously if you have a require only something thats not there, eveything may fail / break before logging is started, which is what you are seeing.

See end of this thread for advice: https://forums.awsgametech.com/t/how-do-i-access-dynamodb-or-lambda-from-a-real-time-server/6839/16

Let me know if that helps as I would guess its how you are packaging the SDK and referencing that is breaking your fleet (and logging)

answered 4 years ago
0

Thanks.

I saw that post a few days ago. I decided to first go through the motions to learn what the Build was all about, as I only launched fleets from Scripts before.

The only thing I don't understand yet is why the build that had the entire node_modules included with it (so no npm install required) still failed to activate the fleet and why it took almost an hour to run the activation. Fleets without the node_modules took less than a minute before failure (which I expected since AWS would have been undefined).

I've got time to continue to look into it. If I find a solution, I'll post.

answered 4 years ago
0

Update:

I created a build that had the absolute bare minimum - a single install.sh file with a "Hello World" echo in to, piped to a file, and a main.js file with nothing but "logger" statements to append to a log file. Does nothing else - no AWS, no game logic, no nothing.

Still, after an hour of Activating, I get a FAILED status with no indication whatsoever what could have gone wrong.

The build boots and works in my local GameLift implementation. I also know that, from previous tries, I can launch the fleet from a script file. But the same script will not work when using a Build.

So I don't know how to even proceed. Without any logs telling me why it failed, I'm at a loss to debug and fix the problem.

Is there any possible way to get a hold of the logs generated during the "Activating" phase?

answered 4 years ago
0

Can you provide your fleet id? I can get the GameLift service team to take a quick for you.

answered 4 years ago
0

Certainly! Here you go: fleet-******

I hope it's just something dumb I've done. :slight_smile:

answered 4 years ago
0

Thanks. I opened an issue with the GameLift service team and hope to get an update soon.

answered 4 years ago
0

From our logs, I found that the executable is /local/game/main.js. For a fleet created from a build to function correctly, the launch path should point to a valid executable. I don't believe the main.js can be executed directly.

You can check locally if a build you want to upload can be successfully executed:

sh /path/to/your/executable arguments

Also, you mentioned that a fleet created from a build with node modules included also went into ERROR state. Can you give me the ID of that fleet?

answered 4 years ago
0

Thank you for you reply. I didn't realize that the build worked differently from using scripts to create your fleet. The fleet that used the full node_modules will show the same problem as the bare-bones fleet, so there's no need for you to investigate - you'll find the same thing.

I did just try this for the launch path: /local/game/startup.sh

...with startup.sh being the following:

#!/bin/bash
node main.js

This produced the same results. I'm going to try again and specify the path to the node executable directly, and see if that makes a difference. I'll let you know.

It may be since I'm using only Node, that I'll just stick to uploading scripts and building fleets from that.

answered 4 years ago
0

An update:

The last thing I tried was having the launch path execute the following:

#!/bin/bash
/local/NodeJS/bin/node /local/game/main.js

And I get the same results.

Since my realtime server is nothing more than JS files, I don't believe that going the Build path is correct for me, as it appears to need an entire server. The whole purpose of me even thinking about using Builds was so that I could, somehow, run npm install from my application and install aws-sdk. That appears not to be possible, at least within my current knowledge of Builds.

But that takes me back to Square 1. I can't load aws-sdk with my server as there is a 5m limit on scripts (mine would be about 6.8). And I can't load in aws-sdk manually after fleet creation because the init() call creates a GameSession, which imports aws-sdk, therefore it must be available upon server boot-up.

At least I know why the fleets are failing. :slight_smile:

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions