Gamelift Realtime (Unity C# SDK) - DTLS Crashing | Reproduction Case

Question

Hello, I've decided to come to the community to see if anyone can make sense of why DTLS isn't working on the Unity SDK of Gamelift Realtime. Instead of trying to explain the code in my project or possibly third party libraries in the project being the cause, I've created a very basic reproduction case that anyone can run and see the problem.

https://github.com/SkyTech6/AWSRealTimeTLSTester

It's a very simple project. On the Manager object there is a toggle for "Using TLS". Then there are two different ways running this project will go:

**Not Using TLS:**

* Click "Create Game Session" Button. That will launch the create game session code as well connect to game session, etc. This will create a fast connection to a NON-cert session.
* Two messages will display on screen that this worked and the server will begin to send "KeepAlive" heart beats (counted in the upper right). 
* Click "Join Group" Button. This will join Group 10, and begin to send you "GroupAlive" heart beats counted above KeepAlive. 
* Type in the Input Field and then click "Send Message". Obviously this will send the message to the server and it will be echoed back to the client and displayed with the other messages.

This for the most part works fine.

**Using TLS:**

This is where things occur differently.

* Create Game Session will work and connect.
* Messages will display and heart beats will begin to count. 
* Clicking "Join Group" will CRASH Unity and give the stack trace attached in error.log.
* Send Message however will work.

When using TLS, _client.JoinGroup(int) will cause Unity to fail.

Interestingly, if you switch SendMessagePayload(string) to use _client.SendEvent instead of a _client.SendMessage().WithDeliveryIntent(DeliveryIntent.Reliable), then Send Message will also crash. Or if you change DeliveryIntent to be Fast, it will crash.

My guess is that the base RTMessage Send function is not correctly identifying the connection type for the messages, nor is it correctly falling back to fast when reliable cannot be made. As well there isn't error handling in place in the SDK to account for this.

[Editor.log|attachment](upload://ea66a1lHwO91APXCXIzYx39b3vn.log) (120.6 KB) [error.log|attachment](upload://ppncfyIqqfLB7FDPWfpa51WIro2.log) (130.3 KB)

Answer

Good morning!

I've followed the process detailed below and taken a look at the logs provided. What I saw in the error log that stood out was:

ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 00000252C78EAEF0)

I did two things:

A. Follow the below exactly within the Unity editor. I saw the same thing - with no TLS everything went as expected. With TLS I could create the game session, send messages / get 200 responses without clicking "Join Group" and get a crash immediately upon clicking "Join Group". My error log also included a similar error to the above:

ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 00000208264361E0)

B. Build the project in Windows and execute it outside of the Unity editor. Re-confirmed that all works as expected for the non-TLS case. Then, with my TLS fleet and NOT being within the Unity editor, I walked through all the steps again. I was able to create the game session, send messages and get 200 responses, click "Join Group" with success and then send messages again, still receiving a 200 response.

The current issue, at least with the latest information and sample project provided in the repository, seems to be an issue with linking the libraries specifically within the Unity editor. Please give the above another try with a built project outside of the Unity editor to see if you see similar success.

Answer

[quote="Sky_Copeland, post:10, topic:9386"]
`    /Users/andres/Library/Developer/CoreSimulator/Devices/91DD50DC-643A-474F-8078-FC9D4AE6BF59/data/Containers/Bundle/Application/4FFC17FF-5C81-4743-B197-D00F9A3354D1/TLSReproductionCase.app/Frameworks/libGameLiftRealtimeNative.dylib: mach-o, but not built for iOS simulator`
[/quote]

This error message looks to be indicating that the library is found now, but can't be used because it isn't built for IOS Simulator.  Can you make sure the build target for the native library build is the same as the build target for the unity-iphone project build. In the case of the screenshots in my post above this is the iphone 11 Prox Max simulator.

Regarding the `NotSupportedException` issue, it does look to be an issue with il2cpp / AOT compilation. More specifially, I believe it is related to SWIG not supporting AOT (il2cpp). 
See issue: https://github.com/swig/swig/issues/1165

Still investigating possible solutions on GameLift end.

Answer

Hey @REDACTEDUSER

I'm unsure how big the differences between Simulator SDK and Device SDK are, but I was unable to get it working on Simulator SDK. I'll probably spend some more time on trying to get that to work tomorrow, but in general I'd like to see this working on-device over anything else.

REMOVEDUPLOAD 
```
dyld: Library not loaded: @REDACTEDUSER
  Referenced from: /Users/andres/Library/Developer/CoreSimulator/Devices/91DD50DC-643A-474F-8078-FC9D4AE6BF59/data/Containers/Bundle/Application/4FFC17FF-5C81-4743-B197-D00F9A3354D1/TLSReproductionCase.app/TLSReproductionCase
  Reason: no suitable image found.  Did find:
    /Users/andres/Library/Developer/CoreSimulator/Devices/91DD50DC-643A-474F-8078-FC9D4AE6BF59/data/Containers/Bundle/Application/4FFC17FF-5C81-4743-B197-D00F9A3354D1/TLSReproductionCase.app/Frameworks/libGameLiftRealtimeNative.dylib: mach-o, but not built for iOS simulator
```

I looked at the iosLogs you've provided

```
NotSupportedException: To marshal a managed method, please add an attribute named 'MonoPInvokeCallback' to the method definition. The method we're attempting to marshal is: GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper::SetPendingApplicationException
```

I checked in the PInvoke file for that method and it is indeed not setup for marshaling

```
static void SetPendingApplicationException(string message) {
      SWIGPendingException.Set(new global::System.ApplicationException(message, SWIGPendingException.Retrieve()));
    }
```

I looked into it, this is specifically a problem unique to IL2CPP and Unity compiling. I've found a GitHub Issue from DiscordRP where an actual Unity IL2CPP developer and community chimed in on the problem and gave a solution that worked for both IL2CPP and Mono JIT (Unity's main option for other platforms).

https://github.com/discord/discord-rpc/issues/186

Something like this does seem to be the fix

```
[MonoPInvokeCallback(typeof(GameLiftRealtimeNativePINVOKE.SetPendingApplicationException))]
    static void SetPendingApplicationException(string message) {
      SWIGPendingException.Set(new global::System.ApplicationException(message, SWIGPendingException.Retrieve()));
    }
```

Although I'm unsure how much of the code you'll end up having to convert to support this to make it all work smoothly for IL2CPP, but according to that Unity developer in the issue discussion, it shouldn't have any affect on non-IL2CPP compiling.

Answer

Thanks for the detailed post and the sample project, which will be helpful.

I've passed this on to the GameLift service team and they should respond asap.

Answer

Has there been any further progress with iOS IL2CPP support?

Answer

@REDACTEDUSER

According to your repo you managed to get iOS working with TLS, did you manage to get past that somehow? If not, is my understanding correct that unsecured TCP communication aren't allowed on iOS AppStore apps, making GameLift Realtime currently not usable for iOS?

Answer

Interestingly, you could just use my built libraries from the github linked at the top (we keep that up to date in-studio as our test harness for GameLift builds). I'd also recommend copying the meta files if you would like their compilation targets to be setup for you as well.

As for your second question, it's not really a discrepancy. iOS and macOS don't use identical systems and these two different types of libraries are required for their respective platforms. However, I've never been able to get a dylib file built that works for macOS unfortunately.

Answer

Hi @REDACTEDUSER

If possible, can you give the following steps for linking the dynamic library on IOS a try:

1. Build the Native DYLIB library targetting IOS specifically using XCODE instead of make.
	a. cd into Native Directory of the Client SDK.
	b. cmake -G Xcode ..
	c. Open XCode Project and Modify Build Settings as necessary to build for IOS.
		Project -> Build Settings -> Change Supported Platforms: IOS
		Product -> Scheme -> Edit Scheme -> Build Configuration -> Release
		Select Device / Simulator to build for and build.

Result should be a libGameLiftRealtimeNative.dylib file in a release folder that is built for IOS.

2. Import the library to the IOS Project. 
	a. After creating the ios xcode project through unity, open it up and drag and drop the .dylib file created in the previous step into the "Frameworks" folder.
	b. Navigate to Unity Iphone Target -> Build Phase and Verify library is present in "Link Binary With Libraries" step.
	c. Add step to "Copy Files" Build Phase to copy the .dylib file during build. Make sure the Destination is also "FrameWorks".

Build / Run the application.

After completing the steps above, the iPhone Binary successfully links with the library on my end, and I stop getting a `DllNotFoundException` , however another issue related to SWIG usage pops up during runtime in the form of a `NotSupportedException` which we are still investigating.

Please let us know if  the steps above to link the Native library work for you.

Answer

Hey it's been about a week, has there been any progress or updates? Just checking in.

Answer

Thanks Pip! I appreciate that a lot.

Answer

>  This error message looks to be indicating that the library is found now, but can’t be used because it isn’t built for IOS Simulator. Can you make sure the build target for the native library build is the same as the build target for the unity-iphone project build. In the case of the screenshots in my post above this is the iphone 11 Prox Max simulator.

REMOVEDUPLOAD

I'm pretty sure I've built them as you have, although I could totally be missing something here on the xcode side... it's really not my area haha.

> Regarding the  `NotSupportedException`  issue, it does look to be an issue with il2cpp / AOT compilation. More specifially, I believe it is related to SWIG not supporting AOT (il2cpp).
See issue: https://github.com/swig/swig/issues/1165

https://github.com/swig/swig/pull/1262

It does seem like this pull request would fix the problem of SWIG not supporting AOT? Although it also looks like swig is blocking it's request for not meeting their standards of documentation...

Perhaps simply building that pull request could be a potential solution? If it's not already on the list haha.

Answer

Oh and regarding the dylib that works for macOS, the instructions on the Readme file (the straightforward cmake calls) did the trick for me when I built them from my M1 Macbook. According to the warning that gets thrown from Unity, looks like your macOS libs are built for 'MacCatalyst' instead of for 'macOS' for some reason. If you want I can share my library binary or give any info on the process I've used if you want.

Answer

Hi @REDACTEDUSER
 
I did this with your repo and the RTS Client SDK from the last public update.

One thing that could be different in our setups is I built for IOS Simulator since I don’t have a device to test on currently. This includes changing the Target SDK in unity to “Simulator SDK”.

I'll provide some screenshots of my build setup below to help narrow down the issue:

xcode project for building Native library:
REMOVEDUPLOAD

xcode project for building the test application:
REMOVEDUPLOAD

Client Log file: [iosLogs.txt|attachment](upload://AMWfuQSsK21kB7miNtAOarI80e.txt) (157.0 KB) 
(Note the `NotSupportedException` instead of the `DllNotFoundException`)

Answer

Hm yea that's one I've seen before... but I forgot what fixes it... maybe it was the link.xml file having all the required assemblies listed? IL2CPP might be stripping Google.Protobuf.

As for TLS, yea on iOS (and Android) our studio app will ship with Secured connections only to satisfy the Apple iOS requirement.

Answer

I followed these instructions and still ended up getting the same DLLNotFoundException.

```DllNotFoundException: Unable to load DLL 'GameLiftRealtimeNative': The specified module could not be found.```

Did you do this with my repo? Did you change anything else to make that work?

---

Just to isolate potentials, has your Native been changed since the last public update from September 2019?

Answer

Hey Michael,

Thanks for giving the reproduction a try. These are interesting findings, I had assumed that if they failed within Unity that a build would likely fail as well (in my experience it's more likely to fail in builds than editor haha).

So I've created builds to test to see if I would get the same results and interestingly enough an x64 Windows standalone application works perfectly with TLS and non-TLS.

However, I also tried to do an x86 build and there the non-TLS worked fine. But, with the TLS it failed to even make connection to the server, but no errors.

I then build for Android and got the same results as the x86 build. (I tested it on a Pixel 3 XL, which uses a x64 architecture, so not sure the actual cause there is being 32bit or not).

Then I had a coworker compile an xcode project to test on iOS. There when trying to make connection to the created game session it gets an app freezing error from "Com.Gamelift.Rt.Proto.PacketReflection" with both TLS and non-TLS. I'll attach that log below since it's the only clear error among these tests.

[iOS_AWSRealTimeTLSTester_Xcode_Log.txt|attachment](upload://o7AQUMYb4JwSV1ouiBL8hNlB5q1.txt) (13.3 KB)

I've also updated the Github Repo with a README.md that displays all the test results in a more readable manner.

Thanks for helping out on this Michael. I have a lot of faith in the Gamelift Realtime product and I'm happy to help trying to make secure connections among different platforms as well-working as possible

Answer

@REDACTEDUSER

Hey guys, just thought I'd give you an update of my findings for today. My focus today was trying to get iOS working. I'm happy to report that I've now got iOS working with a non certification / non-tls connection.

To do this I had to create a link.xml, switch the architecture to arm64, and make sure it was using Net 4.X instead of 3.5 (because Proto.Reflection doesn't support 3.5).

However, that does expose what the actual problem is on iOS getting secured connection to work.

The GameLiftRealtimeNative dll isn't found in ios builds. That's because the PINVOKE script in the SDK is using dynamic linking as shown below.

```
[global::System.Runtime.InteropServices.DllImport("GameLiftRealtimeNative", EntryPoint="CSharp_GameLiftRealtimeNative_new_NativeByteArray")]
  public static extern global::System.IntPtr new_NativeByteArray(int jarg1);

```

iOS (and Xbox) however don't support dynamic linking and instead only work with static linking like below.

```
[global::System.Runtime.InteropServices.DllImport("__Internal", EntryPoint="CSharp_GameLiftRealtimeNative_new_NativeByteArray")]
  public static extern global::System.IntPtr new_NativeByteArray(int jarg1);

```

I've attempted to perform this change myself and simply have two versions of the native plugin, one for iOS and one for other devices, but I've run into problems that aren't really within my wheelhouse and would probably be easier for the team that actually created the system and have access to the SWIG project.

```
DllNotFoundException: Unable to load DLL 'GameLiftRealtimeNative': The specified module could not be found.
  at GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper.SWIGRegisterExceptionCallbacks_GameLiftRealtimeNative (GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate applicationDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate arithmeticDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate divideByZeroDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate indexOutOfRangeDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate invalidCastDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate invalidOperationDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate ioDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate nullReferenceDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate outOfMemoryDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate overflowDelegate, GameLiftRealtimeNative.GameLiftRealtimeNativePINVOKE+SWIGExceptionHelper+ExceptionDelegate systemExceptionDelegate) [0x00000] in <00000000000000000000000000000000>:0 
```

[iOS_TLS_Connection_ErrorLog.txt|attachment](upload://h6nUuBX7gMrsshnC37xKTGsZWRx.txt) (140.2 KB)

The SDK should check to see which type of linking is required and then apply it.

```
string dll = isStatic ? "__Interval" : "GameLiftRealtimeNative";

[global::System.Runtime.InteropServices.DllImport(dll, EntryPoint="CSharp_GameLiftRealtimeNative_new_NativeByteArray")]
  public static extern global::System.IntPtr new_NativeByteArray(int jarg1);
```

Then the Unity Compiler could be checked with Assembly Symbols

```
#if Unity_IOS & !Unity_Editor
Aws.GameliftLiftRealtimeNative.SetStaticLinking(true);
#else
Aws.GameliftLiftRealtimeNative.SetStaticLinking(false);
#endif
```

I've also updated the GitHub Repo with my fixes & new test results.

Answer

I'm also facing the DllNotFoundException: Unable to load DLL issue, trying to get DTLS to work on iOS. I've followed to steps in the Readme to create GameLiftRealtimeNative.framework and have embedded it into my Unity game's xcode project. Also if somebody has a link to a built binary that works for them on iOS that they can share, it could help reduce some possible points of failure.

@REDACTEDUSER

Gamelift Realtime (Unity C# SDK) - DTLS Crashing | Reproduction Case

Relevant content