Using NICE DCV for game streaming - poor fps?

Hi,

there are great examples of how well DCV works for games - e.g. https://www.youtube.com/watch?v=aJ8MO6Sfvpo

Can you check the FPS and bandwidth available which you can find in Settings (wheel to the upper left) -> Open Connection Metrics ?

By default the FPS on the server side might be limited to 25 FPS depending on the OS. Here is more how to tune DCV: https://docs.aws.amazon.com/dcv/latest/adminguide/config-param-ref.html

Best regards,
Karsten

Thanks Karsten for your reply.

I have watched the video and a few others, and this is what prompted me to ask the community how others are achieving these frame rates at 1080p over NICE DCV. I can see in the video he is getting 60-80fps @ 1080p via the Steam fps counter, and the NICE DCV counter is reporting a locked 60fps. Bear in mind he's playing the game on HIGH graphics settings rather than ULTRA. Unfortunately there's no detailed technical info in the video regarding how he configured the NICE DCV instance.

For my testing, I've reviewed the same fps counters and with a similar bandwidth and latency to his (50-70mbps and 15-20ms), the Steam fps counter never goes above 50 and fluctuates down to 30fps. The NICE DCV counter also fluctuates and never locks at 60, it's more like 30-40fps.

Thanks, I also recently learnt about the DCV tuning configuration in the Windows registry. Indeed by default it's capped at 25fps. You need to add the "target-fps" registry entry and specify something like 60 (decimal) to specify that upper limit. You can also set the value to 0 for unlimited fps.

Whilst that registry setting did help, it only took the NICE DCV FPS from 15 to 30fps.

So, I'm still perplexed how these higher locked 60fps frame rates are being achieved. I can tell by the way videos play, and games play, over NICE DCV, that the frame rates are not smooth or consistent. I am hoping there's some other config I'm missing and someone is going to enlighten me :-)

Thanks a lot.

Hi,

in case your network to AWS shows 50-70mbps and 15-20ms (which the DCV client metrics confirm i assume) then with 20ms you should be able to get 50 fps or more.

Are you using the native DCV client (not the web browser) for best performance and can you check if the CPU on your client is maybe the bottleneck (process monitor)?

Best regards,
Karsten

Hi Karsten,

Thanks, that's a very good point. I was using the web client and forgot that the DCV client is more performant ;-) I've now switched to that and it's definately now feeling more "fluid" when navigating with the mouse in-game. The fps reported by the DCV client metrics is a little better, average 40fps, unfortunately still not a consistent 60fps.

I did check the client CPU and that's looking good at 20% utilised. The DCV viewer process is only consuming around 2% CPU.

What I am seeing is better fps for static screens or videos within the game. But once the real gameplay starts, the performance dips and is never consistent. Looking back at that YouTube video where he's playing Tomb Raider, he is seeing similar performance issues actually. Watch it closely... When he stops moving the character, the fps is great. When he starts moving around the world, he also sees drops to 30fps. If a discrete Nvidia GPU on a physical PC had the same vRAM as the Tesla one inside the g4dn instances (16GB vRAM), the performance would be great. But inside AWS via NICE DCV, that GPU vRAM does not seem to translate to consistent frame rates in-game.

I am conducting further tests in an AWS region a little closer to me. I am also trying to obtain more detailed specs on these Tesla T4 cloud GPUs. eg, Are they optimised for 3D modelling/CAD, rather than games where you need the GPU to push out millions of polygons?

Thanks!

Hi,

great to hear the progress! I would assume the T4 are the same as the physical ones you can buy but that is just a guess ;)

You could also check how the load on the CPU is or if there are other bottlenecks on the server side which would explain dropping FPS on the server side when there are strong movements.

If you think we have reached a conclusion please feel free to mark as solved.

Best regards,
Karsten

Thanks for your continued help.

Unfortunately we don't have a solution just yet. I am conducting further in-depth testing and I am finding initially that the type of game really is a factor. Potentially these cloud Teslas are optimised for AI and compute, and in a game with a high number of polygons/character assets when you navigate with the camera, the performance is taking a real hit. The YouTube Tomb Raider video also seems to confirm that.

I did also already review detailed Windows performance counters on the host EC2 instance side, and ruled out any bottleneck when it comes to CPU, RAM, disk, and network. It's the GPU that's reporting 100% utilisation.

Thanks again, will report back when I have more test results.

In case the GPU is the limiting factor did you try a more powerful one like the V100 in the p3.2xl in case?

Thanks for the suggestion.

I believe only the g family instances are suitable for NICE DCV. Using the official AMI reveals they don't support the p family. Looks like that instance family is designed for machine learning, big compute, rather than graphics. Those p instances do have multiple Tesla GPUs, but bear in mind most games will not take advantage of multi-GPU (equivalent of Nvidia SLI).
In fact, when I tested a larger g instance with multiple GPUs, it made no difference to fps/performance for gaming.

Thanks anyway, I think the key thing here is confirming if these cloud-based Tesla GPUs have the relevant polygon/shader/texture support for games. Based on testing so far, they struggle to get above 30fps when there are many character assets/polygons in motion.

Most games it seems will not benefit from multiple GPUs. The P instance with 1 V100 can likely be used for games as well at least with Windows. So it seems in the case of T4 the GPU is the bottleneck - did you check with nvidia-smi -a or other tools which part of the GPU is at 100%?

Thanks, I was not aware of that command. If you have any further detail, I'll check, thanks a lot.

I've run "nvidia-smi -a" when the game streaming fps dropped low, and forwarded that on to AWS Support.

Key performance stats:

Utilization  
    Gpu                               : 98 %  
    Memory                            : 47 %  
    Encoder                           : 3 %  
    Decoder                           : 0 %  
Encoder Stats  
    Active Sessions                   : 1  
    Average FPS                       : 34  
    Average Latency                   : 2227

Great to see you have interesting numbers from nvidia-smi confirming the assumption that the GPU and in this case the utilization seems to be the bottleneck. Please keep us posted on findings.

Hi, How were you able to connect to Nice DCV? I have a G4 instance, with BYOL windows 10 . I have all drivers and I have been able to Stream using parsec, however now that I want to try DCV, I am just not able to connect. Port is open and included in security group, however whenever I type IP or DNS in connection string, it just keeps spinning. Any idea?

Thanks

Hey there,

I don't believe that will work as you need one of the NICE DCV AMI-based instances, these are actually hosted on Windows Server rather than Windows 10.

For example, this is the AMI I've been using:
https://aws.amazon.com/marketplace/pp/B082HYM34K?ref=cns_srchrow

I honestly don't know if Windows 10 will work or is supported. If you scroll down on that web page link, the Usage Instructions may help you. Other than the security group inbound rule on TCP port 8443, did you make sure to grant the instance IAM role permissions to read the NICE DCV licence from the S3 bucket?

Also the usual things to check, such as, is there a Network ACL in the VPC denying inbound/outbound traffic.

Thanks, hope that helps.

Hi,

did you check the DCV logs about any messages? Windows DCV Logfiles: C:\ProgramData\NICE\dcv\log\

More info about installing DCV on Windows is at https://www.ni-sp.com/how-to-install-nice-dcv-on-windows/

Best regards,
Karsten

Nice Karsten, I did not realise you can install this manually onto Windows client OS-based instance if you have BYOL.

One point mentioned in that guide, also make sure the Windows Firewall inbound rule is configured at the OS-level, as well as the security group/NACL at the AWS level.

Thanks for replying. Well, I opened port 8443 for UDP following a tutorial, should I be opening TCP port as well? I also was not aware that it needs to read the license from S3 bucket, however i was able to install it with no issues. Regarding the VPC i am using default VPC, so i guess it should not have an ACL blocking traffic, however i will check.

I am using windows 10 as BYOL as i wanted to try some games available on Microsoft store which was not available thorugh windows server, maybe that could be the issue, that this only works with windows server.

Thanks.

TCP 8443 needs to be opened in the security group for incoming. That seems to be the main issue right now.
S3 bucket missing will give you a notice.

Best regards,
Karsten

Yes, you definately only need that TCP port opened, it works fine for me without needing any UDP port rule.

The NACL for the default VPC is allow all inbound & outbound, but worth checking. Also worth checking which subnet you launched your EC2 into. Obviously it needs a route to the internet (assuming your client is remote somewhere), so either put the instance in a public subnet, or place in private subnet with a NAT gateway.

An update on the original fps performance issue: My support case is ongoing with AWS, they are reviewing...

Edited by: MilkyMagic on Oct 14, 2020 7:39 AM

Thanks for following up guys, I opened TCP port and there is an improve now. Before It was just not able to reach the server, now I get a prompt asking for user, password, It says starting session and then it errors out saying "unable to connect, checkhostname/address and try again" , this is strange, Ihave checked in the server and the service is up and running. I assume the user and password is the one for the PC?

Regarding the VPC, I checked the default VPC, the subnet is using a route table mapped to an internet gateway, so i would think thats not the issue.

I know tit is trying to connect because it disconnects any remote connection i have using RDP or Parsec, so it is definitely reaching the computer, however something is killing the connection at the end.

Hopefully you get a reply from support as i am trying to use this for gaming as well.

Edited by: Rgarza90 on Oct 14, 2020 4:04 PM

Edited by: Rgarza90 on Oct 14, 2020 4:15 PM

Good to hear you are making progress. Your VPC config is good.

I would definately make sure all RDP sessions are LOGGED OFF, and not just disconnected. Also make sure Parsec service is disabled. I know these other products can conflict with NICE DCV. To get a clean test, you may want to deinstall Parsec, logout any RDP sessions, reboot, and then attempt connection to NICE DCV again.

For a simple test, I normally connect to NICE DCV using "Administrator" user. The password will be the one you configured for the local Windows Administrator account. For EC2 instances, you would have a key pair and you have the associated private key. Inside the EC2 console, you would select the Instance, then go to the ACTIONS menu and select "Get Windows Password" - that's the local Administrator password you need to connect over NICE DCV.

There's some detailed info here in the docs around connection configuration, and the steps required to ensure your EC2 can reach the S3 licensing bucket:
https://docs.aws.amazon.com/dcv/latest/adminguide/dcv-ag.pdf

I can see that Windows 10 is indeed supported as a NICE DCV server host, as Karsten commented. Also worth checking, on a Windows Server EC2 instance, the EC2Config service is preconfigured to use the metadata to grab it's credentials and then connect to S3. If you're using your BYOL Windows 10 instance, maybe this EC2Config service is missing and thus causing failure. The steps to check/configure that are in the doc above.

Also, check for error messages in the NICE DCV log here:
C:\ProgramData\NICE\dcv\log\server.log

Hope that helps.

Edited by: MilkyMagic on Oct 15, 2020 12:49 AM

Thanks, I havent tried uninstalling PARSEC, also I dont need to generate dindows password, basically since this is a Virtual machine of windows 10 that I imported to create an AMI, I created my Admin user and password once I created the virtual machine, so all of these steps I can skip.

I have my admin account and this is the one i am using to log in to my windows 10, but howw can you log in to Nice DCV using your user?

Thanks

First off, let's provide an update on the fps frame rate challenges...

The answer to the original topic: Can we get good consistent 60fps via NICE DCV for game streaming, is a "maybe". After extensive analysis it really does depend which game you are trying to stream. The Tesla cloud GPUs are designed for CUDA compute so you'll never get the same experience compared to a physical gaming PC with a consumer graphics card. The NICE DCV protocol itself is extremely efficient with very low latency, it's the GPU that's the bottleneck here for gaming.

I would encourage people to test the specific game(s) you want to stream via NICE DCV. If there are few character assets on-screen at once, you'll most likely get very good fps. However, if the game features hundreds of NPCs/characters you will see the frame rate drop significantly.

This situation most likely will improve over time. We know that AWS are very good at constantly improving their services. Let's hope we see the game-focused optimisations of the NVIDIA 30-series Ampere consumer GPUs make their way over to the cloud GPUs in the near future maybe.

I'll mark this question as answered.

For the other question around Windows 10 BYOL instance and NICE DCV, let's get that set up as a separate discussion, thanks.

Thanks for your comprehensive answer! Did you have the chance to try a V100 based P3 instance for better FPS and performance (the T4 might show half of the performance here compared to the V100)? Should be easy to switch changing the instance type. I guess the 30xx consumer cards will not make it but the A100 was announced.

Hey Karsten, actually I did not test P3 instances, I will spin up and test over the next few days, and report back a performance comparison.

Cheers for the suggestion :-)

Yes, just to confirm, I've tested a p3 instance using both GRID and Tesla drivers. All properly licensed and confirmed detected as NVIDIA Quattro Workstation.

Same fps frame rate performance issues unfortunately. Noteworthy that according to the AWS GPU docs, only the g4 instances are officially supported to use the NVIDIA vGaming licence and gaming driver.

Oh well, was worth a try :-)

Interesting! Would have expected the V100 performs significantly better and produces more FPS - apparently it does'nt. Yes, worth a try ;)

Just providing an update here.

I found an AWS instance config that gives me much better frame rates for the intensive games I was testing. So around ~100fps compared to ~40fps. This is the p3.8xlarge instance with EGA (Elastic Graphics Acceleration) ENI attached. This boosts the GPU with an 8GB vRAM GPU-based ENI, however it only supports OpenGL vs DirectX.

It seems to have fixed a lot of the poor frame rate issues. However, the graphical fidelity takes a hit, most likely because the EGA is not taking care of things via DirectX. For example, the graphics show a lot of “resolving” and gradients when moving around the environment.

So not a perfect solution, but at least we have some options for improved frame rates. AWS also recently announced the first Ampere-based GPU instances in USA, the p4dn instance type. However, it still appears tuned for ML/compute, and there's no NVIDIA gaming-type driver or AMI for that instance type yet.

Hope that helps others.

Edited by: MilkyMagic on Nov 3, 2020 8:09 AM

Thanks for sharing your interesting findings, MilkyMagic!

Are you still using the built-in 4 V100 as well in this scenario included with the p3.8xlarge and EGA - so virtually 4+1 GPUs?

That's correct, Windows Device Manager did detect the 4 x GPUs. The elastic ENI is hidden from Device Manager, you need to follow the AWS docs to load up a utility to support that, plus your instance needs a port rule adding to the security group:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/elastic-graphics.html

The instance type is of course quite expensive, but via spot pricing it's not too bad. Just also note certain regions/AZs may not be successful launching these instances based on capacity at AWS.

Also worth stating I used this AMI: NVIDIA Quadro Virtual Workstation - WinServer 2019
http://aws.amazon.com/marketplace/pp/B07TS3S3ZH?ref=cns_srchrow

Potentially could save a bit of money dropping to the p3.2xlarge instance which also supports this AMI.

Hi!

Just wanted to let you know that the new DCV 2020.2 version is out with QUIC based UDP support making cloud based gaming even more feasible with high FPS on the transport layer. You can enable QUIC in the DCV server and choose in the new DCV 2020.2 client.

Best regards,
Karsten

Ah yes thanks for the heads-up, I just learnt that too.

We'll do another test and report back, let's see what performance we gain from the NICE DCV improvements versus the underlying EC2 GPU hardware remaining the same.

Right, I've tested this latest NICE DCV release, making sure both the server and client were updated.

No improvement in frame rate and performance unfortunately, still inconsistent and averaging around ~40fps. Feels a little worse actually.

To be honest, I believe the NICE DCV protocol was always performant and not the key bottleneck. That bottleneck I believe is still the lack of consumer GPU performance within the cloud instances.

Indeed - the main FPS limiter seemed to be in your case the rendering GPU.

To use QUIC which is offering e.g. 60 FPS in our demo cases also with higher latencies you can enable it in the registry (https://docs.aws.amazon.com/dcv/latest/adminguide/enable-quic.html) and configure to use it in the new DCV client as well.

Here is some background on the configuration: https://www.ni-sp.com/nice-dcv-tips-and-tricks/#h-how-to-enable-the-udp-based-quic-transport-protocol-in-dcv

Good point, I may have spoken too soon! Let me double-check this config, and re-test, thank you.

Edited by: MilkyMagic on Nov 13, 2020 3:57 AM

OK I've retested, no DICE unfortunately (pun intended).

I've enabled QUIC UDP on both the server and client, and I've confirmed via the client's streaming monitor that it's using QUIC.

We are still getting ~40fps locally on the instance, and DCV is reporting ~18fps on the client side via NICE client over QUIC. I believe the QUIC improvements will help with latency over the network. But the issue lies not with the network or the streaming protocol, it's the GPU hardware locally in the instance itself that is not performant for games, it was never designed to be.