awssak program fails in AMI 1.8.1 and 1.9.1

We have two xpfm setup (basically two different versions of the F1 shell) and we have generated bitfiles with them. The name of these two xpfm files are:

• xilinx_aws-vu9p-f1-04261818_dynamic_5_0.xpfm
  *xilinx_aws-vu9p-f1_shell-v04261818_201920_2.xpfm
  I believe the latter is the one which works with Vitis 2019.2 and later versions.
  When we program either of the bitfiles on an F1.2xlarge instance with AMI 1.7.1 (FPGA Developer AMI - 1.7.1-40257ab5-6688-4c95-97d1-e251a40fd1fc-ami-073c0053d4be15233.4 - ami-0c9c61319a32e0692) it passes. But when we program either of the bitfiles on F1.2xlarge instance with AMI 1.8.1 (FPGA Developer AMI - 1.8.1-40257ab5-6688-4c95-97d1-e251a40fd1fc-ami-0c90eb898f6c9f1bd.4 - ami-0209388abb64ef69c) or AMI 1.9.1 (FPGA Developer AMI - 1.9.1-40257ab5-6688-4c95-97d1-e251a40fd1fc-ami-0c3a51e8e73a7fa94.4 - ami-03d255ca72d04ce3e) they fail with the following error messages:

root@ip-172-31-4-120 ~# awssak program -p /tmp/binary_container_1_dynamic.awsxclbin
xclProbe found 1 FPGA slots with xocl driver running
INFO: Found 1 device(s)
IOCTL DRM_IOCTL_XOCL_READ_AXLF Failed: -1
ERROR: awssak program failed.

For newer version bitfile, dmesg shows the following errors:

[Sat Feb 20 00:47:28 2021] xocl 0000:00:1d.0: xocl_read_axlf_helper: did not find platform dtb
[Sat Feb 20 00:47:28 2021] xocl 0000:00:1d.0: xocl_read_axlf_helper: Failed to download xclbin, err: -22

and for the older version bitfile, dmesg shows the following error:

[Sat Feb 20 00:47:18 2021] xocl 0000:00:1d.0: xocl_read_axlf_helper: Failed to download xclbin, err: -110

Can you please guide us why this is happening?

Edited by: shahrzad on Mar 3, 2021 11:52 AM