After load cl_sde AFI, error (error_number=12)

Hello,

Greetings of the day!! Thank you for contacting AWS.

I understand that you are receiving an error while running some performance test with the Amazon FPGA Image and you seek our assistance in resolving the same.

The error indicates a timeout while waiting for descriptor credits during an SDE (Smart Data Engine) H2C (Host to Card) performance test on the FPGA hardware accelerator. The core issue manifests as a descriptor credit timeout error (4099), where the device fails to properly update its descriptor credits, which is typically related to bus mastering configuration problems. Also, It means you are encountering issues with the Amazon FPGA Image (AFI) appears to be related to a DMA (Direct Memory Access) allocation failure. The error code 12 typically corresponds to "ENOMEM" in Linux systems, which indicates "Out of memory" - the system cannot allocate the requested memory. It also

Looking further, the issue starts with "fpga_dma_mem_alloc_huge failed" which suggests that the application is trying to allocate a large amount of memory for DMA operations between the host and the FPGA, but this allocation is failing.

This could be due to several reasons:

1. Insufficient system memory available for the huge page allocation
2. The AFI might not be properly loaded or in the correct state
3. Permission issues with memory allocation (even though you're using sudo)


To troubleshoot this issue:


1. Check the state of your AFI to ensure it's properly loaded and in the "available" state. An AFI can be in one of four states: pending (bitstream generation in progress), available (ready for use), failed (bitstream generation failed), or unavailable (no longer available for use).
2. Verify system memory resources and huge page configuration. The application is trying to allocate 131072 bytes (as specified in your command parameters), but the system might not have enough contiguous memory available.
3. Try reducing the buffer size in your command parameters to see if a smaller allocation succeeds.
4. Check system logs for additional information about memory allocation failures.


The final error "Error: (24) software-problem" is a generic indication that there's an issue with the software stack rather than with the hardware itself.

Further, coming to the the follow-up error,

2025-05-31T01:36:33.602879Z, undefined, ERROR, ../src/sde_lib/sde_mgmt.c +311: sde_mgmt_wait_desc_credit(): Desc Credit Timeout credits_avail=0, num_desc=32, desc_limit=64, desc_consumed=64 2025-05-31T01:36:33.602933Z, undefined, ERROR, ../src/sde_h2c_perf_test.c +94: main(): Error waiting for descriptor credit Error: (4099) descriptor-limit-timeout A descriptor limit timeout was detected. The SDE logic will update the Descriptor Credit "Limit" Counter in local memory. Check that the device has bus mastering enabled.

It might be occurred while executing an SDE host-to-card performance test with parameters set to buffer size 128 bytes. The system encountered a descriptor credit timeout, where the available credits dropped to zero while the system had consumed all 64 possible descriptors against a limit of 64, with 32 descriptors requested for the operation. This exhaustion of descriptor credits triggered error 4099, indicating the SDE logic failed to properly update the Descriptor Credit Limit Counter in local memory.

The system generated this timeout error because it was unable to obtain the necessary descriptor credits to continue operation, suggesting a potential issue with bus mastering. When functioning properly, the SDE logic should continuously update the Descriptor Credit Limit Counter in local memory to maintain data flow. This error typically indicates either a hardware configuration issue, specifically with bus mastering settings, or a communication breakdown between the host system and the FPGA device in managing descriptor credits.

To troubleshoot this issue, you can attempt to run the SDE performance test with modified parameters. First, try reducing the descriptor count by executing the test with a buffer size of 64 bytes using the command 'sudo ./sde_h2c_perf_test 1 64 0'. If this doesn't resolve the issue, you can further reduce the settings to a minimal configuration by running the test with a buffer size of 32 bytes using 'sudo ./sde_h2c_perf_test 1 32 0'. These smaller buffer sizes may help identify if the issue is related to descriptor count limitations or resource constraints in the system.

I hope this information helps you in troubleshooting the issue further. In case if the issue persists, we would require details that are non-public information. Thus, I would request you to kindly open a support case with AWS using the following link - https://console.aws.amazon.com/support/home#/case/create by giving detailed description of the error message along with screenshots/error messages.

Thank you and have a nice day!!