"creating kernel" step took more than 36 hours in hardware system build

Hello!

I'm doing a hardware system build using Vitis on z1d.2xlarge instance. The goal is to accelerate an algorithm with the f1.2xlarge instance. I am using the latest FPGA AMI for this instance.

My application has one kernel in which the core computation-intensive part is two completely unrolled matrix-vector multiplication, each one looks like the following:

#define MAX_ROWS #a number#
#define MAX_COLUMNS #a number#

typedef ap_fixed<2, 2, AP_TRN, AP_WRAP> fixed_point_2t;
typedef ap_fixed<18, 6, AP_TRN, AP_WRAP> fixed_point_16t;
typedef ap_fixed<32, 16, AP_TRN, AP_WRAP> fixed_point_32t;

fixed_point_2t matrix[MAX_ROWS][MAX_COLUMNS] = {0};
fixed_point_32t vector[MAX_COLUMNS] = {0};
fixed_point_32t product[MAX_ROWS] = {0};

for (int i =0; i < NUM_COLUMNS; i ++)
{
#pragma HLS PIPELINE
    for (int j = 0; j < MAX_ROWS; j++)
    {
        product[j] += fixed_point_16t(matrix[j][i]) * fixed_point_16t(vector[i]);
    }
}

NUM_COLUMNS is a variable in the kernel function, and MAX_ROWS is a hardcoded constant. The goal here is to have the kernel support matrix-vector multiplication for N_ROWS, N_COLUMNS < MAX_ROWS, MAX_COLUMNS. All the arrays are partitioned to support the corresponding unrolling. I omitted the memory transferring part from the above code.

Currently, I am compiling a version for MAX_ROWS = 2000 and MAX_COLUMNS = 2000, it has stuck on the "creating kernel" step for 36 hours now. I can see that it is always using 100% of one CPU core, and the memory (RAM) usage on the z1d instance slowly increases over time.

With some back-of-envelope calculations, I am pretty confident that the resources (DSPs, BRAMs) on the FPGA are able to support what the application requires. However, this long building time has made me very worried.

Here are my questions:

1. Is this not the correct way to do large size matrix-vector multiplication (for N >= 2000)? If so, what could be a better way to do it?
2. Is there any place where I can check if the synthesis progress is stuck? It is taking a fairly long time now and I am not sure if it is ever going to finish.

I can share my source code if needed, but I would prefer not posting it on the forum (sorry about that). Thank you for reading through my post, and I am open to any opinion/help on this issue. Please let me know if any part of my question is not clear, and any suggestion is greatly appreciated!

Thank you,

Owen