matrix_multiplication_cudaflow Define a Matrix Multiplication Kernel matrix_multiplication_cudaflow_1GPUAcceleratedMatrixMultiplication Define a cudaFlow for Matrix Multiplication matrix_multiplication_cudaflow_1DefineAcudaFlowForMatrixMultiplication Benchmarking matrix_multiplication_cudaflow_1MatrixMultiplicationcudaFlowBenchmarking Following up on Matrix Multiplication, this page studies how to accelerate a matrix multiplication workload on a GPU using tf::cudaFlow. GPU can perform a lot of parallel computations more than CPUs. It is especially useful for data-intensive computing such as matrix multiplication. With GPU, we express the parallel patterns at a fine-grained level. The kernel, written in CUDA, is described as follows: //CUDAkerneltoperformmatrixmultiplication __global__voidmatmul(int*A,int*B,int*C,intM,intK,intN){ introw=blockIdx.y*blockDim.y+threadIdx.y; intcol=blockIdx.x*blockDim.x+threadIdx.x; intsum=0; if(col<N&&row<M){ for(inti=0;i<K;i++){ sum+=a[row*K+i]*b[i*N+col]; } c[row*N+col]=sum; } } Each CUDA thread corresponds to an element of C and compute its result. Instead of storing each matrix in a 2D array, we use 1D layout to ease the data transfer between CPU and GPU. In a row-major layout, an element (x, y) in the 2D matrix can be addressed at x * width + y in the transformed 1D layout. The next step is to allocate memory for A, B, and C at a GPU. We create three tasks each calling cudaMalloc to allocate space for one matrix. Then, we create a cudaFlow to offload matrix multiplication to a GPU. The entire code is described as follows: voidmatrix_multiplication(int*A,int*B,int*C,intM,intK,intN){ tf::Taskflowtaskflow; tf::Executorexecutor; //allocatethehostandgpustorageforA tf::Taskallocate_a=taskflow.emplace([&](){ cudaMalloc(&da,M*K*sizeof(int)); }).name("allocate_a"); //allocatethehostandgpustorageforB tf::Taskallocate_b=taskflow.emplace([&](){ cudaMalloc(&db,K*N*sizeof(int)); }).name("allocate_b"); //allocatethehostandgpustorageforC tf::Taskallocate_c=taskflow.emplace([&](){ cudaMalloc(&dc,M*N*sizeof(int)); }).name("allocate_c"); //createacudaFlowtasktorunthematrixmultiplication tf::TaskcudaFlow=taskflow.emplace([&](){ tf::cudaFlowcf; //copydatatoda,db,anddc tf::cudaTaskcopy_da=cf.copy(da,A,M*K).name("H2D_A"); tf::cudaTaskcopy_db=cf.copy(db,B,K*N).name("H2D_B"); tf::cudaTaskcopy_hc=cf.copy(C,dc,M*N).name("D2H_C"); dim3grid((K+16-1)/16,(M+16-1)/16); dim3block(16,16); tf::cudaTaskkmatmul=cf.kernel(grid,block,0,matmul,da,db,dc,M,K,N) .name("matmul"); kmatmul.succeed(copy_da,copy_db) .precede(copy_hc); //launchthecudaFlow tf::cudaStreamstream; cf.run(stream); stream.synchronize(); }).name("cudaFlow"); //freethegpustorage autofree=taskflow.emplace([&](){ cudaFree(da); cudaFree(db); cudaFree(dc); }).name("free"); //createdependency cudaFlow.succeed(allocate_a,allocate_b,allocate_c) .precede(free); //dumpthegraphwithoutunfoldingthecudaFlow taskflow.dump(std::cout); //runthetaskflow executor.run(taskflow).wait(); //dumptheentireexecutiongraphincludingunfoldedcudaFlow taskflow.dump(std::cout); } Within the cudaFlow, we create two host-to-device (H2D) tasks that copy data from A and B to da and db, one device-to-host (D2H) task that copies the result from dc to C, and one kernel task that launches matmul on the GPU (by default, GPU 0). H2D tasks precede the kernel and the kernel precedes the D2H task. These GPU operations form a GPU task graph managed by a cudaFlow. The first dump of the taskflow gives the following graph: A cudaFlow encapsulates a GPU task dependency graph similar to a tf::Subflow (see Subflow Tasking). In order to visualize it, we need to execute the graph first and then dump the taskflow. We run three versions of matrix multiplication, sequential CPU, parallel CPUs, and one GPU, on a machine of 12 Intel i7-8700 CPUs at 3.20 GHz and a Nvidia RTX 2080 GPU using various matrix sizes of A, B, and C. A B C CPU Sequential CPU Parallel GPU Parallel 10x10 10x10 10x10 0.142 ms 0.414 ms 82 ms 100x100 100x100 100x100 1.641 ms 0.733 ms 83 ms 1000x1000 1000x1000 1000x1000 1532 ms 504 ms 85 ms 2000x2000 2000x2000 2000x2000 25688 ms 4387 ms 133 ms 3000x3000 3000x3000 3000x3000 104838 ms 16170 ms 214 ms 4000x4000 4000x4000 4000x4000 250133 ms 39646 ms 427 ms As the matrix size increases, the speed-up of GPU over CPUs becomes prominent. For example, at 4000x4000, the GPU runtime is 585.8 times faster than the sequential CPU runtime and is 92.8 times faster than the parallel CPU solutions.