How Fast GPU Computation Can Be

How fast do GPU computation gains compare with CPU? In this article, I am going to test it out using Python and PyTorch Linear Transform functions.

Here are some of my test machine specs:

CPU: Intel i7 6700k (4c/8t)
GPU: RTX 3070 TI (6,144 CUDA cores and 192 Tensor cores)
RAM: 32G
OS: Windows 10

NVIDIA GPU Jargons explained

CUDA is an abbreviation for Compute Unified Device Architecture. You can use CUDA to access the NVIDIA GPU instruction set directly.

Unlike DirectX and OpenGL, which are purposefully designed for building game engines, CUDA does not require users to understand complicated graphics programming language¹.

Tensor Cores are processing units that accelerate the process of matrix multiplication².

For example, multiplying two 4×4 matrices using CPU or CUDA involves 64 multiplications and 48 additions, one operation per clock cycle, while tensor cores can perform multiple operations per clock cycle.

More introductions to Tensor Cores in this video from the Nvidia Developer’s YouTube channel.

What is the relationship between CUDA cores and Tensor cores? Tensor Cores are built into the CUDA cores, those magical cores will be triggered when certain conditions are met.

Test methodology

GPU computation is faster than CPU only in some typical scenarios. In other cases, computation in GPU can be slower than in CPU!

CUDA is vastly used in Machine Learning and Deep Learning because of its particular goodness at parallel matrix multiplication and addition.

In Math equation:

PyTorch’s Linear function torch.nn.Linear do exactly the same operation. For example, you can transform a 2×2 matrice into a 2×3 matrice by the following code:

import torch

in_row,in_f,out_f = 2,2,3
tensor            = torch.randn(in_row,in_f)
l_trans           = torch.nn.Linear(in_f,out_f)
print(l_trans(tensor))

CPU baseline

Before measuring the GPU performance, I need to set up a baseline performance from the CPU.

To give some burdens to the chips and prolong the running time, I increase the in_row, in_f, and out_f numbers, also set the loop operation 10,000 times.

import torch
import torch.nn
import time

in_row, in_f, out_f = 256, 1024, 2048
loop_times = 10000

Now, let’s see how many seconds will take the CPU to finish the 10,000 transformations:

s       = time.time()
tensor  = torch.randn(in_row, in_f).to('cpu')
l_trans = torch.nn.Linear(in_f, out_f).to('cpu')
for _ in range(loop_times):
    l_trans(tensor)
print('cpu take time:',time.time()-s)

Result:

cpu take time: 55.70971965789795

My i7 6700k takes about 55 seconds, to be honest, the result is not bad.

Compute it in GPU

To ask GPU’s CUDA to perform the same computation, I simply replace .to('cpu') to .cude() . Besides, considering the operations in CUDA are asynchronous, I also need to add a synchronization statement to ensure printing the used time after all CUDA tasks are done.

s       = time.time()
tensor  = torch.randn(in_row, in_f).cuda()
l_trans = torch.nn.Linear(in_f, out_f).cuda()
for _ in range(loop_times):
    l_trans(tensor)

torch.cuda.synchronize()
print('CUDA take time:',time.time()-s)

Code changes are highlighted, here is the running result:

CUDA take time: 1.327127456665039

Almost 42x times faster than running in CPU. A model that needs several days’ training in CPU may now take only a few hours in GPU. This is really fast.

Enable Tensor Cores

CUDA is already fast, how about enabling the RTX 3070Ti’s 197 tensor cores? According to this video, In PyTorch, to enable Tensor Cores, all I need to do is reduce the float precision from FP32 to FP16.

s       = time.time()
tensor  = torch.randn(in_row, in_f).cuda().half()
layer   = torch.nn.Linear(in_f, out_f).cuda().half()
for _ in range(loop_times):
    layer(tensor)

torch.cuda.synchronize()
print('CUDA with tensor cores take time:',time.time()-s)

Result:

CUDA with tensor cores take time:0.5381264686584473

Another 2.6x times performance improvement.

Conclusion

In this article, I compared the Linear Transformation operation by calling PyTorch Linear transform function in CPU, GPU CUDA, and GPU CUDA + Tensor Cores. Here is a summarized result: