A Performance Study Depending on Execution Times of Various Frameworks in Machine Learning Inference

Abstract

This work is intended to compare the latency of various frameworks in machine learning inference through an average power calculation model. This model is created in terms of a 2-layer neural network with PyTorch, in Python. Then, it is converted to a traced Torch Script module and also to ONNX file format. Afterwards, the C++ front-end is used for the inference process. The traced model is run with Libtorch on CPU and GPU, the ONNX file is run with ONNX Runtime on both CPU and GPU and it is also run with TensorRT on GPU. The inference execution times for 100 trials are averaged for all cases and it is realized that TensorRT with ONNX file format significantly outperforms its counterparts as expected. Hence, this work highlights the performance of TensorRT in machine learning inference and sheds light into the future by proposing several extensions. © 2021 IEEE.