Neural Speed: Fast Inference on CPU for 4-bit Large Language Models

Up to 40x faster than llama.cpp?

Generate with DALL-E

Running large language models (LLMs) on consumer hardware can be challenging. If the LLM doesn’t fit on the GPU memory, quantization is usually applied to reduce its size. However, even after quantization, the model might still be too large to fit on the GPU. An alternative is to run it on the…