README.md

metadata

title: README
emoji: 💻
colorFrom: purple
colorTo: blue
sdk: static
pinned: false

Software-Delivered AI Inference

Neural Magic helps developers in accelerating deep learning performance using automated model sparsification technologies and inference engines. Download our sparsity-aware inference engines and open source tools for fast model inference.

• NM-vLLM: A high-throughput and memory-efficient inference engine for LLMs, incorporating the latest LLM optimizations like quantization and sparsity
• DeepSparse: Inference runtime offering GPU-class performance on CPUs and APIs to integrate ML into your application
• SparseML: Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models
• SparseZoo: Neural network model repository for highly sparse and sparse-quantized models with matching sparsification recipes

✨NEW✨ DeepSparse LLMs: We are excited to announce our paper on Sparse Fine-Tuning of LLMs, starting with MPT and Llama 2. Check out the paper, models, and usage.