Llm serving

vLLM Ascend Plugin is a community-maintained hardware acceleration plugin for vLLM that enables large language model serving on Huawei Ascend...

RTP-LLM is Alibaba's high-performance LLM inference engine designed for production-scale serving of large language models across multiple business...

TensorRT LLM is an NVIDIA-developed framework that optimizes inference for large language models and visual generation models on NVIDIA GPUs through...

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, using PagedAttention and continuous batching to...

GPUStack is an open-source GPU cluster manager designed for high-performance AI model serving (vLLM, SGLang, TensorRT-LLM) and on-demand GPU instance...

Parallax is a distributed inference framework that enables decentralized LLM serving across heterogeneous devices using P2P communication and...

kvcached is a KV cache management library for LLM serving and training on shared GPUs, using OS-style virtual memory abstraction to enable elastic,...

SkyPilot is a unified control plane for running, managing, and scaling AI workloads across heterogeneous infrastructure—Kubernetes, Slurm, 20+ cloud...

Ray is a distributed computing framework designed for scaling Python and AI workloads across clusters, cloud providers, and Kubernetes. It excels at...

A comprehensive knowledge base and tutorial collection focused on large language model engineering, covering training, inference, compression,...

BentoML is a Python framework for building and deploying AI model inference APIs with features like dynamic batching, multi-model orchestration, and...

Superduper is an end-to-end framework for building AI applications and agents integrated directly with databases (MongoDB, SQL, Snowflake, Redis). It...

OpenLLM enables developers to deploy any open-source LLM (Llama, Qwen, DeepSeek, Mistral, etc.) as an OpenAI-compatible API with a single command,...

Chitu is a production-grade inference framework for large language models optimized for efficiency, flexibility, and operational stability. It...

FastDeploy is a production-grade inference and deployment toolkit specifically designed for large language models (LLMs) and vision language models...

LoRAX is a specialized inference server designed to serve thousands of fine-tuned LoRA adapters efficiently on a single GPU, drastically reducing...