Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.

Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.

InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.

RWKV7 推理服务 | 高并发批量推理，动态扩缩容，兼容OpenAI API，CUDA优化，低显存部署大模型推理 RWKV linear attention batch inference server — auto-queue, dynamic capacity, OpenAI compatible.

KVTide is a Kubernetes-native LLM serving system exploring cache-aware scheduling and proactive peer-to-peer KV mobility.

A high-performance deep learning model inference server based on TensorRT, supporting fast inference for Embedding, Reranker, and NLI models.

Distributed ML training and serving platform

"A practical guide to NLP model optimization and serving with NVIDIA Triton." / "실습으로 익히는 고성능 NLP 모델 최적화 및 NVIDIA Triton 서버 배포."

Online dynamic batching for faster LLM/VLM fine-tuning

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

Production-grade LLM inference API built from scratch. NestJS gateway + Python GPU workers. Scheduling, batching, KV cache, tensor parallelism, multi-modal — all against real GPUs.

cost-aware-inference-cluster: FastAPI inference-serving prototype with Redis queues, worker heartbeats, dynamic batching, autoscaling simulation, and local Docker Compose benchmark evidence.

LM Multi-Bin Dynamic Scheduler Simulator - Implementation combining Multi-Bin batching with SLA-constrained dynamic batching

Zephyros is a lock-free, zero-allocation MPSC (Multi-Producer, Single-Consumer) ring buffer for Go, engineered for applications that demand extreme throughput, predictable latency, and uncompromising efficiency under high concurrency.

Model serving platform with FastAPI, Docker, Prometheus, Grafana, canary releases, A/B testing, drift checks and dynamic batching.

Dynamic batching inference server with FastAPI, dynamic padding, and benchmark visualizations

A lightweight, framework-agnostic middleware that dynamically batches inference requests in real time to maximize GPU/TPU utilization.

High-throughput, low-latency LLM inference platform for LLaMA-3 & Mistral — dynamic batching, KV-cache optimization, FP16/BF16 mixed precision, tensor parallelism, with PyTorch profiling, Prometheus/Grafana observability, Docker & Kubernetes (HPA) deployment.

Production-style model inference platform — versioned registry, dynamic micro-batching (measured 8x throughput), canary + shadow deployments with instant rollback, load shedding (429+Retry-After), Prometheus metrics, async load-test harness

GPU-aware LLM serving runtime with async scheduling, dynamic batching, streaming, admission control, metrics, and mock/llama.cpp/vLLM backends.