Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization [BMVC 2020, ICCV 2021]

Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

PyTorch DataLoader for many VQA datasets

A PyTorch implementation of the Hyena-Y model, a convolution-based multi-hybrid architecture optimized for edge devices.

Predicting a subgraph alongside the answer in a graph based VQA model

MHA, MQA, GQA, MLA 相关原理及简要实现

Vision-Language, Solve GQA(Visual Reasoning in the Real World) dataset.

This is a multimodal model design for the Vision Question Answering (VQA) task. It integrates the Llama2 13B, OWL-ViT, and YOLOv8 models.

A RAG-based question-answering system that processes user queries using local documents. It extracts relevant information to answer questions, falling back to a large language model when local sources are insufficient, ensuring accurate and contextual responses.

[GCPR 2023] Zero-shot Translation of Attention Patterns in VQA Models to Natural Language

LaTeX files for my honours thesis: "Graph Attention Networks for Compositional Visual Question Answering"

⚡ Optimize attention mechanisms with FlashMLA, a library of advanced sparse and dense kernels for DeepSeek models, improving performance and efficiency.

CUDA KV-cache eviction + INT8 quantized paged attention (vLLM-oriented). 26/26 pytest PASS on T1000. Eviction @16k blocks: host 1707.75µs → fused 562µs (~3×). INT8 attn p50 1.76–14.18ms; ~49.9% KV vs fp16. Apache-2.0 · PyTorch · NumPy.

This repository contains an implementation of Group Query Attention (GQA), an efficient variant of multi-head attention used in modern transformer models like LLaMA.

PyTorch implementation of the Transformer architecture (“Attention Is All You Need”) for English–Italian text-to-text translation, featuring encoder–decoder layers, multi-head attention, and training on a subset of the OPUS Books dataset.

LLM 推理链路优化 — Prefill/Decode 基准、Static/Paged KV Cache、GQA/MLA、Speculative Decoding；NumPy 可读参考 + PyTorch SDPA/AMP/compile 性能路径

A toolkit for vision-language processing to support the increasing popularity of mulit-modal transformer-based models

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

SSD-resident INT2/INT4 KV cache with asynchronous staging and fused CUDA dequantizing decode attention for long-context LLM inference.