url = https://github.com/ggerganov/llama.cpp.git

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

- Added first version of the changelog

endif()

Documentation is available at [https://abetlen.github.io/llama-cpp-python](https://abetlen.github.io/llama-cpp-python).

If you have previously installed `llama-cpp-python` through pip and want to upgrade your version or rebuild the package with different compiler options, please add the following flags to ensure that the package is rebuilt correctly:

pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:

Otherwise, while installing it will build the llama.ccp x86 version which will be 10x slower on Apple Silicon (M1) Mac.

CMAKE_ARGS="-DLLAMA_OPENBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

CMAKE_ARGS="-DLLAMA_CLBLAST=on" FORCE_CMAKE=1 pip install llama-cpp-python

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/ggml-model-name.bin ghcr.io/abetlen/llama-cpp-python:latest

>>> tokens = (llama_cpp.llama_token * int(max_tokens))()

ARG IMAGE=python:3-slim-bullseye

FROM ${IMAGE}

ARG IMAGE

RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \

python3 \

python3-pip \

ninja-build \

build-essential

RUN python3 -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette

RUN echo "Image: ${IMAGE}" && \

if [ "${IMAGE}" = "python:3-slim-bullseye" ] ; then \

echo "OpenBLAS install:" && \

apt-get install -y --no-install-recommends libopenblas-dev && \

LLAMA_OPENBLAS=1 pip install llama-cpp-python --verbose; \

else \

echo "CuBLAS install:" && \

LLAMA_CUBLAS=1 pip install llama-cpp-python --verbose; \

fi

RUN rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN echo "Installing model...this can take some time..."

COPY ./model.bin /app/model.bin

COPY ./start_server.sh /app/start_server.sh

RUN chmod +x /app/start_server.sh

ENV HOST=0.0.0.0

EXPOSE 8000

CMD ["/bin/sh", "/app/start_server.sh"]

ARG CUDA_IMAGE="12.1.1-devel-ubuntu22.04"

FROM ${CUDA_IMAGE}

CMD python3 -m llama_cpp.server

- `Dockerfile.openblas_simple` - a simple Dockerfile for non-GPU OpenBLAS

- `Dockerfile.cuda_simple` - a simple Dockerfile for CUDA accelerated CuBLAS

- `hug_model.py` - a Python utility for interactively choosing and downloading the latest `5_1` quantized models from [huggingface.co/TheBloke]( https://huggingface.co/TheBloke)

- `Dockerfile` - a single OpenBLAS and CuBLAS combined Dockerfile that automatically installs a previously downloaded model `model.bin`

You should now have a model in the current directory and `model.bin` symlinked to it for the subsequent Docker build and copy step. e.g.

**Note #1:** Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least

**TWICE** as much disk space as the size of the model:

| Model | Quantized size |

|------:|----------------:|

| 7B | 5 GB |

| 13B | 10 GB |

| 30B | 25 GB |

| 65B | 50 GB |

**Note #2:** If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...`

**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this `README.md` with a PR!

[Install Docker Engine](https://docs.docker.com/engine/install)

Use if you don't have a NVidia GPU. Defaults to `python:3-slim-bullseye` Docker base image and OpenBLAS:

Requires a NVidia GPU with sufficient VRAM (approximately as much as the size above) and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))

import requests

import json

import os

import struct

def make_request(url, params=None):

print(f"Making request to {url}...")

response = requests.get(url, params=params)

if response.status_code == 200:

return json.loads(response.text)

else:

print(f"Request failed with status code {response.status_code}")

return None

def check_magic_and_version(filename):

with open(filename, 'rb') as f:

# Read the first 6 bytes from the file

data = f.read(6)

# Unpack the binary data, interpreting the first 4 bytes as a little-endian unsigned int

# and the next 2 bytes as a little-endian unsigned short

magic, version = struct.unpack('<I H', data)

print(f"magic: 0x{magic:08x}, version: 0x{version:04x}, file: {filename}")

return magic, version

def download_file(url, destination):

print(f"Downloading {url} to {destination}...")

response = requests.get(url, stream=True)

if response.status_code == 200:

with open(destination, 'wb') as f:

total_downloaded = 0

for chunk in response.iter_content(chunk_size=1024):

if chunk: # filter out keep-alive new chunks

f.write(chunk)

total_downloaded += len(chunk)

if total_downloaded >= 10485760: # 10 MB

print('.', end='', flush=True)

total_downloaded = 0

print("\nDownload complete.")

# Creating a symbolic link from destination to "model.bin"

if os.path.isfile("model.bin"):

os.remove("model.bin") # remove the existing link if any

os.symlink(destination, "model.bin")

else:

print(f"Download failed with status code {response.status_code}")

def get_user_choice(model_list):

# Print the enumerated list

print("\n")

for i, (model_id, rfilename) in enumerate(model_list):

print(f"{i+1}: Model ID: {model_id}, RFilename: {rfilename}")

# Get user's choice

choice = input("Choose a model to download by entering the corresponding number: ")

try:

index = int(choice) - 1

if 0 <= index < len(model_list):

# Return the chosen model

return model_list[index]

else:

print("Invalid choice.")

except ValueError:

print("Invalid input. Please enter a number corresponding to a model.")

except IndexError:

print("Invalid choice. Index out of range.")

return None

import argparse

def main():

# Create an argument parser

parser = argparse.ArgumentParser(description='Process the model version.')

parser.add_argument('-v', '--version', type=int, default=0x0003,

help='an integer for the version to be used')

# Parse the arguments

args = parser.parse_args()

# Define the parameters

params = {

"author": "TheBloke", # Filter by author

"tags": "llama"

}

models = make_request('https://huggingface.co/api/models', params=params)

if models is None:

return

model_list = []

# Iterate over the models

for model in models:

model_id = model['id']

model_info = make_request(f'https://huggingface.co/api/models/{model_id}')

if model_info is None:

continue

for sibling in model_info.get('siblings', []):

rfilename = sibling.get('rfilename')

if rfilename and 'q5_1' in rfilename:

model_list.append((model_id, rfilename))

model_choice = get_user_choice(model_list)

if model_choice is not None:

model_id, rfilename = model_choice

url = f"https://huggingface.co/{model_id}/resolve/main/{rfilename}"

download_file(url, rfilename)

_, version = check_magic_and_version(rfilename)

if version != args.version:

print(f"Warning: Expected version {args.version}, but found different version in the file.")

if __name__ == '__main__':

main()

ulimit -l unlimited

if [ "$IMAGE" = "python:3-slim-bullseye" ]; then

python3 -B -m llama_cpp.server --model /app/model.bin

# You may have to reduce --n_gpu_layers=1000 to 20 or less if you don't have enough VRAM

python3 -B -m llama_cpp.server --model /app/model.bin --n_gpu_layers=1000

options:

show_root_heading: true

options:

show_root_heading: true

llama_cpp.llama_sample_top_k(self.ctx, candidates_p, top_k, min_keep=llama_cpp.c_size_t(1))

llama_cpp.llama_sample_tail_free(self.ctx, candidates_p, llama_cpp.c_float(self.params.tfs_z), min_keep=llama_cpp.c_size_t(1))

llama_cpp.llama_sample_typical(self.ctx, candidates_p, llama_cpp.c_float(self.params.typical_p), min_keep=llama_cpp.c_size_t(1))

llama_cpp.llama_sample_top_p(self.ctx, candidates_p, llama_cpp.c_float(self.params.top_p), min_keep=llama_cpp.c_size_t(1))

self.embd.append(id)

for id in self.first_antiprompt[0]:

self.embd.append(id)

else:

# add it to the context

self.embd.append(id)

if pattern & int.from_bytes(cur_char, 'little') == pattern: