[](https://abetlen.github.io/llama-cpp-python)

[](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml)

[](https://pypi.org/project/llama-cpp-python/)

[](https://pypi.org/project/llama-cpp-python/)

[](https://pypi.org/project/llama-cpp-python/)

[](https://pypi.org/project/llama-cpp-python/)

Simple Python bindings for **@ggerganov's** [`llama.cpp`](https://github.com/ggerganov/llama.cpp) library.

This package provides:

- Low-level access to C API via `ctypes` interface.

- High-level Python API for text completion

- OpenAI-like API

- LangChain compatibility

Install from PyPI:

pip install llama-cpp-python

>>> from llama_cpp import Llama

>>> llm = Llama(model_path="./models/7B/ggml-model.bin")

>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)

>>> print(output)

{

"id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",

"object": "text_completion",

"created": 1679561337,

"model": "./models/7B/ggml-model.bin",

"choices": [

{

"text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",

"index": 0,

"logprobs": None,

"finish_reason": "stop"

}

],

"usage": {

"prompt_tokens": 14,

"completion_tokens": 28,

"total_tokens": 42

}

}

`llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API.

This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).

To install the server package and get started:

pip install llama-cpp-python[server]

export MODEL=./models/7B/ggml-model.bin

python3 -m llama_cpp.server

Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.

The low-level API is a direct `ctypes` binding to the C API provided by `llama.cpp`.

The entire API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and should mirror [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h).

This package is under active development and I welcome any contributions.

To get started, clone the repository and install the package in development mode:

git clone git@github.com:abetlen/llama-cpp-python.git

cd llama-cpp-python

git submodule update --init --recursive

pip install --upgrade pip

pip install -e .[all]

This project is licensed under the terms of the MIT license.