name: Bug report

about: Create a report to help us improve

title: ''

labels: ''

assignees: ''

Please answer the following questions for yourself before submitting an issue.

- [ ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.

- [ ] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).

- [ ] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).

- [ ] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

Please provide a detailed written description of what you were trying to do, and what you expected `llama-cpp-python` to do.

Please provide a detailed written description of what `llama-cpp-python` did, instead.

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

* Physical (or virtual) hardware you are using, e.g. for Linux:

* Operating System, e.g. for Linux:

* SDK version, e.g. for Linux:

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1

2. step 2

3. step 3

4. etc.

**Note: Many issues seem to be regarding performance issues / differences with `llama.cpp`. In these cases we need to confirm that you're comparing against the version of `llama.cpp` that was built with your python package, and which parameters you're passing to the context.**

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to **avoid using screenshots** if at all possible. Instead, copy/paste the console output and use [Github's markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) to cleanly format your logs for easy readability.

Example environment info:

name: Feature request

about: Suggest an idea for this project

title: ''

labels: ''

assignees: ''

**Is your feature request related to a problem? Please describe.**

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**

A clear and concise description of what you want to happen.

**Describe alternatives you've considered**

A clear and concise description of any alternative solutions or features you've considered.

**Additional context**

Add any other context or screenshots about the feature request here.

version: 2

updates:

- package-ecosystem: "pip" # See documentation for possible values

directory: "/" # Location of package manifests

schedule:

interval: "weekly"

tags: ghcr.io/abetlen/llama-cpp-python:latest

FROM python:3-slim-bullseye

RUN apt update && apt install -y libopenblas-dev ninja-build build-essential

CMD python3 -m llama_cpp.server

The high-level API provides a simple managed interface through the `Llama` class.

Below is a short example demonstrating how to use the high-level API to generate text:

Linux/MacOS

Windows

The low-level API is a direct [`ctypes`](https://docs.python.org/3/library/ctypes.html) binding to the C API provided by `llama.cpp`.

The entire lowe-level API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and directly mirrors the C API in [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h).

Below is a short example demonstrating how to use the low-level API to tokenize a prompt:

>>> import llama_cpp

>>> import ctypes

>>> params = llama_cpp.llama_context_default_params()

>>> ctx = llama_cpp.llama_init_from_file(b"./models/7b/ggml-model.bin", params)

>>> max_tokens = params.n_ctx

>>> tokens = (llama_cppp.llama_token * int(max_tokens))()

>>> n_tokens = llama_cpp.llama_tokenize(ctx, b"Q: Name the planets in the solar system? A: ", tokens, max_tokens, add_bos=llama_cpp.c_bool(True))

>>> llama_cpp.llama_free(ctx)

Check out the [examples folder](examples/low_level_api) for more examples of using the low-level API.

def __getitem__(self, key: Sequence[llama_cpp.llama_token]) -> "LlamaState":

raise KeyError(f"Key not found: {key}")

eval_logits: Deque[List[float]],

llama_state, # type: llama_cpp.Array[llama_cpp.c_uint8]

self.eval_logits: Deque[List[float]] = deque(

logits: List[List[float]] = [

if float(temp.value) == 0.0:

min_keep=llama_cpp.c_size_t(1),

min_keep=llama_cpp.c_size_t(1),

min_keep=llama_cpp.c_size_t(1),

min_keep=llama_cpp.c_size_t(1),

if reset and len(self.eval_tokens) > 0:

longest_prefix = 0

for a, b in zip(self.eval_tokens, tokens[:-1]):

if a == b:

longest_prefix += 1

else:

break

if longest_prefix > 0:

if self.verbose:

print("Llama.generate: prefix-match hit", file=sys.stderr)

reset = False

tokens = tokens[longest_prefix:]

for _ in range(len(self.eval_tokens) - longest_prefix):

self.eval_tokens.pop()

try:

self.eval_logits.pop()

except IndexError:

pass

all_logprobs = [

Llama.logits_to_logprobs(list(map(float, row)))

for row in self.eval_logits

]

print(

f"Llama.save_state: saving {n_bytes} bytes of llama state",

file=sys.stderr,

)

def logits_to_logprobs(logits: List[float]) -> List[float]:

return [math.log(x / sum_exps) for x in exps]