Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 3feeafe

Browse filesBrowse files
abetlenabetlen
authored
Merge pull request abetlen#499 from abetlen/v0.2-wip
llama-cpp-python v0.2.0
2 parents c7901f1 + bcef9ab commit 3feeafe
Copy full SHA for 3feeafe
Expand file treeCollapse file tree

22 files changed

+242
-2019
lines changed

‎.github/workflows/build-and-release.yaml

Copy file name to clipboardExpand all lines: .github/workflows/build-and-release.yaml
+5-3Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ jobs:
2626

2727
- name: Install dependencies
2828
run: |
29-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools
29+
python -m pip install --upgrade pip
30+
python -m pip install -e .[all]
3031
3132
- name: Build wheels
3233
run: python -m cibuildwheel --output-dir wheelhouse
@@ -46,10 +47,11 @@ jobs:
4647
- uses: actions/setup-python@v3
4748
- name: Install dependencies
4849
run: |
49-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools
50+
python -m pip install --upgrade pip build
51+
python -m pip install -e .[all]
5052
- name: Build source distribution
5153
run: |
52-
python setup.py sdist
54+
python -m build --sdist
5355
- uses: actions/upload-artifact@v3
5456
with:
5557
path: ./dist/*.tar.gz

‎.github/workflows/publish-to-test.yaml

Copy file name to clipboardExpand all lines: .github/workflows/publish-to-test.yaml
+3-2Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,11 @@ jobs:
1919
python-version: "3.8"
2020
- name: Install dependencies
2121
run: |
22-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools
22+
python3 -m pip install --upgrade pip build
23+
python3 -m pip install -e .[all]
2324
- name: Build source distribution
2425
run: |
25-
python setup.py sdist
26+
python3 -m build --sdist
2627
- name: Publish to Test PyPI
2728
uses: pypa/gh-action-pypi-publish@release/v1
2829
with:

‎.github/workflows/publish.yaml

Copy file name to clipboardExpand all lines: .github/workflows/publish.yaml
+3-2Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,11 @@ jobs:
1919
python-version: "3.8"
2020
- name: Install dependencies
2121
run: |
22-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools
22+
python3 -m pip install --upgrade pip build
23+
python3 -m pip install -e .[all]
2324
- name: Build source distribution
2425
run: |
25-
python setup.py sdist
26+
python3 -m build --sdist
2627
- name: Publish distribution to PyPI
2728
# TODO: move to tag based releases
2829
# if: startsWith(github.ref, 'refs/tags')

‎.github/workflows/test-pypi.yaml

Copy file name to clipboardExpand all lines: .github/workflows/test-pypi.yaml
+3-3Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ jobs:
1818
- name: Install dependencies
1919
run: |
2020
python3 -m pip install --upgrade pip
21-
python3 -m pip install --verbose llama-cpp-python[server,test]
21+
python3 -m pip install --verbose llama-cpp-python[all]
2222
- name: Test with pytest
2323
run: |
2424
python3 -c "import llama_cpp"
@@ -38,7 +38,7 @@ jobs:
3838
- name: Install dependencies
3939
run: |
4040
python3 -m pip install --upgrade pip
41-
python3 -m pip install --verbose llama-cpp-python[server,test]
41+
python3 -m pip install --verbose llama-cpp-python[all]
4242
- name: Test with pytest
4343
run: |
4444
python3 -c "import llama_cpp"
@@ -58,7 +58,7 @@ jobs:
5858
- name: Install dependencies
5959
run: |
6060
python3 -m pip install --upgrade pip
61-
python3 -m pip install --verbose llama-cpp-python[server,test]
61+
python3 -m pip install --verbose llama-cpp-python[all]
6262
- name: Test with pytest
6363
run: |
6464
python3 -c "import llama_cpp"

‎.github/workflows/test.yaml

Copy file name to clipboardExpand all lines: .github/workflows/test.yaml
+12-12Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
runs-on: ubuntu-latest
1515
strategy:
1616
matrix:
17-
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
17+
python-version: ["3.8", "3.9", "3.10", "3.11"]
1818

1919
steps:
2020
- uses: actions/checkout@v3
@@ -26,18 +26,18 @@ jobs:
2626
python-version: ${{ matrix.python-version }}
2727
- name: Install dependencies
2828
run: |
29-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi sse-starlette httpx uvicorn pydantic-settings
30-
pip install . -v
29+
python3 -m pip install --upgrade pip
30+
python3 -m pip install .[all] -v
3131
- name: Test with pytest
3232
run: |
33-
pytest
33+
python3 -m pytest
3434
3535
build-windows:
3636

3737
runs-on: windows-latest
3838
strategy:
3939
matrix:
40-
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
40+
python-version: ["3.8", "3.9", "3.10", "3.11"]
4141

4242
steps:
4343
- uses: actions/checkout@v3
@@ -49,18 +49,18 @@ jobs:
4949
python-version: ${{ matrix.python-version }}
5050
- name: Install dependencies
5151
run: |
52-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi sse-starlette httpx uvicorn pydantic-settings
53-
pip install . -v
52+
python3 -m pip install --upgrade pip
53+
python3 -m pip install .[all] -v
5454
- name: Test with pytest
5555
run: |
56-
pytest
56+
python3 -m pytest
5757
5858
build-macos:
5959

6060
runs-on: macos-latest
6161
strategy:
6262
matrix:
63-
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
63+
python-version: ["3.8", "3.9", "3.10", "3.11"]
6464

6565
steps:
6666
- uses: actions/checkout@v3
@@ -72,8 +72,8 @@ jobs:
7272
python-version: ${{ matrix.python-version }}
7373
- name: Install dependencies
7474
run: |
75-
python -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi sse-starlette httpx uvicorn pydantic-settings
76-
pip install . -v
75+
python3 -m pip install --upgrade pip
76+
python3 -m pip install .[all] --verbose
7777
- name: Test with pytest
7878
run: |
79-
pytest
79+
python3 -m pytest

‎CMakeLists.txt

Copy file name to clipboardExpand all lines: CMakeLists.txt
+26-22Lines changed: 26 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,33 +2,37 @@ cmake_minimum_required(VERSION 3.4...3.22)
22

33
project(llama_cpp)
44

5-
option(FORCE_CMAKE "Force CMake build of Python bindings" OFF)
5+
option(LLAMA_BUILD "Build llama.cpp shared library and install alongside python package" ON)
66

7-
set(FORCE_CMAKE $ENV{FORCE_CMAKE})
87

9-
if (UNIX AND NOT FORCE_CMAKE)
10-
add_custom_command(
11-
OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/vendor/llama.cpp/libllama.so
12-
COMMAND make libllama.so
13-
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/vendor/llama.cpp
14-
)
15-
add_custom_target(
16-
run ALL
17-
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/vendor/llama.cpp/libllama.so
18-
)
19-
install(
20-
FILES ${CMAKE_CURRENT_SOURCE_DIR}/vendor/llama.cpp/libllama.so
21-
DESTINATION llama_cpp
22-
)
23-
else()
8+
if (LLAMA_BUILD)
249
set(BUILD_SHARED_LIBS "On")
10+
if (APPLE)
11+
# Need to disable these llama.cpp flags on Apple
12+
# otherwise users may encounter invalid instruction errors
13+
set(LLAMA_AVX "Off" CACHE BOOL "llama: enable AVX" FORCE)
14+
set(LLAMA_AVX2 "Off" CACHE BOOL "llama: enable AVX2" FORCE)
15+
set(LLAMA_FMA "Off" CACHE BOOL "llama: enable FMA" FORCE)
16+
set(LLAMA_F16C "Off" CACHE BOOL "llama: enable F16C" FORCE)
17+
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -march=native -mtune=native")
18+
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native -mtune=native")
19+
endif()
2520
add_subdirectory(vendor/llama.cpp)
2621
install(
2722
TARGETS llama
28-
LIBRARY DESTINATION llama_cpp
29-
RUNTIME DESTINATION llama_cpp
30-
ARCHIVE DESTINATION llama_cpp
31-
FRAMEWORK DESTINATION llama_cpp
32-
RESOURCE DESTINATION llama_cpp
23+
LIBRARY DESTINATION ${SKBUILD_PLATLIB_DIR}/llama_cpp
24+
RUNTIME DESTINATION ${SKBUILD_PLATLIB_DIR}/llama_cpp
25+
ARCHIVE DESTINATION ${SKBUILD_PLATLIB_DIR}/llama_cpp
26+
FRAMEWORK DESTINATION ${SKBUILD_PLATLIB_DIR}/llama_cpp
27+
RESOURCE DESTINATION ${SKBUILD_PLATLIB_DIR}/llama_cpp
28+
)
29+
# Temporary fix for https://github.com/scikit-build/scikit-build-core/issues/374
30+
install(
31+
TARGETS llama
32+
LIBRARY DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp
33+
RUNTIME DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp
34+
ARCHIVE DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp
35+
FRAMEWORK DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp
36+
RESOURCE DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp
3337
)
3438
endif()

‎Makefile

Copy file name to clipboardExpand all lines: Makefile
+11-7Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,26 +5,30 @@ update:
55
update.vendor:
66
cd vendor/llama.cpp && git pull origin master
77

8+
deps:
9+
python3 -m pip install pip
10+
python3 -m pip install -e ".[all]"
11+
812
build:
9-
python3 setup.py develop
13+
python3 -m pip install -e .
1014

1115
build.cuda:
12-
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 python3 setup.py develop
16+
CMAKE_ARGS="-DLLAMA_CUBLAS=on" python3 -m pip install -e .
1317

1418
build.opencl:
15-
CMAKE_ARGS="-DLLAMA_CLBLAST=on" FORCE_CMAKE=1 python3 setup.py develop
19+
CMAKE_ARGS="-DLLAMA_CLBLAST=on" python3 -m pip install -e .
1620

1721
build.openblas:
18-
CMAKE_ARGS="-DLLAMA_OPENBLAS=on" FORCE_CMAKE=1 python3 setup.py develop
22+
CMAKE_ARGS="-DLLAMA_CLBLAST=on" python3 -m pip install -e .
1923

2024
build.blis:
21-
CMAKE_ARGS="-DLLAMA_OPENBLAS=on -DLLAMA_OPENBLAS_VENDOR=blis" FORCE_CMAKE=1 python3 setup.py develop
25+
CMAKE_ARGS="-DLLAMA_OPENBLAS=on -DLLAMA_OPENBLAS_VENDOR=blis" python3 -m pip install -e .
2226

2327
build.metal:
24-
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 python3 setup.py develop
28+
CMAKE_ARGS="-DLLAMA_METAL=on" python3 -m pip install -e .
2529

2630
build.sdist:
27-
python3 setup.py sdist
31+
python3 -m build --sdist
2832

2933
deploy.pypi:
3034
python3 -m twine upload dist/*

‎README.md

Copy file name to clipboardExpand all lines: README.md
+9-8Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# 🦙 Python Bindings for `llama.cpp`
1+
# 🦙 Python Bindings for [`llama.cpp`](https://github.com/ggerganov/llama.cpp)
22

33
[![Documentation Status](https://readthedocs.org/projects/llama-cpp-python/badge/?version=latest)](https://llama-cpp-python.readthedocs.io/en/latest/?badge=latest)
44
[![Tests](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml)
@@ -48,7 +48,6 @@ Otherwise, while installing it will build the llama.ccp x86 version which will b
4848
### Installation with Hardware Acceleration
4949

5050
`llama.cpp` supports multiple BLAS backends for faster processing.
51-
Use the `FORCE_CMAKE=1` environment variable to force the use of `cmake` and install the pip package for the desired BLAS backend.
5251

5352
To install with OpenBLAS, set the `LLAMA_BLAS and LLAMA_BLAS_VENDOR` environment variables before installing:
5453

@@ -208,24 +207,26 @@ If you find any issues with the documentation, please open an issue or submit a
208207

209208
This package is under active development and I welcome any contributions.
210209

211-
To get started, clone the repository and install the package in development mode:
210+
To get started, clone the repository and install the package in editable / development mode:
212211

213212
```bash
214213
git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git
215214
cd llama-cpp-python
216215

216+
# Upgrade pip (required for editable mode)
217+
pip install --upgrade pip
218+
217219
# Install with pip
218220
pip install -e .
219221

220222
# if you want to use the fastapi / openapi server
221223
pip install -e .[server]
222224

223-
# If you're a poetry user, installing will also include a virtual environment
224-
poetry install --all-extras
225-
. .venv/bin/activate
225+
# to install all optional dependencies
226+
pip install -e .[all]
226227

227-
# Will need to be re-run any time vendor/llama.cpp is updated
228-
python3 setup.py develop
228+
# to clear the local build cache
229+
make clean
229230
```
230231

231232
# How does this compare to other Python bindings of `llama.cpp`?

‎docker/cuda_simple/Dockerfile

Copy file name to clipboardExpand all lines: docker/cuda_simple/Dockerfile
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ ENV LLAMA_CUBLAS=1
2121
RUN python3 -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette pydantic-settings
2222

2323
# Install llama-cpp-python (build with cuda)
24-
RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python
24+
RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
2525

2626
# Run the server
2727
CMD python3 -m llama_cpp.server

‎docker/simple/Dockerfile

Copy file name to clipboardExpand all lines: docker/simple/Dockerfile
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ RUN mkdir /app
1919
WORKDIR /app
2020
COPY . /app
2121

22-
RUN python3 -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette pydantic-settings
22+
RUN python3 -m pip install --upgrade pip
2323

24-
RUN make build && make clean
24+
RUN make deps && make build && make clean
2525

2626
# Set environment variable for the host
2727
ENV HOST=0.0.0.0

‎docs/index.md

Copy file name to clipboardExpand all lines: docs/index.md
+4-1Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,12 @@ To get started, clone the repository and install the package in development mode
8282

8383
```bash
8484
git clone git@github.com:abetlen/llama-cpp-python.git
85+
cd llama-cpp-python
8586
git submodule update --init --recursive
8687
# Will need to be re-run any time vendor/llama.cpp is updated
87-
python3 setup.py develop
88+
89+
pip install --upgrade pip
90+
pip install -e .[all]
8891
```
8992

9093
## License

‎docs/install/macos.md

Copy file name to clipboardExpand all lines: docs/install/macos.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ conda activate llama
3030
*(you needed xcode installed in order pip to build/compile the C++ code)*
3131
```
3232
pip uninstall llama-cpp-python -y
33-
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir
33+
CMAKE_ARGS="-DLLAMA_METAL=on" pip install -U llama-cpp-python --no-cache-dir
3434
pip install 'llama-cpp-python[server]'
3535
3636
# you should now have llama-cpp-python v0.1.62 or higher installed

‎llama_cpp/__init__.py

Copy file name to clipboard
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
from .llama_cpp import *
22
from .llama import *
33

4-
from .version import __version__
4+
__version__ = "0.2.0"

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.