Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 0e0c9bb

Browse filesBrowse files
committed
Merge branch 'auto-docker' of github.com:gjmulder/llama-cpp-python-gary into auto-docker
2 parents 0d2cc21 + ec44bda commit 0e0c9bb
Copy full SHA for 0e0c9bb

File tree

Expand file treeCollapse file tree

1 file changed

+11
-5
lines changed
Filter options
Expand file treeCollapse file tree

1 file changed

+11
-5
lines changed

‎docker/README.md

Copy file name to clipboardExpand all lines: docker/README.md
+11-5Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
# Dockerfiles for building the llama-cpp-python server
2+
- `Dockerfile.openblas_simple` - a simple Dockerfile for non-GPU OpenBLAS
3+
- `Dockerfile.cuda_simple` - a simple Dockerfile for CUDA accelerated CuBLAS
4+
- `hug_model.py` - a Python utility for interactively choosing and downloading the latest `5_1` quantized models from [huggingface.co/TheBloke]( https://huggingface.co/TheBloke)
5+
- `Dockerfile` - a single OpenBLAS and CuBLAS combined Dockerfile that automatically installs a previously downloaded model `model.bin`
6+
17
# Get model from Hugging Face
28
`python3 ./hug_model.py`
39

@@ -7,7 +13,7 @@ docker $ ls -lh *.bin
713
-rw-rw-r-- 1 user user 4.8G May 23 18:30 <downloaded-model-file>.q5_1.bin
814
lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> <downloaded-model-file>.q5_1.bin
915
```
10-
**Note #1:** Make sure you have enough disk space to d/l the model. As the model is then copied into the image you will need at least
16+
**Note #1:** Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least
1117
**TWICE** as much disk space as the size of the model:
1218

1319
| Model | Quantized size |
@@ -21,20 +27,20 @@ lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> <downloaded-model-file>.q5
2127

2228
# Install Docker Server
2329

24-
**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this README with a PR!
30+
**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this `README.md` with a PR!
2531

2632
[Install Docker Engine](https://docs.docker.com/engine/install)
2733

2834
# Use OpenBLAS
29-
No NVidia GPU, defaults to `python:3-slim-bullseye` Docker base image and OpenBlAS:
35+
Use if you don't have a NVidia GPU. Defaults to `python:3-slim-bullseye` Docker base image and OpenBLAS:
3036
## Build:
3137
`docker build --build-arg -t openblas .`
3238
## Run:
3339
`docker run --cap-add SYS_RESOURCE -t openblas`
3440

3541
# Use CuBLAS
36-
Requires NVidia GPU and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
42+
Requires a NVidia GPU with sufficient VRAM (approximately as much as the size above) and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
3743
## Build:
38-
`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t opencuda .`
44+
`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t cublas .`
3945
## Run:
4046
`docker run --cap-add SYS_RESOURCE -t cublas`

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.