Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 58279a2

Browse filesBrowse files
authored
Merge branch 'abetlen:main' into main
2 parents 147741e + 1a13d76 commit 58279a2
Copy full SHA for 58279a2

File tree

Expand file treeCollapse file tree

5 files changed

+16
-8
lines changed
Filter options
Expand file treeCollapse file tree

5 files changed

+16
-8
lines changed

‎README.md

Copy file name to clipboardExpand all lines: README.md
+7-1Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,12 @@ pip install llama-cpp-python
2626
The above command will attempt to install the package and build build `llama.cpp` from source.
2727
This is the recommended installation method as it ensures that `llama.cpp` is built with the available optimizations for your system.
2828

29+
Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:
30+
```
31+
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
32+
bash Miniforge3-MacOSX-arm64.sh
33+
```
34+
Otherwise, while installing it will build the llama.ccp x86 version which will be 10x slower on Apple Silicon (M1) Mac.
2935

3036
### Installation with OpenBLAS / cuBLAS / CLBlast
3137

@@ -120,7 +126,7 @@ Below is a short example demonstrating how to use the low-level API to tokenize
120126
>>> ctx = llama_cpp.llama_init_from_file(b"./models/7b/ggml-model.bin", params)
121127
>>> max_tokens = params.n_ctx
122128
# use ctypes arrays for array params
123-
>>> tokens = (llama_cppp.llama_token * int(max_tokens))()
129+
>>> tokens = (llama_cpp.llama_token * int(max_tokens))()
124130
>>> n_tokens = llama_cpp.llama_tokenize(ctx, b"Q: Name the planets in the solar system? A: ", tokens, max_tokens, add_bos=llama_cpp.c_bool(True))
125131
>>> llama_cpp.llama_free(ctx)
126132
```

‎llama_cpp/llama_cpp.py

Copy file name to clipboardExpand all lines: llama_cpp/llama_cpp.py
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,15 +44,17 @@ def _load_shared_library(lib_base_name: str):
4444
_base_path = _lib.parent.resolve()
4545
_lib_paths = [_lib.resolve()]
4646

47+
cdll_args = dict() # type: ignore
4748
# Add the library directory to the DLL search path on Windows (if needed)
4849
if sys.platform == "win32" and sys.version_info >= (3, 8):
4950
os.add_dll_directory(str(_base_path))
51+
cdll_args["winmode"] = 0
5052

5153
# Try to load the shared library, handling potential errors
5254
for _lib_path in _lib_paths:
5355
if _lib_path.exists():
5456
try:
55-
return ctypes.CDLL(str(_lib_path))
57+
return ctypes.CDLL(str(_lib_path), **cdll_args)
5658
except Exception as e:
5759
raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
5860

‎poetry.lock

Copy file name to clipboardExpand all lines: poetry.lock
+4-4Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml

Copy file name to clipboardExpand all lines: pyproject.toml
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ black = "^23.3.0"
2222
twine = "^4.0.2"
2323
mkdocs = "^1.4.3"
2424
mkdocstrings = {extras = ["python"], version = "^0.21.2"}
25-
mkdocs-material = "^9.1.11"
25+
mkdocs-material = "^9.1.12"
2626
pytest = "^7.3.1"
2727
httpx = "^0.24.0"
2828

‎vendor/llama.cpp

Copy file name to clipboard

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.