|
1 | | -# Getting Started |
| 1 | +--- |
| 2 | +title: Getting Started |
| 3 | +--- |
2 | 4 |
|
3 | | -## 🦙 Python Bindings for `llama.cpp` |
4 | | - |
5 | | -[](https://abetlen.github.io/llama-cpp-python) |
6 | | -[](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml) |
7 | | -[](https://pypi.org/project/llama-cpp-python/) |
8 | | -[](https://pypi.org/project/llama-cpp-python/) |
9 | | -[](https://pypi.org/project/llama-cpp-python/) |
10 | | -[](https://pypi.org/project/llama-cpp-python/) |
11 | | - |
12 | | -Simple Python bindings for **@ggerganov's** [`llama.cpp`](https://github.com/ggerganov/llama.cpp) library. |
13 | | -This package provides: |
14 | | - |
15 | | -- Low-level access to C API via `ctypes` interface. |
16 | | -- High-level Python API for text completion |
17 | | - - OpenAI-like API |
18 | | - - LangChain compatibility |
19 | | - |
20 | | -## Installation |
21 | | - |
22 | | -Install from PyPI: |
23 | | - |
24 | | -```bash |
25 | | -pip install llama-cpp-python |
26 | | -``` |
27 | | - |
28 | | -## High-level API |
29 | | - |
30 | | -```python |
31 | | ->>> from llama_cpp import Llama |
32 | | ->>> llm = Llama(model_path="./models/7B/ggml-model.bin") |
33 | | ->>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True) |
34 | | ->>> print(output) |
35 | | -{ |
36 | | - "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", |
37 | | - "object": "text_completion", |
38 | | - "created": 1679561337, |
39 | | - "model": "./models/7B/ggml-model.bin", |
40 | | - "choices": [ |
41 | | - { |
42 | | - "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.", |
43 | | - "index": 0, |
44 | | - "logprobs": None, |
45 | | - "finish_reason": "stop" |
46 | | - } |
47 | | - ], |
48 | | - "usage": { |
49 | | - "prompt_tokens": 14, |
50 | | - "completion_tokens": 28, |
51 | | - "total_tokens": 42 |
52 | | - } |
53 | | -} |
54 | | -``` |
55 | | - |
56 | | -## Web Server |
57 | | - |
58 | | -`llama-cpp-python` offers a web server which aims to act as a drop-in replacement for the OpenAI API. |
59 | | -This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc). |
60 | | - |
61 | | -To install the server package and get started: |
62 | | - |
63 | | -```bash |
64 | | -pip install llama-cpp-python[server] |
65 | | -export MODEL=./models/7B/ggml-model.bin |
66 | | -python3 -m llama_cpp.server |
67 | | -``` |
68 | | - |
69 | | -Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation. |
70 | | - |
71 | | -## Low-level API |
72 | | - |
73 | | -The low-level API is a direct `ctypes` binding to the C API provided by `llama.cpp`. |
74 | | -The entire API can be found in [llama_cpp/llama_cpp.py](https://github.com/abetlen/llama-cpp-python/blob/master/llama_cpp/llama_cpp.py) and should mirror [llama.h](https://github.com/ggerganov/llama.cpp/blob/master/llama.h). |
75 | | - |
76 | | - |
77 | | -## Development |
78 | | - |
79 | | -This package is under active development and I welcome any contributions. |
80 | | - |
81 | | -To get started, clone the repository and install the package in development mode: |
82 | | - |
83 | | -```bash |
84 | | -git clone git@github.com:abetlen/llama-cpp-python.git |
85 | | -cd llama-cpp-python |
86 | | -git submodule update --init --recursive |
87 | | -# Will need to be re-run any time vendor/llama.cpp is updated |
88 | | - |
89 | | -pip install --upgrade pip |
90 | | -pip install -e .[all] |
91 | | -``` |
92 | | - |
93 | | -## License |
94 | | - |
95 | | -This project is licensed under the terms of the MIT license. |
| 5 | +-8<- "README.md" |
0 commit comments