Skip to content

Commit 727d6dc

Browse files
bsboddenrbs333tylerhutcherson
authored
Poetry for Dependency Management and TestContainers for Integration Testing (#134)
In this PR: - Uses Poetry for Dependency Management - Replaces the Makefile with Python scripts that can be run from Poetry - Implements TestContainers with a Docker Compose file for integration testing --------- Co-authored-by: Robert Shelton <robert.b.shelton.42@gmail.com> Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
1 parent 5fad799 commit 727d6dc

24 files changed

+5151
-288
lines changed

‎.github/workflows/lint.yml‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
# Starting new jobs is also relatively slow,
2222
# so linting on fewer versions makes CI faster.
2323
python-version:
24-
- "3.8"
24+
- "3.9"
2525
- "3.11"
2626

2727
steps:
@@ -30,19 +30,17 @@ jobs:
3030
uses: actions/setup-python@v2
3131
with:
3232
python-version: ${{ matrix.python-version }}
33+
- name: Install Poetry
34+
uses: snok/install-poetry@v1
3335
- name: Install dependencies
3436
run: |
35-
python -m pip install --upgrade pip
36-
pip install .[dev,all]
37-
37+
poetry install --all-extras
3838
- name: check-sort-import
3939
run: |
40-
make check-sort-imports
41-
40+
poetry run check-sort-imports
4241
- name: check-black-format
4342
run: |
44-
make check-format
45-
43+
poetry run check-format
4644
- name: check-mypy
4745
run: |
48-
make mypy
46+
poetry run mypy

‎.github/workflows/publish.yml‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,15 @@ jobs:
1818
with:
1919
python-version: "3.10"
2020

21+
- name: Install Poetry
22+
uses: snok/install-poetry@v1
23+
2124
- name: Install dependencies
2225
run: |
23-
python -m pip install --upgrade pip
24-
python -m pip install setuptools wheel
26+
poetry install --all-extras
2527
26-
- name: create python package
27-
run: python setup.py sdist bdist_wheel
28+
- name: Build package
29+
run: poetry build
2830

2931
- name: Create Release
3032
uses: ncipollo/release-action@v1
@@ -36,8 +38,7 @@ jobs:
3638
tag: ${{ github.event.inputs.version }}
3739
commit: main
3840

39-
- name: deploy
40-
uses: pypa/gh-action-pypi-publish@release/v1
41-
with:
42-
user: __token__
43-
password: ${{ secrets.PYPI }}
41+
- name: Publish to PyPI
42+
env:
43+
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI }}
44+
run: poetry publish

‎.github/workflows/run_tests.yml‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ jobs:
1515
strategy:
1616
fail-fast: false
1717
matrix:
18-
python-version: [3.8, 3.9, '3.10', 3.11]
18+
python-version: [3.9, '3.10', 3.11]
1919
connection: ['hiredis', 'plain']
20-
redis-stack-version: ['6.2.6-v9', 'latest']
20+
redis-stack-version: ['6.2.6-v9', 'latest', 'edge']
2121

2222
services:
2323
redis:
@@ -33,20 +33,21 @@ jobs:
3333
python-version: ${{ matrix.python-version }}
3434
cache: 'pip'
3535

36+
- name: Install Poetry
37+
uses: snok/install-poetry@v1
38+
3639
- name: Install dependencies
3740
run: |
38-
python -m pip install --upgrade pip
39-
pip install .[dev,all]
41+
poetry install --all-extras
4042
4143
- name: Install hiredis if needed
4244
if: matrix.connection == 'hiredis'
4345
run: |
44-
pip install hiredis
46+
poetry add hiredis
4547
46-
- name: Start Redis
48+
- name: Set Redis version
4749
run: |
48-
REDIS_URL=redis://localhost:6379
49-
echo REDIS_URL=$REDIS_URL >> $GITHUB_ENV
50+
echo "REDIS_VERSION=${{ matrix.redis-stack-version }}" >> $GITHUB_ENV
5051
5152
- name: Authenticate to Google Cloud
5253
uses: google-github-actions/auth@v1
@@ -65,12 +66,12 @@ jobs:
6566
AZURE_OPENAI_DEPLOYMENT_NAME: ${{secrets.AZURE_OPENAI_DEPLOYMENT_NAME}}
6667
OPENAI_API_VERSION: ${{secrets.OPENAI_API_VERSION}}
6768
run: |
68-
make test-cov
69+
poetry run test-cov
6970
7071
- name: Run tests
7172
if: matrix.connection != 'plain' || matrix.redis-stack-version != 'latest'
7273
run: |
73-
SKIP_VECTORIZERS=True make test-cov
74+
SKIP_VECTORIZERS=True poetry run test-cov
7475
7576
- name: Run notebooks
7677
if: matrix.connection == 'plain' && matrix.redis-stack-version == 'latest'
@@ -84,10 +85,11 @@ jobs:
8485
AZURE_OPENAI_DEPLOYMENT_NAME: ${{secrets.AZURE_OPENAI_DEPLOYMENT_NAME}}
8586
OPENAI_API_VERSION: ${{secrets.OPENAI_API_VERSION}}
8687
run: |
87-
cd docs/ && treon -v --exclude="./examples/openai_qna.ipynb"
88+
cd docs/ && poetry run treon -v --exclude="./examples/openai_qna.ipynb"
8889
8990
- name: Publish coverage results
90-
uses: codecov/codecov-action@v2
91+
uses: codecov/codecov-action@v4
9192
with:
9293
token: ${{ secrets.CODECOV_TOKEN }}
93-
fail_ci_if_error: true
94+
files: ./coverage.xml
95+
fail_ci_if_error: true

‎.github/workflows/test-publish.yml‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,20 +11,22 @@ jobs:
1111
build_release:
1212
name: Build Release
1313
runs-on: ubuntu-latest
14+
1415
steps:
1516
- uses: actions/checkout@v3
17+
1618
- name: Set up Python 3.10
1719
uses: actions/setup-python@v4
1820
with:
1921
python-version: "3.10"
2022

21-
- name: Install dependencies
22-
run: |
23-
python -m pip install --upgrade pip
24-
python -m pip install setuptools wheel
23+
- name: Install Poetry
24+
uses: snok/install-poetry@v1
2525

26-
- name: create python package
27-
run: python setup.py sdist bdist_wheel
26+
- name: Build package
27+
run: |
28+
poetry version ${{ github.event.inputs.version }}
29+
poetry build
2830
2931
- name: Create Release
3032
uses: ncipollo/release-action@v1
@@ -36,9 +38,9 @@ jobs:
3638
tag: ${{ github.event.inputs.version }}
3739
commit: main
3840

39-
- name: deploy
41+
- name: Publish to TestPyPI
4042
uses: pypa/gh-action-pypi-publish@release/v1
4143
with:
4244
user: __token__
43-
password: ${{ secrets.PYPI }}
45+
password: ${{ secrets.TEST_PYPI_API_TOKEN }}
4446
repository_url: https://test.pypi.org/legacy/

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@ __pycache__/
22
redisvl.egg-info/
33
.coverage.*
44
.coverage
5+
coverage.xml
56
scratch
67
.DS_Store
78
*.csv
89
wiki_schema.yaml
910
docs/_build/
10-
.venv
11+
.venv
12+
coverage.xml

‎CONTRIBUTING.md‎

Lines changed: 23 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -28,45 +28,49 @@ Here's how to get started with your code contribution:
2828
pull request.
2929

3030
### Dev Environment
31-
There is a provided `requirements.txt` and `requirements-dev.txt` file you can use to install required libraries with `pip` into your virtual environment.
31+
RedisVL uses [Poetry](https://python-poetry.org/) for dependency management.
32+
33+
Follow the instructions to [install Poetry](https://python-poetry.org/docs/#installation).
34+
35+
Then install the required libraries:
3236

33-
Or use the local package editable install method:
3437
```bash
35-
python -m venv .venv
36-
source .venv/bin/activate
37-
pip install -e '.[all,dev]'
38+
poetry install --all-extras
3839
```
3940

40-
Then to deactivate the env:
41-
```
42-
source deactivate
43-
```
4441

4542
### Linting and Tests
4643

4744
Check formatting, linting, and typing:
4845
```bash
49-
make check
46+
poetry run format
47+
poetry run sort-imports
48+
poetry run mypy
5049
```
5150

51+
#### TestContainers
52+
53+
RedisVL uses Testcontainers Python for integration tests. Testcontainers is an open-source framework for provisioning throwaway, on-demand containers for development and testing use cases.
54+
55+
To run Testcontainers-based tests you need a local Docker installation such as:
56+
- [Docker Desktop](https://www.docker.com/products/docker-desktop/)
57+
- [Docker Engine on Linux](https://docs.docker.com/engine/install/)
58+
59+
#### Running the Tests
60+
5261
Tests (with vectorizers):
5362
```bash
54-
make test-cov
63+
poetry run test-cov
5564
```
5665

57-
**NOTE**: Some tests require the `REDIS_URL` environment variable to be set (e.g. `export REDIS_URL=redis://localhost:6379`).
58-
5966
Tests w/out vectorizers:
6067
```bash
61-
SKIP_VECTORIZERS=true make test-cov
68+
SKIP_VECTORIZERS=true poetry run test-cov
6269
```
6370

64-
> Dev requirements are needed here to be able to run tests and linting.
65-
> See other commands in the [Makefile](Makefile)
66-
67-
### Docker Tips
71+
### Getting Redis
6872

69-
Make sure to have [Redis](https://redis.io) accessible with Search & Query features enabled on [Redis Cloud](https://redis.com/try-free) or locally in docker with [Redis Stack](https://redis.io/docs/getting-started/install-stack/docker/):
73+
In order for your applications to use RedisVL, you must have [Redis](https://redis.io) accessible with Search & Query features enabled on [Redis Cloud](https://redis.com/try-free) or locally in docker with [Redis Stack](https://redis.io/docs/getting-started/install-stack/docker/):
7074

7175
```bash
7276
docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

‎Makefile‎

Lines changed: 0 additions & 102 deletions
This file was deleted.

0 commit comments

Comments
 (0)