Skip to content

Commit e2b6484

Browse files
m1sojamesbhobbscoderabbitai[bot]
authored
chore(docs): Clean up readme and contributing guide (#14)
* chore(docs): Clean up readme and contributing guide * chore(docs): Update readme with CLI quickstart instead of local dev * chore: Incorporate PR review suggestions * chore(docs): improve suggestion to try extensions first Co-authored-by: James Hobbs <15235276+jamesbhobbs@users.noreply.github.com> * chore(docs): remove trailing space Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: James Hobbs <15235276+jamesbhobbs@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
1 parent 48b0a54 commit e2b6484

File tree

3 files changed

+293
-263
lines changed

3 files changed

+293
-263
lines changed

CONTRIBUTING.md

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
# Contributing
2+
3+
All contributions are highly appreciated!
4+
Start by [forking the repository](https://github.com/deepnote/deepnote-toolkit/fork) on GitHub and setting up Deepnote Toolkit for local development.
5+
6+
## Local development setup
7+
8+
#### Option 1: Using mise (Recommended)
9+
10+
[mise](https://mise.jdx.dev/) automatically manages Python, Java, and other tool versions:
11+
12+
1. Install mise: [Getting started](https://mise.jdx.dev/getting-started.html)
13+
2. Run setup:
14+
15+
```bash
16+
mise install # Installs Python 3.12 and Java 11
17+
mise run setup # Installs dependencies and pre-commit hooks
18+
```
19+
20+
#### Option 2: Manual setup
21+
22+
1. Install poetry: [Installation](https://python-poetry.org/docs/#installation)
23+
2. Install Java 11 (required for PySpark tests):
24+
- macOS: `brew install openjdk@11`
25+
- Ubuntu/Debian: `sudo apt-get install openjdk-11-jdk`
26+
- RHEL/Fedora: `sudo dnf install java-11-openjdk-devel`
27+
3. Set up venv for development package:
28+
29+
```bash
30+
# If Python 3.10 is available, point Poetry to it
31+
$ poetry env use 3.10
32+
```
33+
34+
4. Verify the virtual environment location:
35+
36+
```bash
37+
$ poetry env info
38+
```
39+
40+
5. Install dependencies:
41+
42+
```bash
43+
$ poetry install
44+
```
45+
46+
6. Install Poe Poetry add-on:
47+
48+
```bash
49+
$ poetry self add 'poethepoet[poetry_plugin]'
50+
```
51+
52+
7. Install pre-commit hooks:
53+
54+
```bash
55+
$ poetry poe setup-hooks
56+
```
57+
58+
8. Verify installation:
59+
60+
```bash
61+
$ poetry poe lint
62+
$ poetry poe format
63+
```
64+
65+
### Setup troubleshooting
66+
67+
1. If `poetry install` fails with error `library 'ssl' not found`:
68+
69+
```bash
70+
env LDFLAGS="-I/opt/homebrew/opt/openssl/include -L/opt/homebrew/opt/openssl/lib" poetry install
71+
```
72+
73+
2. If `poetry install` fails installing `pymssql`, install `freetds` via Homebrew.
74+
75+
## Testing
76+
77+
Tests run against all supported Python versions using nox in Docker for reproducible environments.
78+
79+
### Local Testing
80+
81+
#### Using mise (Recommended)
82+
83+
```bash
84+
# Run unit tests (no coverage by default)
85+
mise run test
86+
87+
# Run unit tests with coverage
88+
mise run test:coverage
89+
90+
# Run tests quickly without nox/coverage overhead
91+
mise run test:quick tests/unit/test_file.py
92+
mise run test:quick tests/unit/test_file.py::TestClass::test_method -v
93+
94+
# Pass custom arguments (including --coverage)
95+
mise run test -- --coverage tests/unit/test_file.py
96+
```
97+
98+
#### Using nox directly
99+
100+
```bash
101+
# Run unit tests without coverage
102+
poetry run nox -s unit
103+
104+
# Run unit tests with coverage
105+
poetry run nox -s unit -- --coverage
106+
107+
# Run specific test file
108+
poetry run nox -s unit -- tests/unit/test_file.py
109+
```
110+
111+
#### Using Docker
112+
```bash
113+
# Run unit tests
114+
TEST_TYPE="unit" TOOLKIT_VERSION="local-build" ./bin/test
115+
116+
# Run integration tests
117+
TEST_TYPE="integration" TOOLKIT_VERSION="local-build" TOOLKIT_INDEX_URL="http://localhost:8000" ./bin/test
118+
119+
# Or use the test-local script for both unit tests and integration tests
120+
./bin/test-local
121+
122+
# Run a specific file with test-local
123+
./bin/test-local tests/unit/test_file.py
124+
125+
# ... or specific test
126+
./bin/test-local tests/unit/test_file.py::TestClass::test_method
127+
```
128+
129+
### Adding Dependencies
130+
131+
- Kernel dependencies: Add to `[tool.poetry.dependencies]` in pyproject.toml
132+
133+
```bash
134+
# Add a package to kernel bundle (available in notebooks)
135+
$ poetry add pandas
136+
137+
# Add a package with specific version
138+
$ poetry add "pandas>=2.0.0"
139+
```
140+
141+
```bash
142+
# Add a development dependency
143+
$ poetry add --group dev pytest
144+
```
145+
146+
After adding dependencies, run tests to verify compatibility:
147+
148+
```bash
149+
$ ./bin/test-local
150+
```
151+
152+
## Development Workflow for Deepnote maintainers
153+
154+
### Using in Deepnote Projects
155+
156+
When you push a commit, a new version of `deepnote/jupyter-for-local` is built with your commit hash (shortened!). Use it in projects by updating `common.yml`:
157+
158+
```yaml
159+
jupyter:
160+
image: "deepnote/jupyter-for-local:SHORTENED_COMMIT_SHA"
161+
```
162+
163+
Alternatively, to develop against a local copy of Deepnote Toolkit, first run this command to build the image:
164+
165+
```bash
166+
docker build \
167+
--build-arg "FROM_PYTHON_TAG=3.11" \
168+
-t deepnote/deepnote-toolkit-local-hotreload \
169+
-f ./dockerfiles/jupyter-for-local-hotreload/Dockerfile .
170+
```
171+
172+
Then start the container:
173+
174+
```bash
175+
# To include server logs in the output add this argument
176+
# -e WITH_SERVER_LOGS=1 \
177+
178+
# Some toolkit features (e.g. feature flags support) require
179+
# DEEPNOTE_PROJECT_ID to be set to work correctly. Add this
180+
# argument with your project id
181+
# -e DEEPNOTE_PROJECT_ID=981af2c1-fe8b-41b7-94bf-006b74cf0641 \
182+
183+
docker run \
184+
-v "$(pwd)":/deepnote-toolkit \
185+
-v /tmp/deepnote-mounts:/deepnote-mounts:shared \
186+
-p 8888:8888 \
187+
-p 2087:2087 \
188+
-p 8051:8051 \
189+
-w /deepnote-toolkit \
190+
--add-host=localstack.dev.deepnote.org:host-gateway \
191+
--rm \
192+
--name deepnote-toolkit-local-hotreload-container \
193+
deepnote/deepnote-toolkit-local-hotreload
194+
```
195+
196+
This will start a container with Deepnote Toolkit mounted inside and expose all required ports. If you change code that runs in the kernel (e.g. you updated the DataFrame formatter), you only need to restart the kernel from Deepnote's UI. If you update code that starts Jupyter itself, you need to restart the container. And if you add or modify dependencies you need to rebuild the image.
197+
198+
Now, you need to modify `common.yml` in the Deepnote app. First, replace `jupyter` service with noop image:
199+
200+
```yml
201+
jupyter:
202+
image: 'screwdrivercd/noop-container'
203+
```
204+
205+
And change `JUPYTER_HOST` variable of executor to point to host machine:
206+
207+
```yml
208+
executor:
209+
environment:
210+
JUPYTER_HOST: host.docker.internal
211+
```
212+
213+
### Review Applications
214+
215+
Each PR creates a review application for testing. Access it via GitHub checks. Monitor logs in Grafana:
216+
217+
```
218+
{pod="p-PROJECT_ID", container="notebook"}
219+
```
220+
221+
222+
### Docker Usage
223+
224+
We use Docker to ensure reproducible environments due to Jupyter libraries' binary dependencies:
225+
226+
- `builder.Dockerfile`: Creates Python package bundles for different versions (3.9-3.12), generates kernel and server bundles, and packages the toolkit for distribution using Poetry.
227+
228+
- `test.Dockerfile`: Provides consistent test environment for running unit and integration tests across Python versions using nox. Used both locally and in CI/CD pipeline.
229+
230+
- `jupyter-for-local.Dockerfile`: Creates development environment with Jupyter integration, used for local development from docker-compose used in main monorepo.
231+
232+
### Production Releases
233+
234+
To release a new version to production:
235+
236+
1. Merge your changes to main. This will automatically trigger a GitHub Actions workflow that runs the test suite and a staging deployment.
237+
2. Trigger a new [GitHub Release](https://github.com/deepnote/deepnote-toolkit/releases) in the GitHub UI.
238+
3. Monitor [the GitHub Actions workflows](https://github.com/deepnote/deepnote-toolkit/actions) and ensure a successful production deployment.
239+
240+
Note: The production release pipeline automatically creates two PRs in the ops and app-config repositories:
241+
242+
- A staging PR that updates staging values and is auto-merged
243+
- A production PR that updates production values and requires manual approval and merge
244+
245+
Important: Always test the changes in the staging environment before approving and merging the production PR to ensure everything works as expected.

0 commit comments

Comments
 (0)