Major overhaul, discriminator pattern for models, client updates (#132)

* Moving clients into Origami

* wip

* WIP, pushing branch up in order to work on it across machines

* move models into api folder

* fix imports

* add path into File url

* make file path a pathlib.Path

* move kernels models out of api, fix imports

* move kernels file

* fix import

* warning on downloading presigned url error

* revert logging

* temporary hack to support Skaffold dev

* add mimetype to File model

* update file model

* more minio hacks

* more info on warnings

* don't use event in logging extrasgit status

* fix file susbcribe reply parsing

* fix imports and tests

* update ci

* remove deleted files

* bump to alpha rc

* ci

* flake8

* make linter happy

* yaml lint

* default kernel state

* fix for file subscribe parsing

* logging

* add cells to bottom of notebook by default

* new queue execution method

* slight tweak in syntax

* run all test

* add RTUClient timeout on file subscribes

* move logging around

* debugging RTUClient not initializing right

* more debug

* revert debug logs

* updated README

* docs

* rename queue_execution

* fix tests

* update docs from Kyles feedback

Author: Matt Kafonek

.bumpversion.cfg

@@ -17,37 +17,40 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v3
-      - name: Setup nox
-        uses: daisylb/setup-nox@v2.1.0
       - name: Setup poetry
         uses: Gr1N/setup-poetry@v8
         with:
           poetry-version: 1.4.2
       - name: Install dependencies
-        run: |
-          poetry install --all-extras --with dev
-          pip install nox-poetry
-      - name: Run tests
-        run: nox -s test-${{ matrix.python_version }}
-      - name: Generate coverage XML
-        run: nox -s generate_coverage_xml
-      - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v3
+        run: poetry install --all-extras --with dev
+
+      # Tests in Origami 1.0 alpha rc are end-to-end, need a live Noteable app.
+      # Commenting tests out until we have time to configure Github repo with service account
+      # access token and other settings so it can interact with Noteable from CI tests
+
+      # - name: Run tests with pytest
+      #   run: pytest -v --cov=origami
+
+      # - name: Upload coverage to Codecov
+      #   uses: codecov/codecov-action@v3
 
   lint:
     runs-on: ubuntu-20.04
     steps:
       - name: Checkout code
         uses: actions/checkout@v3
-      - name: Setup nox
-        uses: daisylb/setup-nox@v2.1.0
       - name: Setup poetry
         uses: Gr1N/setup-poetry@v8
         with:
           poetry-version: 1.4.2
       - name: Install dependencies
-        run: |
-          poetry install --all-extras --with dev
-          pip install nox-poetry
-      - name: Run lint
-        run: nox -s lint
+        run: pip install flake8 black isort
+
+      - name: Lint with flake8
+        run: flake8 origami --count --show-source --statistics --benchmark
+
+      - name: Check with black
+        run: black --check origami
+
+      - name: Check with isort
+        run: isort --diff --check origami

.github/workflows/ci.yaml

@@ -29,14 +29,20 @@ The project strucute follows this pattern:
 
 ```
    pyproject.toml # The repository toml for setup instructions
-   mkdocs.yml     # The configuration file
+   mkdocs.yml     # Docs configuration file
    docs/
       index.md    # The documentation homepage.
       ...         # Other markdown pages, images and other files
+   tests/
+      ...         # End-to-end tests against a Noteable cluster
    origami/
-      defs/       # Definitions for various pydantic types used
-      tests/      # Unittests for the library
-      client.py   # The primary client module
-      format.py   # Notebook formatting helpers
-      pathing.py  # Pathing utilities
-```
+      clients/
+         api.py    # HTTP API Client for CRUD resources
+         rtu.py    # RTU Client for live Notebook document model updates and cell execution
+         cache.py  # RTU Client cache for interacting with multiple Notebooks
+      models/
+         rtu/      # Real-time-update websocket payload models
+         deltas/   # Document model updates within RTU Delta payloads
+      notebook/
+         ...       # In-memory Notebook builder that squashes RTU/Deltas
+```

CONTRIBUTING.md

@@ -43,16 +43,18 @@ Python 3.8+
 
 ## Installation
 
-### Poetry
+For stable release:
+```bash
+pip install noteable-origami
+```
 
-```shell
+```bash
 poetry add noteable-origami
 ```
 
-### Pip
-
-```shell
-pip install noteable-origami
+For alpha pre-release:
+```bash
+pip install noteable-origami --pre
 ```
 
 <!-- --8<-- [end:install] -->
@@ -61,56 +63,73 @@ pip install noteable-origami
 
 ## Getting Started
 
-Get your API token from Noteable within user settings.
-Within user settings, go to the API Token page, and generate a new token.
+> **Warning**
+Developer note: this documentation is written for the 1.0 alpha release. For stable release, see [pre-1.0 README](https://github.com/noteable-io/origami/blob/release/0.0.35/README.md)
 
-```python
-from origami.client import NoteableClient
 
-token = ''  # Your API token from Noteable
+### API Tokens
 
-# Establish a connection to the realtime API
-async with NoteableClient(api_token=token) as client:
-    await client.ping_rtu()
-```
+The Noteable API requires an authentication token. You can manage tokens at the Noteable user settings page.
 
-### Token via Environment Variable
+1. Log in to Noteable (sign up is free)
+2. In the User Settings tab, navigate to `API Tokens` and generate a new token
 
-Alternatively you can set the environment variable:
+### Usage
 
-```bash
-NOTEABLE_TOKEN=xxxx
-```
-
-and skip assigning the token:
+The example below shows how to create a Notebook, launch a Kernel, add new cells, and execute code. 
 
 ```python
-async with NoteableClient() as client:
-    await client.ping_rtu()
-```
+# Grab a project_id from the Noteable UI, the url will look like: app.noteable.io/p/....
+api_token = '...'
 
-### Custom Domain
+# Client for interacting with Noteables REST API
+from origami.clients.api import APIClient
+api_client = APIClient(api_token)
 
-```bash
-NOTEABLE_TOKEN=xxxx
-NOTEABLE_DOMAIN=app.noteable.io
-```
+# Sanity check your user information
+user = await api_client.user_info()
 
-And the client will use that particular domain, for custom deployment location. This value defaults to `app.noteable.io`.
+# Choose a project to create the notebook in, here using the ChatGPT plugin default project
+project_id = user.origamist_default_project_id
 
-```python
-async with NoteableClient() as client:
-    await client.ping_rtu()
+# Create a new Notebook
+file = await api_client.create_notebook(project_id=project_id, path="Demo.ipynb")
+
+# Start a Kernel
+await api_client.launch_kernel(file.id)
+
+# Client for Real-time Updates (RTU), used with Notebooks
+rtu_client = await api_client.rtu_client(file.id)
+
+# Add a new cell
+from origami.models.notebook import CodeCell
+cell = CodeCell(source="print('Hello World')")
+await rtu_client.add_cell(cell)
+
+# Execute the cell
+queued_execution = await rtu_client.queue_execution(cell.id)
+
+# Wait for the execution to be complete, cell is an updated instance of CodeCell with metadata/outputs
+cell = await queued_execution
+
+# Grab the output
+output_collection = await api_client.get_output_collection(cell.output_collection_id)
+print(output_collection.outputs[0].content.raw) # 'Hello World\n'
 ```
 
+
 <!-- --8<-- [end:start] -->
 
+
+## 1.0 Roadmap
+
+Origami is heading towards a 1.0 release. The alpha release candidate is on Pypi now, installable with a `--pre` flag. The 1.0 release represents a major refactor of the Origami using the best practices and lessons learned from creating multiple production API and RTU clients, including our ChatGPT plugin. It will likely come out of alpha once all of our internal applications are using the Origami 1.0 syntax.
+
 ## Contributing
 
 See [CONTRIBUTING.md](./CONTRIBUTING.md).
 
 ---
-
 <p align="center">Open sourced with ❤️ by <a href="https://noteable.io">Noteable</a> for the community.</p>
 
-<img href="https://pages.noteable.io/private-beta-access" src="https://assets.noteable.io/github/2022-07-29/noteable.png" alt="Boost Data Collaboration with Notebooks">
+<img href="https://pages.noteable.io/private-beta-access" src="https://assets.noteable.io/github/2022-07-29/noteable.png" alt="Boost Data Collaboration with Notebooks">

README.md

@@ -0,0 +1,3 @@
+from importlib_metadata import version
+
+__version__ = version("noteable-origami")