Merge pull request #194 from labthings/drop-dependencies-feedback

Drop dependencies

This PR replaces dependencies with several new features: `lt.thing_slot`, a `ThingServerInterface`, and `lt.cancellable_sleep`. These are all described in the documentation in more detail.

Dependencies and DirectThingClient are still in the codebase: these should be deprecated for the next release and removed in the following one.

Author: rwb27

docs/source/actions.rst

@@ -5,7 +5,7 @@ Actions
 
 Actions are the way `.Thing` objects are instructed to do things. In Python
 terms, any method of a `.Thing` that we want to be able to call over HTTP
-should be decorated as an Action, using :deco:`.thing_action`.
+should be decorated as an Action, using `.thing_action`.
 
 This page gives an overview of how actions are implemented in LabThings-FastAPI.
 :ref:`wot_cc` includes a section on :ref:`wot_actions` that introduces the general concept.
@@ -58,6 +58,51 @@ The first is ``self`` (the first positional argument), which is always the
 supply resources needed by the action. Most often, this is a way of accessing
 other `.Things` on the same server.
 
+.. action_logging:
+Logging from actions
+--------------------
+Action code should use `.Thing.logger` to log messages. This will be configured
+to handle messages on a per-invocation basis and make them available when the action
+is queried over HTTP.
+
+This may be used to display status updates to the user when an action takes
+a long time to run, or it may simply be a helpful debugging aid. 
+
+See :mod:`.logs` for details of how this is implemented.
+
+.. action_cancellation:
+Cancelling actions
+------------------
+If an action could run for a long time, it is useful to be able to cancel it
+cleanly. LabThings makes provision for this by allowing actions to be cancelled
+using a ``DELETE`` HTTP request. In order to allow an action to be cancelled,
+you must give LabThings opportunities to interrupt it. This is most often done
+by replacing a `time.sleep()` statement with `.cancellable_sleep()` which
+is equivalent,  but will raise an exception if the action is cancelled.
+
+For more advanced options, see `.invocation_contexts` for detail.
+
+.. invocation_context:
+Invocation contexts
+-------------------
+Cancelling actions and capturing their logs requires action code to use a
+specific logger and check for cancel events. This is done using `contextvars`
+such that the action code can use module-level symbols rather than needing
+to explicitly pass the logger and cancel hook as arguments to the action
+method.
+
+Usually, you don't need to consider this mechanism: simply use `.Thing.logger`
+or `.cancellable_sleep` as explained above. However, if you want to run actions
+outside of the server (for example, for testing purposes) or if you want to
+call one action from another action, but not share the cancellation signal
+or log, functions are provided in `.invocation_contexts` to manage this.
+
+If you start a new thread from an action, code running in that thread will
+not have an invocation ID set in a context variable. A subclass of
+`threading.Thread` is provided to do this, `.ThreadWithInvocationID`\ .
+This may be useful for test code, or if you wish to run actions in the
+background, with the option of cancelling them.
+
 Raising exceptions
 ------------------
 If an action raises an unhandled exception, the action will terminate with an Error

docs/source/concurrency.rst

@@ -11,7 +11,7 @@ In the case of properties, the HTTP response is only returned once the `.Thing`
 
 Many of the functions that handle HTTP requests are asynchronous, running in an :mod:`anyio` event loop. This enables many HTTP connections to be handled at once with good efficiency. The `anyio documentation`_ describes the functions that link between async and threaded code. When the LabThings server is started, we create an :class:`anyio.from_thread.BlockingPortal`, which allows threaded code to run code asynchronously in the event loop.
 
-An action can obtain the blocking portal using the `~labthings_fastapi.dependencies.blocking_portal.BlockingPortal` dependency, i.e. by declaring an argument of that type. This avoids referring to the blocking portal through a global variable, which could lead to confusion if there are multiple event loops, e.g. during testing.
+An action can run async code using its server interface. See `.ThingServerInterface.start_async_task_soon` for details.
 
 There are relatively few occasions when `.Thing` code will need to consider this explicitly: more usually the blocking portal will be obtained by a LabThings function, for example the `.MJPEGStream` class.
 
@@ -22,3 +22,11 @@ Calling Things from other Things
 
 When one `Thing` calls the actions or properties of another `.Thing`, either directly or via a `.DirectThingClient`, no new threads are spawned: the action or property is run in the same thread as the caller. This mirrors the behaviour of the `.ThingClient`, which blocks until the action or property is complete. See :doc:`using_things` for more details on how to call actions and properties of other Things.
 
+Invocations and concurrency
+---------------------------
+
+Each time an action is run ("invoked" in :ref:`wot_cc`), we create a new thread to run it. This thread has a context variable set, such that ``lt.cancellable_sleep`` and ``lt.get_invocation_logger`` are aware of which invocation is currently running. If an action spawns a new thread (e.g. using `threading.Thread`\ ), this new thread will not have an invocation ID, and consequently the two invocation-specific functions mentioned will not work.
+
+Usually, the best solution to this problem is to generate a new invocation ID for the thread. This means only the original action thread will receive cancellation events, and only the original action thread will log to the invocation logger. If the action is cancelled, you must cancel the background thread. This is the behaviour of `.ThreadWithInvocationID`\ .
+
+It is also possible to copy the current invocation ID to a new thread. This is often a bad idea, as it's ill-defined whether the exception will arise in the original thread or the new one if the invocation is cancelled. Logs from the two threads will also be interleaved. If it's desirable to log from the background thread, the invocation logger may safely be passed as an argument, rather than accessed via ``lt.get_invocation_logger``\ .

docs/source/dependencies/dependencies.rst

@@ -3,11 +3,19 @@
 Dependencies
 ============
 
+.. warning::
+
+    The use of dependencies is now deprecated. See :ref:`thing_slots` and `.ThingServerInterface` for a more intuitive way to access that functionality.
+
 LabThings makes use of the powerful "dependency injection" mechanism in FastAPI. You can see the `FastAPI documentation`_ for more information. In brief, FastAPI dependencies are annotated types that instruct FastAPI to supply certain function arguments automatically. This removes the need to set up resources at the start of a function, and ensures everything the function needs is declared and typed clearly. The most common use for dependencies in LabThings is where an action needs to make use of another `.Thing` on the same `.ThingServer`.
 
 Inter-Thing dependencies
 ------------------------
 
+.. warning::
+
+    These dependencies are deprecated - see :ref:`thing_slots` instead.
+
 Simple actions depend only on their input parameters and the `.Thing` on which they are defined. However, it's quite common to need something else, for example accessing another `.Thing` instance on the same LabThings server. There are two important principles to bear in mind here:
 
 * Other `.Thing` instances should be accessed using a `.DirectThingClient` subclass if possible. This creates a wrapper object that should work like a `.ThingClient`, meaning your code should work either on the server or in a client script. This makes the code much easier to debug.

docs/source/dependencies/example.py

@@ -5,7 +5,7 @@
 import labthings_fastapi as lt
 from labthings_fastapi.example_things import MyThing
 
-MyThingClient = lt.deps.direct_thing_client_class(MyThing, "/mything/")
+MyThingClient = lt.deps.direct_thing_client_class(MyThing, "mything")
 MyThingDep = Annotated[MyThingClient, Depends()]
 
 
@@ -18,9 +18,12 @@ def increment_counter(self, my_thing: MyThingDep) -> None:
         my_thing.increment_counter()
 
 
-server = lt.ThingServer()
-server.add_thing(MyThing(), "/mything/")
-server.add_thing(TestThing(), "/testthing/")
+server = lt.ThingServer(
+    {
+        "mything": MyThing,
+        "testthing": TestThing,
+    }
+)
 
 if __name__ == "__main__":
     import uvicorn

docs/source/index.rst

@@ -7,10 +7,11 @@ Documentation for LabThings-FastAPI
 
    quickstart/quickstart.rst
    wot_core_concepts.rst
-   lt_core_concepts.rst
+   structure.rst
    tutorial/index.rst
    examples.rst
    actions.rst
+   thing_slots.rst
    dependencies/dependencies.rst
    blobs.rst
    concurrency.rst
@@ -19,26 +20,21 @@ Documentation for LabThings-FastAPI
 
    autoapi/index
 
-`labthings-fastapi` implements a Web of Things interface for laboratory hardware using Python. This is a ground-up rewrite of python-labthings_, replacing Flask 1 and Marshmallow with FastAPI and Pydantic. It is the underlying framework for v3 of the `OpenFlexure Microscope software <https://gitlab.com/openflexure/openflexure-microscope-server/>`_.
+`labthings-fastapi` is a Python library to simplify the process of making laboratory instruments available via a HTTP. It aims to create an API that is usable from any modern programming language, with API documentation in both :ref:`openapi` and :ref:`gen_td` formats. It is the underlying framework for v3 of the `OpenFlexure Microscope software <https://gitlab.com/openflexure/openflexure-microscope-server/>`_. Key features and design aims are:
 
-`labthings-fastapi` aims to simplify the process of making laboratory instruments available via an HTTP API. Key features and design aims are below:
-
-* Functionality together in `Thing` subclasses, which represent units of hardware or software (see :doc:`wot_core_concepts`)
-* Methods and properties of `Thing` subclasses may be added to the HTTP API and Thing Description using decorators
+* The functionality of a unit of hardware or software is described using `.Thing` subclasses.
+* Methods and properties of `.Thing` subclasses may be added to the HTTP API and associated documentation using decorators.
+* Datatypes of action input/outputs and properties are defined with Python type hints.
+* Actions are decorated methods of a `.Thing` class. There is no need for separate schemas or endpoint definitions.
+* Properties are defined either as typed attributes (similar to `pydantic` or `dataclasses`) or with a `property`\ -like decorator.
+* Lifecycle and concurrency are appropriate for hardware: `Thing` code is always run in a thread, and each `Thing` is instantiated, started up, and shut down only once.
 * Vocabulary and concepts are aligned with the `W3C Web of Things <https://www.w3.org/WoT/>`_ standard (see :doc:`wot_core_concepts`)
-    - Things are classes, with properties and actions defined exactly once
-    - Thing Descriptions are automatically generated, and validated with `pydantic`
-    - OpenAPI documentation is automatically generated by FastAPI
-* We follow FastAPI_'s lead and try to use standard Python features to minimise unnecessary code
-    - Datatypes of action input/outputs and properties are defined with Python type hints
-    - Actions are defined exactly once, as a method of a `Thing` class
-    - Properties and actions are declared using decorators (or descriptors if that's preferred)
-    - FastAPI_ "Dependency injection" is used to manage relationships between Things and dependency on the server
-* Lifecycle and concurrency are appropriate for hardware: `Thing` code is always run in a thread, and each `Thing` is instantiated and shut down only once.
-    - Starlette (used by FastAPI) can handle requests asynchronously - this improves performance and enables websockets and other long-lived connections.
-    - `Thing` code is still, for now, threaded. In the future it may become possible to us other concurrency models in `Thing` code.
-
-Compared to `python-labthings`_, this framework updates dependencies, shrinks the codebase, and simplifies the API  (see :doc:`lt_core_concepts`).
+
+Previous version
+----------------
+
+This is a ground-up rewrite of python-labthings_, replacing Flask 1 and Marshmallow with FastAPI and Pydantic. 
+Compared to `python-labthings`_, this framework updates dependencies, shrinks the codebase, and simplifies the API  (see :doc:`lt_structure`).
 * FastAPI more or less completely eliminates OpenAPI generation code from our codebase
 * Marshmallow schemas and endpoint classes are replaced with Python type hints, eliminating double- or triple-definition of actions and their inputs/outputs.
 * Thing Description generation is very much simplified by the new structure (multiple Things instead of one massive Thing with many extensions)

docs/source/lt_core_concepts.rst

@@ -31,10 +31,7 @@ def slowly_increase_counter(self) -> None:
 if __name__ == "__main__":
     import uvicorn
 
-    server = lt.ThingServer()
-
-    # The line below creates a TestThing instance and adds it to the server
-    server.add_thing(TestThing(), "/counter/")
+    server = lt.ThingServer({"counter": TestThing})
 
     # We run the server using `uvicorn`:
     uvicorn.run(server.app, port=5000)