Data Applications Serverside Impl (#1728)

* v3 proto

* v3 proto

* v3 proto

* update proto

* Add definitions for new Application and Function HTTP objects

+ added route for registering/updating application and parsing
its manifest.

* update executor api v3

* update executor api

* update proto

* Add metadata_offset and metadata_size to DataPayload

* Add positional_args and keyword_args to TaskAllocation

* update proto

* update proto

* update proto for indexify v3

* add missing fields to proto

* remove initial_value from ReduceOp

* update to proto

* function sdk implementation

* fixed tests

* removing example.py

* removing task cache

* fix: update state store to use new driver

* review comments

* review comments

* lint

* Add id to ReduceOp

* add comments to ReduceOp

* Fix the reducer code path

* added a new RPC call to make blocking function calls

* Rename Executor V3 API proto fields to use consistent new SDK terms

Renaming only at the API boundary as this is most problematic to
refactor later. Helps me to not write new code in Executor that
uses old and new terms inconsistently.

* More Server API protos cleanups

Just again renaming a few things and removing not used stuff.

* Main Executor side changes to work with new Server and FE APIs

Executor starts up, didn't run anything besides this.

* Remove is_reducer from ApplicationFunction

There are no statically defined reducer functions anymore.
User decides to run a reduce operation using a function in runtime.

* Fixes in app deployment and run code, Executor dummy bugfixes

* A few more improvements

* Minor Server fixes + function call tree output value workaround

* Add root_function_call_id to ExecutionPlanUpdates

* Tracking the root function call id in the graph updates

* Fixes for complex graph

* Propagate value output up through the call tree to its consumer functions
* Set node id of propagated data payloads correctly so Executor can restore
  the tree even with incorrect id in propagated data payload metadata.
* Work with user supplied DataPayload content types.

* Optimizing how we store the source of output when FunctionRuns return execution plan updates

* Removed un-necessary code

* updated code to use the new scheduler update request

* lint

* fix rust tests

* fixed tests

* Fix function calls without parameters and rename "tasks" to "requests"

* propagate output of function runs to consumers

* fixed reduce op

* Reduce fixes, fix idempotency check, added comments to code paths

* Fix FE initialization failure handling code

We were ignoring FE initilization failures previously and were
failing when running alloc instead.

* updated indexify tests

* Add class instance init timeout feature and fix a bunch of tests

* More test fixes

* Fix the rest of the Indexify tests

* Remove not implemented settings from tensorlake.Retries

* Dynamically override function output serializer if it's outputs are API outputs

Indexify side of the change.

* Fixing app update

* clippy fixes

* update deps

* clippy fixes

* clippy fixes

* clippy fixes

* fmt

* fmt

* fmt

---------

Co-authored-by: Eugene Batalov <eugene@tensorlake.ai>

Author: diptanu

Cargo.lock

@@ -1,8 +1,11 @@
 Unified tags for indexify / PE / FE stack. All events/metrics/spans should use these attribute names when relevant.
 
-- graph - Name of graph.
-- graph_version - Version of the graph
-- invocation_id = ID of the invocation.
+- graph - Name of graph. (Deprecated by application)
+- graph_version - Version of the graph (Deprecated by app_version)
+- invocation_id = ID of the invocation. (Deprecated by request_id)
+- app - Name of application.
+- app_version - Version of the application
+- request_id = ID of the request.
 - fn - Name of the compute function
 - allocation_id - ID of the allocation
 - task_id - ID of the task.

docs/tags.md

@@ -6,13 +6,13 @@ build: build_proto
 	@poetry build
 
 SERVER_API_PY_CLIENT_PROTO_DIR_PATH=indexify/proto
-SERVER_API_PROTO_DIR_PATH=../server/proto
+SERVER_API_PROTO_PATH=../server/proto/executor_api_v3.proto
 
-build_proto: ${SERVER_API_PROTO_DIR_PATH}/executor_api.proto
+build_proto: ${SERVER_API_PROTO_PATH}
 	@poetry install
 	@# .proto file and generated Python files have to be in the same directory.
 	@# See known issue https://github.com/grpc/grpc/issues/29459.
-	@cp ${SERVER_API_PROTO_DIR_PATH}/executor_api.proto src/${SERVER_API_PY_CLIENT_PROTO_DIR_PATH}/executor_api.proto
+	@cp ${SERVER_API_PROTO_PATH} src/${SERVER_API_PY_CLIENT_PROTO_DIR_PATH}/executor_api.proto
 	@cd src && poetry run python -m grpc_tools.protoc \
 		--proto_path=. \
 		--python_out=. \

indexify/Makefile

@@ -29,9 +29,9 @@ boto3 = "^1.40.15"
 structlog = "25.4.0"
 # Adds function-executor binary, utils lib, sdk used in indexify-cli commands.
 # We need to specify the tensorlake version exactly because pip install doesn't respect poetry.lock files.
-tensorlake = "0.2.47"
+# tensorlake = "0.2.48"
 # Uncomment the next line to use local tensorlake package (only for development!)
-# tensorlake = { path = "../tensorlake", develop = true }
+tensorlake = { path = "../tensorlake", develop = true }
 # grpcio is provided by tensorlake
 # grpcio-tools is provided by tensorlake
 

indexify/poetry.lock

@@ -18,8 +18,6 @@
 import structlog
 
 from indexify.executor.blob_store.blob_store import BLOBStore
-from indexify.executor.blob_store.local_fs_blob_store import LocalFSBLOBStore
-from indexify.executor.blob_store.s3_blob_store import S3BLOBStore
 from indexify.executor.executor import Executor
 from indexify.executor.function_executor.server.subprocess_function_executor_server_factory import (
     SubprocessFunctionExecutorServerFactory,
@@ -68,7 +66,7 @@
     default=[],
     multiple=True,
     help="Functions that the executor will run "
-    "specified as <namespace>:<workflow>:<function>:<version>"
+    "specified as <namespace>:<application>:<function>:<version>"
     "version is optional, not specifying it will make the server send any version"
     "of the function. Any number of --function arguments can be passed.",
 )

indexify/pyproject.toml

@@ -2,7 +2,7 @@
 import signal
 from pathlib import Path
 from socket import gethostname
-from typing import Dict, List, Optional
+from typing import Dict, List
 
 import structlog
 
@@ -48,7 +48,7 @@ def __init__(
         function_executor_server_factory: FunctionExecutorServerFactory,
         server_addr: str,
         grpc_server_addr: str,
-        config_path: Optional[str],
+        config_path: str | None,
         monitoring_server_host: str,
         monitoring_server_port: int,
         blob_store: BLOBStore,
@@ -100,8 +100,8 @@ def __init__(
             reported_state_handler=ReportedStateHandler(self._state_reporter),
             desired_state_handler=DesiredStateHandler(self._state_reconciler),
         )
-        self._run_aio_task: Optional[asyncio.Task] = None
-        self._shutdown_aio_task: Optional[asyncio.Task] = None
+        self._run_aio_task: asyncio.Task | None = None
+        self._shutdown_aio_task: asyncio.Task | None = None
 
         executor_info: Dict[str, str] = {
             "id": id,