Standardize Environment Directory Structure #145

[rycerzes]

This PR standardizes the OpenEnv environment structure to support standalone development, multi-mode deployment, and better dependency management using pyproject.toml and uv.

Key Changes

Standardized Environment Structure

• Migrated from requirements.txt to pyproject.toml for dependency management
• Added openenv.yaml manifest file for environment metadata
• Introduced standardized outputs/ directory structure (logs, evals) - gitignored
• Each environment is now independently packageable and deployable

New CLI Commands

• openenv build - Build Docker images with automatic context detection (standalone vs in-repo)
• openenv validate - Validate environment configuration for multi-mode deployment
• openenv serve - Run environments locally without Docker (placeholder implementation)

Enhanced Existing Commands

• openenv init - Now generates pyproject.toml instead of requirements.txt, includes openenv.yaml manifest
• openenv push - Updated to work with new structure, improved HuggingFace deployment flow

Dependency Management Overhaul

• Root pyproject.toml now contains only minimal shared dependencies (fastapi, pydantic, uvicorn, CLI tools)
• Heavy dependencies (torch, numpy, smolagents) moved to individual environment pyproject.toml files
• Added uv.lock files for reproducible builds
• Version bump: 0.1.0 → 0.1.1
• Added dependencies: tomli>=2.3.0, tomli-w>=1.2.0 for TOML manipulation

Updated Templates

• New environment template includes pyproject.toml with proper structure
• Dockerfile templates updated for uv integration and multi-stage builds
• FastAPI app templates modernized with better error handling
• Requirements.txt now auto-generated from pyproject.toml during build

Core Infrastructure

• Added _validation.py module for environment validation logic
• Enhanced _cli_utils.py with additional utility functions
• Core container Dockerfile updated for better layer caching and uv support
• Core dependencies refined in src/core/pyproject.toml

Documentation and Migration Tools

• Added scripts/CONVERT.md - comprehensive guide for converting existing environments
• Added scripts/convert_env.sh - automated conversion script for environments
• Updated main README.md with new dependency management workflow and development patterns
• Added documentation on development workflows (pip, uv, Docker)

Echo Environment Updated

• Migrated echo_env to new standard as reference implementation
• Added pyproject.toml, openenv.yaml, and uv.lock
• Updated client, models, server code for new structure
• Modernized Dockerfile with multi-stage build

[meta-cla]

Hi @rycerzes!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[rycerzes]

@zkwentz @burtenshaw @init27 @jspisak this is the PR for #145
For now, I have created a draft. Please let me know, what changes will I need to make

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[zkwentz]

Amazing @rycerzes I took a cursory look and impressive to see convert in the list of newly added commands! I need to get my hands dirty with this one to provide a proper review for ya, will do so this weekend.

One immediate thing that stands out, can you move your build-docker script to openenv build?

And thank you for jumping on this so fast, you're helping us get out of the woods :)

[rycerzes]

One immediate thing that stands out, can you move your build-docker script to openenv build?

Added 👍

[burtenshaw]

Really nice draft PR @rycerzes. Thanks for the contribution. I haven't gone team yet I'm just exploring at a high level. My first question is why do we need convert?

I understand that we need to do a one time conversion of the legacy envs in this repo and in the wild, but I'm not sure if we need to maintain that code within the cli. We could just keep the functionality within ./scripts and do the conversion. IMO, this would be ok because this is still an experimental project.

[rycerzes]

My first question is why do we need convert?

It's a temporary command, which I will remove before the PR is merged. I had previously planned to migrate the existing envs in a different PR but it would be wise to update the envs in this PR itself

[zkwentz]

Okay played around with things a bit. Love the changes you have made to standard template.

One thing I think we haven't made clear enough, and we should make clear once this PR is merged is this:

The new approach is for this repository to be the specification and CLI repo with some canonical examples of interesting examples (criteria for a good example environment is to demonstrate new specification or openenv core functionality).

Going forward, all other environments, will be assumed to have been started with openenv init and will live in their own repo. openenv push will allow you to push that environment (assuming it is conforming ... openenv validate) to the Hugging Face Env Hub with appropriate tagging and build for discovery on Hugging Face itself.

All CLI commands will be assumed to be running from the root of an openenv initialized environment.

So for your newly added commands can you remove the <env_name> argument? As in:

• openenv build <env_name> -> openenv build
• openenv validate <env_name> -> openenv validate
• openenv start <env_name> -> openenv start

etc.

Additional feedback:

• +1 to removing convert, it's cool to see but a one-time tax to pay with the new CLI-based workflow going forward.

I'll add inline comments now that are in pursuit of this direction.

[zkwentz]

One last thing, though you are killing off openenv convert command, we do need good conversion instructions to land with this standard.

@rycerzes since you're closest to the new standard, can you write up CONVERT.md instructions for environment authors to follow to convert their current src/envs/<env_name> to their own separate environment that is openenv_cli compatible (e.g. it can be openenv pushed).

Things to cover that come to mind:

• Manually adding HF frontmatter to their readme for openenv push
• Changes to dockerfile
• Changes to app.py
• Migrate from requirements.txt -> pyproject.toml (if applicable)
• some quick inline script to move their environment to its own directory with new git repo instantiated

[rycerzes]

@zkwentz Thanks for the detailed review! I understand the new CLI-first approach and will make all the requested changes

Questions:

• For the registry push in openenv push --registry, should this be a separate flag or integrated with the existing push functionality?
• Any specific format preferences for the CONVERT.md structure?

I'll have these changes ready in some time :)

[zkwentz]

For the registry push in openenv push --registry, should this be a separate flag or integrated with the existing push functionality?

I see what you're saying, it's a bit wonky because we do a custom build for Hugging Face... it will be like that for some time as well. It's making sense to me now why you included a registry push on build instead of push itself.

Would it be weird to have three optional arguments:

• --interface (optional, default: false if registry present)
• --no-interface (optional, default: true if registry present)
• --registry (optional, url, default: null)

Resulting in the following options:

• openenv push - pushes to HF Env Hub with web interface (default behavior)
• openenv push --no-interface pushes to HF Env Hub without web interface
• openenv push --registry <registry> pushes to a custom registry without web interface
• openenv push --registry <registry> --interface pushed to a custom registry with web interface
• openenv push --registry <registry> --no-interface pushed to a custom registry without web interface

Any specific format preferences for the CONVERT.md structure?

No preferences :) Thank you!

[burtenshaw]

Really good work so far @rycerzes. Since we're dependent on this PR, I think we should limit it down and return to some features in subsequent PRs.

[zkwentz]

+1 @burtenshaw, we need this PR to land asap. Standardize template, convert, update other PRs. @rycerzes if you can descope this to land it more quickly please do.

[rycerzes]

@zkwentz @burtenshaw Thanks for the feedback! I completely understand the need to land this quickly. I'll descope this PR to focus on the core standardization.

@burtenshaw - Re: port exposure - yes, I'll add port as a CLI argument to the app.py entrypoint.

@burtenshaw - Re: dependencies - the downstream images will inherit the base dependencies and add their own in their respective pyproject.toml files.

What's staying in this PR:

• Standardized environment directory structure
• pyproject.toml as primary dependency specification
• Updated base Dockerfile with uv
• CLI updates: build, validate, push (all without env_name arguments)
• CONVERT.md documentation
• Validation utilities

What's being removed:

• openenv serve command (will revisit in a separate PR)
• openenv convert command (will handle migration manually)

For the push command flags, I'll implement the structure you proposed: --interface, --no-interface, and --registry options.

I'll have these changes pushed shortly.

[zkwentz]

Thanks so much @rycerzes sorry to put pressure on like this, but we've got a backlog and two imminent hackathons to get ahead of :)

[rycerzes]

All good @zkwentz! I’ve already pushed the updates for review so we can move this forward quickly :)

[burtenshaw]

Thanks @rycerzes . This is Pro work. I'm reviewing now.

[zkwentz]

@rycerzes thank you for quickly jumping in and getting this updated.

It looks good to me, with one note: this does not convert the other environments and it may in fact break them given the removal of some downstream dependencies in the simplified base image (as noted by @burtenshaw).

I think it's fine to land this without completing that effort and to let those that authored the environments come in and follow your wonderful conversion guideline (and script) as a follow up exercise.

[rycerzes]

It looks good to me, with one note: this does not convert the other environments and it may in fact break them given the removal of some downstream dependencies in the simplified base image (as noted by @burtenshaw).

Yes, only the echo_env is migrated to the new standard. for the rest, I will create a separate PR which will integrate all the existing envs to be compatible with the new standard.

[zkwentz]

Thank you @rycerzes 🙏

@burtenshaw good to ship it?

[burtenshaw]

LGTM! Amazing work @rycerzes and thanks again.