Merge branch 'main' into main-cml-pr-37b8a4fe

Author: yathomasi

packages/example/content/docs/examples.md

@@ -85,3 +85,14 @@ To push the changes upstream, run:
     git push origin cv-class#dev#1 #this is a comment
 #This is a comment
 ```
+
+## Highlight --options-with-hypthen
+
+```cli
+$ cml runner launch \
+  --cloud=aws \
+  --cloud-region=us-west \
+  --cloud-type=m+t4 \
+  --cloud-permission-set=arn:aws:iam::1234567890:instance-profile/dvc-s3-access \
+  --labels=cml-gpu
+```

packages/example/content/docs/gto/command-reference/annotate.md

@@ -0,0 +1,86 @@
+# annotate
+
+Update artifact metadata annotations.
+
+## Synopsis
+
+```usage
+usage: gto annotate [-r <text>] [--type <text>] [--path <text>]
+                    [-e] [--label <text>] [-d <text>] [--commit]
+                    [--push] [-h]
+                    name
+
+arguments:
+  name             Artifact name
+```
+
+## Description
+
+The command adds the artifact annotation to `artifacts.yaml` file. Feel free to
+modify the file directly!
+
+For example,
+
+```cli
+$ gto annotate awesome-model \
+  --type model \
+  --path models/neural_network.h5 \
+  --label ml
+  --label cool
+  --description "This model is very cool"
+```
+
+will create
+
+```yaml
+awesome-model:
+  description: This model is very cool
+  labels:
+    - ml
+    - cool
+  path: models/neural_network.h5
+  type: model
+```
+
+This information can be later retrieved by running `gto describe` command.
+
+> Don't forget to commit `artifacts.yaml` with Git to associate it with the
+> latest artifact version and stage in any copy of the repo.
+
+By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
+GTO the artifact file is committed to Git.
+
+<details>
+
+### Virtual vs. Physical artifacts
+
+- Physical files/directories are committed to the repo. When you create a new
+  version or assign a stage to it, Git guarantees that it's immutable -- you can
+  return a year later and get the same artifact by providing a version.
+
+- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
+  local path to a metafile representing an externally stored artifact file (as
+  [with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
+  can't pin versions to a physical state of the artifact and guarantee its
+  immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
+  know it, nor have a way to recover the original file.
+
+> In future versions, we will support additional enrichments: useful information
+> that other tools like [DVC](https://dvc.org/) and [MLEM](https://mlem.ai/) can
+> provide about the artifacts. This will allow treating DVC repo outputs as
+> usual artifacts instead of `virtual` ones.
+
+</details>
+
+## Options
+
+- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
+- `--type <text>` - Artifact type
+- `--path <text>` - Artifact path
+- `-e`, `--must-exist` - Verify artifact is committed to Git
+- `--label <text>` - Labels to add to artifact
+- `-d <text>`, `--description <text>` - Artifact description
+- `--commit` - Automatically commit changes due to this command (experimental)
+- `--push` - Push created commit automatically (experimental) - will set
+  commit=True
+- `-h`, `--help` - Show this message and exit.

packages/example/content/docs/gto/command-reference/assign.md

@@ -0,0 +1,61 @@
+# assign
+
+Assign stage to specific artifact version.
+
+## Synopsis
+
+```usage
+usage: gto assign [-r <text>] [--version <text>]
+                  [--stage <text>] [-m <text>]
+                  [--simple <text>] [--force] [--push] [--sr]
+                  [-h]
+                  name [ref]
+
+arguments:
+  name             Artifact name
+  [ref]            Git reference to use
+```
+
+## Description
+
+To assign an actionable stage for a specific artifact version use the same
+`gto assign` command. Stages can mark the artifact readiness for a specific
+consumer. You can plug in a real downsteam system via CI/CD or web hooks, e.g.
+to redeploy an ML model.
+
+```cli
+$ gto assign awesome-model --version v0.0.1 --stage prod
+Created git tag 'awesome-model#prod#1' that assigns stage to 'v0.0.1'
+```
+
+GTO creates a special Git tag in
+[the standard format](/doc/gto/user-guide/git-tags).
+
+## Options
+
+- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
+- `--version <text>` - If you provide REF, this will be used to name new version
+- `--stage <text>` - Stage to assign
+- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
+- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
+  [supported values: auto, true, false] [default: auto]
+- `--force` - Create the Git tag even if it already exists and is in effect
+- `--push` - Push created tag automatically (experimental)
+- `--sr`, `--skip-registration` - Don't register a version at specified commit
+- `-h`, `--help` - Show this message and exit.
+
+## Examples
+
+Assign artifact "nn" to "prod" at specific Git ref instead of supplying artifact
+version (note that this will also register a version if it doesn't exist):
+
+```cli
+$ gto assign nn abcd123 --stage prod
+```
+
+Assign stage at specific Git ref and name the version explicitly (this assumes
+that version was not registered yet):
+
+```cli
+$ gto assign nn abcd123 --version v1.0.0 --stage prod
+```

packages/example/content/docs/gto/command-reference/check-ref.md

@@ -0,0 +1,36 @@
+# check-ref
+
+Find out the artifact version registered/assigned with ref.
+
+## Synopsis
+
+```usage
+usage: gto check-ref [-r <text>] [--json] [--name] [--version]
+                     [--event] [--stage] [-h]
+                     ref
+
+arguments:
+  ref              Git reference to analyze
+```
+
+## Description
+
+You can use `gto check-ref` to interpret a Git tag:
+
+```cli
+$ gto check-ref -r build/example-gto churn#prod#3
+✅  Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
+```
+
+For machine-consumable format, use `--json` flag or output specific pieces of
+information with `--name`, `--version`, `--stage` or `--event`.
+
+## Options
+
+- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
+- `--json` - Print output in json format
+- `--name` - Show artifact name
+- `--version` - Output artifact version
+- `--event` - Show event
+- `--stage` - Show artifact stage
+- `-h`, `--help` - Show this message and exit.

packages/example/content/docs/gto/command-reference/deprecate.md

@@ -0,0 +1,141 @@
+# deprecate
+
+Deprecate artifact, deregister a version, or unassign a stage.
+
+## Synopsis
+
+```usage
+usage: gto deprecate [-r <text>] [-m <text>] [--simple <text>]
+                     [--force] [-d] [--push] [-h]
+                     name [version] [stage]
+
+arguments:
+  name             Artifact name
+  [version]        Artifact version
+  [stage]          Stage to unassign
+```
+
+## Description
+
+The command supports three use cases:
+
+```cli
+# deprecate an artifact:
+$ gto deprecate nn
+
+# deprecate a version:
+$ gto deprecate nn v0.0.1
+
+# unassign a stage:
+$ gto deprecate nn v0.0.1 prod
+```
+
+### Unassigning a stage
+
+Sometimes you need to mark an artifact version no longer ready for a specific
+consumer, and maybe signal a downstream system about this. You can use
+`gto deprecate` for that:
+
+```cli
+$ gto deprecate awesome-model v0.0.1 prod
+Created git tag 'awesome-model#prod!#2' that unassigns a stage from 'v0.0.1'
+```
+
+<details>
+
+### Unassigning a stage: some details and options
+
+GTO creates a special Git tag in
+[the standard format](/doc/gto/user-guide/git-tags).
+
+Note, that later you can create this stage again, if you need to, by calling
+`$ gto assign` again.
+
+You also may want to delete the git tag instead of creating a new one. This is
+useful if you don't want to keep extra tags in you Git repo, don't need history
+and don't want to trigger a CI/CD or another downstream system. For that, you
+can use:
+
+```cli
+$ gto deprecate awesome-model v0.0.1 prod --delete
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'
+To push the changes upstream, run:
+git push origin awesome-model#prod#1 --delete
+```
+
+</details>
+
+### Deregister a version
+
+Sometimes you need mark a specific artifact version as a no longer ready for
+usage. You could just delete a git tag, but if you want to preserve a history of
+the actions, you may again use `gto deprecate`.
+
+```cli
+$ gto deprecate awesome-model v0.0.1
+Created git tag 'awesome-model@v0.0.1!' that deregistered a version.
+```
+
+<details>
+
+### Deregister a version: some details and options
+
+If you want to deregister the version by deleting the Git tags itself, you could
+use
+
+```cli
+$ gto deprecate awesome-model v0.0.1 --delete
+Deleted git tag 'awesome-model@v0.0.1' that registered a version.
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
+Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
+To push the changes upstream, run:
+git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
+```
+
+This includes all Git tags related to the version: a tag that registered it and
+all tags that assigned stages to it.
+
+</details>
+
+### Deprecating an artifact
+
+Sometimes you need to need to mark the artifact as "deprecated", usually meaning
+it's outdated and will no longer be developed. To do this, you could run:
+
+```cli
+$ gto deprecate awesome-model
+Created Git tag 'awesome-model@deprecated' that deprecates an artifact.
+```
+
+<details>
+
+### Deprecating an artifact: some details and options
+
+With `awesome-model@deprecated` Git tag the artifact will be considered
+deprecated until you register a new version or assign a new stage to it after
+the deprecation.
+
+If you want to deprecate an artifact by deleting git tags, you'll need to delete
+all of them for the artifact. You could do that with
+
+```cli
+$ gto deprecate awesome-model --delete
+Deleted git tag 'awesome-model@v0.0.1' that registered a version.
+Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
+Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
+To push the changes upstream, run:
+git push origin awesome-model@v0.0.1 awesome-model#prod#1 awesome-model#prod!#2 --delete
+```
+
+</details>
+
+## Options
+
+- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
+- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
+- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
+  [supported values: auto, true, false] [default: auto]
+- `--force` - Create the Git tag even if it already exists and is in effect
+- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
+- `--push` - Push created tag automatically (experimental)
+- `-h`, `--help` - Show this message and exit.

packages/example/content/docs/gto/command-reference/describe.md

@@ -0,0 +1,38 @@
+# describe
+
+Display enrichments for an artifact.
+
+## Synopsis
+
+```usage
+usage: gto describe [-r <text>] [--rev <text>] [--type] [--path]
+                    [--description] [-h]
+                    name
+
+arguments:
+  name             Artifact name
+```
+
+## Description
+
+To get details about an artifact (from `artifacts.yaml`) use `gto describe`:
+
+```cli
+$ gto describe churn -r https://github.com/iterative/example-gto
+{
+    "type": "model",
+    "path": "models/churn.pkl",
+    "virtual": false
+}
+```
+
+The output is in JSON format for ease of parsing programatically.
+
+## Options
+
+- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
+- `--rev <text>` - Repo revision to use
+- `--type` - Show type
+- `--path` - Show path
+- `--description` - Show description
+- `-h`, `--help` - Show this message and exit.