Develop merge submodule (#77)

* Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Arthur Juliani <awjuliani@gmail.com>

* Update changelog

* [MLA-1762] reduce memory allocations from DiscreteActionOutputApplier (#4922)

* add option for Burst inference (#4925)

* surface specific GRPC errors more visibly (#4930)

* Add additional logic to avoid load being called on every advance (#4934)

* [MLA-1767] Refactor communicator connection exceptions (#4935)

* Update changelog for release 13. (#4938)

* Update master versions for release 13. (#4945)

* Release 13 versions. (#4946)

- updated release tag validation script to automate the updating of files with release tags that need to be changed as part of the pre-commit operation.

* Update docs to pass doc validation. (#4953)

* update defines, compile out Initialize body on non-desktop (#4957)

* Adding references to the Extensions package to help promote it. (#4967)

* Adding references to the Extensions package to help promote it.

* Updating header formatting to match the rest of the page

* Adding references to the Extensions package to help promote it. (#4967) (#4968)

Co-authored-by: Marwan Mattar <marwan@unity3d.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* InputActuatorComponent to allow the generation of an action space from an InputActionAsset (#4881)

* InputActuatorComponent to allow the generation of an action space from an InputActionAsset (#4881) (#4974)

* pass sensor dimension flags to analytics (#4954)

* Communicator factory (#4965)

* Update barracuda, swtich Agents in Sorter use Burst. (#4979)

* Update barracuda, swtich Agents in Sorter use Burst. (#4979) (#4981)

* Set ignore done=False in GAIL (#4971)

* MultiAgentGroup Interface (#4923)

* add SimpleMultiAgentGroup

* add group reward field to agent and proto

* Make TrainingAnalyticsSideChannel internal (#4999)

* [MLA-1783] built-in actuator type (#4950)

* Add component menues for some sensors and actuators. (#5001)

* Add component menues for some sensors and actuators. (#5001) (#5002)

* Merge master -> release_13_branch-to-master

* Fix RpcCommunicator merge.

* master -> main. (#5010)

* Adding a name field to BufferSensorComponent

* Adding a note to the CHANGELOG about var len obs

* Adding a helper method for creating observation placeholder names and removed the _h and _c placeholders

* Adding a custom editor for BufferSensorComponent

* adding inheritdoc

* Update cattrs dependencies to support python3.9 (#4821)

* Fix issue with queuing input events that stomp on others. (#5034)

* Update cattrs dependencies to support python3.9 (#4821)

* Fix issue with queuing input events that stomp on others. (#5034)

* Update versions for release 14 hotfix. (#5040)

* master -> main. (#5010) (#5044)

* Update changelog. (#5045)

* [MLA-1809] catch mismatched observation sizes (#5030)

* Update changelog. (#5055)

* Fix xml docs. (#5057)

* pass sensor name through to ObservationSpec (#5036)

* Remove unused allocation (#5068)

* Automatically generate samples based on placement of mlagents-sample.json files in our examples. (#5077)

* Update barracuda to 1.3.2-preview. (#5084)

* POCA trainer (#5005)

Co-authored-by: Ervin Teng <ervin@unity3d.com>
Co-authored-by: Ruo-Ping Dong <ruoping.dong@unity3d.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>
Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com>

* [docs] Documentation for POCA and cooperative behaviors (#5056)

Co-authored-by: Ruo-Ping Dong <ruoping.dong@unity3d.com>

* [docs] Update changelog (#5095)

* Add multiAgentGroup capabilities flag (#5096)

* Add multiAgentGroup capabilities flag

* Add proto

* Fix compiler error

* Add warning for multiagent group

* Add comment

* Fix spelling mistake

* Fix ghost curriculum and make steps private (#5098)

* use get step to determine curriculum

* add to CHANGELOG

* Make step in trainer private (#5099)

Co-authored-by: Ervin T <ervin@unity3d.com>

* [cherry-pick] Fix ghost curriculum and make steps private (#5098)

* use get step to determine curriculum

* add to CHANGELOG

* Make step in trainer private (#5099)

Co-authored-by: Ervin T <ervin@unity3d.com>

* Update changelog for samples. (#5103)

* Update changelog for samples. (#5103) (#5106)

* Update versions on main (#5102)

Increment versions after release 15 branch split

* [release_15] Release 15 update versions (#5101)

* Update versions

* Fix for validate release links

* Update release tag and docs

* [docs] Add missing package docs (#5108)

* Add missing package docs

* Fix grammar

* Update comment on Dispose

* Fix typo in doc

* [docs] Update readme and changelog (#5107)

* Make analytics module an optional dependency. (#5109)

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* Redo dotnet format (#5119)

* [docs] Add links to example envs for Buffer Sensor and MultiAgentGroup (#5116)

* Update changelog with links to example envs

* Cooperative push block link update

* [cherry-pick] Integrate Group Manager to soccer/retrain with POCA (#5115) (#5121)

* Integrate Group Manager to soccer/retrain with POCA (#5115)

* Add Soccer env to changelog

Co-authored-by: andrewcoh <54679309+andrewcoh@users.noreply.github.com>

* [cherry-pick][docs] Add Dungeon Escape Environment (#5133)

* Add DungeonEscape POCA Environment (#5128)

* Add DungeonEscape assets from working branch

* Add Dungeon Escape docs

* Create dungeon_escape.png

* Add to docs

Co-authored-by: Hunter-Unity <hunter@unity3d.com>

* update pre-commit instructions for CONTRIBUTING.md (#5130)

* Remove duplicated changelog

* Fix another duplicated line in the changelog...

* Removing Obsolete methods from the package (#5024)

* Removing Obsolete methods from the package

* Missing depecration and modified changelog

* Readding the obsolete BrainParameter methods, will need a larger discussion on these

* Removing Action Masker, readding the warining when using a non-implemented Heuristic, Removing NumAction from Brain Parameters

* removing documentation and some calls to deprecated methods in the extensions package

* Editing the Changelog to put the unreleased on top

* non-IEnumerable interface for action masking (#5060)

* V2 staging new model version (#5080)

* Make modelCheck have flavors of error messages

* ONNX exporter v3

* Using a better CheckType and a switch statement

* Removing unused message

* More tests

* Use an enum for valid versions and use GetVersion on model directly

* Maybe the model export version a static constant in Python

* Use static constructor for FailedCheck

* Use static constructor for FailedCheck

* Modifying the docstrings

* renaming LegacyDiscreteActionOutputApplier

* removing testing code

* better warning message

* Nest the CheckTypeEnum into the FailedCheck class

* Update com.unity.ml-agents/Runtime/Inference/BarracudaModelParamLoader.cs

* Adding a line explaining that legacy tensor checks are for versions 1.X only

* Modifying the changelog

* Exporting all the branches size instead of omly the sum (#5092)

* addressing comments

* Update com.unity.ml-agents/Runtime/Inference/BarracudaModelParamLoader.cs

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* readding tests

* Adding a comment around the new DiscreteOutputSize method

* Clearer warning : Model contains unexpected input > Model requires unknown input

* Fixing a bug in the case where the discrete action tensor does not exist

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* Make IActuator implement IHeuristicProvider. (#5110)

* Adding more tests for v2 models (#5138)

* Adding more tests for v2 models

* formatting

* fixing formatting

* [MLA-1634] Add ObservationSpec and update ISensor interfaces (#5127)

* Add custom package settings (#5027)

* cleanup: yamato triggers, changelog, docstrings

* clean up merge

* fix compile error

* update c# version to 2.0.0-exp.1 (#5160)

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* Move physics/sensor tests that fail in edit mode depending on what's in the scene to playmode tests. (#5162)

* update package version

* remove defines that are now always true

* changelog

* Remove all old analytics defines. (#5168)

* [MLA-1634] Compression spec (#5164)

* Low hanging fruit tests for coverage. (#5170)

* Update Policies to conditionally compile their analytics method. (#5171)

* Turns physics modules into optional dependencies. (#5112)

* [MLA-1634] Remove SensorComponent.GetObservationShape() (#5172)

* PR Feedback. (#5173)

* change default barracuda behavior (#5175)

* [MLA-1824] make SensorComponent return ISensor[] (#5181)

* Make SensorComponent return an array

* split match3 sensors, partial retrain

* docstrings, migration, changelog, cleanup

* Adding the goal conditioning sensors with the new observation specs (#5159)

* Fixing networks.py for the merge

* fix compile error

* Adding the goal conditioning sensors with the new observation specs

* addressing feedback

* I forgot to change the m_observationType

* Renaming Goal to GoalSignal (#5190)

* Renaming GOAL to GOAL_SIGNAL

* VectorSensorComponent to use new API

* Adding docstrings

* verbose pytest on github action

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* [MLA-1879] culture-invariant sorting for sensors and actuators (#5194)

* Goal conditioning grid world : Example of goal conditioning (#5193)

* Aded the Goal conditioned GridWorld to replace regular gridworld

* adding missing files

* Code improvements

* Documentation change on gridworld

* resolving conflicts

* new model

* Addressing comments

* comments and renames

* Update docs/Learning-Environment-Examples.md

Co-authored-by: Ervin T. <ervin@unity3d.com>

* adding reference to gridworld in docs about goal signal

Co-authored-by: Chris Elion <chris.elion@unity3d.com>
Co-authored-by: Ervin T. <ervin@unity3d.com>

* [MLA-1584] Match3 variable board size (#5189)

* [bug-fix] When agent isn't training, don't clear update buffer (#5205)

* Don't clear update buffer, but don't append to it either

* Update changelog

* Address comments

* Make experience replay buffer saving more verbose

* Changelog for goal conditioning (#5195)

* Changelog for goal conditioning

* adding a line about the changes to the gridworld example

* Addressing comments

* Change capitalization

* [bug fix] Fix warning using demo recorder (#5216)

* Making VectorActionSize and VectorActionSpaceType internal (#5214)

Made sure the editor does not complain;

* Turning some logger.info into logger.debug and remove some logging overhead when not using debug (#5211)

* turning some logger.info into logger.debug and remove some logging overhead when not using debug

* Addressing comments

* Adding to changelog

* rearrange match3 docs (#5215)

* rearrange match3 docs

* changelog and migration from previous PR

* [bug-fix] Fix POCA LSTM, pad sequences in the back (#5206)

* Pad buffer at the end

* Fix padding in optimizer value estimate

* Fix additional bugs and POCA

* Fix groupmate obs, add tests

* Update changelog

* Improve tests

* Address comments

* Fix poca test

* Fix buffer test

* Increase entropy for Hallway

* Add EOF newline

* Fix Behavior Name

* Address comments

* [MLA-1880] Raycast sensor interface improvements (#5222)

* WIP

* remove debug info struct

* cleanup + add to test

* changelog

* fix unit tests

* PR feedback

* Load individual elements if state dict load fails (#5213)


Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com>
Co-authored-by: Ervin T. <ervin@unity3d.com>

* [MLA-1159] Add virtual methods to DecisionRequester (#5223)

* [Bug Fix] Serialization of vectorActionSpace (#5220)

* [Bug Fix] Serialization of vectorActionSpace

* adding a test

* Clear ActionBuffers before Heuristic calls (#5227)

* Add ObservationType to analytics ObservationSpec (#5218)

* [MLA-1909] Match3 and Camera/RenderTexture sensor GC improvements (#5233)

* avoid empty set iteration, avoid Debug.AssertFormat (#5246)

* avoid empty set iteration, avoid Debug.AssertFormat

* changelog

* Update to barracuda 1.3.3 and changes to the model inputs and outputs for LSTM (#5236)

* Initial commit

* making it work with hallway. Added a new model version

* addressing feedback

* Adding a comment

* formatting

* modifying the changelog

* Adding descriptions on the model version descriptions

* Update com.unity.ml-agents/Runtime/Inference/GeneratorImpl.cs

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* addressing comments

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* [Release 16] Release 16 Merge Back to Main (#5255)

Update versions and documentation for Release 16. 

Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com>
Co-authored-by: andrewcoh <54679309+andrewcoh@users.noreply.github.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* fix link, run link check in quiet mode (#5257)

* [MLA-1912] Move Match3 to com.unity.ml-agents (#5259)

* Move GridSensor into main package (#5256)

* move OneHotGridSensor into main package

* changelog and migration guide

* remove old doc

* check if physics module presents

* Add advanced installation guide for com.unity.ml-agents (#5260)

* Fix errors when creating new RayCast sensor (#5261)

* check if tag and angle is null

* add test

* changelog

* fix all PVS and doc generation warnings (#5262)

* Fix --results-dir (#5269)

* Fix GridSensorComponent bug (#5270)

* LSTM models from 1.x will be incompatible with MLA 2.x (#5254)

* LSTM models from 1.x will be incompatible with MLA 2.x

* Adding a test and a new v2 model

* Make the Model Runner raise an error if using 1.0 model with LSTM

* adding a new model for hallway trained with 2.0

* reword error messages

* Only raise if error, not if warning

* Addressing comments: The legacy Barrauda memory generator and applier were removed. All code that checked for (memories + v1.X) have been removed since these will no longer be supported

* Modifying the changelog and the migrating guide with this change

* Fixing the merge issues

Co-authored-by: Chris Elion <chris.elion@unity3d.com>

* [MLA-1939] Match3 Custom Editor (#5263)

* Rider suggested cleanup, part 1  (#5265)

* Fix the attention module embedding size (#5272)

* Fix the attention module embedding size

* editing the changelog

* [bug-fix] Fix null ref exception in the demo recorder (#5274)

* Fix null ref exception in the demo recorder

* Update changelog

* Clearer logic

* Modifying the changelog for release 17 (#5277)

* Modifying the changelog for release 17

* Change version to 2.0.0-exp.1

* Removing the [Unreleased] section

* check for missing AbstractBoard, display warning (#5276)

* Updating the barracuda 1.4.0 (#5291)

* [debug] Require all behavior names to have a matching YAML entry (#5210)

* Add strict check to settings.py

* Remove warning from trainer factory, add test

* Add changelog

* Fix test

* Update changelog

* Remove strict CLI options

* Remove strict option, rename, make strict default

* Remove newline

* Update comments

* Set default dict to actually default to a default dict

* Fix tests

* Fix tests again

* Default trainer dict to requiring all fields

* Fix settings typing

* Use logger

* Add default_settings to error

* Version bump main (#5279)

* Version bump for the main branch following branching of release_17

* Messed up the extensions package version

* Bring back root reference in grid sensor (#5300)

* [MLA-1952] Add optional seed for gym action spaces (#5303)

* add optional seed for action spaces

* add optional seed for action spaces

* changelog

* undo packages-lock.json change

* Release 17 version bumps and docs version bumps (#5280)

* Migration guide - list removed APIs with replacements (#5307)

* list removed APIs with replacements

* fixes

* [docs] Update changelog (#5308)

* [MLA-1952] Add optional seed for gym action spaces (#5303) (#5315)

* Fix stacked grid sensor (#5335)

* Update to Unity Package licenses (#5340)

* Update to Unity Package licenses

Per Dan

* Updating the main repo license dates.

* Adding a fully connected visual encoder for super small visual input + tests (#5351)

* initial commit for a fully connected visual encoder

* adding a test

* addressing comments

* Fixing error with minimal size of fully connected network

* adding documentation and changelog

* Make OverlapChecker an interface (#5324)

* Add stacking option to VectorSensorComponent (#5376)

* [WIP] [Fix] Fixing collect observation called on done (#5375)

* [WIP] [Fix] Fixing collect observation called on done

* Update com.unity.ml-agents/Runtime/Agent.cs

* ⚠️ Modifying the test of stacking sensor when the agent is done

* modifying the documentation for BufferSensor to specify to call AddObservation in the CollectObservations method

* Fix NullReferenceException for Behavior Parameters without Agent (#5382)

* check agent existence. add warning box

* changelog

* [WIP] 2.0 verified to main (#5385)

* Warn if no joints on Root Body (#5387)

* Modifying the changelog for the continuous action SAC target entropy fix (#5394)

* Editing the GridSensor documentation for 2D use case (#5396)

* Editing the GridSensor documentation for 2D use case

* changing chagelog

* [MLA-2017] Move colab notebooks to github (#5399)

* Fix the reporting of histogram stats and adding a test (#5410)

* Fix the reporting of histogram stats and adding a test

* Appending to the Changelog

* Update changelog before Release 18 branch (#5412)

* Update package versions for main branch (#5413)

* [Release 18] Update versions and links (#5414)

* add changelog entries for PRs that were missing them before. (#5416)

* fix cref error found by doc validation (#5421)

* fix release versions and changelog section

* [bug-fix] Fix when group terminal steps are deleted, robust test (#5441)

* Fix when terminal steps are deleted, robust test

* Update changelog

* Fix test comment

* Fix torch device option in argparser (#5467)

* [FIX] prioritize cli options over yaml (#5495)

* prioritize cli over yaml in checkpointSettings

* prioritize resume if both set in one place

* fixed test_commandline_args

* addressing comments:renaming, commenting

* more tests and referenced in change_log

* Only update the native library and the c# calls that need to be made to clean up channels. (#5283)

* Fix Mac backcompat test (#5519)

* Initialize-from custom  checkpoints  (#5525)

* init from any checkpoint including older ones
* moving init_path logic ahead to learn.py
* fixing pytest to take the full path
* doc & changelog

* Update gym version to 0.20.0 (#5540)

* Fixed failing test due to gym 0.20.0 release. Updated gym-unity package to fix gym version to 0.20.0.

* Updated CHANGELOG.

* Fix VAIL (#5546)

* Custom trainer editor analytics (#5511)

* Custom trainer editor analytics
* inherit the default_training_analytics_side_channel

* Feature/mla 2205 separate schedule lr beta epsilon (#5538)

* Added initial (untested) support for separating schedule for beta, LR, and epsilon.

* Added support for beta and epsilon schedule in test config conversion

* Updated docs. Set epsilon to be linear by default.

* Updated configs to add beta and epsilon schedule where appropriate.

* Addressed CR change requests.

* Minor update.

* Updated docs.

* Added changes to CHANGELOG.

* Updated CHANGELOG.

* Fixed typo in Training-Configuration-File.md

* Reverted config changes to remove beta and epsilon schedules.

* mypy error fixed from pre-commit.

* Fixed a bug that was causing a settings test to fail.

* Addressed CR comments.

Co-authored-by: Miguel Alonso Jr <miguel.alonso@miguel.alonso-00MM>
Co-authored-by: Miguel Alonso Jr <miguelalonsojr>

* Harden user PII protection logic and extend TrainingAnalytics to expose detailed configuration parameters. (#5512)

* Hash128 is not a cryptographic hash, replace with HMAC-SHA256.

* Extend TrainingAnalytics side channel to expose configuration details

* Change member function scopes and hash demo_paths

* Extract tbiEvent hashing method and add test coverage

* Restart crashed Unity environments (#5553)

### Change(s)

Update the SubprocessEnvManager to restart workers when the underlying Unity environments crash. 
When a worker receives an ENV_EXITED signal it will now:

1. Record all failures coming through the step queue and drop all other messages.
2. Purge any pending trajectories as they may belong to a crashed worker or be corrupted.
3. Restart all failed workers (up to a configurable limit)

This behavior can be limited via a rate limit, max lifetime limit, or both. The configuration options for both are shown below with their default values. 

⚠️  Each of these options applies to a single environment, if num_envs > 1 then the limit will apply separately to each replica (num_envs = 2 will spawn 2 Unity environments which can each be restarted 10 times).

```yaml
env_settings:
  # Can restart 10 times over the lifetime of the experiment.
  max_lifetime_restarts: 10
  # Rate limit of 1 failure per 60s
  restarts_rate_limit_n: 1
  restarts_rate_limit_period_s: 60
```

They can of course be passed via CLI arguments as well

```bash
--max-lifetime-restarts
--restarts-rate-limit-n
--restarts-rate-limit-period-s
```

### Disabling this feature
* Rate limiting can be turned off by setting `--restarts-rate-limit-n=-1`
* Lifetime limiting can be turned off by setting `--max-lifetime-restarts=-1`

* Develop add training area replicator (#5568)

* Added training area replicator to com.unity.ml-agents package.

* Added num_areas to Unity RL Initialization proto. Added cli and config file support for num_areas.

* Changed training area replicator to size grid automatically from number of areas.

* Added tests for the training area replicator.

* Added setup for tests for the training area replicator.

* Added comments and updated create tutorial for training area replicator.

* Updated CHANGELOG.

* Fixed some failing tests.

* Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>

* Update docs/Learning-Environment-Create-New.md

Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>

* Update com.unity.ml-agents/Runtime/Areas/TrainingAreaReplicator.cs

Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>

* Addressed CR comments.

Co-authored-by: Miguel Alonso Jr <miguelalonsojr>
Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>

* Fixing the analytics side-channel for curriculum learning. (#5586)

* Fixing the analytics side-channel for curriculum learning.

* Made a more robust test.

* Update the changelog.

* Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Method to return stacked observations (#5547)

* Method to return stacked observations

* Added testing and returning staked observation flat.

* Update the comment lines.

* Remove brainstorm commits.

* Upgrade Barracuda to 2.3.1-preview (#5591)

* Upgrade to 2.3.1-preview

* Fix for critic normalization bug (#5595)

* Added normalization to critic during training for poca, ppo, and sac.

* Fixed critic normalization bug. Added new tests to cover.

* Updated CHANGELOG.

* Fixed typo in test_trainers.py

* Deterministic actions python training (#5619)

* Progress on propagating the setting to the action model.

* Added the _sample_action logic and tests.

* Add information to the changelog.

* Prioritize the CLI over the configuration file.

* Update documentation for config file.

* CR refactor.

* Update docs/Training-Configuration-File.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Update ml-agents/mlagents/trainers/settings.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Update ml-agents/mlagents/trainers/cli_utils.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Fix CR requests

* Add tests for discrete.

* Update ml-agents/mlagents/trainers/torch/distributions.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Added more stable test.

* Return deterministic actions for training (#5615)

* Added more stable test.

* Fix the tests.

* Fix pre-commit

* Fix help line to pass precommit.

* support for deterministic inference in onnx (#5593)

* Init: actor.forward outputs separate deterministic actions

* changelog

* Renaming

* Add more tests

* Package changes to support deterministic inference (#5599)

* Init: actor.forward outputs separate deterministic actions

* fix tensor shape for discrete actions

* Add test and editor flag

- Add tests for deterministic sampling
- update editor and tooltips

* Reverting to "Deterministic Inference"

* dissect tests

* Update docs

* Update CHANGELOG.md

Co-authored-by: Chingiz Mardanov <chingiz.mardanov@unity3d.com>
Co-authored-by: cmard <87716492+cmard@users.noreply.github.com>

* Revert "Deterministic actions python training (#5619)" (#5622)

This reverts commit 9ea962027d4c581de7c45a07d669f96453236c5c.

* Deterministic actions python training (#5626)

* Progress on propagating the setting to the action model.

* Added the _sample_action logic and tests.

* Add information to the changelog.

* Prioritize the CLI over the configuration file.

* Update documentation for config file.

* CR refactor.

* Update docs/Training-Configuration-File.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Miguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Update com.unity.ml-agents/CHANGELOG.md

Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Update ml-agents/mlagents/trainers/settings.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Update ml-agents/mlagents/trainers/cli_utils.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Fix CR requests

* Add tests for discrete.

* Update ml-agents/mlagents/trainers/torch/distributions.py

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Added more stable test.

* Return deterministic actions for training (#5615)

* Added more stable test.

* Fix the tests.

* Fix pre-commit

* Fix help line to pass precommit.

* support for deterministic inference in onnx (#5593)

* Init: actor.forward outputs separate deterministic actions

* changelog

* Renaming

* Add more tests

* Package changes to support deterministic inference (#5599)

* Init: actor.forward outputs separate deterministic actions

* fix tensor shape for discrete actions

* Add test and editor flag

- Add tests for deterministic sampling
- update editor and tooltips

* Reverting to "Deterministic Inference"

* dissect tests

* Update docs

* Update CHANGELOG.md

* Fix the deterministic showing up all the tiime (#5621)

Co-authored-by: Chingiz Mardanov <chingiz.mardanov@unity3d.com>
Co-authored-by: cmard <87716492+cmard@users.noreply.github.com>

* Changelog cleanup. (#5627)

* Release cleanup for changelog

* Update CHANGELOG.md

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Fix analytics error when compiling XBOX and PS5 (#5628)

* Add ENABLE_CLOUD_SERVICES_ANALYTICS Guards

To use analytics safely you must wrap usage with 
`#if ENABLE_CLOUD_SERVICES_ANALYTICS` 
See https://docs.unity3d.com/ScriptReference/Analytics.Analytics.html

* Update DQN Colab example to support export for Barracuda inference (#5643)

* Update DQN GridWorld colab example to be capable of outputting an ONNX model that works for inference.
-Allow GridWorld to work without a GoalSensor.  Defaults to the GreenPlus target in that case.
-Add GridWorldColab.onnx file, trained and generated with the colab.

* Upgrade DQN GridWorld Colab to produce model that can work with (modified) GridWorld Scene.
-permute observation images from NHWC to NCHW right after env_step.  Everything following is NCHW.
-update input_shape indexes and remove equivalent permute from the VisualQNetwork
-fix reference to global qnet inside generate_trajectories (use the passed in version)
-create wrapper network to add required constants and dummy input masks required by the
 runtime inference (Barracuda)
-export the wrapped network to ONNX format compatible with Barracuda

* Add additional GridWorld scene called GridWorldColab that has the goal sensors removed (target is always green plus) and masks disabled.  The purpose of this is to have a scene compatible with the DQN GridWorld Colab example.

* Minor edit in Colab docs.

* Delete unnecessary .lighting file.

* Update changelog.

* Update colab/Colab_UnityEnvironment_2_Train.ipynb

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Fix error in colab metadata name.

* Add suggested comments and fix incorrect parameter ordering.

* Update docs for Elo curriculum completion measure.

Co-authored-by: Maryam Honari <honari.m94@gmail.com>

* Changelog prepped for release (#5629)

* Changelog prepped for release

* Added missing PRs to the changelog.

* Added missing CHANGELOG item.

* enable 3D Ball sample import button in package manager for unity sdk (#5665)

* enable 3D Ball sample import button in package manager for unity sdk

* add Samples folder

* add meta file

* fix git ignore config

* Updated versions for main. Also updated validate versions script to r… (#5668)

* Updated versions for main. Also updated validate versions script to remove exp.1 from package version.

* Reverted validate versions.

* Updated versions.

* Bumped package version for patch.

* Removed unreleased section. (#5667)

* Updated release versions and release links. (#5669)

* Updates for package validation errors. (#5671)

* Release 19 branch lts deprecation (#5673)

* Updated Project and DevProject to 2020.3 LTS.

* Updated supported LTS version to 2020.3.

* Updated changelog.

* Bumped up CI Unity version from 2020.2 to 2020.3.

* Update CHANGELOG.md

* Removing VS Studio dependency from manifest.json.

* Update packages-lock.json

* Update CHANGELOG.md

* Bumped package version for patch.

* Fixed build-docker and updated CHANGELOG to add unreleased section. (#5682)

* Fixed build-docker and updated CHANGELOG to add unreleased section.

* Updated backward compat mac training yamato test. (#5685)

* Develop deprecate python 3.6 (#5)

* Dropped support for python 3.6

* Pinning python 3.9.9 for tests due to typing issues with 3.9.10

* Testing new bokken image.

* Testing new bokken image.

* Updated yamato standalone build test.

* Updated yamato standalone build test.

* Updated standalone build test.

* Updated yamato configs to use mla bokken vm.

* Bug fixes for yamato yml files.

* Fixed com.unity.ml-agents-test.yml

* Bumped min python version to 3.7.2

* Updated failing yamato jobs.

* Updated github workflow for colab tests.

* Updated github workflow for colab tests.

* Updated github workflow for colab tests.

* Develop python api ga (#6)

* Dropped support for python 3.6

* Pinning python 3.9.9 for tests due to typing issues with 3.9.10

* Testing new bokken image.

* Testing new bokken image.

* Updated yamato standalone build test.

* Updated yamato standalone build test.

* Updated standalone build test.

* Updated yamato configs to use mla bokken vm.

* Bug fixes for yamato yml files.

* Fixed com.unity.ml-agents-test.yml

* Bumped min python version to 3.7.2

* pettingzoo api prototype

* add example

* update file names

* support multiple behavior names

* fix multi behavior action index

* add install in colab

* add setup

* update colab

* fix __init__

* clone single branch

* import tags only

* import in init

* catch import error

* update colab

* move colab and add readme

* handle agent dying

* add tests

* update doc

* add info

* add action mask

* fix action mask

* update action masks in colab

* change default env

* set version

* fix hybrid action

* fix colab for hybrid actions

* add note on auto reset

* Updated colab name.

* Update README.md

* Following petting_zoo registry API (#5557)

* init petting_zoo registry

* cherrypick Custom trainer editor analytics (#5511)

* cherrypick "Update dotnet-format to address breaking changes introduced by upstream changes (#5528)"

* Update colab to match pettingZoo import api

* ToRevert: pull exp-petting-registry branch

* Add init file to tests

* Install pettingzoo-unity requirements for pytest

* update pytest command

* Add docstrings and comments

* update coverage to pettingzoo folder

* unset log level

* update env string

* Two small bugfixes (#5589)

1. Add the missing `_cumulative_rewards` property
2. Update `agent_selection` to not error out when an agent finishes an episode.

* Updated gym to 0.21.0 and petting zoo to 1.13.1, fixed bugs with AEC wrapper for gym and PZ updates. API tests are passing.

* Some refactoring.

* Finished inital implementation of parallel. Tests not passing.

* Finished parallel API implementation and refactor. All PZ tests passing.

* Cleanup.

* Refactoring.

* Pinning numpy version.

* add metadata and behavior_specs initialization

* addressing behaviour_spec issues

* Bumped PZ version to 1.14.0. Fixed failing tests.

* Refactored gym-unity and petting-zoo into ml-agents-envs

* Added TODO to pydoc-config.yaml

* Refactored gym and pz to be under a subpackage in mlagents_env package

* Refactored ml-agents-envs docs.

* Minor update to PZ API doc.

* Updated mlagents_envs docs and colab.

* Updated pytest gh workflow to remove ref to gym and pz.

* Refactored to remove some test coupling between trainers and envs.

* Updated installation doc.

* Update ml-agents-envs/README.md

Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com>

* Updated failing yamato jobs.

* pettingzoo api prototype

* add example

* update file names

* support multiple behavior names

* fix multi behavior action index

* add install in colab

* add setup

* update colab

* fix __init__

* clone single branch

* import tags only

* import in init

* catch import error

* update colab

* move colab and add readme

* handle agent dying

* add tests

* update doc

* add info

* add action mask

* fix action mask

* update action masks in colab

* change default env

* set version

* fix hybrid action

* fix colab for hybrid actions

* add note on auto reset

* Updated colab name.

* Update README.md

* Following petting_zoo registry API (#5557)

* init petting_zoo registry

* cherrypick Custom trainer editor analytics (#5511)

* cherrypick "Update dotnet-format to address breaking changes introduced by upstream changes (#5528)"

* Update colab to match pettingZoo import api

* ToRevert: pull exp-petting-registry branch

* Add init file to tests

* Install pettingzoo-unity requirements for pytest

* update pytest command

* Add docstrings and comments

* update coverage to pettingzoo folder

* unset log level

* update env string

* Two small bugfixes (#5589)

1. Add the missing `_cumulative_rewards` property
2. Update `agent_selection` to not error out when an agent finishes an episode.

* Updated gym to 0.21.0 and petting zoo to 1.13.1, fixed bugs with AEC wrapper for gym and PZ updates. API tests are passing.

* Some refactoring.

* Finished inital implementation of parallel. Tests not passing.

* Finished parallel API implementation and refactor. All PZ tests passing.

* Cleanup.

* Refactoring.

* Pinning numpy version.

* add metadata and behavior_specs initialization

* addressing behaviour_spec issues

* Bumped PZ version to 1.14.0. Fixed failing tests.

* Refactored gym-unity and petting-zoo into ml-agents-envs

* Added TODO to pydoc-config.yaml

* Refactored gym and pz to be under a subpackage in mlagents_env package

* Refactored ml-agents-envs docs.

* Minor update to PZ API doc.

* Updated mlagents_envs docs and colab.

* Updated pytest gh workflow to remove ref to gym and pz.

* Refactored to remove some test coupling between trainers and envs.

* Updated installation doc.

* Update ml-agents-envs/README.md

Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com>

* Updated CHANGELOG.

* Updated Migration guide.

* Doc updates based on CR.

* Updated github workflow for colab tests.

* Updated github workflow for colab tests.

* Updated github workflow for colab tests.

* Fixed yamato import error.

Co-authored-by: Ruo-Ping Dong <ruoping.dong@unity3d.com>
Co-authored-by: Miguel Alonso Jr <miguelalonsojr>
Co-authored-by: jmercado1985 <75792879+jmercado1985@users.noreply.github.com>
Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>
Co-authored-by: mahon94 <maryam.honari@unity3d.com>
Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com>

* Refactoring for extraction to separate repo.

* removing unused github workflows.

* Updates for pre-commit and github actions.

* Fixed bug with validate_meta_files.py missing a file to exclude.

* Updated precommit gh action.

* Refactored yamato pack.

* Added yamato test yaml.

* Updated to CHANGELOG and VectorSensor from mono repo before swap.

* Pin dotnet version for GHA formatting.

* Refactor for MLA pro support. (#1)

* Refactored for MLA pro support.

* Fixed bug in yamato test.

* Updated the CHANGELOG.

* Fixed trigger logic in yamato test.

* Fixed failing tests. (#2)

* Fixed failing tests.

* Adding meta files.

* Fix execution order (#4)

* Fixed execution order.

* Fixed bug with CRT build.

* precommit formatting update.

* Updated agent Awake docstring.

* Set execution order to be negative to make sure scripts run before default time. (#5)

* Added #if to exclude communicator registration in the Agent awake script (#6)

* Add visibility to pro package (#7)

* add coverage publish promotion yamato jobs (#3)

* bump barracuda to 3.0.0 (#9)

upgrade to barracuda 3.0.0

* Updated README. (#8)

* add validation test dep. to publish job (#12)

* add validation test dep. to publish job
* fix package path

* Update automated trigger rules to support oneflow develop branch (#11)

* Update automated trigger rules to support oneflow develop branch

* Fix release branch name for github actions

* Release 2.3.0 exp.1 (#13)

* upgrade package to 2.3.0-exp.1

* rebase on develop branch

* update changelog

* remove unreleased section (#16)

* Updated experimental prerelease version and changelog. (#18)

* Added submodule update gha. (#19)

* fix bugs from sonar-scanner (#21)

* add ml-agents-pro as a upstream for submodule update (#22)

* Added support for standalone use of Academy with refactored communicator factory. (#24)

* clean commandline Arguments (#25)

* clean commandline Arguments
* fix precommit

* update unrelease section in changelog (#27)

* MLA-2537 Reorder RayPerceptionSensor output to reflect its spatial structure. (#26)

* Changes to support in order rays.  GUI for switching to this.  Updated tests.

* Updated README. (#29)

* Fixed samples. (#30)

* Added camera sensor component checkbox for runtime enablement. update… (#31)

* Added camera sensor component checkbox for runtime enablement. updated camera sensor to only be active when enabled for runtime.

* Updated yamato job.

* Updated CHANGELOG.

* Fixed issue with Academy not building for webgl builds. (#32)

* Dev bump min ver (#33)

* Bumped min version of Unity to 2021.3.

* Updated min version of python in pre-commit. Updated changelog.

* Moving files into packages/com.unity.ml-agents directoy.

* Remove com.unity.ml-agents submodule.

* Added back in ml-agents package.

* Removed packages temp folder.

* Updated yamato protobuf gen.

* Updated yamato to remove submodule update.

* Updated linux tool chain.

* Updated Project package dependencies.

* Updated flake8 repo link in precommit.

Co-authored-by: Vincent-Pierre BERGES <vincentpierre@unity3d.com>
Co-authored-by: Arthur Juliani <awjuliani@gmail.com>
Co-authored-by: Chris Elion <chris.elion@unity3d.com>
Co-authored-by: andrewcoh <54679309+andrewcoh@users.noreply.github.com>
Co-authored-by: Chris Goy <christopherg@unity3d.com>
Co-authored-by: Marwan Mattar <marwan@unity3d.com>
Co-authored-by: Ruo-Ping Dong <ruoping.dong@unity3d.com>
Co-authored-by: Ervin Teng <ervin@unity3d.com>
Co-authored-by: Hunter-Unity <hunter@unity3d.com>
Co-authored-by: Maryam Honari <honari.m94@gmail.com>
Co-authored-by: Miguel Alonso Jr <76960110+miguelalonsojr@users.noreply.github.com>
Co-authored-by: Miguel Alonso Jr <miguel.alonso@miguel.alonso-00MM>
Co-authored-by: Jason Bowman <jasonb@unity3d.com>
Co-authored-by: Henry Peteet <henry.peteet@unity3d.com>
Co-authored-by: cmard <87716492+cmard@users.noreply.github.com>
Co-authored-by: Chingiz Mardanov <chingiz.mardanov@unity3d.com>
Co-authored-by: Jason Rupert <92821409+jrupert-unity@users.noreply.github.com>
Co-authored-by: Xun Cao <xun.cao@outlook.com>
Co-authored-by: jmercado1985 <75792879+jmercado1985@users.noreply.github.com>
Co-authored-by: mahon94 <maryam.honari@unity3d.com>
Co-authored-by: Andrew Cohen <andrew.cohen@unity3d.com>
Co-authored-by: Xun Cao <xun.cao@unity3d.com>
Co-authored-by: Jason Rupert <jason.rupert@unity3d.com>

Author: 19 people

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "com.unity.ml-agents"]
-	path = com.unity.ml-agents
-	url = ../com.unity.ml-agents.git

.pre-commit-config.yaml

@@ -25,7 +25,7 @@ repos:
         exclude: ".*_pb2.py"
         args: [--ignore-missing-imports, --disallow-incomplete-defs, --no-strict-optional]
         additional_dependencies: [types-PyYAML, types-attrs, types-protobuf, types-setuptools, types-filelock]
--   repo: https://gitlab.com/pycqa/flake8
+-   repo: https://github.com/PyCQA/flake8
     rev: 3.9.2
     hooks:
     -   id: flake8

.yamato/com.unity.ml-agents-coverage.yml

@@ -12,7 +12,6 @@ test_coverage_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
         image: {{ platform.image }}
         flavor: {{ platform.flavor}}
     commands:
-        - git submodule update --init --recursive
         - npm install upm-ci-utils@1.27.0 -g --registry https://artifactory.prd.cds.internal.unity3d.com/artifactory/api/npm/upm-npm
         - upm-ci project test -u {{ editor.version }} --type project-tests --project-path {{ editor.testProject }} --package-filter {{ package.name }} {{ coverageOptions }} --extra-utr-arg "reruncount=2"
         - |

.yamato/com.unity.ml-agents-pack.yml

@@ -5,7 +5,6 @@ pack:
     image: ml-agents/ml-agents-ubuntu-18.04:latest
     flavor: b1.small
   commands:
-    - git submodule update --init --recursive
     - |
       eval "$($HOME/anaconda/bin/conda shell.bash hook)"
       conda activate python3.8

.yamato/com.unity.ml-agents-performance.yml

@@ -12,7 +12,6 @@ Run_Mac_Perfomance_Tests{{ editor.version }}:
   variables:
     UNITY_VERSION: {{ editor.version }}
   commands:
-    - git submodule update --init --recursive
     - python3 -m pip install unity-downloader-cli --index-url https://artifactory.prd.it.unity3d.com/artifactory/api/pypi/pypi/simple --upgrade
     - unity-downloader-cli -u {{ editor.version }} -c editor --wait --fast
     - curl -s https://artifactory.prd.it.unity3d.com/artifactory/unity-tools-local/utr-standalone/utr --output utr

.yamato/com.unity.ml-agents-test.yml

@@ -79,7 +79,6 @@ test_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
         image: {{ platform.image }}
         flavor: {{ platform.flavor}}
     commands:
-        - git submodule update --init --recursive
         - npm install upm-ci-utils@1.27.0 -g --registry https://artifactory.prd.cds.internal.unity3d.com/artifactory/api/npm/upm-npm
         - upm-ci package test -u {{ editor.version }} --package-path {{ package.name }} {{ noDefaultPackagesOptions }} --warnings-as-errors --extra-utr-arg "reruncount=2"
     artifacts:
@@ -125,7 +124,6 @@ test_{{ package.name }}_{{ platform.name }}_trunk:
         image: {{ platform.image }}
         flavor: {{ platform.flavor}}
     commands:
-        - git submodule update --init --recursive
         - |
 {% if platform.name == "linux" %}
           conda activate python3.8

.yamato/compressed-sensor-test.yml

@@ -10,7 +10,6 @@ test_compressed_obs_{{ editor.version }}:
   variables:
     UNITY_VERSION: {{ editor.version }}
   commands:
-    - git submodule update --init --recursive
     - |
       eval "$($HOME/anaconda/bin/conda shell.bash hook)"
       conda activate python3.8

.yamato/gym-interface-test.yml

@@ -10,7 +10,6 @@ test_gym_interface_{{ editor.version }}:
   variables:
     UNITY_VERSION: {{ editor.version }}
   commands:
-    - git submodule update --init --recursive
     - |
       eval "$($HOME/anaconda/bin/conda shell.bash hook)"
       conda activate python3.8

.yamato/protobuf-generation-test.yml

@@ -6,9 +6,8 @@ test_linux_protobuf_generation:
     flavor: b1.large
   variables:
     GRPC_VERSION: "1.14.1"
-    CS_PROTO_PATH: "Runtime/Grpc/CommunicatorObjects"
+    CS_PROTO_PATH: "com.unity.ml-agents/Runtime/Grpc/CommunicatorObjects"
   commands:
-    - git submodule update --init --recursive
     - |
       sudo apt-get update && sudo apt-get install -y nuget
       eval "$($HOME/anaconda/bin/conda shell.bash hook)"
@@ -23,7 +22,6 @@ test_linux_protobuf_generation:
       popd
       mkdir -p artifacts
       touch artifacts/proto.patch
-      cd com.unity.ml-agents
       git diff --exit-code -- :/ ":(exclude,top)$CS_PROTO_PATH/*.meta"                          \
       || { GIT_ERR=$?; echo "protobufs need to be regenerated, apply the patch uploaded to artifacts."; \
       echo "Apply the patch with the command 'git apply proto.patch'";                                  \

.yamato/pytest-gpu.yml

@@ -5,7 +5,6 @@ pytest_gpu:
     image: ml-agents/ml-agents-ubuntu-18.04:latest
     flavor: b1.large
   commands:
-    - git submodule update --init --recursive
     - |
       eval "$($HOME/anaconda/bin/conda shell.bash hook)"
       conda activate python3.8