Fix dry run mode (#2027)

Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.12.0)
(oldest at bottom):
* #2029
* #2030
* #2028
* __->__ #2027
* #2026

Dry run mode works but it doesn't exit gracefully for all cases. This PR
fixes it

```
DRY_RUN=1 CONFIG_FILE="./torchtitan/models/llama3/train_configs/debug_model.toml" ./run_train.sh   --training.steps=10 --activation_checkpoint.mode="none"
--debug.deterministic --debug.seed=42
```

Author: fegin

scripts/dry_run.py

@@ -151,6 +151,9 @@ def __init__(self, job_config: JobConfig):
         logger.info("Configuration is ready for training execution.")
         logger.info("=" * 80)
 
+    def train(self):
+        return
+
 
 if __name__ == "__main__":
     main(DryRunTrainer)

torchtitan/train.py

@@ -735,7 +735,8 @@ def main(trainer_class: type[Trainer]) -> None:
         raise
     else:
         trainer.close()
-        torch.distributed.destroy_process_group()
+        if torch.distributed.is_initialized():
+            torch.distributed.destroy_process_group()
         logger.info("Process group destroyed")