feat: add delete command, make cancel abort for k8s (#1156)

[clumsy]

Make k8s scheduler cancel abort the job, instead of removing it completely.

Adding delete command for all schedulers to override, default implementation calls cancel to mimic the current behavior.

Test plan:
[x] added unit tests

[clumsy]

Please check this proposal, @kiukchung
Adding -d to cancel is not the best option I think since we either break all custom schedulers or we need to use introspection.

Let me know if you want to limit the scope to just change the behavior of cancel for k8s.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.63%. Comparing base (dd683f4) to head (4430d18).

Files with missing lines	Patch %	Lines
torchx/runner/api.py	83.33%	1 Missing ⚠️
torchx/schedulers/api.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1158   +/-   ##
=======================================
  Coverage   91.63%   91.63%           
=======================================
  Files          84       85    +1     
  Lines        6595     6624   +29     
=======================================
+ Hits         6043     6070   +27     
- Misses        552      554    +2     

Flag	Coverage Δ
unittests	91.63% <93.10%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[clumsy]

Some testing evidence @kiukchung

• torchx run ...:

...
kubernetes://torchx/some-namespace:some-app
torchx 2025-10-31 15:29:30 INFO     Launched app: kubernetes://torchx/some-namespace:some-app
torchx 2025-10-31 15:29:30 INFO     AppStatus:
    State: UNKNOWN
    Num Restarts: -1
    Roles: 
    Msg: <NONE>
    Structured Error Msg: <NONE>
    UI URL: None

• torchx status kubernetes://torchx/some-namespace:some-app:

AppStatus:
    State: PENDING
    Num Restarts: -1
    Roles: 
    Msg: <NONE>
    Structured Error Msg: <NONE>
    UI URL: None

• torchx cancel kubernetes://torchx/some-namespace:some-app:

• torchx status kubernetes://torchx/some-namespace:some-app: (Looks like k8s:Aborted is already mapped to CANCELLED!)

AppStatus:
    State: CANCELLED
    Num Restarts: -1
    Roles: 
    Msg: <NONE>
    Structured Error Msg: <NONE>
    UI URL: None

• torchx delete kubernetes://torchx/some-namespace:some-app:

• torchx status kubernetes://torchx/some-namespace:some-app:

torchx 2025-10-31 15:45:43 ERROR    AppDef: some-namespace:some-app, does not exist or has been removed from kubernetes's data plane

• torchx delete kubernetes://torchx/some-namespace:some-app:

• torchx cancel kubernetes://torchx/some-namespace:some-app:

If the above looks good I guess the only question is if we want to print anything when we try to delete and do it, or skip if it does not exists.

[meta-codesync]

@kiukchung has imported this pull request. If you are a Meta employee, you can view this in D85961055.

[clumsy]

Updated the docstring, @kiukchung

[clumsy]

Hm, that Meta Internal-Only Changes Check is failing again, did I do something wrong, @kiukchung ?

[clumsy]

I've templetized delete using _delete_existing similar to _cancel_existing works.

Please let me know if you have any remaining concerns, @kiukchung

[kiukchung]

Hm, that Meta Internal-Only Changes Check is failing again, did I do something wrong, @kiukchung ?

nope, the PR just needs to be imported as a "Diff" internally to keep things in sync. I'll do that and merge.

[clumsy]

I can't see what failed in internal checks, @kiukchung Are we good for this PR?

[AbishekS]

Looks good to me!