[feat] Improve handling of maintenance reservations in Slurm

[ekouts]

Add MAINTENANCE to the available states for nodes that belong in a reservation with the MAINT flag, when the user explicitly requests for this reservation.

Fixes #3572 .

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.22%. Comparing base (a71c68b) to head (c0d9fb3).
⚠️ Report is 20 commits behind head on develop.

Files with missing lines	Patch %	Lines
reframe/core/schedulers/slurm.py	50.00%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3573      +/-   ##
===========================================
- Coverage    91.26%   91.22%   -0.05%     
===========================================
  Files           62       62              
  Lines        13534    13540       +6     
===========================================
  Hits         12352    12352              
- Misses        1182     1188       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ekouts]

I wasn't sure if we want to fail here or silently ignore it?

[vkarak]

I'd rather simply log it with self.log().

[vkarak]

I agree with the overall fix; I have just some suggestions on the implementation.

[vkarak]

Suggested change

	if 'MAINT' in flags_match[1].split(','):
	if 'MAINT' in flags_match.group(1).split(','):

[vkarak]

I would rather pass the flags and let the __init__() decide what to do with them, in which case you don't need the addl_avail_states. Simply, add the MAINTENANCE state in the available_states if the flag is set.

[ekouts]

Okay, just to make sure I understand: _create_nodes will get a flags argument (instead of addl_avail_states), and then pass it in the __init__() of each _SlurmNode and MAINTENANCE will be added in the allowed states of the node?
My thinking was that eventually we may want to allow the user to pass even more additional "available states", through the cli, for some reason. But I am okay with what you are suggesting too.

[vkarak]

Okay, just to make sure I understand: _create_nodes will get a flags argument (instead of addl_avail_states), and then pass it in the init() of each _SlurmNode and MAINTENANCE will be added in the allowed states of the node?

Yes.

My thinking was that eventually we may want to allow the user to pass even more additional "available states", through the cli, for some reason. But I am okay with what you are suggesting too.

Let's keep it clean at the moment. If we want define custom available states, maybe we should do it with a configuration parameter.

[vkarak]

I'd rather simply log it with self.log().

[vkarak]

We would also need a unit test.