Pause crawls instead of stopping when quotas are reached or archiving is disabled

[tw4l]

Fixes #2957

Full backend and frontend implementation, with a new email notification to org admins when a crawl is paused because an org quota has been reached.

Backend changes

• Modify operator to auto-pause crawls when quotas are reached or archiving is disabled rather than stopping the crawls
• Add new crawl states: paused_storage_quota_reached, paused_time_quota_reached, paused_org_readonly
• Add uploaded WACZs to org storage totals immediately after upload so that auto-paused crawls will actually put the org's bytesStored above the storage quota
• Send an email from new template to all org admins when a crawl is auto-paused with information about what to do
• Fix datetime deprecation in tests

Updated nightly tests all pass: https://github.com/webrecorder/browsertrix/actions/runs/19684324914

Frontend changes

• Add new paused crawl states
• Update checks throughout frontend for whether crawl is paused to compare against all paused states

Dependencies

Relies on crawler changes introduced in webrecorder/browsertrix-crawler#919

Out of scope

Crawl workflow counts are a bit off, counting all crawls that complete as successful regardless of state and sometimes incrementing workflow storage counts incorrectly. I started trying to address that in this branch but it's a bit involved and may require a migration so best handled separately, I think. Issue: #3011

[tw4l]

Suggested change

	# sizes = await redis.hkeys(f"{crawl.id}:size")
	# for size in sizes:
	# await redis.hmset(f"{crawl.id}:size", {size: 0 for size in sizes})

Remove before merging

[tw4l]

Suggested change

	print(f"pending size: {pending_size}", flush=True)
	print(f"status.filesAdded: {status.filesAdded}", flush=True)
	print(f"status.filesAddedSize: {status.filesAddedSize}", flush=True)
	print(f"total: {total_size}", flush=True)
	print(
	f"org quota: {crawl.org.bytesStored + stats.size} <= {crawl.org.quotas.storageQuota}",
	flush=True,
	)

Remove before merging, useful for testing

[tw4l]

Tagging @emma-sg @SuaYoo for review in addition to @ikreymer , with particular interest in getting your eyes on the frontend, email, and email copy parts of this. Thanks!

[SuaYoo]

Nice! Still doing manual testing, my initial impression is it's probably worth adding an isPaused helper to utils/crawler.

export function isPaused({ state }: { state: string | null }) {
  return state && (PAUSED_STATES as readonly string[]).includes(state);
}

[ikreymer]

We want to send the e-mails multiple times, if a crawl reaches quota, then is resumed, then reaches quota again, right?
If so, should also clear autoPausedEmailsSent when crawl is running again

[tw4l]

Nice! Still doing manual testing, my initial impression is it's probably worth adding an isPaused helper to utils/crawler.

export function isPaused({ state }: { state: string | null }) {
  return state && (PAUSED_STATES as readonly string[]).includes(state);
}

I added a helper but made it except a string or null rather than an object with state property, as none of the uses of this take an object with that key. Take a look and let me know what you think.

[tw4l]

We want to send the e-mails multiple times, if a crawl reaches quota, then is resumed, then reaches quota again, right? If so, should also clear autoPausedEmailsSent when crawl is running again

Done, and now storing this state in the db to be more reliable.

[SuaYoo]

Frontend portion looks good!

[emma-sg]

Email language looks good! Left a few suggestions, one splitting a sentence into two and a few just using curly quotes/removing unused code. Nice work!

I'll take another look for frontend & backend changes, just wanted to get you some feedback on the email template now.