Run unit tests with real LLM calls

[TomeHirata]

This PR enables running unit tests that require real LLM calls, which have been skipped in CI.
The model for testing is configurable through LLM_MODEL env variable, and we use ollama/llama3.2:3b in the branch build to balance quality and latency. The Ollama model pulling only introduces ~13s latency, so this PR just enables real LLM tests in the branch build instead of nightly tests.

[chenmoneygithub]

@TomeHirata This is awesome! One thing I want to discuss.

I compared the time cost of running test before and after this PR, seems we are seeing a significant increase from 2m => 4m:

Before:

After:

4m seems to be all right, but that's the outcome of only converting 6 test cases into using Ollama, which means it could become much slower over time. I am thinking about the following:

1. Use pytest markers to split tests requiring actual LM into its own group, like @pytest.mark.real_lm_call. These tests will be skipped in normal github tests, and only run in the dedicated action Run Tests with Real LM. We may not need to test across all python versions, because essentially it's an supplementary to the normal test to test against actual LM behavior.
2. Prefer mocking over real LM testing in unit tests. Only when mocking goes too complex like optimizer, or unreliable over time like streaming, we switch to using the real LM.

Let me know what you think!

[TomeHirata]

Hi, @chenmoneygithub. I agree with both points. I've split LLM call tests into a separate job. I completely agree with #2. We should limit the usage of LLM call in unit tests because of latency and potential flakiness.

[chenmoneygithub]

The setup looks good!

[chenmoneygithub]

LGTM!