Add TTIT tracking array to RequestStateStats to trunk

[jaydenyuu]

Summary:
This allows us to track TTIT's at token generation time and fix a weird discrepancy whereTTFT is saved to RequestStateStats but not TTIT

Test Plan: N/A (this only adds a field)

Differential Revision: D86552981

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request aims to add tracking for inter-token latencies (TTIT) to RequestStateStats. The implementation correctly collects the latencies during token generation. However, I've found a critical issue: the collected inter_token_latencies are discarded when a request finishes, as they are not transferred from RequestStateStats to FinishedRequestStats. This makes the collected data unusable. My review includes a comment detailing how to fix this by updating FinishedRequestStats and the logic in update_from_finished_request. Additionally, this change lacks tests. Adding tests would have likely caught this issue and is crucial for verifying that metrics are collected and reported correctly.

[gemini-code-assist]

The inter_token_latencies are collected in RequestStateStats but are not transferred to FinishedRequestStats when a request completes. This means the collected latency data is lost when RequestState is discarded upon request completion.

To fix this, you should also add inter_token_latencies to the FinishedRequestStats dataclass and populate it in IterationStats.update_from_finished_request.

1. Add the field to FinishedRequestStats:

@dataclass
class FinishedRequestStats:
    # ... existing fields
    is_corrupted: bool = False
    inter_token_latencies: list[float] = field(default_factory=list)

2. Populate it in update_from_finished_request:

        finished_req = FinishedRequestStats(
            # ... existing assignments
            is_corrupted=req_stats.is_corrupted,
            inter_token_latencies=req_stats.inter_token_latencies,
        )