[Feature] Enable Prompt caching for anthropic / openai

[onel]

What feature would you like to see?

I couldn't find anything in the docs / code regarding a way to enable prompt caching with a model provider.
Is this possible with dspy?

Sending cache=True to dspy.LM enables local caching but not prompt caching.

This is extremely useful when using dspy.ReAct() with the same prompt being sent multiple times

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

Using dspy 2.6

[TomeHirata]

Are you looking for a way to enable provider cache through the cache_control parameter in the message? https://docs.litellm.ai/docs/completion/prompt_caching

[onel]

Yes, exactly. I haven't found a way to tell dspy to tell liteLLM to enable that. Any pointers?

[TomeHirata]

No, it's not supported yet. You can still customize LM and pass it to dspy.settings.configure.

[onel]

Got it. I'm willing to work on this if dspy wants to support it

[onel]

For people on version 2.*, I've customized the LM like this. seems to work ok.
Only for anthropic, but can be extended.

import dspy

class CachedLM(dspy.LM):
    """
    Custom LM class that extends dspy.LM to add Anthropic prompt caching support.
    Adds cache_control to messages to enable prompt caching for Anthropic models.
    """

    def _add_cache_control_to_messages(self, messages):
        """
        Add cache_control to messages for Anthropic prompt caching.
        Caches all messages except the last user message.
        """
        # Only apply caching for Anthropic models
        if not self.model.startswith('anthropic/'):
            return messages

        if not messages or len(messages) == 0:
            return messages

        # Create a copy to avoid modifying the original
        cached_messages = []

        for i, msg in enumerate(messages):
            msg_copy = msg.copy()

            # Add cache control to all messages except the last one
            if i < len(messages) - 1:
                content = msg_copy.get('content', '')

                # Handle both string content and structured content
                if isinstance(content, str):
                    # Convert string to structured format with cache_control
                    msg_copy['content'] = [
                        {
                            "type": "text",
                            "text": content,
                            "cache_control": {"type": "ephemeral"}
                        }
                    ]
                elif isinstance(content, list):
                    # Add cache_control to the last content block
                    if len(content) > 0:
                        content_copy = [c.copy() for c in content]
                        content_copy[-1]["cache_control"] = {"type": "ephemeral"}
                        msg_copy['content'] = content_copy

            cached_messages.append(msg_copy)

        return cached_messages

    def forward(self, prompt=None, messages=None, **kwargs):
        """
        Override forward to add cache_control before sending to LiteLLM.
        """
        messages = messages or [{"role": "user", "content": prompt}]
        messages = self._add_cache_control_to_messages(messages)
        return super().forward(prompt=prompt, messages=messages, **kwargs)

    async def aforward(self, prompt=None, messages=None, **kwargs):
        """
        Async version of forward with cache_control support for Anthropic models.
        """
        messages = messages or [{"role": "user", "content": prompt}]
        messages = self._add_cache_control_to_messages(messages)
        return await super().aforward(prompt=prompt, messages=messages, **kwargs)

[TomeHirata]

Thanks, let's keep this issue open. We are able to officially support it if there are high demands.

[onel]

I honestly think dspy should already support this because it makes using the framework much more expensive than alternatives.

In my case, I was thinking of going with alternatives because this feature was missing. It wouldn't have made sense for me to use in prod without it.

[hung-phan]

My team is using bedrock and have the same use case. The prompt is large so we do need this capability

[gauravkumar37]

+1 Is there a way for us to inject the cache control ephemeral in dspy v3?

[TomeHirata]

It seems LiteLLM supports cache_control_injection_points option. Can you all try passing the parameter to dspy.LM?

lm = dspy.LM(
    "anthropic/claude-3-5-sonnet-20240620",
    cache_control_injection_points=[
        {
            "location": "message",
            "role": "system",
        }
    ],
)
dspy.configure(lm=lm)

[gauravkumar37]

2 alternatives:

1. DSPy automatically knows the best practices and auto-injects cache control param after tools, system and messages
2. Advanced users can add their own cache control breakpoints
   Ultimately, when to add has a lot to do with how much prompt caching happens. [This is like saving intermediate stages to disk to avoid recomputation, users should be able to choose this] See also https://docs.claude.com/en/docs/build-with-claude/prompt-caching#structuring-your-prompt

[TomeHirata]

Thanks, I believe my previous comment is #2. Can you check if that works for you?

[gauravkumar37]

Yes that is what we want but how do we specify "cache_control": {"type": "ephemeral"} in DSPy?
https://docs.litellm.ai/docs/completion/prompt_caching#anthropic-example

[TomeHirata]

LiteLLM automatically injects the cashe control item when cache_control_injection_points is passed. Can you take a look at https://docs.litellm.ai/docs/tutorials/prompt_caching#configuration?