Merge pull request stanfordnlp#3 from stanfordnlp/main

update

Author: GFarnon

README.md

@@ -72,11 +72,11 @@ Or open our intro notebook in Google Colab: [<img align="center" src="https://co
 
 By default, DSPy installs the latest `openai` from pip. However, if you install old version before OpenAI changed their API `openai~=0.28.1`, the library will use that just fine. Both are supported.
 
-For the optional (alphabetically sorted) [Chromadb](https://github.com/chroma-core/chroma), [Qdrant](https://github.com/qdrant/qdrant), [Marqo](https://github.com/marqo-ai/marqo), Pinecone, [Weaviate](https://github.com/weaviate/weaviate),
+For the optional (alphabetically sorted) [Chromadb](https://github.com/chroma-core/chroma), [Qdrant](https://github.com/qdrant/qdrant), [Marqo](https://github.com/marqo-ai/marqo), Pinecone, [Snowflake](https://github.com/snowflakedb/snowpark-python) [Weaviate](https://github.com/weaviate/weaviate),
 or [Milvus](https://github.com/milvus-io/milvus) retrieval integration(s), include the extra(s) below:
 
 ```
-pip install dspy-ai[chromadb]  # or [qdrant] or [marqo] or [mongodb] or [pinecone] or [weaviate] or [milvus]
+pip install dspy-ai[chromadb]  # or [qdrant] or [marqo] or [mongodb] or [pinecone] or [snowflake] or [weaviate] or [milvus]
 ```
 
 ## 2) Documentation
@@ -101,7 +101,7 @@ The DSPy documentation is divided into **tutorials** (step-by-step illustration
 
 - [DSPy talk at ScaleByTheBay Nov 2023](https://www.youtube.com/watch?v=Dt3H2ninoeY).
 - [DSPy webinar with MLOps Learners](https://www.youtube.com/watch?v=im7bCLW2aM4), a bit longer with Q&A.
-- Hands-on Overviews of DSPy by the community: [DSPy Explained! by Connor Shorten](https://www.youtube.com/watch?v=41EfOY0Ldkc), [DSPy explained by code_your_own_ai](https://www.youtube.com/watch?v=ycfnKPxBMck), [DSPy Crash Course by AI Bites](https://youtu.be/5-zgASQKkKQ?si=3gnmVouT5_rpk_nu)
+- Hands-on Overviews of DSPy by the community: [DSPy Explained! by Connor Shorten](https://www.youtube.com/watch?v=41EfOY0Ldkc), [DSPy explained by code_your_own_ai](https://www.youtube.com/watch?v=ycfnKPxBMck), [DSPy Crash Course by AI Bites](https://youtu.be/5-zgASQKkKQ?si=3gnmVouT5_rpk_nu), [DSPy Paper Explained by Unify](https://youtu.be/kFB8kFchCH4?si=FuM6L5H5lweanckz)
 - Interviews: [Weaviate Podcast in-person](https://www.youtube.com/watch?v=CDung1LnLbY), and you can find 6-7 other remote podcasts on YouTube from a few different perspectives/audiences.
 - **Tracing in DSPy** with Arize Phoenix: [Tutorial for tracing your prompts and the steps of your DSPy programs](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/tracing/dspy_tracing_tutorial.ipynb)
 - [DSPy: Not Your Average Prompt Engineering](https://jina.ai/news/dspy-not-your-average-prompt-engineering), why it's crucial for future prompt engineering, and yet why it is challenging for prompt engineers to learn.

docs/api/language_model_clients/Cloudflare.md

@@ -0,0 +1,42 @@
+---
+sidebar_position: 10
+---
+
+# dspy.CloudflareAI
+
+### Usage
+
+```python
+lm = dspy.CloudflareAI(model="@hf/meta-llama/meta-llama-3-8b-instruct")
+```
+
+### Constructor
+
+The constructor initializes the base class `LM` and verifies the `api_key` and `account_id` for using Cloudflare AI API.
+The following environment variables are expected to be set or passed as arguments:
+
+- `CLOUDFLARE_ACCOUNT_ID`: Account ID for Cloudflare.
+- `CLOUDFLARE_API_KEY`: API key for Cloudflare.
+
+```python
+class CloudflareAI(LM):
+  def __init__(
+          self,
+          model: str = "@hf/meta-llama/meta-llama-3-8b-instruct",
+          account_id: Optional[str] = None,
+          api_key: Optional[str] = None,
+          system_prompt: Optional[str] = None,
+          **kwargs,
+      ):
+```
+
+**Parameters:**
+
+- `model` (_str_): Model hosted on Cloudflare. Defaults to `@hf/meta-llama/meta-llama-3-8b-instruct`.
+- `account_id` (_Optional[str]_, _optional_): Account ID for Cloudflare. Defaults to None. Reads from environment variable `CLOUDFLARE_ACCOUNT_ID`.
+- `api_key` (_Optional[str]_, _optional_): API key for Cloudflare. Defaults to None. Reads from environment variable `CLOUDFLARE_API_KEY`.
+- `system_prompt` (_Optional[str]_, _optional_): System prompt to use for generation.
+
+### Methods
+
+Refer to [`dspy.OpenAI`](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI) documentation.

docs/api/language_model_clients/Groq.md

@@ -12,7 +12,7 @@ lm = dspy.GROQ(model='mixtral-8x7b-32768', api_key ="gsk_***" )
 
 ### Constructor
 
-The constructor initializes the base class `LM` and verifies the provided arguments like the `api_key` for GROQ api retriver. The `kwargs` attribute is initialized with default values for relevant text generation parameters needed for communicating with the GPT API, such as `temperature`, `max_tokens`, `top_p`, `frequency_penalty`, `presence_penalty`, and `n`.
+The constructor initializes the base class `LM` and verifies the provided arguments like the `api_key` for GROQ api retriver. The `kwargs` attribute is initialized with default values for relevant text generation parameters needed for communicating with the GROQ API, such as `temperature`, `max_tokens`, `top_p`, `frequency_penalty`, `presence_penalty`, and `n`.
 
 ```python
 class GroqLM(LM):
@@ -42,10 +42,10 @@ Internally, the method handles the specifics of preparing the request prompt and
 After generation, the generated content look like `choice["message"]["content"]`. 
 
 **Parameters:**
-- `prompt` (_str_): Prompt to send to OpenAI.
+- `prompt` (_str_): Prompt to send to GROQ.
 - `only_completed` (_bool_, _optional_): Flag to return only completed responses and ignore completion due to length. Defaults to True.
 - `return_sorted` (_bool_, _optional_): Flag to sort the completion choices using the returned averaged log-probabilities. Defaults to False.
 - `**kwargs`: Additional keyword arguments for completion request.
 
 **Returns:**
-- `List[Dict[str, Any]]`: List of completion choices.
+- `List[Dict[str, Any]]`: List of completion choices.

docs/api/language_model_clients/Snowflake.md

@@ -0,0 +1,45 @@
+---
+sidebar_position: 
+---
+
+# dspy.Snowflake
+
+### Usage
+
+```python
+import dspy
+import os
+
+connection_parameters = {
+
+    "account": os.getenv('SNOWFLAKE_ACCOUNT'),
+    "user": os.getenv('SNOWFLAKE_USER'),
+    "password": os.getenv('SNOWFLAKE_PASSWORD'),
+    "role": os.getenv('SNOWFLAKE_ROLE'),
+    "warehouse": os.getenv('SNOWFLAKE_WAREHOUSE'),
+    "database": os.getenv('SNOWFLAKE_DATABASE'),
+    "schema": os.getenv('SNOWFLAKE_SCHEMA')}
+
+lm = dspy.Snowflake(model="mixtral-8x7b",credentials=connection_parameters)
+```
+
+### Constructor
+
+The constructor inherits from the base class `LM` and verifies the `credentials` for using Snowflake API.
+
+```python
+class Snowflake(LM):
+    def __init__(
+        self, 
+        model,
+        credentials,
+        **kwargs):
+```
+
+**Parameters:**
+- `model` (_str_): model hosted by [Snowflake Cortex](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability).
+- `credentials`  (_dict_): connection parameters required to initialize a [snowflake snowpark session](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.Session)
+
+### Methods
+
+Refer to [`dspy.Snowflake`](https://dspy-docs.vercel.app/api/language_model_clients/Snowflake) documentation.

docs/api/retrieval_model_clients/SnowflakeRM.md

@@ -0,0 +1,79 @@
+---
+sidebar_position:
+---
+
+# retrieve.SnowflakeRM
+
+### Constructor
+
+Initialize an instance of the `SnowflakeRM` class, with the option to use `e5-base-v2` or `snowflake-arctic-embed-m` embeddings or any other Snowflake Cortex supported embeddings model.
+
+```python
+SnowflakeRM(
+     snowflake_table_name: str,
+     snowflake_credentials: dict,
+     k: int = 3,
+     embeddings_field: str,
+     embeddings_text_field:str,
+     embeddings_model: str = "e5-base-v2",
+)
+```
+
+**Parameters:**
+
+- `snowflake_table_name (str)`: The name of the Snowflake table containing embeddings.
+- `snowflake_credentials (dict)`: The connection parameters needed to initialize a Snowflake Snowpark Session.
+- `k (int, optional)`: The number of top passages to retrieve. Defaults to 3.
+- `embeddings_field (str)`: The name of the column in the Snowflake table containing the embeddings.
+- `embeddings_text_field (str)`: The name of the column in the Snowflake table containing the passages.
+- `embeddings_model (str)`: The model to be used to convert text to embeddings
+
+### Methods
+
+#### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction`
+
+Search the Snowflake table for the top `k` passages matching the given query or queries, using embeddings generated via the default `e5-base-v2` model or the specified `embedding_model`.
+
+**Parameters:**
+
+- `query_or_queries` (_Union[str, List[str]]_): The query or list of queries to search for.
+- `k` (_Optional[int]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization.
+
+**Returns:**
+
+- `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"id": str, "score": float, "long_text": str, "metadatas": dict }]`
+
+### Quickstart
+
+To support passage retrieval, it assumes that a Snowflake table has been created and populated with the passages in a column `embeddings_text_field` and the embeddings in another column `embeddings_field`
+
+SnowflakeRM uses `e5-base-v2` embeddings model by default or any Snowflake Cortex supported embeddings model.
+
+#### Default OpenAI Embeddings
+
+```python
+from dspy.retrieve.snowflake_rm import SnowflakeRM
+import os
+
+connection_parameters = {
+
+    "account": os.getenv('SNOWFLAKE_ACCOUNT'),
+    "user": os.getenv('SNOWFLAKE_USER'),
+    "password": os.getenv('SNOWFLAKE_PASSWORD'),
+    "role": os.getenv('SNOWFLAKE_ROLE'),
+    "warehouse": os.getenv('SNOWFLAKE_WAREHOUSE'),
+    "database": os.getenv('SNOWFLAKE_DATABASE'),
+    "schema": os.getenv('SNOWFLAKE_SCHEMA')}
+
+retriever_model = SnowflakeRM(
+    snowflake_table_name="<YOUR_SNOWFLAKE_TABLE_NAME>",
+    snowflake_credentials=connection_parameters,
+    embeddings_field="<YOUR_EMBEDDINGS_COLUMN_NAME>",
+    embeddings_text_field= "<YOUR_PASSAGE_COLUMN_NAME>"
+    )
+
+results = retriever_model("Explore the meaning of life", k=5)
+
+for result in results:
+    print("Document:", result.long_text, "\n")
+```

docs/docs/building-blocks/6-optimizers.md

@@ -27,40 +27,54 @@ DSPy programs consist of multiple calls to LMs, stacked together as [DSPy module
 
 Given a metric, DSPy can optimize all of these three with multi-stage optimization algorithms. These can combine gradient descent (for LM weights) and discrete LM-driven optimization, i.e. for crafting/updating instructions and for creating/validating demonstrations. DSPy Demonstrations are like few-shot examples, but they're far more powerful. They can be created from scratch, given your program, and their creation and selection can be optimized in many effective ways.
 
-In many cases, we found that compiling leads to better prompts than humans write. Not because DSPy optimizers are more creative than humans, but simply because they can try more things, much more systematically, and tune the metrics directly.
+In many cases, we found that compiling leads to better prompts than human writing. Not because DSPy optimizers are more creative than humans, but simply because they can try more things, much more systematically, and tune the metrics directly.
 
 
 ## What DSPy Optimizers are currently available?
 
+<!-- The following diagram was generated by: -->
+<!-- 1. Running symilar on the teleprompter module to extract the python hierarchy as a Graphviz dot file -->
+<!-- 2. Hand-editing the resulting dot file to remove classes that are not teleprompters/optimizers (e.g., classes for data structures manipulated by optimizers). -->
+<!-- 3. Using dot to compile the `.dot` file into a PNG -->
+<!-- Robert Goldman [2024/05/11:rpg] -->
+
+[Subclasses of Teleprompter](figures/teleprompter-classes.png)
+
 All of these can be accessed via `from dspy.teleprompt import *`.
 
 #### Automatic Few-Shot Learning
 
-1. **`LabeledFewShot`**: Simply constructs few-shot examples from provided labeled Q/A pairs.
+These optimizers extend the signature by automatically generating and including **optimized** examples within the prompt sent to the model, implementing few-shot learning.
+
+1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled input and output data points.  Requires `k` (number of examples for the prompt) and `trainset` to randomly select `k` examples from.
+
+2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program, along with labeled examples in `trainset`. Parameters include `max_labeled_demos` (the number of demonstrations randomly selected from the `trainset`) and `max_bootstrapped_demos` (the number of additional examples generated by the `teacher`). The bootstrapping process employs the metric to validate demonstrations, including only those that pass the metric in the "compiled" prompt. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.
 
-2. **`BootstrapFewShot`**: Uses your program to self-generate complete demonstrations for every stage of your program. Will simply use the generated demonstrations (if they pass the metric) without any further optimization. Advanced: Supports using a teacher program (a different DSPy program that has compatible structure) and a teacher LM, for harder tasks.
+3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of `BootstrapFewShot`, with the addition of `num_candidate_programs`, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, `LabeledFewShot` optimized program, `BootstrapFewShot` compiled program with unshuffled examples and `num_candidate_programs` of `BootstrapFewShot` compiled programs with randomized example sets. 
 
-3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program.
+4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` with Optuna optimization across demonstration sets, running trials to maximize evaluation metrics and selecting the best demonstrations. 
 
-4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics. 
+5. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm to pick a diverse set of examples from different clusters.  Vectorizes the examples, and then clusters them, using cluster centers with `BootstrapFewShot` for bootstrapping/selection process.  This will be  useful when there's a lot of data over random spaces: using KNN helps optimize the `trainset`  for `BootstrapFewShot`.  See [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/knn.ipynb) for an example.
 
 
 #### Automatic Instruction Optimization
 
-4. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent.
+These optimizers produce optimal instructions for the prompt and, in the case of MIPRO also optimize the set of few-shot demonstrations.
 
-5. **`MIPRO`**: Generates instructions and few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
+6. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). Parameters include `depth` which is the number of iterations of prompt improvement the optimizer runs over.
+
+7. **`MIPRO`**: Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
 
 
 #### Automatic Finetuning
 
+This optimizer is used to fine-tune the underlying LLM(s).
+
 6. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.
 
 
 #### Program Transformations
 
-7. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm integrating `BootstrapFewShot` for bootstrapping/selection process. 
-
 8. **`Ensemble`**: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.
 
 
@@ -90,7 +104,7 @@ from dspy.teleprompt import BootstrapFewShotWithRandomSearch
 
 # Set up the optimizer: we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps.
 # The optimizer will repeat this 10 times (plus some initial attempts) before selecting its best attempt on the devset.
-config = dict(max_bootstrapped_demos=3, max_labeled_demos=3, num_candidate_programs=10, num_threads=4)
+config = dict(max_bootstrapped_demos=4, max_labeled_demos=4, num_candidate_programs=10, num_threads=4)
 
 teleprompter = BootstrapFewShotWithRandomSearch(metric=YOUR_METRIC_HERE, **config)
 optimized_program = teleprompter.compile(YOUR_PROGRAM_HERE, trainset=YOUR_TRAINSET_HERE)
@@ -115,4 +129,4 @@ To load a program from a file, you can instantiate an object from that class and
 ```python
 loaded_program = YOUR_PROGRAM_CLASS()
 loaded_program.load(path=YOUR_SAVE_PATH)
-```
+```

docs/docs/building-blocks/figures/teleprompter-classes.png

@@ -7,6 +7,7 @@
 from .azure_openai import AzureOpenAI
 from .cache_utils import *
 from .clarifai import *
+from .cloudflare import *
 from .cohere import *
 from .colbertv2 import ColBERTv2
 from .databricks import *
@@ -22,4 +23,6 @@
 from .pyserini import *
 from .sbert import *
 from .sentence_vectorizer import *
+from .snowflake import *
 from .watsonx import *
+

dsp/modules/__init__.py

@@ -62,7 +62,7 @@ def __init__(
         }
         self.kwargs["model"] = model
         self.history: list[dict[str, Any]] = []
-        self.client = Anthropic(api_key=api_key)
+        self.client = Anthropic(api_key=api_key, base_url=api_base)
 
     def log_usage(self, response):
         """Log the total tokens from the Anthropic API response."""

dsp/modules/anthropic.py

@@ -3,6 +3,31 @@
 from abc import ABC, abstractmethod
 from typing import Any, Optional
 
+import backoff
+
+try:
+    import boto3
+    from botocore.exceptions import ClientError
+    ERRORS = (ClientError,)
+
+except ImportError:
+    ERRORS = (Exception,)
+
+
+def backoff_hdlr(details):
+    """Handler from https://pypi.org/project/backoff/."""
+    print(
+        "Backing off {wait:0.1f} seconds after {tries} tries "
+        "calling function {target} with kwargs "
+        "{kwargs}".format(**details),
+    )
+
+
+def giveup_hdlr(details):
+    """Wrapper function that decides when to give up on retry."""
+    if "max retries" in details.args[0]:
+        return False
+    return True
 
 class AWSProvider(ABC):
     """This abstract class adds support for AWS model providers such as Bedrock and SageMaker.
@@ -52,6 +77,14 @@ def get_provider_name(self) -> str:
         return self.__class__.__name__
 
     @abstractmethod
+    @backoff.on_exception(
+        backoff.expo,
+        ERRORS,
+        max_time=1000,
+        max_tries=8,
+        on_backoff=backoff_hdlr,
+        giveup=giveup_hdlr,
+    )
     def call_model(self, model_id: str, body: str) -> str:
         """Call the model and return the response."""
 
@@ -119,6 +152,14 @@ def __init__(
         """
         super().__init__(region_name, "runtime.sagemaker", profile_name)
 
+    @backoff.on_exception(
+        backoff.expo,
+        ERRORS,
+        max_time=1000,
+        max_tries=8,
+        on_backoff=backoff_hdlr,
+        giveup=giveup_hdlr,
+    )
     def call_model(self, model_id: str, body: str) -> str:
         return self.predictor.invoke_endpoint(
             EndpointName=model_id,