WIP: Assertions FAQ

[okhat]

Gathering questions about assertions and suggestions from different users.

This will be most relevant for @Shangyint @manishshettym @arnavsinghvi11.

1. What's changing between different versions? What will be the final stable API for everything below?

2. Do users need to manually set trace=[]. Can this be automated?

3. Does dspy.Assert retry? If not, shouldn't it?

4. What happens if I don't set target_module? Does it directly re-run the last module? What's confusing people here currently.

5. Is it true that you can't really pass a module to the target_module , it must be it's signature?

6. I saved my compiled program with assertions. When I load it (like code below), I get DSPySuggestionError.

loaded_compiled_module = MyModuleWithAssertions()
loaded_compiled_module.load(path="compiled_mods_assertions/BootstrapFewShotWithRandomSearch_With_Assertions.json")
# MyModuleWithAssertions embeds dspy.Suggest 
module_with_assertions = assert_transform_module(MyModuleWithAssertions().map_named_predictors(Retry), backtrack_handler) 

[arnavsinghvi11]

1. Assertions API is currently consistent with the existing documentation, which will be updated if any other changes are made!

2. Yes this should actually be configured by default, just modifying trace=[] here which will resolve issues like Assert transform module doesn't work? #432. @okhat let me know if this change can be added without breaking other existing behavior!

3. dspy.Assert performs the same retrying attempts as dspy.Suggest, with the difference only lying in if the validation function is not passed even when the max_backtracking_attempts are reached, dspy.Assert will trigger a DSPyAssertionError while dspy.Suggestion will log this error but continue program execution.

One thing to note is once an DSPyAssertionError/DSPySuggestionError is triggered at any point, this is considered as an "attempt." This can have some implications with multiple Assertion statements as if a particular statement continually produces an invalid response, the result will error out on that corresponding statement(if using dspy.Assert statements) or continue to the next examples (if using dspy.Suggest statements) without having the output fully tested on the rest of the Assertion statements.
This behavior is due to the "best-effort" nature of the Assertions backtracking in trying to ensure the constraints are passed in sequential order. The user is always alerted of which constraint it has failed at when using dspy.Assert as indicated by the feedback message.

4. The default behavior without setting target_module is to re-run the last module stored in the trace. We recommend setting this however to ensure controlled behavior of the assertions backtracking, especially when DSPy programs that involve multiple modules calls

5. So technically you can't pass a module as the target_module but internally, we are identifying the corresponding trace module with the Retry wrapped module by the signature (since we check equality through this - e.g. ChainOfThought(GenAnswer) vs Retry(GenAnswer)), hence how/why specifying a signature type identifier from your module corresponds to it being the target_module within assertions internals. I get that the naming is a bit funky so open to any suggestions! 😉

6. You'll actually want to load in the saved compiled program with assertions to module_with_assertions, not loaded_compiled_module as any program using Assertions has to be initialized with the assert_transform_module.

These are great FAQs for Assertions! Feel free to drop any follow-ups or additional questions!

[neoxelox]

Hey @arnavsinghvi11 thanks for answering so fast these questions. I want to comment about 3 & 5.

3. I can't get Assert working in my programs. I am getting this error: AttributeError: 'DSPyAssertionError' object has no attribute 'target_module' everytime even after setting the target module manually. After taking a look at the underlying code, seems that Assert misses the functionality to retry, I guess that is unexpected? I see that DSPyAssertionError is actually not raised with the stored target_module in the Constraint parent class: dspy/primitives/assertions.py:99:17 :

DSPyAssertionError(
    id=self.id,
    msg=self.msg,
    state=dsp.settings.trace,
    is_metric=self.is_metric,
)

this causes dspy/primitives/assertions.py:242:29 : e.target_module to crash.

5. It would be very helpful to either rename target_module to target_signature or to be able to pass a real instantiated module to the target_module, this would make the API more clear. I prefer the second one because I guess that different modules can have the same signature, so how does DSPy discern between those? I see that this check is made at dspy/primitives/assertions.py:257:37 : if mod.signature == suggest_target_module:. If I understood you correctly, mod becomes an instance of Retry when doing .map_named_predictors(dspy.Retry) to activate the assertions. However, I can see that Retry does store the underlying wrapped module in self.module so we could compare mod.module with target_module instead of comparing signatures.

[arnavsinghvi11]

Hey @neoxelox

For #3, I believe this was patched in a recent commit. Could you pull or maybe rebuild from the repo?

Great point regarding #5. I think in the case of multiple modules with the same signature, it would be advised to keep the corresponding Assertion statement after the relevant module, since by default, it will trace back to the most recent one. (However, naturally, there are edge cases where the multiple modules have unrelated outputs but the same signature so we'll have to integrate a stronger comparison in that case). leaving this open for now.