New approach to multimodal document ingestion (#2558)

* Prepare change for multimodal, rm old vision approach stuff

* Add LLM-based media describer

* Prepdocs progress

* Fix media description with OpenAI

* More prepdocs improvements for image handling

* Store bbox as list of pixel floats, add storage container just for extracted images

* Getting image citations almost working

* More progress on multimodal approach

* Update more tests

* Fix up more app tests

* Add test for upload_document_image

* Add media describer and embeddings tests

* Fix tests for vision, work on vectorizer

* Add font, rename multimodal doc

* Update links to multimodal

* Fix import

* Doc fixes

* Fix f-string syntax

* Markdown lint issues

* mypy fixes and reasoning fixes

* Rename vision variables, fix mypy

* Mypy fixes

* Fix all mypy issues

* Fixes to sidebar so that it all fits

* Fixes to sidebar so that it all fits

* Integrated vectorization and user upload work

* Progress on user upload support

* changes needed for user upload

* Update tests

* Integrated vectorization progress

* Fix tests

* Use ImageEmbeddings client directly

* Change frontend for vector fields

* Use boolean parameters in the backend as well, for vector fields

* Updated translations

* Change frontend for LLM inputs

* Change from LLM inputs to booleans

* Working on tests

* Blob manager improvements/tests

* Change to a global client that we close in lifespan

* Add latest int vect changes

* Update the tests

* Add as_bytes option

* Mypy fixes

* Mypy fixes

* More mypy fixes

* More mypy fixes

* Address more TODOs

* Fix E2E tests

* Add more tests for blobmanger

* Markdown fix, more coverage

* Fix broken MD link

* Increase coverage

* Increase test coverage

* Add diff-cover step to python test

* Fix diff-cover action

* Fetch origin main for diff-cover

* Increase test coverage

* More tests, Windows check

* Better copilot instructions

* Updated merge

* Revert integrated vectorization changes, using a different strategy

* Remove unused identity in int vect

* Refactor get_sources_content to return DataPoints

* Fix multimodal bicep variable error

* Fix prepdocs to properly close async clients

* better CSS for image URLs and images in Thought Process and Supporting Content

* Revert logging level to WARNING as before

* Update text splitter chunking logic and add full test coverage

* Use single token char

* Fix mypy error

* Add some helper functions and modules to improve code clarity for textsplitter

* Update splitting algorithm with better overlap algorithm, rename SplitPage to Chunk

* markdown issues

* Revise multimodal doc to be clearer

* Rephrase fragment shift to be more grokkable

* Reword duplicate part of textsplitter doc

Author: pamelafox

.azdo/pipelines/azure-dev.yml

@@ -77,11 +77,6 @@ steps:
       AZURE_OPENAI_EMB_DEPLOYMENT_VERSION: $(AZURE_OPENAI_EMB_DEPLOYMENT_VERSION)
       AZURE_OPENAI_EMB_DEPLOYMENT_SKU: $(AZURE_OPENAI_EMB_DEPLOYMENT_SKU)
       AZURE_OPENAI_EMB_DIMENSIONS: $(AZURE_OPENAI_EMB_DIMENSIONS)
-      AZURE_OPENAI_GPT4V_MODEL: $(AZURE_OPENAI_GPT4V_MODEL)
-      AZURE_OPENAI_GPT4V_DEPLOYMENT: $(AZURE_OPENAI_GPT4V_DEPLOYMENT)
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY: $(AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY)
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION: $(AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION)
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU: $(AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU)
       AZURE_OPENAI_DISABLE_KEYS: $(AZURE_OPENAI_DISABLE_KEYS)
       OPENAI_HOST: $(OPENAI_HOST)
       OPENAI_API_KEY: $(OPENAI_API_KEY)
@@ -91,13 +86,13 @@ steps:
       AZURE_APPLICATION_INSIGHTS_DASHBOARD: $(AZURE_APPLICATION_INSIGHTS_DASHBOARD)
       AZURE_LOG_ANALYTICS: $(AZURE_LOG_ANALYTICS)
       USE_VECTORS: $(USE_VECTORS)
-      USE_GPT4V: $(USE_GPT4V)
+      USE_MULTIMODAL: $(USE_MULTIMODAL)
       AZURE_VISION_ENDPOINT: $(AZURE_VISION_ENDPOINT)
       VISION_SECRET_NAME: $(VISION_SECRET_NAME)
-      AZURE_COMPUTER_VISION_SERVICE: $(AZURE_COMPUTER_VISION_SERVICE)
-      AZURE_COMPUTER_VISION_RESOURCE_GROUP: $(AZURE_COMPUTER_VISION_RESOURCE_GROUP)
-      AZURE_COMPUTER_VISION_LOCATION: $(AZURE_COMPUTER_VISION_LOCATION)
-      AZURE_COMPUTER_VISION_SKU: $(AZURE_COMPUTER_VISION_SKU)
+      AZURE_VISION_SERVICE: $(AZURE_VISION_SERVICE)
+      AZURE_VISION_RESOURCE_GROUP: $(AZURE_VISION_RESOURCE_GROUP)
+      AZURE_VISION_LOCATION: $(AZURE_VISION_LOCATION)
+      AZURE_VISION_SKU: $(AZURE_VISION_SKU)
       ENABLE_LANGUAGE_PICKER: $(ENABLE_LANGUAGE_PICKER)
       USE_SPEECH_INPUT_BROWSER: $(USE_SPEECH_INPUT_BROWSER)
       USE_SPEECH_OUTPUT_BROWSER: $(USE_SPEECH_OUTPUT_BROWSER)
@@ -126,6 +121,10 @@ steps:
       AZURE_CONTAINER_APPS_WORKLOAD_PROFILE: $(AZURE_CONTAINER_APPS_WORKLOAD_PROFILE)
       USE_CHAT_HISTORY_BROWSER: $(USE_CHAT_HISTORY_BROWSER)
       USE_MEDIA_DESCRIBER_AZURE_CU: $(USE_MEDIA_DESCRIBER_AZURE_CU)
+      RAG_SEARCH_TEXT_EMBEDDINGS: $(RAG_SEARCH_TEXT_EMBEDDINGS)
+      RAG_SEARCH_IMAGE_EMBEDDINGS: $(RAG_SEARCH_IMAGE_EMBEDDINGS)
+      RAG_SEND_TEXT_SOURCES: $(RAG_SEND_TEXT_SOURCES)
+      RAG_SEND_IMAGE_SOURCES: $(RAG_SEND_IMAGE_SOURCES)
   - task: AzureCLI@2
     displayName: Deploy Application
     inputs:

.github/chatmodes/fixer.chatmode.md

@@ -26,7 +26,12 @@ You MUST check task output readiness before debugging, testing, or declaring wor
 	- If watchers seem stuck or output stops updating, stop the tasks and run the "Development" task again.
 - To interact with a running application, use the Playwright MCP server. If testing login, you will need to navigate to 'localhost' instead of '127.0.0.1' since that's the URL allowed by the Entra application.
 
+## Running Python scripts
+
+If you are running Python scripts that depend on installed requirements, you must run them using the virtual environment in `.venv`.
+
 ## Committing the change
 
 When change is complete, offer to make a new branch, git commit, and pull request.
+(DO NOT check out a new branch unless explicitly confirmed - sometimes user is already in a branch)
 Make sure the PR follows the PULL_REQUEST_TEMPLATE.md format, with all sections filled out and appropriate checkboxes checked.

.github/copilot-instructions.md

@@ -1,3 +1,33 @@
+# Overall code layout
+
+* app: Contains the main application code, including frontend and backend.
+  * app/backend: Contains the Python backend code, written with Quart framework.
+    * app/backend/approaches: Contains the different approaches
+      * app/backend/approaches/approach.py: Base class for all approaches
+      * app/backend/approaches/retrievethenread.py: Ask approach, just searches and answers
+      * app/backend/approaches/chatreadretrieveread.py: Chat approach, includes query rewriting step first
+      * app/backend/approaches/prompts/ask_answer_question.prompty: Prompt used by the Ask approach to answer the question based off sources
+      * app/backend/approaches/prompts/chat_query_rewrite.prompty: Prompt used to rewrite the query based off search history into a better search query
+      * app/backend/approaches/prompts/chat_query_rewrite_tools.json: Tools used by the query rewriting prompt
+      * app/backend/approaches/prompts/chat_answer_question.prompty: Prompt used by the Chat approach to actually answer the question based off sources
+    * app/backend/app.py: The main entry point for the backend application.
+  * app/frontend: Contains the React frontend code, built with TypeScript, built with vite.
+    * app/frontend/src/api: Contains the API client code for communicating with the backend.
+    * app/frontend/src/components: Contains the React components for the frontend.
+    * app/frontend/src/locales: Contains the translation files for internationalization.
+      * app/frontend/src/locales/da/translation.json: Danish translations
+      * app/frontend/src/locales/en/translation.json: English translations
+      * app/frontend/src/locales/es/translation.json: Spanish translations
+      * app/frontend/src/locales/fr/translation.json: French translations
+      * app/frontend/src/locales/it/translation.json: Italian translations
+      * app/frontend/src/locales/ja/translation.json: Japanese translations
+      * app/frontend/src/locales/nl/translation.json: Dutch translations
+      * app/frontend/src/locales/ptBR/translation.json: Portuguese translations
+      * app/frontend/src/locales/tr/translation.json: Turkish translations
+    * app/frontend/src/pages: Contains the main pages of the application
+* infra: Contains the Bicep templates for provisioning Azure resources.
+* tests: Contains the test code, including e2e tests, app integration tests, and unit tests.
+
 # Adding new data
 
 New files should be added to the `data` folder, and then either run scripts/prepdocs.sh or scripts/prepdocs.ps1 to ingest the data.

.github/workflows/azure-dev.yml

@@ -37,10 +37,10 @@ jobs:
       AZURE_DOCUMENTINTELLIGENCE_RESOURCE_GROUP: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_RESOURCE_GROUP }}
       AZURE_DOCUMENTINTELLIGENCE_SKU: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_SKU }}
       AZURE_DOCUMENTINTELLIGENCE_LOCATION: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_LOCATION }}
-      AZURE_COMPUTER_VISION_SERVICE: ${{ vars.AZURE_COMPUTER_VISION_SERVICE }}
-      AZURE_COMPUTER_VISION_RESOURCE_GROUP: ${{ vars.AZURE_COMPUTER_VISION_RESOURCE_GROUP }}
-      AZURE_COMPUTER_VISION_LOCATION: ${{ vars.AZURE_COMPUTER_VISION_LOCATION }}
-      AZURE_COMPUTER_VISION_SKU: ${{ vars.AZURE_COMPUTER_VISION_SKU }}
+      AZURE_VISION_SERVICE: ${{ vars.AZURE_VISION_SERVICE }}
+      AZURE_VISION_RESOURCE_GROUP: ${{ vars.AZURE_VISION_RESOURCE_GROUP }}
+      AZURE_VISION_LOCATION: ${{ vars.AZURE_VISION_LOCATION }}
+      AZURE_VISION_SKU: ${{ vars.AZURE_VISION_SKU }}
       AZURE_SEARCH_INDEX: ${{ vars.AZURE_SEARCH_INDEX }}
       AZURE_SEARCH_SERVICE: ${{ vars.AZURE_SEARCH_SERVICE }}
       AZURE_SEARCH_SERVICE_RESOURCE_GROUP: ${{ vars.AZURE_SEARCH_SERVICE_RESOURCE_GROUP }}
@@ -67,11 +67,6 @@ jobs:
       AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY: ${{ vars.AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY }}
       AZURE_OPENAI_EMB_DEPLOYMENT_VERSION: ${{ vars.AZURE_OPENAI_EMB_DEPLOYMENT_VERSION }}
       AZURE_OPENAI_EMB_DIMENSIONS: ${{ vars.AZURE_OPENAI_EMB_DIMENSIONS }}
-      AZURE_OPENAI_GPT4V_MODEL: ${{ vars.AZURE_OPENAI_GPT4V_MODEL }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU }}
       USE_EVAL: ${{ vars.USE_EVAL }}
       AZURE_OPENAI_EVAL_MODEL: ${{ vars.AZURE_OPENAI_EVAL_MODEL }}
       AZURE_OPENAI_EVAL_MODEL_VERSION: ${{ vars.AZURE_OPENAI_EVAL_MODEL_VERSION }}
@@ -87,7 +82,7 @@ jobs:
       AZURE_APPLICATION_INSIGHTS_DASHBOARD: ${{ vars.AZURE_APPLICATION_INSIGHTS_DASHBOARD }}
       AZURE_LOG_ANALYTICS: ${{ vars.AZURE_LOG_ANALYTICS }}
       USE_VECTORS: ${{ vars.USE_VECTORS }}
-      USE_GPT4V: ${{ vars.USE_GPT4V }}
+      USE_MULTIMODAL: ${{ vars.USE_MULTIMODAL }}
       AZURE_VISION_ENDPOINT: ${{ vars.AZURE_VISION_ENDPOINT }}
       VISION_SECRET_NAME: ${{ vars.VISION_SECRET_NAME }}
       ENABLE_LANGUAGE_PICKER: ${{ vars.ENABLE_LANGUAGE_PICKER }}
@@ -116,6 +111,10 @@ jobs:
       USE_CHAT_HISTORY_BROWSER: ${{ vars.USE_CHAT_HISTORY_BROWSER }}
       USE_MEDIA_DESCRIBER_AZURE_CU: ${{ vars.USE_MEDIA_DESCRIBER_AZURE_CU }}
       USE_AI_PROJECT: ${{ vars.USE_AI_PROJECT }}
+      RAG_SEARCH_TEXT_EMBEDDINGS: ${{ vars.RAG_SEARCH_TEXT_EMBEDDINGS }}
+      RAG_SEARCH_IMAGE_EMBEDDINGS: ${{ vars.RAG_SEARCH_IMAGE_EMBEDDINGS }}
+      RAG_SEND_TEXT_SOURCES: ${{ vars.RAG_SEND_TEXT_SOURCES }}
+      RAG_SEND_IMAGE_SOURCES: ${{ vars.RAG_SEND_IMAGE_SOURCES }}
     steps:
       - name: Checkout
         uses: actions/checkout@v5

.github/workflows/evaluate.yaml

@@ -35,10 +35,10 @@ jobs:
       AZURE_DOCUMENTINTELLIGENCE_RESOURCE_GROUP: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_RESOURCE_GROUP }}
       AZURE_DOCUMENTINTELLIGENCE_SKU: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_SKU }}
       AZURE_DOCUMENTINTELLIGENCE_LOCATION: ${{ vars.AZURE_DOCUMENTINTELLIGENCE_LOCATION }}
-      AZURE_COMPUTER_VISION_SERVICE: ${{ vars.AZURE_COMPUTER_VISION_SERVICE }}
-      AZURE_COMPUTER_VISION_RESOURCE_GROUP: ${{ vars.AZURE_COMPUTER_VISION_RESOURCE_GROUP }}
-      AZURE_COMPUTER_VISION_LOCATION: ${{ vars.AZURE_COMPUTER_VISION_LOCATION }}
-      AZURE_COMPUTER_VISION_SKU: ${{ vars.AZURE_COMPUTER_VISION_SKU }}
+      AZURE_VISION_SERVICE: ${{ vars.AZURE_VISION_SERVICE }}
+      AZURE_VISION_RESOURCE_GROUP: ${{ vars.AZURE_VISION_RESOURCE_GROUP }}
+      AZURE_VISION_LOCATION: ${{ vars.AZURE_VISION_LOCATION }}
+      AZURE_VISION_SKU: ${{ vars.AZURE_VISION_SKU }}
       AZURE_SEARCH_INDEX: ${{ vars.AZURE_SEARCH_INDEX }}
       AZURE_SEARCH_SERVICE: ${{ vars.AZURE_SEARCH_SERVICE }}
       AZURE_SEARCH_SERVICE_RESOURCE_GROUP: ${{ vars.AZURE_SEARCH_SERVICE_RESOURCE_GROUP }}
@@ -62,11 +62,6 @@ jobs:
       AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY: ${{ vars.AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY }}
       AZURE_OPENAI_EMB_DEPLOYMENT_VERSION: ${{ vars.AZURE_OPENAI_EMB_DEPLOYMENT_VERSION }}
       AZURE_OPENAI_EMB_DIMENSIONS: ${{ vars.AZURE_OPENAI_EMB_DIMENSIONS }}
-      AZURE_OPENAI_GPT4V_MODEL: ${{ vars.AZURE_OPENAI_GPT4V_MODEL }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_CAPACITY }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_VERSION }}
-      AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU: ${{ vars.AZURE_OPENAI_GPT4V_DEPLOYMENT_SKU }}
       USE_EVAL: ${{ vars.USE_EVAL }}
       AZURE_OPENAI_EVAL_MODEL: ${{ vars.AZURE_OPENAI_EVAL_MODEL }}
       AZURE_OPENAI_EVAL_MODEL_VERSION: ${{ vars.AZURE_OPENAI_EVAL_MODEL_VERSION }}
@@ -82,7 +77,7 @@ jobs:
       AZURE_APPLICATION_INSIGHTS_DASHBOARD: ${{ vars.AZURE_APPLICATION_INSIGHTS_DASHBOARD }}
       AZURE_LOG_ANALYTICS: ${{ vars.AZURE_LOG_ANALYTICS }}
       USE_VECTORS: ${{ vars.USE_VECTORS }}
-      USE_GPT4V: ${{ vars.USE_GPT4V }}
+      USE_MULTIMODAL: ${{ vars.USE_MULTIMODAL }}
       AZURE_VISION_ENDPOINT: ${{ vars.AZURE_VISION_ENDPOINT }}
       VISION_SECRET_NAME: ${{ vars.VISION_SECRET_NAME }}
       ENABLE_LANGUAGE_PICKER: ${{ vars.ENABLE_LANGUAGE_PICKER }}

.github/workflows/python-test.yaml

@@ -64,7 +64,7 @@ jobs:
           run: black . --check --verbose
         - name: Run Python tests
           if: runner.os != 'Windows'
-          run: pytest -s -vv --cov --cov-report=xml --cov-fail-under=89
+          run: pytest -s -vv --cov --cov-report=xml --cov-fail-under=90
         - name: Check diff coverage
           if: runner.os != 'Windows'
           run: |

CONTRIBUTING.md

@@ -73,7 +73,7 @@ Once tests are passing, generate a coverage report to make sure your changes are
 
 ```shell
 pytest --cov --cov-report=xml && \
-diff-cover coverage.xml --format html:coverage_report.html && \
+diff-cover coverage.xml --html-report coverage_report.html && \
 open coverage_report.html
 ```
 

README.md

@@ -61,7 +61,7 @@ The repo includes sample data so it's ready to try end to end. In this sample ap
 - Renders citations and thought process for each answer
 - Includes settings directly in the UI to tweak the behavior and experiment with options
 - Integrates Azure AI Search for indexing and retrieval of documents, with support for [many document formats](/docs/data_ingestion.md#supported-document-formats) as well as [integrated vectorization](/docs/data_ingestion.md#overview-of-integrated-vectorization)
-- Optional usage of [GPT-4 with vision](/docs/gpt4v.md) to reason over image-heavy documents
+- Optional usage of [multimodal models](/docs/multimodal.md) to reason over image-heavy documents
 - Optional addition of [speech input/output](/docs/deploy_features.md#enabling-speech-inputoutput) for accessibility
 - Optional automation of [user login and data access](/docs/login_and_acl.md) via Microsoft Entra
 - Performance tracing and monitoring with Application Insights
@@ -92,7 +92,7 @@ However, you can try the [Azure pricing calculator](https://azure.com/e/e3490de2
 - Azure AI Search: Basic tier, 1 replica, free level of semantic search. Pricing per hour. [Pricing](https://azure.microsoft.com/pricing/details/search/)
 - Azure Blob Storage: Standard tier with ZRS (Zone-redundant storage). Pricing per storage and read operations. [Pricing](https://azure.microsoft.com/pricing/details/storage/blobs/)
 - Azure Cosmos DB: Only provisioned if you enabled [chat history with Cosmos DB](docs/deploy_features.md#enabling-persistent-chat-history-with-azure-cosmos-db). Serverless tier. Pricing per request unit and storage. [Pricing](https://azure.microsoft.com/pricing/details/cosmos-db/)
-- Azure AI Vision: Only provisioned if you enabled [GPT-4 with vision](docs/gpt4v.md). Pricing per 1K transactions. [Pricing](https://azure.microsoft.com/pricing/details/cognitive-services/computer-vision/)
+- Azure AI Vision: Only provisioned if you enabled [multimodal approach](docs/multimodal.md). Pricing per 1K transactions. [Pricing](https://azure.microsoft.com/pricing/details/cognitive-services/computer-vision/)
 - Azure AI Content Understanding: Only provisioned if you enabled [media description](docs/deploy_features.md#enabling-media-description-with-azure-content-understanding). Pricing per 1K images. [Pricing](https://azure.microsoft.com/pricing/details/content-understanding/)
 - Azure Monitor: Pay-as-you-go tier. Costs based on data ingested. [Pricing](https://azure.microsoft.com/pricing/details/monitor/)
 
@@ -255,7 +255,7 @@ You can find extensive documentation in the [docs](docs/README.md) folder:
   - [Enabling optional features](docs/deploy_features.md)
     - [All features](docs/deploy_features.md)
     - [Login and access control](docs/login_and_acl.md)
-    - [GPT-4 Turbo with Vision](docs/gpt4v.md)
+    - [Multimodal](docs/multimodal.md)
     - [Reasoning](docs/reasoning.md)
     - [Private endpoints](docs/deploy_private.md)
     - [Agentic retrieval](docs/agentic_retrieval.md)