feat: receive pre-messages and adapt download on demand #7431

Simon-Laux · 2025-11-12T15:07:45Z

This PR implements receiving of pre-message, download of full messages and the new logic for deciding what to fetch when from IMAP.

This is part of #7367

progress:

specific aspects that should become reality:

In receive_imf, the full-message can be handled similarly to Chat-Edit (grep for "ChatEdit" in receive_imf.rs).
As opposed to the current download-on-demand, we won't replace the whole msgs row, because this way, it will just work if an edit-message is received, and then later the user requests to download the attachment.
- Need to change Viewtype, probably metadata params like image size
pre-message gets Viewtype::Text in the database until downloaded
background_fetch should not download messages with the Chat-Is-Full-Message header.
~~background_fetch: If a large message without Chat-Is-Full-Message header. is received, the UI should be notified via an event.~~ postponed
A read receipt should be shown already when the user saw the pre-message. feat: pre messages can now get read receipts #7433

API Changes:

all: change Message.get_filebytes - if message is not downloaded pre message, and metadata is available, then it returns the size that the file has after download, so ui can use this to show download size
duration and dimensions (width, heigth) are already set in pre-messages (if they contain that metadata)
rust: add Message.get_full_message_viewtype to get view type that message will have after being downloaded.
jsonrpc: add full_message_view_type to Message and MessageInfo

move `get_msg_id_by_rfc724_mid` to `MsgId::get_by_rfc724_mid`

`message::rfc724_mid_exists`

side-effects.

pre_message

already fill from the metadata. Also add a new api to check what viewtype the message will have once downloaded.

`MessageInfo`

full-message metadata params

header in the unencrypted headers before)

replacement for full messages

src/message.rs

src/test_utils.rs

the pre-message

test_webxdc_update_for_not_downloaded_instance

Simon-Laux · 2025-11-30T20:12:52Z

src/message.rs

+                "DELETE FROM download WHERE rfc724_mid=?",
+                (&msg.rfc724_mid,),
+            )?;
+            // TODO: is the following nessesary?


TODO: answer the question

Hocuri · 2025-12-01T11:48:37Z

background_fetch: If a large message without Chat-Is-Full-Message header. is received, the UI should be notified via an event.

rust: add Message.get_full_message_viewtype to get view type that message will have after being downloaded.

jsonrpc: add full_message_view_type to Message and MessageInfo

In the first iteration, it's not necessary to emit an event, or add any new API. (not sure how the event landed in the "steps" at #7367, but it already is in the "Future Possibilities")

src/context.rs

Simon-Laux · 2025-12-01T18:12:54Z

src/download.rs

+
+/// Download known full messages without pre_message
+/// in order to guard against lost pre-messages:
+// TODO better fn name


any ideas for a better function name?

Simon-Laux · 2025-12-01T18:15:21Z

src/imap.rs


        chat::mark_old_messages_as_noticed(context, received_msgs).await?;

+        // TODO: is there correct place for this?


address to do comment

src/mimefactory.rs

Simon-Laux · 2025-12-01T18:18:32Z

src/mimeparser.rs

+            if let Some(full_msg_rfc724_mid) =
+                mail.headers.get_header_value(HeaderDef::ChatFullMessageId)
+            {
+                // TODO: is there a better method for this task? (removing `<>` delimiters of RFC message ID)


address to do comment

Simon-Laux · 2025-12-01T18:43:27Z

src/scheduler.rs

            .context("delete_expired_imap_messages")?;
+
+        //-------
+        // TODO: verify that this is the correct position for this call


address todo comment

Simon-Laux · 2025-12-01T18:59:07Z

Progress of the tests

Overview about the recycled(♻️) and dropped(🗑️) tests of the tests that I removed in #7373

🗑️test_reaction_to_partially_fetched_msg
🗑️ test_partial_download_and_ephemeral
🗑️ test_status_update_expands_to_nothing
♻️ test_partial_download_trashed
♻️ test_partial_download_and_reaction
♻️ test_partial_group_consistency
🗑️ test_partial_download_key_contact_lookup
♻️ test_webxdc_update_for_not_downloaded_instance
🗑️test_no_partial_calls
♻️ test_markseen_not_downloaded_msg
♻️ test_msg_seen_on_imap_when_downloaded
♻️ test_create_group_with_big_msg
♻️ test_download_later
🗑️ test_prefer_references_to_downloaded_msgs
♻️test_download_on_demand
♻️test_download_limit_chat_assignment
🗑️test_webxdc_download_on_demand

All the recycled tests were already recycled/re-made except for some which are still to do:

test_partial_group_consistency
test_msg_seen_on_imap_when_downloaded
test_download_limit_chat_assignment

Furthermore there need to be new tests to test the downloading/scheduling changes.

This was referenced Nov 12, 2025

[Tracking Issue] Pre-Messages #7367

Open

[Branch] pre-messages / next version of download on demand #7371

Draft

Simon-Laux force-pushed the simon/sending-pre-message branch 3 times, most recently from 2b900c3 to 305498d Compare November 16, 2025 20:07

Simon-Laux force-pushed the simon/receive-pre-messages branch 4 times, most recently from b94964a to ce97ba3 Compare November 23, 2025 02:27

Simon-Laux force-pushed the simon/receive-pre-messages branch from cd716c5 to 4c14566 Compare November 26, 2025 14:13

Base automatically changed from simon/sending-pre-message to pre-messages November 26, 2025 14:14

Simon-Laux and others added 17 commits November 26, 2025 16:09

fix python lint errors

0110d6d

receive pre-mesages, start with changes to imap loop.

e7751ce

simplify code a bit

31a91d2

refactor: move download code from scheduler.rs to download.rs, also

c928c4b

move `get_msg_id_by_rfc724_mid` to `MsgId::get_by_rfc724_mid`

MAX_FETCH_MSG_SIZE is no longer unused

6985536

Parse if it is a pre-message or full-message

45de7cd

start with receiving logic

fa063da

get rid of MsgId::get_by_rfc724_mid because it was a duplicate of

a5d9ed2

`message::rfc724_mid_exists`

docs: add hint to MimeMessage::from_bytes stating that it has

511d4d2

side-effects.

receiving full message

38e2745

send and receive attachment_size and set viewtype to text in

07a8d2f

pre_message

metadata as struct in pre-message in header. And fill params that we can

c260576

already fill from the metadata. Also add a new api to check what viewtype the message will have once downloaded.

api: jsonrpc: add full_message_view_type to Message and

7f3d2f5

`MessageInfo`

apply clippy suggestion

ea511d4

make PreMsgMetadata.to_header_value not consume self/PreMsgMetadata

ce7e62e

add api to merge params

f5577d9

on download full message: merge new params into old params and remove

ca6bf65

full-message metadata params

Simon-Laux force-pushed the simon/receive-pre-messages branch from 4c14566 to ca6bf65 Compare November 26, 2025 15:09

move tests to src/tests/pre_messages.rs

b9a4c43

Simon-Laux added 8 commits November 30, 2025 04:52

dynamically allocate test attachment bytes

f1f9868

fix detection of pre-messages. (it looked for the ChatFullMessageId

01cee7e

header in the unencrypted headers before)

fix setting dl state to avaiable on pre-messages

59b55e3

fix: save pre message with rfc724_mid of full message als disable

91e8f67

replacement for full messages

add some receiving tests and update test todo for premessage metadata

d361eeb

test: process full message before pre-message

5230aba

test receive normal message

99610b2

some serialization tests for PreMsgMetadata

15fa49b

Simon-Laux commented Nov 30, 2025

View reviewed changes

src/message.rs Show resolved Hide resolved

Simon-Laux added 3 commits November 30, 2025 07:54

remove outdated todo comment

6042d17

test that pre-message contains message text

9cff662

PreMsgMetadata: test_build_from_file_msg and test_build_from_file_msg

1ae1a34

Simon-Laux commented Nov 30, 2025

View reviewed changes

src/test_utils.rs Show resolved Hide resolved

Simon-Laux added 10 commits November 30, 2025 09:32

test: test_receive_pre_message_image

b5629e9

Test receiving the full message after receiving an edit after receiving

156d5b8

the pre-message

test_reaction_on_pre_message

957e967

test_full_download_after_trashed

4bbe0c1

test_webxdc_update_for_not_downloaded_instance

59ca831

simplify fake webxdc generation in

477e2ee

test_webxdc_update_for_not_downloaded_instance

test_markseen_pre_msg

5e499d8

test_pre_msg_can_start_chat and test_full_msg_can_start_chat

9803b30

test_download_later_keeps_message_order

0934daf

test_chatlist_event_on_full_msg_download

d70bfcd

Simon-Laux commented Nov 30, 2025

View reviewed changes

Hocuri reviewed Dec 1, 2025

View reviewed changes

src/context.rs Show resolved Hide resolved

Simon-Laux marked this pull request as ready for review December 1, 2025 15:46

Simon-Laux commented Dec 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: receive pre-messages and adapt download on demand #7431

feat: receive pre-messages and adapt download on demand #7431

Uh oh!

Simon-Laux commented Nov 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Simon-Laux Nov 30, 2025

Uh oh!

Hocuri commented Dec 1, 2025

Uh oh!

Uh oh!

Simon-Laux Dec 1, 2025

Uh oh!

Simon-Laux Dec 1, 2025

Uh oh!

Uh oh!

Simon-Laux Dec 1, 2025

Uh oh!

Simon-Laux Dec 1, 2025

Uh oh!

Simon-Laux commented Dec 1, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		chat::mark_old_messages_as_noticed(context, received_msgs).await?;

		// TODO: is there correct place for this?

Uh oh!

feat: receive pre-messages and adapt download on demand #7431

Are you sure you want to change the base?

feat: receive pre-messages and adapt download on demand #7431

Uh oh!

Conversation

Simon-Laux commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

progress:

specific aspects that should become reality:

API Changes:

Uh oh!

Uh oh!

Uh oh!

Simon-Laux Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

Hocuri commented Dec 1, 2025

Uh oh!

Uh oh!

Simon-Laux Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Laux Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Simon-Laux Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Laux Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Laux commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress of the tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Simon-Laux commented Nov 12, 2025 •

edited

Loading

Simon-Laux commented Dec 1, 2025 •

edited

Loading