Parallelize message processing and batch IMAP fetches #3

New issue

Closed

opened 2026-05-11 00:31:33 +02:00 by heiko · 0 comments

heiko commented

2026-05-11 00:31:33 +02:00

Owner

Problem

Messages are processed sequentially: fetch → parse MIME → scan attachments → store flag. For large mailboxes this is slow because each FetchMessage and AddFlag is a synchronous, blocking IMAP command.

Proposal

Batch initial fetch: replace N individual UID FETCH <uid> BODY[] calls with a single UID FETCH <set> BODY[] stream — the library returns messages as they arrive
Parallelize detection: XRechnung detection (XML/PDF parsing) is CPU-bound and can run concurrently across fetched messages
Batch flag store: accumulate matched UIDs during a scan, then issue a single UID STORE <set> +FLAGS ($xrechnung) at the end

Notes

The IMAP connection uses a mutex (sync.Mutex) so concurrent IMAP commands need careful sequencing or a command pipeline
IDLE loop handles one message at a time — parallelism only helps the initial full-mailbox scan
Benchmark before optimising; for typical invoice volumes the current approach may be fast enough

## Problem Messages are processed sequentially: fetch → parse MIME → scan attachments → store flag. For large mailboxes this is slow because each `FetchMessage` and `AddFlag` is a synchronous, blocking IMAP command. ## Proposal - **Batch initial fetch**: replace N individual `UID FETCH <uid> BODY[]` calls with a single `UID FETCH <set> BODY[]` stream — the library returns messages as they arrive - **Parallelize detection**: XRechnung detection (XML/PDF parsing) is CPU-bound and can run concurrently across fetched messages - **Batch flag store**: accumulate matched UIDs during a scan, then issue a single `UID STORE <set> +FLAGS ($xrechnung)` at the end ## Notes - The IMAP connection uses a mutex (`sync.Mutex`) so concurrent IMAP commands need careful sequencing or a command pipeline - IDLE loop handles one message at a time — parallelism only helps the initial full-mailbox scan - Benchmark before optimising; for typical invoice volumes the current approach may be fast enough

heiko added this to the v0.5.0 milestone

2026-05-11 00:31:33 +02:00

heiko referenced this issue from a commit

2026-05-11 18:38:15 +02:00

fix: IDLE not waking on new messages, add --debug flag

heiko referenced this issue from a commit

2026-05-11 18:44:51 +02:00

fix: IDLE not waking on new messages, add --debug flag

heiko closed this issue

2026-05-11 18:45:19 +02:00

heiko referenced this issue from a commit

2026-05-13 17:55:20 +02:00