my weekend project: a streaming POP3 fetcher, replacing fetchmail/getmail

[Cameron Simpson]

On 06Apr2021 23:12, Kurt Hackenberg <kh at panix.com> wrote:
>On Wed, Apr 07, 2021 at 09:43:36AM +1000, Cameron Simpson wrote:
>>My new tool streams the fetches: it issues RETRs for every message up
>>front at maximum network speed - fully buffered and with no waits. A
>>parallel worker thread collects the messages as they come in at full
>>speed (the upstream server likely also gets to fully buffer); it issues
>>DELEtes as each message is saved, also fully buffered.
>
>Slick.  Clearly the right way to handle that high latency.

Just tried it on the satellite link with an overnight load of messages, 
normally a 10 minute exercise with getmail (give or take). 411 messages, 
8.5 seconds.

>Have you ever tried the program fdm?  It fetches, filters, and
>delivers mail, like getmail and procmail combined.  I haven't tried
>it, but it looks interesting.

I have not.

My mail filer is decoupled from my fetcher: it monitors spool Maildirs 
(which also means I can refile a message just by saving it to a spool 
Maildir).  And it has its own syntax to my liking; other tools 
inherently will not :-)

And looking at the conf file, it seems that (like procmail, which I 
abandoned years ago) it matches using regexps. These are appalling for 
email addresses. When testing addresses, my filer does a correct address 
parse and compares the inner component (eg "cs at cskk.id.au" from "Cameron 
Simpson <cs at cskk.id.au>"), and can do set membership tests on that eg 
"is this address in this group?".

>This paragraph in its manual sounds like it might stream fetching like
>your program:
>
>"fdm tries to queue a number of mails simultaneously, so that older
>can be delivered while waiting for the server to provide the next. The
>maximum length of the queue for each account is set by the
>'queue-high' option (the default is two) and the maximum mail size
>accepted by the 'maximum-size' option (the default is 32 MB). In
>addition, the 'rewrite' action requires an additional temporary
>mail. Although fdm will fail rather than dropping mail if the disk
>becomes full, users should bear in mind the possibility and set the
>size of the temporary directory and the fdm options according to their
>needs."

I've been reading the manual. I think it does not. I think it actually 
allows the filing/saving to proceeed while requesting/fetching the next 
message. So a simple form of parallelism, but not one which reduces the 
fetch latency between requests.

>fdm is at github:
><https://github.com/nicm/fdm/>
>
>The paragraph quoted above is at about line 300 in the manual, which is here:
><https://github.com/nicm/fdm/blob/master/MANUAL>

Thanks. An interesting read.

Cheers,
Cameron Simpson <cs at cskk.id.au>