my weekend project: a streaming POP3 fetcher, replacing fetchmail/getmail

[Cameron Simpson]

Like several here, I fetch email from my ISP mail spool(s) and file 
messages locally. If my laptop's been offline overnight there can be 
hundreds of messages to fetch when I wake it up. On a satellite link 
(geostationary) with a ping time of over 600ms this can be many minutes 
of tedium.

The raw bandwidth is fine and my filing process is pretty expeditious;
the root cause of that tedium is network latency and the synchronous 
behaviour of getmail. Its cycle is like this:

- RETRieve the message, collect and save
- DELEte the message
- repeat for all the messages, then QUIT to commit the deletes

Each of steps 1 and 2 above incurs over a second just in network 
latency. That scales up over hundreds of messages.

My new tool streams the fetches: it issues RETRs for every message up 
front at maximum network speed - fully buffered and with no waits. A 
parallel worker thread collects the messages as they come in at full 
speed (the upstream server likely also gets to fully buffer); it issues 
DELEtes as each message is saved, also fully buffered.

The code's here:

    https://hg.sr.ht/~cameron-simpson/css/browse/lib/python/cs/pop3.py?rev=tip

The cs.pop3 modules on PyPI here:

    https://pypi.org/project/cs.pop3/

and can be installed with:

    pip install cs.pop3

which also provides a "pop3" command in your Python environment.

Typical use is:

    pop3 dl mylogin at mail.cskk.id.au ~/var/mail/spool

specifying my internet mail spool and the local Maildir to receive the 
messages.

Cheers,
Cameron Simpson <cs at cskk.id.au>

If you cannot, in the long run, tell everyone what you have been doing, 
your doing has been worthless.       - Erwin Schrodinger