mutt workflow and mail filtering

[Cameron Simpson]

On 11May2019 09:38, benfitzg at gmail.com <benfitzg at gmail.com> wrote:
>On Mon, May 06, 2019 at 03:59:04PM +1000, Cameron Simpson wrote:
>> Ny mail filtering is maildir based. I fetch from my ISP with getmail 
>> which
>> simply delivers to my "+spool" maildir. My mail filer watches that directory
>> and filters anything which lands there.
>>
>> To use such a scheme for gmail I'd imagine watching the local INBOX maildir
>> and filtering everything which shows up; one side effect is that INBOX would
>> stop being your inbox, and something else would need to become so.
>>
>> My mail filer is here:
>>
>>  https://pypi.org/project/cs.app.mailfiler/
>
>This is really interesting, I'll check it out.

Apologies for the typos in the description page; I'm tidying that up.  
And I need to update the package to include the mailfiler(1cs) and 
mailfiler(5cs) man pages (command line and rule syntax respectively).

If it's any comfort, I've been running mailfiler on my email for years.

>My main concern is the resync
>back up to the IMAP server. OfflineImap keeps metadata on each mail, in the form
>of a UID.

I would think that should still work. This is an untested belief.

However, offlineimap might push all the refiled email back up into the 
respective folders; I don't know if it is smart enough to see the UID 
move and issue an IMAP message move instruction.

What happens if you manually move a local message with mutt and rerun 
offlineimap?

>I *think* that if I were to use your mailfiler on my $Maildir, leaving
>concurrent access issues aside for a moment (I'd ensure one runs after the
>other), that the behaviour would be:
>
>- run offlineimap sync, download new mail to $Maildir/INBOX
>- run mailfiler against $INBOX, rules move a matching message to $Maildir/other (eg UID 1234)
>- offlineimap sees the deletion of message UID 1234 and syncs this deletion back to the server
>- offlineimap sees the addition of a new msg to folder other, UID 1234, syncs to the server

That sounds likely.

Note that I run mailfiler in daemon mode, where it monitors maildirs 
continuously. By you could easily run it in just-once mode from the 
offlineimap post hook, or via a wrapper script to run offline imap, 
mailfiler-once, offlineimap.

But I expect offlineimap expects to work concurrently with other maildir 
clients, that's what maildir is for. So I wouldn't worry much.

One thing that occurs to me is that mailfiler could do with a mode to 
explicitly leave messages behind (to keep your INBOX as an inbox).  
Currently it expects to refile every message: my flow empties the spool 
maildirs into my inbox and list folders. In its present state you'd need 
to start treating a secondary folder as your inbox.

It already has half this mode, in that if there's an exception filing a 
message it "lurks" it, and ignores that particular message file from 
then on - this avoids firing the same failure on every subsequent scan 
of the folder.  The lurking is filename based, so all you need to do to 
unlurk a message is to resave it from mutt (save to same maildir), or of 
course just restart mailfiler - the lurking is just an in-memory table.

(Horrible thought: you could have a gmail rule to filter _all_ new mail 
to an "unfiled" folder and have mailfiler move things from there _into_ 
INBOX.  Urgh. But the gmail INBOX is a logical concept and I'm not sure 
how well that would work. How well does that work doing it by hand, eg 
from mutt on your local copy?)

>Not being able to process rules locally is basically driving me insane 
>at work, as I have to use Exchange rules, which are just dire.

Aye. I'm pretty sure I've had exchange outright ignore some perfectly 
legal and correct rules, too.

>Plus I cannot edit them on the fly. Ideally I'd have a mutt macro that 
>would take the core header fields out the mail, append them to 
>mailfiler rules, let me edit out what I don't need then save and quit.  
>In Outlook this is time consuming and brittle, plus there is no way of 
>seeing a log of what matches what, so undesirable matches are hard to 
>catch, potentially meaning I miss important mails.

Mailfiler writes a log of its actions for each folder it monitors.  
Example:

  2019-05-12 02:35:28,760 INFO 2019-05-12 02:35:28 Alex Dunn->fvwmorg/fvwm github,Subscribed: Re: [fvwmorg/fvwm] FIX: EdgeCommand is sometimes not triggered. (#52)
  2019-05-12 02:35:28,762 INFO   +spool-in/new/1557592528.#4587M757606P3357 <fvwmorg/fvwm/pull/52/c491525676 at github.com>
  2019-05-12 02:35:28,876 INFO     OK >> +fvwm

That's just 3 lines if your mail reader folds it:

Line 1: summary line of message: timestamp, from->to, subject
Line 2: source message location
Line 3 onward: status (OK) and where the message was filed, on per 
filing.

Regarding adding rules:

- rule files are just text files, and most rules are just one line.

- mailfiler sees modifications to these files and reloads them if you 
  change them.

- mailfiler has a command line test mode to read a message from stdin 
  and run a set of rules

- the log shows what happens to any message, and can of course be tailed

- if I want to change the rules I often write the new rule, then just 
  save (or copy) the problematic message straight into the spool folder 
  where it will get refiled according to the new rules. Tail the log 
  file to verify.

- I've got a few mutt macros to add knowledge based on a message. By far 
  the commonest one is to add a message's author as "known", which lands 
  them in my inbox if no other rules match. Then I can (a) see the 
  message regardless and (b) hand write a rule for it and its kin if 
  that is appropriate. I use this macro mostly from my "unknown" folder, 
  where messages which match _no_ rules land. (Of course, I also use 
  mailfiler on my "+spool-out" folder, which crossfiles sent email and 
  also adds the recipients to the "sent" list, making them "known".)

>I really have to fix this as it's making my working life miserable, as I'm in a
>company with really, really bad email culture.

A common circumstance, sadly.

Cheers,
Cameron Simpson <cs at cskk.id.au>