now you're gonna die wearing that stupid little hat, how does it feel? (part 3)

i de-branched the webmail folders code the other week to make it live on the dls website. about 5 months in the making (just working on it here and there whenever i had spare time at work), it implements some features that probably seem pretty basic for a web-based e-mail client:

• a basic concept of folders to separate messages into logical groups and allow the user to create new folders
• a "sent" folder to put a copy of all messages sent from the interface
• a "trash" folder in which deleted messages are moved to, allowing the undeletion of individual messages or emptying of the trash folder all at once
• fast re-sorting of a folder by the from address, subject, date, etc., ascending or descending
• mark messages that have been sent, replied to, and/or are new (unread)

but of course, those had to be implemented with some heavy restrictions:

• must use the pop3 protocol; cannot use any non-standard server extensions, the imap4 protocol, or do direct manipulation of a mailbox or its messages (in some cases we may be talking to a customer-owned pop3 server that we can't control)
• must leave messages on the server untouched to allow compatibility with normal pop3 clients, and cope with external clients modifying the mailbox
• sent messages must be contained in the pop3 mailbox to allow downloading to pop3 clients and enforce server-side mailbox size quotas
• must be fast; every page view of the message index or message contents has to login to the pop3 server, check for new mail, and present the list of messages with new-mail counts in each folder on a sidebar

so how does one finagle folder support into a client for a protocol that doesn't support folders? with a sql cache, of course.

when the webmail client logs in to the pop3 server, it issues the "UIDL" command to list each message and cache its corresponding message number and UIDL. this list is compared to the sql cache and any messages that are on the pop3 server but not in sql are read and cached, any on the sql server that aren't on the pop3 server are deleted from sql, and the rest have nothing done on them.

then when the message list is viewed, it pulls this sql cache which contains the from address, subject, date, size, and attributes (sent, replied to, new/seen) of each message. each also contains a field to store the id of its folder that the user has moved it to. as each message is looked at in this way, each is shuffled into its correct folder and the page is drawn.

it sounds like a lot of processing to be doing, but it's relatively quick. when the mailbox is unchanged, the only pop3 traffic is logging in and issuing the UIDL command to grab the message-id-to-uidl mapping. then each UIDL is looked up in sql which is very fast. when a user moves a message to another folder, the only thing that changes is the sql cache record to have its folder id changed. same for viewing a new message and marking it seen, deleting a message (which just changes its folder id to that of the trash folder), and marking a message as replied to.

sent messages are handled by adding a "bcc" header to outbound messages to copy the message back to the logged in user's address. an additional "X-" header is written to the outgoing message so when it's seen by webmail when it comes back into the pop3 mailbox, it knows to move that message to the "sent" folder right away and mark it as sent (and read) before the user ever sees it in the inbox.

so while all of this is relatively quick for browsing mailboxes that don't change much, logging into a mailbox with ~800 messages for the first time means waiting for the headers of all 800 of those messages to be read from the pop3 server and inserted into sql. users don't like waiting that long, so the solution is to fork that loading the background, wait only a few seconds so that a few messages have been loaded into the cache, and then show the user the mailbox with a few messages in it. then you show them that the mailbox is still loading, and auto-refresh the mailbox index every ~10 seconds so that newly cached messages are visible. that lets the user view newer messages (which they probably care more about right now than old messages) and quickly move to the compose screen if they just want to send a message from a remote computer.

so that's it. there are a lot of technical details to the system, but that's basically how to implement folder support in a pop3 webmail system while keeping it quick and responsive, and without requiring any control of the pop3 server.