imapext-2007
view docs/rfc/rfc4549.txt @ 0:ada5e610ab86
imap-2007e
| author | yuuji@gentei.org |
|---|---|
| date | Mon, 14 Sep 2009 15:17:45 +0900 |
| parents | |
| children |
line source
7 Network Working Group A. Melnikov, Ed.
8 Request for Comments: 4549 Isode Ltd.
9 Category: Informational June 2006
12 Synchronization Operations for Disconnected IMAP4 Clients
14 Status of This Memo
16 This memo provides information for the Internet community. It does
17 not specify an Internet standard of any kind. Distribution of this
18 memo is unlimited.
20 Copyright Notice
22 Copyright (C) The Internet Society (2006).
24 Abstract
26 This document attempts to address some of the issues involved in
27 building a disconnected IMAP4 client. In particular, it deals with
28 the issues of what might be called the "driver" portion of the
29 synchronization tool: the portion of the code responsible for issuing
30 the correct set of IMAP4 commands to synchronize the disconnected
31 client in the way that is most likely to make the human who uses the
32 disconnected client happy.
34 This note describes different strategies that can be used by
35 disconnected clients and shows how to use IMAP protocol in order to
36 minimize the time of the synchronization process.
38 This note also lists IMAP extensions that a server should implement
39 in order to provide better synchronization facilities to disconnected
40 clients.
58 Melnikov Informational [Page 1]
60 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
63 Table of Contents
65 1. Introduction ....................................................3
66 1.1. Conventions Used in This Document ..........................3
67 2. Design Principles ...............................................3
68 3. Overall Picture of Synchronization ..............................4
69 4. Mailbox Synchronization Steps and Strategies ....................7
70 4.1. Checking UID Validity ......................................7
71 4.2. Synchronizing Local Changes with the Server ................8
72 4.2.1. Uploading Messages to the Mailbox ...................8
73 4.2.2. Optimizing "move" and "copy" Operations .............9
74 4.2.3. Replaying Local Flag Changes .......................14
75 4.2.4. Processing Mailbox Compression (EXPUNGE) Requests ..15
76 4.2.5. Closing a Mailbox ..................................17
77 4.3. Details of "Normal" Synchronization of a Single Mailbox ...18
78 4.3.1. Discovering New Messages and Changes to Old
79 Messages ...........................................18
80 4.3.2. Searching for "Interesting" Messages. ..............20
81 4.3.3. Populating Cache with "Interesting" Messages. ......21
82 4.3.4. User-Initiated Synchronization .....................22
83 4.4. Special Case: Descriptor-Only Synchronization .............22
84 4.5. Special Case: Fast New-Only Synchronization ...............23
85 4.6. Special Case: Blind FETCH .................................23
86 5. Implementation Considerations ..................................24
87 5.1. Error Recovery during Playback ............................26
88 5.2. Quality of Implementation Issues ..........................28
89 5.3. Optimizations .............................................28
90 6. IMAP Extensions That May Help ..................................30
91 6.1. CONDSTORE Extension .......................................30
92 7. Security Considerations ........................................33
93 8. References .....................................................33
94 8.1. Normative References ......................................33
95 8.2. Informative References ....................................34
96 9. Acknowledgements ...............................................34
114 Melnikov Informational [Page 2]
116 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
119 1. Introduction
121 Several recommendations presented in this document are generally
122 applicable to all types of IMAP clients. However, this document
123 tries to concentrate on disconnected mail clients [IMAP-MODEL]. It
124 also suggests some IMAP extensions* that should be implemented by
125 IMAP servers in order to make the life of disconnected clients
126 easier. In particular, the [UIDPLUS] extension was specifically
127 designed to streamline certain disconnected operations, like
128 expunging, uploading, and copying messages (see Sections 4.2.1,
129 4.2.2.1, and 4.2.4).
131 Readers of this document are also strongly advised to read RFC 2683
132 [RFC2683].
134 * Note that the functionality provided by the base IMAP protocol
135 [IMAP4] is sufficient to perform basic synchronization.
137 1.1. Conventions Used in This Document
139 In examples, "C:" and "S:" indicate lines sent by the client and
140 server, respectively. Long lines in examples are broken for
141 editorial clarity.
143 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
144 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
145 document are to be interpreted as described in RFC 2119 [KEYWORDS].
147 Let's call an IMAP command idempotent if the result of executing the
148 command twice sequentially is the same as the result of executing the
149 command just once.
151 2. Design Principles
153 All mailbox state or content information stored on the disconnected
154 client should be viewed strictly as a cache of the state of the
155 server. The "master" state remains on the server, just as it would
156 with an interactive IMAP4 client. The one exception to this rule is
157 that information about the state of the disconnected client's cache
158 (the state includes flag changes while offline and during scheduled
159 message uploads) remains on the disconnected client: that is, the
160 IMAP4 server is not responsible for remembering the state of the
161 disconnected IMAP4 client.
163 We assume that a disconnected client is a client that, for whatever
164 reason, wants to minimize the length of time that it is "on the
165 phone" to the IMAP4 server. Often this will be because the client is
166 using a dialup connection, possibly with very low bandwidth, but
170 Melnikov Informational [Page 3]
172 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
175 sometimes it might just be that the human is in a hurry to catch an
176 airplane, or some other event beyond our control. Whatever the
177 reason, we assume that we must make efficient use of the network
178 connection, both in the usual sense (not generating spurious traffic)
179 and in the sense that we would prefer not to have the connection
180 sitting idle while the client and/or the server is performing
181 strictly local computation or I/O. Another, perhaps simpler way of
182 stating this is that we assume that network connections are
183 "expensive".
185 Practical experience with disconnected mail systems has shown that
186 there is no single synchronization strategy that is appropriate for
187 all cases. Different humans have different preferences, and the same
188 human's preference will vary depending both on external circumstance
189 (how much of a hurry the human is in today) and on the value that the
190 human places on the messages being transferred. The point here is
191 that there is no way that the synchronization program can guess
192 exactly what the human wants to do, so the human will have to provide
193 some guidance.
195 Taken together, the preceding two principles lead to the conclusion
196 that the synchronization program must make its decisions based on
197 some kind of guidance provided by the human, by selecting the
198 appropriate options in the user interface or through some sort of
199 configuration file. Almost certainly, it should not pause for I/O
200 with the human in the middle of the synchronization process. The
201 human will almost certainly have several different configurations for
202 the synchronization program, for different circumstances.
204 Since a disconnected client has no way of knowing what changes might
205 have occurred to the mailbox while it was disconnected, message
206 numbers are not useful to a disconnected client. All disconnected
207 client operations should be performed using UIDs, so that the client
208 can be sure that it and the server are talking about the same
209 messages during the synchronization process.
211 3. Overall Picture of Synchronization
213 The basic strategy for synchronization is outlined below. Note that
214 the real strategy may vary from one application to another or may
215 depend on a synchronization mode.
217 a) Process any "actions" that were pending on the client that were
218 not associated with any mailbox. (In particular sending messages
219 composed offline with SMTP. This is not part of IMAP
220 synchronization, but it is mentioned here for completeness.)
226 Melnikov Informational [Page 4]
228 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
231 b) Fetch the current list of "interesting" mailboxes. (The
232 disconnected client should allow the user to skip this step
233 completely.)
235 c) "Client-to-server synchronization": for each IMAP "action" that
236 was pending on the client, do the following:
238 1) If the action implies opening a new mailbox (any operation that
239 operates on messages), open the mailbox. Check its UID
240 validity value (see Section 4.1 for more details) returned in
241 the UIDVALIDITY response code. If the UIDVALIDITY value
242 returned by the server differs, the client MUST empty the local
243 cache of the mailbox and remove any pending "actions" that
244 refer to UIDs in that mailbox (and consider them failed). Note
245 that this doesn't affect actions performed on client-generated
246 fake UIDs (see Section 5).
248 2) Perform the action. If the action is to delete a mailbox
249 (DELETE), make sure that the mailbox is closed first (see also
250 Section 3.4.12 of [RFC2683]).
252 d) "Server-to-client synchronization": for each mailbox that requires
253 synchronization, do the following:
255 1) Check the mailbox UIDVALIDITY (see Section 4.1 for more
256 details) with SELECT/EXAMINE/STATUS.
258 If UIDVALIDITY value returned by the server differs, the client
259 MUST
261 * empty the local cache of that mailbox;
262 * remove any pending "actions" that refer to UIDs in that
263 mailbox and consider them failed; and
264 * skip step 2-II.
266 2) Fetch the current "descriptors";
268 I) Discover new messages.
270 II) Discover changes to old messages.
272 3) Fetch the bodies of any "interesting" messages that the client
273 doesn't already have.
275 e) Close all open mailboxes not required for further operations (if
276 staying online) or disconnect all open connections (if going
277 offline).
282 Melnikov Informational [Page 5]
284 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
287 Terms used:
289 "Actions" are queued requests that were made by the human to the
290 client's Mail User Agent (MUA) software while the client was
291 disconnected.
293 We define "descriptors" as a set of IMAP4 FETCH data items.
294 Conceptually, a message's descriptor is that set of information that
295 allows the synchronization program to decide what protocol actions
296 are necessary to bring the local cache to the desired state for this
297 message; since this decision is really up to the human, this
298 information probably includes at least a few header fields intended
299 for human consumption. Exactly what will constitute a descriptor
300 depends on the client implementation. At a minimum, the descriptor
301 contains the message's UID and FLAGS. Other likely candidates are
302 the RFC822.SIZE, RFC822.HEADER, BODYSTRUCTURE, or ENVELOPE data
303 items.
305 Comments:
307 1) The list of actions should be ordered. For example, if the human
308 deletes message A1 in mailbox A, then expunges mailbox A, and then
309 deletes message A2 in mailbox A, the human will expect that
310 message A1 is gone and that message A2 is still present but is now
311 deleted.
313 By processing all the actions before proceeding with
314 synchronization, we avoid having to compensate for the local MUA's
315 changes to the server's state. That is, once we have processed
316 all the pending actions, the steps that the client must take to
317 synchronize itself will be the same no matter where the changes to
318 the server's state originated.
320 2) Steps a and b can be performed in parallel. Alternatively, step a
321 can be performed after d.
323 3) On step b, the set of "interesting" mailboxes pretty much has to
324 be determined by the human. What mailboxes belong to this set may
325 vary between different IMAP4 sessions with the same server,
326 client, and human. An interesting mailbox can be a mailbox
327 returned by LSUB command (see Section 6.3.9 of [IMAP4]). The
328 special mailbox "INBOX" SHOULD be in the default set of mailboxes
329 that the client considers interesting. However, providing the
330 ability to ignore INBOX for a particular session or client may be
331 valuable for some mail filtering strategies.
338 Melnikov Informational [Page 6]
340 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
343 4) On step d-2-II, the client also finds out about changes to the
344 flags of messages that the client already has in its local cache,
345 and about messages in the local cache that no longer exist on the
346 server (i.e., messages that have been expunged).
348 5) "Interesting" messages are those messages that the synchronization
349 program thinks the human wants to have cached locally, based on
350 the configuration and the data retrieved in step b.
352 6) A disconnected IMAP client is a special case of an IMAP client, so
353 it MUST be able to handle any "unexpected" unsolicited responses,
354 like EXISTS and EXPUNGE, at any time. The disconnected client MAY
355 ignore EXPUNGE response during "client-to-server" synchronization
356 phase (step c).
358 The rest of this discussion will focus primarily on the
359 synchronization issues for a single mailbox.
361 4. Mailbox Synchronization Steps and Strategies
363 4.1. Checking UID Validity
365 The "UID validity" of a mailbox is a number returned in an
366 UIDVALIDITY response code in an OK untagged response at mailbox
367 selection time. The UID validity value changes between sessions when
368 UIDs fail to persist between sessions.
370 Whenever the client selects a mailbox, the client must compare the
371 returned UID validity value with the value stored in the local cache.
372 If the UID validity values differ, the UIDs in the client's cache are
373 no longer valid. The client MUST then empty the local cache of that
374 mailbox and remove any pending "actions" that refer to UIDs in that
375 mailbox. The client MAY also issue a warning to the human. The
376 client MUST NOT cancel any scheduled uploads (i.e., APPENDs) for the
377 mailbox.
379 Note that UIDVALIDITY is not only returned on a mailbox selection.
380 The COPYUID and APPENDUID response codes defined in the [UIDPLUS]
381 extension (see also 4.2.2) and the UIDVALIDITY STATUS response data
382 item also contain a UIDVALIDITY value for some other mailbox. The
383 client SHOULD behave as described in the previous paragraph (but it
384 should act on the other mailbox's cache), no matter how it obtained
385 the UIDVALIDITY value.
394 Melnikov Informational [Page 7]
396 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
399 4.2. Synchronizing Local Changes with the Server
401 4.2.1. Uploading Messages to the Mailbox
403 Two of the most common examples of operations resulting in message
404 uploads are:
406 1) Saving a draft message
408 2) Copying a message between remote mailboxes on two different IMAP
409 servers or a local mailbox and a remote mailbox.
411 Message upload is performed with the APPEND command. A message
412 scheduled to be uploaded has no UID associated with it, as all UIDs
413 are assigned by the server. The APPEND command will effectively
414 associate a UID with the uploaded message that can be stored in the
415 local cache for future reference. However, [IMAP4] doesn't describe
416 a simple mechanism to discover the message UID by just performing the
417 APPEND command. In order to discover the UID, the client can do one
418 of the following:
420 1) Remove the uploaded message from cache. Then, use the mechanism
421 described in 4.3 to fetch the information about the uploaded
422 message as if it had been uploaded by some other client.
424 2) Try to fetch header information as described in 4.2.2 in order to
425 find a message that corresponds to the uploaded message. One
426 strategy for doing this is described in 4.2.2.
428 Case 1 describes a not particularly smart client.
430 C: A003 APPEND Drafts (\Seen $MDNSent) {310}
431 S: + Ready for literal data
432 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
433 C: From: Fred Foobar <foobar@blt.example.COM>
434 C: Subject: afternoon meeting
435 C: To: mooch@owatagu.siam.edu
436 C: Message-Id: <B27397-0100000@blt.example.COM>
437 C: MIME-Version: 1.0
438 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
439 C:
440 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
441 C:
442 S: A003 OK APPEND Completed
444 Fortunately, there is a simpler way to discover the message UID in
445 the presence of the [UIDPLUS] extension:
450 Melnikov Informational [Page 8]
452 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
455 C: A003 APPEND Drafts (\Seen $MDNSent) {310}
456 S: + Ready for literal data
457 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
458 C: From: Fred Foobar <foobar@blt.example.COM>
459 C: Subject: afternoon meeting
460 C: To: mooch@owatagu.siam.edu
461 C: Message-Id: <B27397-0100000@blt.example.COM>
462 C: MIME-Version: 1.0
463 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
464 C:
465 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
466 C:
467 S: A003 OK [APPENDUID 1022843275 77712] APPEND completed
469 The UID of the appended message is the second parameter of APPENDUID
470 response code.
472 4.2.2. Optimizing "move" and "copy" Operations
474 Practical experience with IMAP and other mailbox access protocols
475 that support multiple mailboxes suggests that moving a message from
476 one mailbox to another is an extremely common operation.
478 4.2.2.1. Moving a Message between Two Mailboxes on the Same Server
480 In IMAP4, a "move" operation between two mailboxes on the same server
481 is really a combination of a COPY operation and a STORE +FLAGS
482 (\Deleted) operation. This makes good protocol sense for IMAP, but
483 it leaves a simple-minded disconnected client in the silly position
484 of deleting and possibly expunging its cached copy of a message, then
485 fetching an identical copy via the network.
487 However, the presence of the UIDPLUS extension in the server can
488 help:
490 C: A001 UID COPY 567,414 "Interesting Messages"
491 S: A001 OK [COPYUID 1022843275 414,567 5:6] Completed
493 This tells the client that the message with UID 414 in the current
494 mailbox was successfully copied to the mailbox "Interesting Messages"
495 and was given the UID 5, and that the message with UID 567 was given
496 the UID 6.
498 In the absence of UIDPLUS extension support in the server, the
499 following trick can be used. By including the Message-ID: header and
500 the INTERNALDATE data item as part of the descriptor, the client can
501 check the descriptor of a "new" message against messages that are
502 already in its cache and avoid fetching the extra copy. Of course,
506 Melnikov Informational [Page 9]
508 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
511 it's possible that the cost of checking to see if the message is
512 already in the local cache may exceed the cost of just fetching it,
513 so this technique should not be used blindly. If the MUA implements
514 a "move" command, it makes special provisions to use this technique
515 when it knows that a copy/delete sequence is the result of a "move"
516 command.
518 Note that servers are not required (although they are strongly
519 encouraged with "SHOULD language") to preserve INTERNALDATE when
520 copying messages.
522 Also note that since it's theoretically possible for this algorithm
523 to find the wrong message (given sufficiently malignant Message-ID
524 headers), implementers should provide a way to disable this
525 optimization, both permanently and on a message-by-message basis.
527 Example 1: Copying a message in the absence of UIDPLUS extension.
529 At some point in time the client has fetched the source message and
530 some information was cached:
532 C: C021 UID FETCH <uids> (BODY.PEEK[] INTERNALDATE FLAGS)
533 ...
534 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
535 FLAGS (\Draft $MDNSent) BODY[] {1036}
536 S: ...
537 S: Message-Id: <20040903110856.22a127cd@chardonnay>
538 S: ...
539 S: ...message body...
540 S: )
541 ...
542 S: C021 OK fetch completed
544 Later on, the client decides to copy the message:
546 C: C035 UID COPY 123 "Interesting Messages"
547 S: C035 OK Completed
549 As the server hasn't provided the COPYUID response code, the client
550 tries the optimization described above:
552 C: C036 SELECT "Interesting Messages"
553 ...
554 C: C037 UID SEARCH ON 31-May-2002 HEADER
555 "Message-Id" "20040903110856.22a127cd@chardonnay"
556 S: SEARCH 12368
557 S: C037 OK completed
562 Melnikov Informational [Page 10]
564 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
567 Note that if the server has returned multiple UIDs in the SEARCH
568 response, the client MUST NOT use any of the returned UID.
570 4.2.2.2. Moving a Message from a Remote Mailbox to a Local
572 Moving a message from a remote mailbox to a local is done with FETCH
573 (that includes FLAGS and INTERNALDATE) followed by UID STORE <uid>
574 +FLAGS.SILENT (\Deleted):
576 C: A003 UID FETCH 123 (BODY.PEEK[] INTERNALDATE FLAGS)
577 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
578 FLAGS (\Seen $MDNSent) BODY[]
579 S: ...message body...
580 S: )
581 S: A003 OK UID FETCH completed
582 C: A004 UID STORE <uid> +FLAGS.SILENT (\Deleted)
583 S: A004 STORE completed
585 Note that there is no reason to fetch the message during
586 synchronization if it's already in the client's cache. Also, the
587 client SHOULD preserve delivery date in the local cache.
589 4.2.2.3. Moving a Message from a Local Mailbox to a Remote
591 Moving a message from a local mailbox to a remote is done with
592 APPEND:
594 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
595 {310}
596 S: + Ready for literal data
597 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
598 C: From: Fred Foobar <foobar@blt.example.COM>
599 C: Subject: afternoon meeting
600 C: To: mooch@owatagu.siam.edu
601 C: Message-Id: <B27397-0100000@blt.example.COM>
602 C: MIME-Version: 1.0
603 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
604 C:
605 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
606 C:
607 S: A003 OK [APPENDUID 1022843275 77712] completed
609 The client SHOULD specify the delivery date from the local cache in
610 the APPEND.
612 If the [LITERAL+] extension is available, the client can save a
613 round-trip*:
618 Melnikov Informational [Page 11]
620 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
623 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
624 {310+}
625 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
626 C: From: Fred Foobar <foobar@blt.example.COM>
627 C: Subject: afternoon meeting
628 C: To: mooch@owatagu.siam.edu
629 C: Message-Id: <B27397-0100000@blt.example.COM>
630 C: MIME-Version: 1.0
631 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
632 C:
633 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
634 C:
635 S: A003 OK [APPENDUID 1022843275 77712] completed
637 * Note that there is a risk that the server will reject the message
638 due to its size. If this happens, the client will waste bandwidth
639 transferring the whole message. If the client wouldn't have used
640 the LITERAL+, this could have been avoided:
642 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2004 05:26:59 -0600"
643 {16777215}
644 S: A003 NO Sorry, message is too big
646 4.2.2.4. Moving a Message between Two Mailboxes on Different Servers
648 Moving a message between two mailbox on two different servers is a
649 combination of the operations described in 4.2.2.2 followed by the
650 operations described in 4.2.2.3.
652 4.2.2.5. Uploading Multiple Messages to a Remote Mailbox with
653 MULTIAPPEND
655 When there is a need to upload multiple messages to a remote mailbox
656 (e.g., as per 4.2.2.3), the presence of certain IMAP extensions may
657 significantly improve performance. One of them is [MULTIAPPEND].
659 For some mail stores, opening a mailbox for appending might be
660 expensive. [MULTIAPPEND] tells the server to open the mailbox once
661 (instead of opening and closing it "n" times per "n" messages to be
662 uploaded) and to keep it open while a group of messages is being
663 uploaded to the server.
665 Also, if the server supports both [MULTIAPPEND] and [LITERAL+]
666 extensions, the entire upload is accomplished in a single
667 command/response round-trip.
674 Melnikov Informational [Page 12]
676 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
679 Note: Client implementers should be aware that [MULTIAPPEND] performs
680 append of multiple messages atomically. This means, for example, if
681 there is not enough space to save "n"-th message (or the message has
682 invalid structure and is rejected by the server) after successful
683 upload of "n-1" messages, the whole upload operation fails, and no
684 message will be saved in the mailbox. Although this behavior might
685 be desirable in certain situations, it might not be what you want.
686 Otherwise, the client should use the regular APPEND command (Section
687 4.2.2.3), possibly utilizing the [LITERAL+] extension. See also
688 Section 5.1 for discussions about error recovery.
690 Note: MULTIAPPEND can be used together with the UIDPLUS extension in
691 a way similar to what was described in Section 4.2.1. [MULTIAPPEND]
692 extends the syntax of the APPENDUID response code to allow for
693 multiple message UIDs in the second parameter.
695 Example 2:
697 This example demonstrates the use of MULTIAPPEND together with
698 UIDPLUS (synchronization points where the client waits for
699 confirmations from the server are marked with "<--->"):
701 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
702 {310}
703 <--->
704 S: + Ready for literal data
705 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
706 C: From: Fred Foobar <foobar@blt.example.COM>
707 C: Subject: afternoon meeting
708 C: To: mooch@owatagu.siam.edu
709 C: Message-Id: <B27397-0100000@blt.example.COM>
710 C: MIME-Version: 1.0
711 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
712 C:
713 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
714 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286}
715 <--->
716 S: + Ready for literal data
717 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
718 C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
719 C: Subject: Re: afternoon meeting
720 C: To: foobar@blt.example.com
721 C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
722 C: MIME-Version: 1.0
723 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
724 C:
725 C: 3:30 is fine with me.
726 C:
730 Melnikov Informational [Page 13]
732 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
735 S: A003 OK [APPENDUID 1022843275 77712,77713] completed
737 The upload takes 3 round-trips.
739 Example 3:
741 In this example, Example 2 was modified for the case when the server
742 supports MULTIAPPEND, LITERAL+, and UIDPLUS. The upload takes only 1
743 round-trip.
745 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
746 {310+}
747 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
748 C: From: Fred Foobar <foobar@blt.example.COM>
749 C: Subject: afternoon meeting
750 C: To: mooch@owatagu.siam.edu
751 C: Message-Id: <B27397-0100000@blt.example.COM>
752 C: MIME-Version: 1.0
753 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
754 C:
755 C: Hello Joe, do you think we can meet at 3:30 tomorrow?
756 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286+}
757 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
758 C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
759 C: Subject: Re: afternoon meeting
760 C: To: foobar@blt.example.com
761 C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
762 C: MIME-Version: 1.0
763 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
764 C:
765 C: 3:30 is fine with me.
766 C:
767 S: A003 OK [APPENDUID 1022843275 77712,77713] completed
769 4.2.3. Replaying Local Flag Changes
771 The disconnected client uses the STORE command to synchronize local
772 flag state with the server. The disconnected client SHOULD use
773 +FLAGS.SILENT or -FLAGS.SILENT in order to set or unset flags
774 modified by the user while offline. The FLAGS form MUST NOT be used,
775 as there is a risk that this will overwrite flags on the server that
776 have been changed by some other client.
778 Example 4:
780 For the message with UID 15, the disconnected client stores the
781 following flags \Seen and $Highest. The flags were modified on the
782 server by some other client: \Seen, \Answered, and $Highest. While
786 Melnikov Informational [Page 14]
788 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
791 offline, the user requested that the $Highest flags be removed and
792 that the \Deleted flag be added. The flag synchronization sequence
793 for the message should look like:
795 C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted)
796 S: A001 STORE completed
797 C: A002 UID STORE 15 -FLAGS.SILENT ($Highest)
798 S: A002 STORE completed
800 If the disconnected client is able to store an additional binary
801 state information (or a piece of information that can take a value
802 from a predefined set of values) in the local cache of an IMAP
803 mailbox or in a local mailbox (e.g., message priority), and if the
804 server supports storing of arbitrary keywords, the client MUST use
805 keywords to store this state on the server.
807 Example 5:
809 Imagine a speculative mail client that can mark a message as one of
810 work-related ($Work), personal ($Personal), or spam ($Spam). In
811 order to mark a message as personal, the client issues:
813 C: A001 UID STORE 15 +FLAGS.SILENT ($Personal)
814 S: A001 STORE completed
815 C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam)
816 S: A002 STORE completed
818 In order to mark the message as not work, not personal and not spam,
819 the client issues:
821 C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam)
822 S: A003 STORE completed
824 4.2.4. Processing Mailbox Compression (EXPUNGE) Requests
826 A naive disconnected client implementation that supports compressing
827 a mailbox while offline may decide to issue an EXPUNGE command to the
828 server in order to expunge messages marked \Deleted. The problem
829 with this command during synchronization is that it permanently
830 erases all messages with the \Deleted flag set, i.e., even those
831 messages that were marked as \Deleted on the server while the user
832 was offline. Doing this might result in an unpleasant surprise for
833 the user.
835 Fortunately the [UIDPLUS] extension can help in this case as well.
836 The extension introduces UID EXPUNGE command, that, unlike EXPUNGE,
837 takes a UID set parameter, that lists UIDs of all messages that can
838 be expunged. When processing this command the server erases only
842 Melnikov Informational [Page 15]
844 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
847 messages with \Deleted flag listed in the UID list. Thus, messages
848 not listed in the UID set will not be expunged even if they have the
849 \Deleted flag set.
851 Example 6:
853 While the user was offline, 3 messages with UIDs 7, 27, and 65 were
854 marked \Deleted when the user requested to compress the open mailbox.
855 Another client marked a message \Deleted on the server (UID 34).
856 During synchronization, the disconnected client issues:
858 C: A001 UID EXPUNGE 7,27,65
859 S: * ... EXPUNGE
860 S: * ... EXPUNGE
861 S: * ... EXPUNGE
862 S: A001 UID EXPUNGE completed
864 If another client issues UID SEARCH DELETED command (to find all
865 messages with the \Deleted flag) before and after the UID EXPUNGE, it
866 will get:
868 Before:
870 C: B001 UID SEARCH DELETED
871 S: * SEARCH 65 34 27 7
872 S: B001 UID SEARCH completed
874 After:
876 C: B002 UID SEARCH DELETED
877 S: * SEARCH 34
878 S: B002 UID SEARCH completed
880 In the absence of the [UIDPLUS] extension, the following sequence of
881 commands can be used as an approximation. Note: It's possible for
882 another client to mark additional messages as deleted while this
883 sequence is being performed. In this case, these additional messages
884 will be expunged as well.
886 1) Find all messages marked \Deleted on the server.
888 C: A001 UID SEARCH DELETED
889 S: * SEARCH 65 34 27 7
890 S: A001 UID SEARCH completed
892 2) Find all messages that must not be erased (for the previous
893 example the list will consist of the message with UID 34).
898 Melnikov Informational [Page 16]
900 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
903 3) Temporarily remove \Deleted flag on all messages found in step 2.
905 C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted)
906 S: A002 UID STORE completed
908 4) Expunge the mailbox.
910 C: A003 EXPUNGE
911 S: * 20 EXPUNGE
912 S: * 7 EXPUNGE
913 S: * 1 EXPUNGE
914 S: A003 EXPUNGE completed
916 Here, the message with UID 7 has message number 1, with UID 27 has
917 message number 7, and with UID 65 has message number 20.
919 5) Restore \Deleted flag on all messages found when performing step
920 2.
922 C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted)
923 S: A004 UID STORE completed
925 4.2.5. Closing a Mailbox
927 When the disconnected client has to close a mailbox, it should not
928 use the CLOSE command, because CLOSE does a silent EXPUNGE. (Section
929 4.2.4 explains why EXPUNGE should not be used by a disconnected
930 client.) It is safe to use CLOSE only if the mailbox was opened with
931 EXAMINE.
933 If the mailbox was opened with SELECT, the client can use one of the
934 following commands to implicitly close the mailbox and prevent the
935 silent expunge:
937 1) UNSELECT - This is a command described in [UNSELECT] that works as
938 CLOSE, but doesn't cause the silent EXPUNGE. This command is
939 supported by the server if it reports UNSELECT in its CAPABILITY
940 list.
942 2) SELECT <another_mailbox> - SELECT causes implicit CLOSE without
943 EXPUNGE.
945 3) If the client intends to issue LOGOUT after closing the mailbox,
946 it may just issue LOGOUT, because LOGOUT causes implicit CLOSE
947 without EXPUNGE as well.
949 4) SELECT <non_existing_mailbox> - If the client knows a mailbox that
950 doesn't exist or can't be selected, it MAY SELECT it.
954 Melnikov Informational [Page 17]
956 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
959 If the client opened the mailbox with SELECT and just wants to avoid
960 implicit EXPUNGE without closing the mailbox, it may also use the
961 following:
963 5) EXAMINE <mailbox> - Reselect the same mailbox in read-only mode.
965 4.3. Details of "Normal" Synchronization of a Single Mailbox
967 The most common form of synchronization is where the human trusts the
968 integrity of the client's copy of the state of a particular mailbox
969 and simply wants to bring the client's cache up to date so that it
970 accurately reflects the mailbox's current state on the server.
972 4.3.1. Discovering New Messages and Changes to Old Messages
974 Let <lastseenuid> represent the highest UID that the client knows
975 about in this mailbox. Since UIDs are allocated in strictly
976 ascending order, this is simply the UID of the last message in the
977 mailbox that the client knows about. Let <lastseenuid+1> represent
978 <lastseenuid>'s UID plus one. Let <descriptors> represent a list
979 consisting of all the FETCH data item items that the implementation
980 considers part of the descriptor; at a minimum this is just the FLAGS
981 data item, but it usually also includes BODYSTRUCTURE and
982 RFC822.SIZE. At this step, <descriptors> SHOULD NOT include RFC822.
984 With no further information, the client can issue the following two
985 commands:
987 tag1 UID FETCH <lastseenuid+1>:* <descriptors>
988 tag2 UID FETCH 1:<lastseenuid> FLAGS
990 The first command will request some information about "new" messages
991 (i.e., messages received by the server since the last
992 synchronization). It will also allow the client to build a message
993 number to UID map (only for new messages). The second command allows
994 the client to
996 1) update cached flags for old messages;
998 2) find out which old messages got expunged; and
1000 3) build a mapping between message numbers and UIDs (for old
1001 messages).
1003 The order here is significant. We want the server to start returning
1004 the list of new message descriptors as fast as it can, so that the
1005 client can start issuing more FETCH commands, so we start out by
1006 asking for the descriptors of all the messages we know the client
1010 Melnikov Informational [Page 18]
1012 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1015 cannot possibly have cached yet. The second command fetches the
1016 information we need to determine what changes may have occurred to
1017 messages that the client already has cached. Note that the former
1018 command should only be issued if the UIDNEXT value cached by the
1019 client differs from the one returned by the server. Once the client
1020 has issued these two commands, there's nothing more the client can do
1021 with this mailbox until the responses to the first command start
1022 arriving. A clever synchronization program might use this time to
1023 fetch its local cache state from disk or to start the process of
1024 synchronizing another mailbox.
1026 The following is an example of the first FETCH:
1028 C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
1029 RFC822.SIZE)
1031 Note 1: The first FETCH may result in the server's sending a huge
1032 volume of data. A smart disconnected client should use message
1033 ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is
1034 able to execute a different operation between fetching information
1035 for a group of new messages.
1037 Example 7:
1039 Knowing the new UIDNEXT returned by the server on SELECT or EXAMINE
1040 (<uidnext>), the client can split the UID range
1041 <lastseenuid+1>:<uidnext> into groups, e.g., 100 messages. After
1042 that, the client can issue:
1044 C: A011 UID fetch <lastseenuid+1>:<lastseenuid+100>
1045 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
1046 ...
1047 C: A012 UID fetch <lastseenuid+101>:<lastseenuid+200>
1048 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
1049 ...
1050 ...
1051 C: A0FF UID fetch <lastseenuid+901>:<uidnext>
1052 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
1054 Note that unless a SEARCH command is issued, it is impossible to
1055 determine how many messages will fall into a subrange, as UIDs are
1056 not necessarily contiguous.
1058 Note 2: The client SHOULD ignore any unsolicited EXPUNGE responses
1059 received during the first FETCH command. EXPUNGE responses contain
1060 message numbers that are useless to a client that doesn't have the
1061 message-number-to-UID translation table.
1066 Melnikov Informational [Page 19]
1068 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1071 The second FETCH command will result in zero or more untagged fetch
1072 responses. Each response will have a corresponding UID FETCH data
1073 item. All messages that didn't have a matching untagged FETCH
1074 response MUST be removed from the local cache.
1076 For example, if the <lastseenuid> had a value 15000 and the local
1077 cache contained 3 messages with the UIDs 12, 777, and 14999,
1078 respectively, then after receiving the following responses from the
1079 server, the client must remove the message with UID 14999 from its
1080 local cache.
1082 S: * 1 FETCH (UID 12 FLAGS (\Seen))
1083 S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted))
1085 Note 3: If the client is not interested in flag changes (i.e., the
1086 client only wants to know which old messages are still on the
1087 server), the second FETCH command can be substituted with:
1089 tag2 UID SEARCH UID 1:<lastseenuid>
1091 This command will generate less traffic. However, an implementor
1092 should be aware that in order to build the mapping table from message
1093 numbers to UIDs, the output of the SEARCH command MUST be sorted
1094 first, because there is no requirement for a server to return UIDs in
1095 SEARCH response in any particular order.
1097 4.3.2. Searching for "Interesting" Messages.
1099 This step is performed entirely on the client (from the information
1100 received in the step described in 4.3.1), entirely on the server, or
1101 on some combination of both. The decision on what is an
1102 "interesting" message is up to the client software and the human.
1103 One easy criterion that should probably be implemented in any client
1104 is whether the message is "too big" for automatic retrieval, where
1105 "too big" is a parameter defined in the client's configuration.
1107 Another commonly used criterion is the age of a message. For
1108 example, the client may choose to download only messages received in
1109 the last week (in this case, <date> would be today's date minus 7
1110 days):
1112 tag3 UID SEARCH UID <uidset> SINCE <date>
1114 Keep in mind that a date search disregards time and time zone. The
1115 client can avoid doing this search if it specified INTERNALDATE in
1116 <descriptors> on the step described in 4.3.1. If the client did, it
1117 can perform the local search on its message cache.
1122 Melnikov Informational [Page 20]
1124 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1127 At this step, the client also decides what kind of information about
1128 a particular message to fetch from the server. In particular, even
1129 for a message that is considered "too big", the client MAY choose to
1130 fetch some part(s) of it. For example, if the message is a
1131 multipart/mixed containing a text part and a MPEG attachment, there
1132 is no reason for the client not to fetch the text part. The decision
1133 of which part should or should not be fetched can be based on the
1134 information received in the BODYSTRUCTURE FETCH response data item
1135 (i.e., if BODYSTRUCTURE was included in <descriptors> on the step
1136 described in 4.3.1).
1138 4.3.3. Populating Cache with "Interesting" Messages.
1140 Once the client has found out which messages are "interesting", it
1141 can start issuing appropriate FETCH commands for "interesting"
1142 messages or parts thereof.
1144 Note that fetching a message into the disconnected client's local
1145 cache does NOT imply that the human has (or even will) read the
1146 message. Thus, the synchronization program for a disconnected client
1147 should always be careful to use the .PEEK variants of the FETCH data
1148 items that implicitly set the \Seen flag.
1150 Once the last descriptor has arrived and the last FETCH command has
1151 been issued, the client simply needs to process the incoming fetch
1152 items and use them to update the local message cache.
1154 In order to avoid deadlock problems, the client must give processing
1155 of received messages priority over issuing new FETCH commands during
1156 this synchronization process. This may necessitate temporary local
1157 queuing of FETCH requests that cannot be issued without causing a
1158 deadlock. In order to achieve the best use of the "expensive"
1159 network connection, the client will almost certainly need to pay
1160 careful attention to any flow-control information that it can obtain
1161 from the underlying transport connection (usually a TCP connection).
1163 Note: The requirement stated in the previous paragraph might result
1164 in an unpleasant user experience, if followed blindly. For example,
1165 the user might be unwilling to wait for the client to finish
1166 synchronization before starting to process the user's requests. A
1167 smart disconnected client should allow the user to perform requested
1168 operations in between IMAP commands that are part of the
1169 synchronization process. See also Note 1 in Section 4.3.1.
1178 Melnikov Informational [Page 21]
1180 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1183 Example 8:
1185 After fetching a message BODYSTRUCTURE, the client discovers a
1186 complex MIME message. Then, it decides to fetch MIME headers of the
1187 nested MIME messages and some body parts.
1189 C: A011 UID fetch 11 (BODYSTRUCTURE)
1190 S: ...
1191 C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME]
1192 BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME]
1193 BODY[5.MIME] BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME]
1194 BODY[10.MIME] BODY[11.MIME] BODY[12.MIME] BODY[13.MIME]
1195 BODY[14.MIME] BODY[15.MIME] BODY[16.MIME] BODY[17.MIME]
1196 BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] BODY[21.MIME])
1197 S: ...
1198 C: A013 UID fetch 11 (BODY[1.1] BODY[1.2])
1199 S: ...
1200 C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8]
1201 BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16]
1202 BODY[21])
1203 S: ...
1205 4.3.4. User-Initiated Synchronization
1207 After the client has finished the main synchronization process as
1208 described in Sections 4.3.1-4.3.3, the user may optionally request
1209 additional synchronization steps while the client is still online.
1210 This is not any different from the process described in Sections
1211 4.3.2 and 4.3.3.
1213 Typical examples are:
1215 1) fetch all messages selected in UI.
1216 2) fetch all messages marked as \Flagged on the server.
1218 4.4. Special Case: Descriptor-Only Synchronization
1220 For some mailboxes, fetching the descriptors might be the entire
1221 synchronization step. Practical experience with IMAP has shown that
1222 a certain class of mailboxes (e.g., "archival" mailboxes) are used
1223 primarily for long-term storage of important messages that the human
1224 wants to have instantly available on demand but does not want
1225 cluttering up the disconnected client's cache at any other time.
1226 Messages in this kind of mailbox would be fetched exclusively by
1227 explicit actions queued by the local MUA. Thus, the only
1228 synchronization desirable on this kind of mailbox is fetching enough
1229 descriptor information for the user to be able to identify messages
1230 for subsequent download.
1234 Melnikov Informational [Page 22]
1236 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1239 Special mailboxes that receive messages from a high volume, low
1240 priority mailing list might also be in this category, at least when
1241 the human is in a hurry.
1243 4.5. Special Case: Fast New-Only Synchronization
1245 In some cases, the human might be in such a hurry that he or she
1246 doesn't care about changes to old messages, just about new messages.
1247 In this case, the client can skip the UID FETCH command that obtains
1248 the flags and UIDs for old messages (1:<lastseenuid>).
1250 4.6. Special Case: Blind FETCH
1252 In some cases, the human may know (for whatever reason) that he or
1253 she always wants to fetch any new messages in a particular mailbox,
1254 unconditionally. In this case, the client can just fetch the
1255 messages themselves, rather than just the descriptors, by using a
1256 command like:
1258 tag1 UID FETCH <lastseenuid+1>:* (FLAGS BODY.PEEK[])
1260 Note that this example ignores the fact that the messages can be
1261 arbitrary long. The disconnected client MUST always check for
1262 message size before downloading, unless explicitly told otherwise. A
1263 well-behaved client should instead use something like the following:
1265 1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS RFC822.SIZE)".
1267 2) From the message sizes returned in step 1, construct UID set
1268 <required_messages>.
1270 3) Issue "tag2 UID FETCH <required_messages> (BODY.PEEK[])".
1272 or
1274 1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS)".
1276 2) Construct UID set <old_uids> from the responses of step 1.
1278 3) Issue "tag2 SEARCH UID <old_uids> SMALLER <message_limit>".
1279 Construct UID set <required_messages> from the result of the
1280 SEARCH command.
1282 4) Issue "tag3 UID FETCH <required_messages> (BODY.PEEK[])".
1290 Melnikov Informational [Page 23]
1292 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1295 or
1297 1) Issue "tag1 UID FETCH <lastseenuid+1>:* (FLAGS
1298 BODY.PEEK[]<0.<length>>)", where <length> should be replaced with
1299 the maximal message size the client is willing to download.
1301 Note: In response to such a command, the server will only return
1302 partial data if the message is longer than <length>. It will
1303 return the full message data for any message whose size is smaller
1304 than or equal to <length>. In the former case, the client will
1305 not be able to extract the full MIME structure of the message from
1306 the truncated data, so the client should include BODYSTRUCTURE in
1307 the UID FETCH command as well.
1309 5. Implementation Considerations
1311 Below are listed some common implementation pitfalls that should be
1312 considered when implementing a disconnected client.
1314 1) Implementing fake UIDs on the client.
1316 A message scheduled to be uploaded has no UID, as UIDs are
1317 selected by the server. The client may implement fake UIDs
1318 internally in order to reference not-yet-uploaded messages in
1319 further operations. (For example, a message could be scheduled to
1320 be uploaded, but subsequently marked as deleted or copied to
1321 another mailbox). Here, the client MUST NOT under any
1322 circumstances send these fake UIDs to the server. Also, client
1323 implementers should be reminded that according to [IMAP4] a UID is
1324 a 32-bit unsigned integer excluding 0. So, both 4294967295 and
1325 2147483648 are valid UIDs, and 0 and -1 are both invalid. Some
1326 disconnected mail clients have been known to send negative numbers
1327 (e.g., "-1") as message UIDs to servers during synchronization.
1329 Situation 1: The user starts composing a new message, edits it,
1330 saves it, continues to edit it, and saves it again.
1332 A disconnected client may record in its replay log (log of
1333 operations to be replayed on the server during synchronization)
1334 the sequence of operations as shown below. For the purpose of
1335 this situation, we assume that all draft messages are stored in
1336 the mailbox called Drafts on an IMAP server. We will also use the
1337 following conventions: <old_uid> is the UID of the intermediate
1338 version of the draft when it was saved for the first time. This
1339 is a fake UID generated on the client. <new_uid> is the UID of
1340 the final version of the draft. This is another fake UID
1341 generated on the client.
1346 Melnikov Informational [Page 24]
1348 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1351 1) APPEND Drafts (\Seen $MDNSent \Drafts) {<nnn>}
1352 ...first version of the message follows...
1354 2) APPEND Drafts (\Seen $MDNSent \Drafts) {<mmm>}
1355 ...final version of the message follows...
1357 3) STORE <old_uid> +FLAGS (\Deleted)
1359 Step 1 corresponds to the first attempt to save the draft message,
1360 step 2 corresponds to the second attempt to save the draft
1361 message, and step 3 deletes the first version of the draft message
1362 saved in step 1.
1364 A naive disconnected client may send the command in step 3 without
1365 replacing the fake client generated <old_uid> with the value
1366 returned by the server in step 1. A server will probably reject
1367 this command, which will make the client believe that the
1368 synchronization sequence has failed.
1370 2) Section 5.1 discusses common implementation errors related to
1371 error recovery during playback.
1373 3) Don't assume that the disconnected client is the only client used
1374 by the user.
1376 Situation 2: Some clients may use the \Deleted flag as an
1377 indicator that the message should not appear in the user's view.
1378 Usage of the \Deleted flag for this purpose is not safe, as other
1379 clients (e.g., online clients) might EXPUNGE the mailbox at any
1380 time.
1382 4) Beware of data dependencies between synchronization operations.
1384 It might be very tempting for a client writer to perform some
1385 optimizations on the playback log. Such optimizations might
1386 include removing redundant operations (for example, see
1387 optimization 2 in Section 5.3), or their reordering.
1389 It is not always safe to reorder or remove redundant operations
1390 during synchronization because some operations may have
1391 dependencies (as Situation 3 demonstrates). So, if in doubt,
1392 don't do this.
1394 Situation 3: The user copies a message out of a mailbox and then
1395 deletes the mailbox.
1402 Melnikov Informational [Page 25]
1404 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1407 C: A001 SELECT Old-Mail
1408 S: ...
1409 C: A002 UID COPY 111 ToDo
1410 S: A002 OK [COPYUID 1022843345 111 94] Copy completed
1411 ...
1412 C: A015 CLOSE
1413 S: A015 OK Completed
1414 C: A016 DELETE Old-Mail
1415 S: A016 OK Mailbox deletion completed successfully
1417 If the client performs DELETE (tag A016) first and COPY (tag A002)
1418 second, then the COPY fails. Also, the message that the user so
1419 carefully copied into another mailbox has been lost.
1421 5.1. Error Recovery during Playback
1423 Error recovery during synchronization is one of the trickiest parts
1424 to get right. Below, we will discuss certain error conditions and
1425 suggest possible choices for handling them.
1427 1) Lost connection to the server.
1429 The client MUST remember the current position in the playback
1430 (replay) log and replay it starting from the interrupted operation
1431 (the last command issued by the client, but not acknowledged by
1432 the server) the next time it successfully connects to the same
1433 server. If the connection was lost while executing a non-
1434 idempotent IMAP command (see the definition in Section 1), then
1435 when the client is reconnected, it MUST make sure that the
1436 interrupted command was indeed not executed. If it wasn't
1437 executed, the client must restart playback from the interrupted
1438 command, otherwise from the following command.
1440 Upon reconnect, care must be taken in order to properly reapply
1441 logical operations that are represented by multiple IMAP commands,
1442 e.g., UID EXPUNGE emulation when UID EXPUNGE is not supported by
1443 the server (see Section 4.2.4).
1445 Once the client detects that the connection to the server was
1446 lost, it MUST stop replaying its log. There are existing
1447 disconnected clients that, to the great annoyance of users, pop up
1448 an error dialog for each and every playback operation that fails.
1450 2) Copying/appending messages to a mailbox that doesn't exist. (The
1451 server advertises this condition by sending the TRYCREATE response
1452 code in the tagged NO response to the APPEND or COPY command.)
1458 Melnikov Informational [Page 26]
1460 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1463 The user should be advised about the situation and be given one of
1464 the following choices:
1466 a) Try to recreate a mailbox.
1467 b) Copy/upload messages to another mailbox.
1468 c) Skip copy/upload.
1469 d) Abort replay.
1471 3) Copying messages from a mailbox that doesn't exist, or renaming or
1472 getting/changing ACLs [ACL] on a mailbox that doesn't exist:
1474 a) Skip operation.
1475 b) Abort replay.
1477 4) Deleting mailboxes or deleting/expunging messages that no longer
1478 exist.
1480 This is actually is not an error and should be ignored by the
1481 client.
1483 5) Performing operations on messages that no longer exist.
1485 a) Skip operation.
1486 b) Abort replay.
1488 In the case of changing flags on an expunged message, the client
1489 should silently ignore the error.
1491 Note 1: Several synchronization operations map to multiple IMAP
1492 commands (for example, "move" described in 4.2.2). The client must
1493 guarantee atomicity of each such multistep operation. For example,
1494 when performing a "move" between two mailboxes on the same server, if
1495 the server is unable to copy messages, the client MUST NOT attempt to
1496 set the \Deleted flag on the messages being copied, let alone expunge
1497 them. However, the client MAY consider that move operation to have
1498 succeeded even if the server was unable to set the \Deleted flag on
1499 copied messages.
1501 Note 2: Many synchronization operations have data dependencies. A
1502 failed operation must cause all dependent operations to fail as well.
1503 The client should check this and MUST NOT try to perform all
1504 dependent operations blindly (unless the user corrected the original
1505 problem). For example, a message may be scheduled to be appended to
1506 a mailbox on the server and later on the appended message may be
1507 copied to another mailbox. If the APPEND operation fails, the client
1508 must not attempt to COPY the failed message later on. (See also
1509 Section 5, Situation 3).
1514 Melnikov Informational [Page 27]
1516 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1519 5.2. Quality of Implementation Issues
1521 Below, some quality of implementation issues are listed for
1522 disconnected clients. They will help to write a disconnected client
1523 that works correctly, performs synchronization as quickly as possible
1524 (and thus can make the user happier as well as save her some money),
1525 and minimizes the server load:
1527 1) Don't lose information.
1529 No matter how smart your client is in other areas, if it loses
1530 information, users will get very upset.
1532 2) Don't do work unless explicitly asked. Be flexible. Ask all
1533 questions BEFORE starting synchronization, if possible.
1535 3) Minimize traffic.
1537 The client MUST NOT issue a command if the client already received
1538 the required information from the server.
1540 The client MUST make use of UIDPLUS extension if it is supported
1541 by the server.
1543 See also optimization 1 in Section 5.3.
1545 4) Minimize the number of round-trips.
1547 Round-trips kill performance, especially on links with high
1548 latency. Sections 4.2.2.5 and 5.2 give some advice on how to
1549 minimize the number of round-trips.
1551 See also optimization 1 in Section 5.3.
1553 5.3. Optimizations
1555 Some useful optimizations are described in this section. A
1556 disconnected client that supports the recommendations listed below
1557 will give the user a more pleasant experience.
1559 1) The initial OK or PREAUTH responses may contain the CAPABILITY
1560 response code as described in Section 7.1 of [IMAP4]. This
1561 response code gives the same information as returned by the
1562 CAPABILITY command*. A disconnected client that pays attention to
1563 this response code can avoid sending CAPABILITY command and will
1564 save a round-trip.
1570 Melnikov Informational [Page 28]
1572 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1575 * Note: Some servers report in the CAPABILITY response code
1576 extensions that are only relevant in unauthenticated state or in
1577 all states. Such servers usually send another CAPABILITY
1578 response code upon successful authentication using LOGIN or
1579 AUTHENTICATE command (that negotiates no security layer; see
1580 Section 6.2.2 of [IMAP4]). The CAPABILITY response code sent
1581 upon successful LOGIN/AUTHENTICATE might be different from the
1582 CAPABILITY response code in the initial OK response, as
1583 extensions only relevant for unauthenticated state will not be
1584 advertised, and some additional extensions available only in
1585 authenticated and/or selected state will be.
1587 Example 9:
1589 S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS
1590 AUTH=DIGEST-MD5 AUTH=SRP] imap.example.com ready
1591 C: 2 authenticate DIGEST-MD5
1592 S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN
1593 SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND]
1594 User authenticated (no layer)
1596 2) An advanced disconnected client may choose to optimize its replay
1597 log. For example, there might be some operations that are
1598 redundant (the list is not complete):
1600 a) an EXPUNGE followed by another EXPUNGE or CLOSE;
1601 b) changing flags (other than the \Deleted flag) on a message that
1602 gets immediately expunged;
1603 c) opening and closing the same mailbox.
1605 When optimizing, be careful about data dependencies between commands.
1606 For example, if the client is wishing to optimize (see case b, above)
1608 tag1 UID STORE <uid1> +FLAGS (\Deleted)
1609 ...
1610 tag2 UID STORE <uid1> +FLAGS (\Flagged)
1611 ...
1612 tag3 UID COPY <uid1> "Backup"
1613 ...
1614 tag4 UID EXPUNGE <uid1>
1616 it can't remove the second UID STORE command because the message is
1617 being copied before it gets expunged.
1619 In general, it might be a good idea to keep mailboxes open during
1620 synchronization (see case c above), if possible. This can be more
1621 easily achieved in conjunction with optimization 3 described below.
1626 Melnikov Informational [Page 29]
1628 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1631 3) Perform some synchronization steps in parallel, if possible.
1633 Several synchronization steps don't depend on each other and thus
1634 can be performed in parallel. Because the server machine is
1635 usually more powerful than the client machine and can perform some
1636 operations in parallel, this may speed up the total time of
1637 synchronization.
1639 In order to achieve such parallelization, the client will have to
1640 open more than one connection to the same server. Client writers
1641 should not forget about non-trivial cost associated with
1642 establishing a TCP connection and performing an authentication.
1643 The disconnected client MUST NOT use one connection per mailbox.
1644 In most cases, it is sufficient to have two connections. The
1645 disconnected client SHOULD avoid selecting the same mailbox in
1646 more than one connection; see Section 3.1.1 of [RFC2683] for more
1647 details.
1649 Any mailbox synchronization MUST start with checking the
1650 UIDVALIDITY as described in Section 4.1 of this document. The
1651 client MAY use STATUS command to check UID Validity of a non-
1652 selected mailbox. This is preferable to opening many connections
1653 to the same server to perform synchronization of multiple
1654 mailboxes simultaneously. As described in Section 5.3.10 of
1655 [IMAP4], this SHOULD NOT be used on the selected mailbox.
1657 6. IMAP Extensions That May Help
1659 The following extensions can save traffic and/or the number of
1660 round-trips:
1662 1) The use of [UIDPLUS] is discussed in Sections 4.1, 4.2.1, 4.2.2.1
1663 and 4.2.4.
1665 2) The use of the MULTIAPPEND and LITERAL+ extensions for uploading
1666 messages is discussed in Section 4.2.2.5.
1668 3) Use the CONDSTORE extension (see Section 6.1) for quick flag
1669 resynchronization.
1671 6.1. CONDSTORE Extension
1673 An advanced disconnected mail client should use the [CONDSTORE]
1674 extension when it is supported by the server. The client must cache
1675 the value from HIGHESTMODSEQ OK response code received on mailbox
1676 opening and update it whenever the server sends MODSEQ FETCH data
1677 items.
1682 Melnikov Informational [Page 30]
1684 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1687 If the client receives NOMODSEQ OK untagged response instead of
1688 HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value from
1689 its cache and follow the more general instructions in Section 3.
1691 When the client opens the mailbox for synchronization, it first
1692 compares UIDVALIDITY as described in step d-1 in Section 3. If the
1693 cached UIDVALIDITY value matches the one returned by the server, the
1694 client MUST compare the cached value of HIGHESTMODSEQ with the one
1695 returned by the server. If the cached HIGHESTMODSEQ value also
1696 matches the one returned by the server, then the client MUST NOT
1697 fetch flags for cached messages, as they hasn't changed. If the
1698 value on the server is higher than the cached one, the client MAY use
1699 "SEARCH MODSEQ <cached-value>" to find all messages with flags
1700 changed since the last time the client was online and had the mailbox
1701 opened. Alternatively, the client MAY use "FETCH 1:* (FLAGS)
1702 (CHANGEDSINCE <cached-value>)". The latter operation combines
1703 searching for changed messages and fetching new information.
1705 In all cases, the client still needs to fetch information about new
1706 messages (if requested by the user) as well as discover which
1707 messages have been expunged.
1709 Step d ("Server-to-client synchronization") in Section 4 in the
1710 presence of the CONDSTORE extension is amended as follows:
1712 d) "Server-to-client synchronization" - For each mailbox that
1713 requires synchronization, do the following:
1715 1a) Check the mailbox UIDVALIDITY (see section 4.1 for more
1716 details) with SELECT/EXAMINE/STATUS.
1718 If the UIDVALIDITY value returned by the server differs, the
1719 client MUST
1721 * empty the local cache of that mailbox;
1722 * "forget" the cached HIGHESTMODSEQ value for the mailbox;
1723 * remove any pending "actions" that refer to UIDs in that
1724 mailbox (note that this doesn't affect actions performed on
1725 client-generated fake UIDs; see Section 5); and
1726 * skip steps 1b and 2-II;
1728 1b) Check the mailbox HIGHESTMODSEQ. If the cached value is the
1729 same as the one returned by the server, skip fetching message
1730 flags on step 2-II, i.e., the client only has to find out
1731 which messages got expunged.
1738 Melnikov Informational [Page 31]
1740 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1743 2) Fetch the current "descriptors".
1745 I) Discover new messages.
1747 II) Discover changes to old messages and flags for new messages
1748 using
1749 "FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>)" or
1750 "SEARCH MODSEQ <cached-value>".
1752 Discover expunged messages; for example, using
1753 "UID SEARCH 1:<lastseenuid>". (All messages not returned
1754 in this command are expunged.)
1756 3) Fetch the bodies of any "interesting" messages that the client
1757 doesn't already have.
1759 Example 10:
1761 The UIDVALIDITY value is the same, but the HIGHESTMODSEQ value
1762 has changed on the server while the client was offline.
1764 C: A142 SELECT INBOX
1765 S: * 172 EXISTS
1766 S: * 1 RECENT
1767 S: * OK [UNSEEN 12] Message 12 is first unseen
1768 S: * OK [UIDVALIDITY 3857529045] UIDs valid
1769 S: * OK [UIDNEXT 201] Predicted next UID
1770 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
1771 S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
1772 S: * OK [HIGHESTMODSEQ 20010715194045007]
1773 S: A142 OK [READ-WRITE] SELECT completed
1775 After that, either:
1777 C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001)
1778 S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted))
1779 S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk
1780 $AutoJunk $MDNSent))
1781 ...
1782 S: A143 OK FETCH completed
1784 or:
1794 Melnikov Informational [Page 32]
1796 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1799 C: A143 UID SEARCH MODSEQ 20010715194032001 UID 1:20
1800 S: * SEARCH 6 9 11 12 18 19 20 23 (MODSEQ 20010917162500)
1801 S: A143 OK Search complete
1802 C: A144 UID SEARCH 1:20
1803 S: * SEARCH 6 9 ...
1804 S: A144 OK FETCH completed
1806 7. Security Considerations
1808 It is believed that this document does not raise any new security
1809 concerns that are not already present in the base [IMAP4] protocol,
1810 and these issues are discussed in [IMAP4]. Additional security
1811 considerations may be found in different extensions mentioned in this
1812 document; in particular, in [UIDPLUS], [LITERAL+], [CONDSTORE],
1813 [MULTIAPPEND], and [UNSELECT].
1815 Implementers are also reminded about the importance of thorough
1816 testing.
1818 8. References
1820 8.1. Normative References
1822 [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
1823 Requirement Levels", BCP 14, RFC 2119, March 1997.
1825 [IMAP4] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL -
1826 VERSION 4rev1", RFC 3501, March 2003.
1828 [UIDPLUS] Crispin, M., "Internet Message Access Protocol (IMAP) -
1829 UIDPLUS extension", RFC 4315, December 2005.
1831 [LITERAL+] Myers, J., "IMAP4 non-synchronizing literals", RFC
1832 2088, January 1997.
1834 [CONDSTORE] Melnikov, A. and S. Hole, "IMAP Extension for
1835 Conditional STORE Operation or Quick Flag Changes
1836 Resynchronization", RFC 4551, June 2006.
1838 [MULTIAPPEND] Crispin, M., "Internet Message Access Protocol (IMAP) -
1839 MULTIAPPEND Extension", RFC 3502, March 2003.
1841 [UNSELECT] Melnikov, A., "Internet Message Access Protocol (IMAP)
1842 UNSELECT command", RFC 3691, February 2004.
1844 [RFC2683] Leiba, B., "IMAP4 Implementation Recommendations", RFC
1845 2683, September 1999.
1850 Melnikov Informational [Page 33]
1852 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1855 8.2. Informative References
1857 [ACL] Melnikov, A., "IMAP4 Access Control List (ACL)
1858 Extension", RFC 4314, December 2005.
1860 [IMAP-MODEL] Crispin, M., "Distributed Electronic Mail Models in
1861 IMAP4", RFC 1733, December 1994.
1863 9. Acknowledgements
1865 This document is based on version 01 of the text written by Rob
1866 Austein in November 1994.
1868 The editor appreciates comments posted by Mark Crispin to the IMAP
1869 mailing list and the comments/corrections/ideas received from Grant
1870 Baillie, Cyrus Daboo, John G. Myers, Chris Newman, and Timo Sirainen.
1872 The editor would also like to thank the developers of Netscape
1873 Messenger and Mozilla mail clients for providing examples of
1874 disconnected mail clients that served as a base for many
1875 recommendations in this document.
1877 Editor's Address
1879 Alexey Melnikov
1880 Isode Limited
1881 5 Castle Business Village
1882 36 Station Road
1883 Hampton, Middlesex
1884 TW12 2BX
1885 United Kingdom
1887 Phone: +44 77 53759732
1888 EMail: alexey.melnikov@isode.com
1906 Melnikov Informational [Page 34]
1908 RFC 4549 Synch Ops for Disconnected IMAP4 Clients June 2006
1911 Full Copyright Statement
1913 Copyright (C) The Internet Society (2006).
1915 This document is subject to the rights, licenses and restrictions
1916 contained in BCP 78, and except as set forth therein, the authors
1917 retain all their rights.
1919 This document and the information contained herein are provided on an
1920 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
1921 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
1922 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
1923 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
1924 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
1925 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
1927 Intellectual Property
1929 The IETF takes no position regarding the validity or scope of any
1930 Intellectual Property Rights or other rights that might be claimed to
1931 pertain to the implementation or use of the technology described in
1932 this document or the extent to which any license under such rights
1933 might or might not be available; nor does it represent that it has
1934 made any independent effort to identify any such rights. Information
1935 on the procedures with respect to rights in RFC documents can be
1936 found in BCP 78 and BCP 79.
1938 Copies of IPR disclosures made to the IETF Secretariat and any
1939 assurances of licenses to be made available, or the result of an
1940 attempt made to obtain a general license or permission for the use of
1941 such proprietary rights by implementers or users of this
1942 specification can be obtained from the IETF on-line IPR repository at
1943 http://www.ietf.org/ipr.
1945 The IETF invites any interested party to bring to its attention any
1946 copyrights, patents or patent applications, or other proprietary
1947 rights that may cover technology that may be required to implement
1948 this standard. Please address the information to the IETF at
1949 ietf-ipr@ietf.org.
1951 Acknowledgement
1953 Funding for the RFC Editor function is provided by the IETF
1954 Administrative Support Activity (IASA).
1962 Melnikov Informational [Page 35]