1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/docs/rfc/rfc4978.txt	Mon Sep 14 15:17:45 2009 +0900
     1.3 @@ -0,0 +1,507 @@
     1.4 +
     1.5 +
     1.6 +
     1.7 +
     1.8 +
     1.9 +
    1.10 +Network Working Group                                     A. Gulbrandsen
    1.11 +Request for Comments: 4978                        Oryx Mail Systems GmbH
    1.12 +Category: Standards Track                                    August 2007
    1.13 +
    1.14 +
    1.15 +                      The IMAP COMPRESS Extension
    1.16 +
    1.17 +Status of this Memo
    1.18 +
    1.19 +   This document specifies an Internet standards track protocol for the
    1.20 +   Internet community, and requests discussion and suggestions for
    1.21 +   improvements.  Please refer to the current edition of the "Internet
    1.22 +   Official Protocol Standards" (STD 1) for the standardization state
    1.23 +   and status of this protocol.  Distribution of this memo is unlimited.
    1.24 +
    1.25 +Abstract
    1.26 +
    1.27 +   The COMPRESS extension allows an IMAP connection to be effectively
    1.28 +   and efficiently compressed.
    1.29 +
    1.30 +   Table of Contents
    1.31 +
    1.32 +   1. Introduction and Overview .......................................2
    1.33 +   2. Conventions Used in This Document ...............................2
    1.34 +   3. The COMPRESS Command ............................................3
    1.35 +   4. Compression Efficiency ..........................................4
    1.36 +   5. Formal Syntax ...................................................6
    1.37 +   6. Security Considerations .........................................6
    1.38 +   7. IANA Considerations .............................................6
    1.39 +   8. Acknowledgements ................................................7
    1.40 +   9. References ......................................................7
    1.41 +      9.1. Normative References .......................................7
    1.42 +      9.2. Informative References .....................................7
    1.43 +
    1.44 +
    1.45 +
    1.46 +
    1.47 +
    1.48 +
    1.49 +
    1.50 +
    1.51 +
    1.52 +
    1.53 +
    1.54 +
    1.55 +
    1.56 +
    1.57 +
    1.58 +
    1.59 +
    1.60 +
    1.61 +Gulbrandsen                 Standards Track                     [Page 1]
    1.62 +
    1.63 +RFC 4978              The IMAP COMPRESS Extension            August 2007
    1.64 +
    1.65 +
    1.66 +1.  Introduction and Overview
    1.67 +
    1.68 +   A server which supports the COMPRESS extension indicates this with
    1.69 +   one or more capability names consisting of "COMPRESS=" followed by a
    1.70 +   supported compression algorithm name as described in this document.
    1.71 +
    1.72 +   The goal of COMPRESS is to reduce the bandwidth usage of IMAP.
    1.73 +
    1.74 +   Compared to PPP compression (see [RFC1962]) and modem-based
    1.75 +   compression (see [MNP] and [V42BIS]), COMPRESS offers much better
    1.76 +   compression efficiency.  COMPRESS can be used together with Transport
    1.77 +   Security Layer (TLS) [RFC4346], Simple Authentication and Security
    1.78 +   layer (SASL) encryption, Virtual Private Networks (VPNs), etc.
    1.79 +   Compared to TLS compression [RFC3749], COMPRESS has the following
    1.80 +   (dis)advantages:
    1.81 +
    1.82 +   - COMPRESS can be implemented easily both by IMAP servers and
    1.83 +     clients.
    1.84 +
    1.85 +   - IMAP COMPRESS benefits from an intimate knowledge of the IMAP
    1.86 +     protocol's state machine, allowing for dynamic and aggressive
    1.87 +     optimization of the underlying compression algorithm's parameters.
    1.88 +
    1.89 +   - When the TLS layer implements compression, any protocol using that
    1.90 +     layer can transparently benefit from that compression (e.g., SMTP
    1.91 +     and IMAP).  COMPRESS is specific to IMAP.
    1.92 +
    1.93 +   In order to increase interoperation, it is desirable to have as few
    1.94 +   different compression algorithms as possible, so this document
    1.95 +   specifies only one.  The DEFLATE algorithm (defined in [RFC1951]) is
    1.96 +   standard, widely available and fairly efficient, so it is the only
    1.97 +   algorithm defined by this document.
    1.98 +
    1.99 +   In order to increase interoperation, IMAP servers that advertise this
   1.100 +   extension SHOULD also advertise the TLS DEFLATE compression mechanism
   1.101 +   as defined in [RFC3749].  IMAP clients MAY use either COMPRESS or TLS
   1.102 +   compression, however, if the client and server support both, it is
   1.103 +   RECOMMENDED that the client choose TLS compression.
   1.104 +
   1.105 +   The extension adds one new command (COMPRESS) and no new responses.
   1.106 +
   1.107 +2.  Conventions Used in This Document
   1.108 +
   1.109 +   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   1.110 +   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   1.111 +   document are to be interpreted as described in [RFC2119].
   1.112 +
   1.113 +   Formal syntax is defined by [RFC4234] as modified by [RFC3501].
   1.114 +
   1.115 +
   1.116 +
   1.117 +Gulbrandsen                 Standards Track                     [Page 2]
   1.118 +
   1.119 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.120 +
   1.121 +
   1.122 +   In the examples, "C:" and "S:" indicate lines sent by the client and
   1.123 +   server respectively. "[...]" denotes elision.
   1.124 +
   1.125 +3.  The COMPRESS Command
   1.126 +
   1.127 +   Arguments: Name of compression mechanism: "DEFLATE".
   1.128 +
   1.129 +   Responses: None
   1.130 +
   1.131 +   Result: OK The server will compress its responses and expects the
   1.132 +              client to compress its commands.
   1.133 +           NO Compression is already active via another layer.
   1.134 +          BAD Command unknown, invalid or unknown argument, or COMPRESS
   1.135 +              already active.
   1.136 +
   1.137 +   The COMPRESS command instructs the server to use the named
   1.138 +   compression mechanism ("DEFLATE" is the only one defined) for all
   1.139 +   commands and/or responses after COMPRESS.
   1.140 +
   1.141 +   The client MUST NOT send any further commands until it has seen the
   1.142 +   result of COMPRESS.  If the response was OK, the client MUST compress
   1.143 +   starting with the first command after COMPRESS.  If the server
   1.144 +   response was BAD or NO, the client MUST NOT turn on compression.
   1.145 +
   1.146 +   If the server responds NO because it knows that the same mechanism is
   1.147 +   active already (e.g., because TLS has negotiated the same mechanism),
   1.148 +   it MUST send COMPRESSIONACTIVE as resp-text-code (see [RFC3501],
   1.149 +   Section 7.1), and the resp-text SHOULD say which layer compresses.
   1.150 +
   1.151 +   If the server issues an OK response, the server MUST compress
   1.152 +   starting immediately after the CRLF which ends the tagged OK
   1.153 +   response.  (Responses issued by the server before the OK response
   1.154 +   will, of course, still be uncompressed.)  If the server issues a BAD
   1.155 +   or NO response, the server MUST NOT turn on compression.
   1.156 +
   1.157 +   For DEFLATE (as for many other compression mechanisms), the
   1.158 +   compressor can trade speed against quality.  When decompressing there
   1.159 +   isn't much of a tradeoff.  Consequently, the client and server are
   1.160 +   both free to pick the best reasonable rate of compression for the
   1.161 +   data they send.
   1.162 +
   1.163 +   When COMPRESS is combined with TLS (see [RFC4346]) or SASL (see
   1.164 +   [RFC4422]) security layers, the sending order of the three extensions
   1.165 +   MUST be first COMPRESS, then SASL, and finally TLS.  That is, before
   1.166 +   data is transmitted it is first compressed.  Second, if a SASL
   1.167 +   security layer has been negotiated, the compressed data is then
   1.168 +   signed and/or encrypted accordingly.  Third, if a TLS security layer
   1.169 +   has been negotiated, the data from the previous step is signed and/or
   1.170 +
   1.171 +
   1.172 +
   1.173 +Gulbrandsen                 Standards Track                     [Page 3]
   1.174 +
   1.175 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.176 +
   1.177 +
   1.178 +   encrypted accordingly.  When receiving data, the processing order
   1.179 +   MUST be reversed.  This ensures that before sending, data is
   1.180 +   compressed before it is encrypted, independent of the order in which
   1.181 +   the client issues COMPRESS, AUTHENTICATE, and STARTTLS.
   1.182 +
   1.183 +   The following example illustrates how commands and responses are
   1.184 +   compressed during a simple login sequence:
   1.185 +
   1.186 +        S: * OK [CAPABILITY IMAP4REV1 STARTTLS COMPRESS=DEFLATE]
   1.187 +        C: a starttls
   1.188 +        S: a OK TLS active
   1.189 +
   1.190 +            From this point on, everything is encrypted.
   1.191 +
   1.192 +        C: b login arnt tnra
   1.193 +        S: b OK Logged in as arnt
   1.194 +        C: c compress deflate
   1.195 +        S: d OK DEFLATE active
   1.196 +
   1.197 +            From this point on, everything is compressed before being
   1.198 +            encrypted.
   1.199 +
   1.200 +   The following example demonstrates how a server may refuse to
   1.201 +   compress twice:
   1.202 +
   1.203 +        S: * OK [CAPABILITY IMAP4REV1 STARTTLS COMPRESS=DEFLATE]
   1.204 +        [...]
   1.205 +        C: c compress deflate
   1.206 +        S: c NO [COMPRESSIONACTIVE] DEFLATE active via TLS
   1.207 +
   1.208 +4.  Compression Efficiency
   1.209 +
   1.210 +   This section is informative, not normative.
   1.211 +
   1.212 +   IMAP poses some unusual problems for a compression layer.
   1.213 +
   1.214 +   Upstream is fairly simple.  Most IMAP clients send the same few
   1.215 +   commands again and again, so any compression algorithm that can
   1.216 +   exploit repetition works efficiently.  The APPEND command is an
   1.217 +   exception; clients that send many APPEND commands may want to
   1.218 +   surround large literals with flushes in the same way as is
   1.219 +   recommended for servers later in this section.
   1.220 +
   1.221 +   Downstream has the unusual property that several kinds of data are
   1.222 +   sent, confusing all dictionary-based compression algorithms.
   1.223 +
   1.224 +
   1.225 +
   1.226 +
   1.227 +
   1.228 +
   1.229 +Gulbrandsen                 Standards Track                     [Page 4]
   1.230 +
   1.231 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.232 +
   1.233 +
   1.234 +   One type is IMAP responses.  These are highly compressible; zlib
   1.235 +   using its least CPU-intensive setting compresses typical responses to
   1.236 +   25-40% of their original size.
   1.237 +
   1.238 +   Another type is email headers.  These are equally compressible, and
   1.239 +   benefit from using the same dictionary as the IMAP responses.
   1.240 +
   1.241 +   A third type is email body text.  Text is usually fairly short and
   1.242 +   includes much ASCII, so the same compression dictionary will do a
   1.243 +   good job here, too.  When multiple messages in the same thread are
   1.244 +   read at the same time, quoted lines etc. can often be compressed
   1.245 +   almost to zero.
   1.246 +
   1.247 +   Finally, attachments (non-text email bodies) are transmitted, either
   1.248 +   in binary form or encoded with base-64.
   1.249 +
   1.250 +   When attachments are retrieved in binary form, DEFLATE may be able to
   1.251 +   compress them, but the format of the attachment is usually not IMAP-
   1.252 +   like, so the dictionary built while compressing IMAP does not help.
   1.253 +   The compressor has to adapt its dictionary from IMAP to the
   1.254 +   attachment's format, and then back.  A few file formats aren't
   1.255 +   compressible at all using deflate, e.g., .gz, .zip, and .jpg files.
   1.256 +
   1.257 +   When attachments are retrieved in base-64 form, the same problems
   1.258 +   apply, but the base-64 encoding adds another problem.  8-bit
   1.259 +   compression algorithms such as deflate work well on 8-bit file
   1.260 +   formats, however base-64 turns a file into something resembling 6-bit
   1.261 +   bytes, hiding most of the 8-bit file format from the compressor.
   1.262 +
   1.263 +   When using the zlib library (see [RFC1951]), the functions
   1.264 +   deflateInit2(), deflate(), inflateInit2(), and inflate() suffice to
   1.265 +   implement this extension.  The windowBits value must be in the range
   1.266 +   -8 to -15, or else deflateInit2() uses the wrong format.
   1.267 +   deflateParams() can be used to improve compression rate and resource
   1.268 +   use.  The Z_FULL_FLUSH argument to deflate() can be used to clear the
   1.269 +   dictionary (the receiving peer does not need to do anything).
   1.270 +
   1.271 +   A client can improve downstream compression by implementing BINARY
   1.272 +   (defined in [RFC3516]) and using FETCH BINARY instead of FETCH BODY.
   1.273 +   In the author's experience, the improvement ranges from 5% to 40%
   1.274 +   depending on the attachment being downloaded.
   1.275 +
   1.276 +   A server can improve downstream compression if it hints to the
   1.277 +   compressor that the data type is about to change strongly, e.g., by
   1.278 +   sending a Z_FULL_FLUSH at the start and end of large non-text
   1.279 +   literals (before and after '*CHAR8' in the definition of literal in
   1.280 +   RFC 3501, page 86).  Small literals are best left alone.  A possible
   1.281 +   boundary is 5k.
   1.282 +
   1.283 +
   1.284 +
   1.285 +Gulbrandsen                 Standards Track                     [Page 5]
   1.286 +
   1.287 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.288 +
   1.289 +
   1.290 +   A server can improve the CPU efficiency both of the server and the
   1.291 +   client if it adjusts the compression level (e.g., using the
   1.292 +   deflateParams() function in zlib) at these points, to avoid trying to
   1.293 +   compress incompressible attachments.  A very simple strategy is to
   1.294 +   change the level to 0 at the start of a literal provided the first
   1.295 +   two bytes are either 0x1F 0x8B (as in deflate-compressed files) or
   1.296 +   0xFF 0xD8 (JPEG), and to keep it at 1-5 the rest of the time.  More
   1.297 +   complex strategies are possible.
   1.298 +
   1.299 +5.  Formal Syntax
   1.300 +
   1.301 +   The following syntax specification uses the Augmented Backus-Naur
   1.302 +   Form (ABNF) notation as specified in [RFC4234].  This syntax augments
   1.303 +   the grammar specified in [RFC3501].  [RFC4234] defines SP and
   1.304 +   [RFC3501] defines command-auth, capability, and resp-text-code.
   1.305 +
   1.306 +   Except as noted otherwise, all alphabetic characters are case-
   1.307 +   insensitive.  The use of upper or lower case characters to define
   1.308 +   token strings is for editorial clarity only.  Implementations MUST
   1.309 +   accept these strings in a case-insensitive fashion.
   1.310 +
   1.311 +       command-auth =/ compress
   1.312 +
   1.313 +       compress    = "COMPRESS" SP algorithm
   1.314 +
   1.315 +       capability  =/ "COMPRESS=" algorithm
   1.316 +                     ;; multiple COMPRESS capabilities allowed
   1.317 +
   1.318 +       algorithm   = "DEFLATE"
   1.319 +
   1.320 +       resp-text-code =/ "COMPRESSIONACTIVE"
   1.321 +
   1.322 +   Note that due the syntax of capability names, future algorithm names
   1.323 +   must be atoms.
   1.324 +
   1.325 +6.  Security Considerations
   1.326 +
   1.327 +   As for TLS compression [RFC3749].
   1.328 +
   1.329 +7.  IANA Considerations
   1.330 +
   1.331 +   The IANA has added COMPRESS=DEFLATE to the list of IMAP capabilities.
   1.332 +
   1.333 +
   1.334 +
   1.335 +
   1.336 +
   1.337 +
   1.338 +
   1.339 +
   1.340 +
   1.341 +Gulbrandsen                 Standards Track                     [Page 6]
   1.342 +
   1.343 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.344 +
   1.345 +
   1.346 +8.  Acknowledgements
   1.347 +
   1.348 +   Eric Burger, Dave Cridland, Tony Finch, Ned Freed, Philip Guenther,
   1.349 +   Randall Gellens, Tony Hansen, Cullen Jennings, Stephane Maes, Alexey
   1.350 +   Melnikov, Lyndon Nerenberg, and Zoltan Ordogh have all helped with
   1.351 +   this document.
   1.352 +
   1.353 +   The author would also like to thank various people in the rooms at
   1.354 +   meetings, whose help is real, but not reflected in the author's
   1.355 +   mailbox.
   1.356 +
   1.357 +9.  References
   1.358 +
   1.359 +9.1.  Normative References
   1.360 +
   1.361 +   [RFC1951]  Deutsch, P., "DEFLATE Compressed Data Format Specification
   1.362 +              version 1.3", RFC 1951, May 1996.
   1.363 +
   1.364 +   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
   1.365 +              Requirement Levels", BCP 14, RFC 2119, March 1997.
   1.366 +
   1.367 +   [RFC3501]  Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION
   1.368 +              4rev1", RFC 3501, March 2003.
   1.369 +
   1.370 +   [RFC4234]  Crocker, D. and P. Overell, "Augmented BNF for Syntax
   1.371 +              Specifications: ABNF", RFC 4234, October 2005.
   1.372 +
   1.373 +9.2.  Informative References
   1.374 +
   1.375 +   [RFC1962]  Rand, D., "The PPP Compression Control Protocol (CCP)",
   1.376 +              RFC 1962, June 1996.
   1.377 +
   1.378 +   [RFC3516]  Nerenberg, L., "IMAP4 Binary Content Extension", RFC 3516,
   1.379 +              April 2003.
   1.380 +
   1.381 +   [RFC3749]  Hollenbeck, S., "Transport Layer Security Protocol
   1.382 +              Compression Methods", RFC 3749, May 2004.
   1.383 +
   1.384 +   [RFC4346]  Dierks, T. and E. Rescorla, "The Transport Layer Security
   1.385 +              (TLS) Protocol Version 1.1", RFC 4346, April 2006.
   1.386 +
   1.387 +   [RFC4422]  Melnikov, A. and  K. Zeilenga, "Simple Authentication and
   1.388 +              Security Layer (SASL)", RFC 4422, June 2006.
   1.389 +
   1.390 +   [V42BIS]   ITU, "V.42bis: Data compression procedures for data
   1.391 +              circuit-terminating equipment (DCE) using error correction
   1.392 +              procedures", http://www.itu.int/rec/T-REC-V.42bis, January
   1.393 +              1990.
   1.394 +
   1.395 +
   1.396 +
   1.397 +Gulbrandsen                 Standards Track                     [Page 7]
   1.398 +
   1.399 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.400 +
   1.401 +
   1.402 +   [MNP]      Gilbert Held, "The Complete Modem Reference", Second
   1.403 +              Edition, Wiley Professional Computing, ISBN 0-471-00852-4,
   1.404 +              May 1994.
   1.405 +
   1.406 +Author's Address
   1.407 +
   1.408 +    Arnt Gulbrandsen
   1.409 +    Oryx Mail Systems GmbH
   1.410 +    Schweppermannstr. 8
   1.411 +    D-81671 Muenchen
   1.412 +    Germany
   1.413 +
   1.414 +    Fax: +49 89 4502 9758
   1.415 +    EMail: arnt@oryx.com
   1.416 +
   1.417 +
   1.418 +
   1.419 +
   1.420 +
   1.421 +
   1.422 +
   1.423 +
   1.424 +
   1.425 +
   1.426 +
   1.427 +
   1.428 +
   1.429 +
   1.430 +
   1.431 +
   1.432 +
   1.433 +
   1.434 +
   1.435 +
   1.436 +
   1.437 +
   1.438 +
   1.439 +
   1.440 +
   1.441 +
   1.442 +
   1.443 +
   1.444 +
   1.445 +
   1.446 +
   1.447 +
   1.448 +
   1.449 +
   1.450 +
   1.451 +
   1.452 +
   1.453 +Gulbrandsen                 Standards Track                     [Page 8]
   1.454 +
   1.455 +RFC 4978              The IMAP COMPRESS Extension            August 2007
   1.456 +
   1.457 +
   1.458 +Full Copyright Statement
   1.459 +
   1.460 +   Copyright (C) The IETF Trust (2007).
   1.461 +
   1.462 +   This document is subject to the rights, licenses and restrictions
   1.463 +   contained in BCP 78, and except as set forth therein, the authors
   1.464 +   retain all their rights.
   1.465 +
   1.466 +   This document and the information contained herein are provided on an
   1.467 +   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   1.468 +   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   1.469 +   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   1.470 +   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   1.471 +   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   1.472 +   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
   1.473 +
   1.474 +Intellectual Property
   1.475 +
   1.476 +   The IETF takes no position regarding the validity or scope of any
   1.477 +   Intellectual Property Rights or other rights that might be claimed to
   1.478 +   pertain to the implementation or use of the technology described in
   1.479 +   this document or the extent to which any license under such rights
   1.480 +   might or might not be available; nor does it represent that it has
   1.481 +   made any independent effort to identify any such rights.  Information
   1.482 +   on the procedures with respect to rights in RFC documents can be
   1.483 +   found in BCP 78 and BCP 79.
   1.484 +
   1.485 +   Copies of IPR disclosures made to the IETF Secretariat and any
   1.486 +   assurances of licenses to be made available, or the result of an
   1.487 +   attempt made to obtain a general license or permission for the use of
   1.488 +   such proprietary rights by implementers or users of this
   1.489 +   specification can be obtained from the IETF on-line IPR repository at
   1.490 +   http://www.ietf.org/ipr.
   1.491 +
   1.492 +   The IETF invites any interested party to bring to its attention any
   1.493 +   copyrights, patents or patent applications, or other proprietary
   1.494 +   rights that may cover technology that may be required to implement
   1.495 +   this standard.  Please address the information to the IETF at
   1.496 +   ietf-ipr@ietf.org.
   1.497 +
   1.498 +
   1.499 +
   1.500 +
   1.501 +
   1.502 +
   1.503 +
   1.504 +
   1.505 +
   1.506 +
   1.507 +
   1.508 +
   1.509 +Gulbrandsen                 Standards Track                     [Page 9]
   1.510 +