One post tagged with "parse"

Parsing E-mail messages in Rust

November 2, 2021 · 5 min read

Project Maintainer

Today we released mail-parser, an e-mail parsing library written in Rust that fully conforms to the Internet Message Format standard (RFC 5322), the Multipurpose Internet Mail Extensions (MIME; RFC 2045–2049) as well as other internet messaging RFCs.

It also supports decoding messages in 41 different character sets including obsolete formats such as UTF-7. All Unicode (UTF-*) and single-byte character sets are handled internally by the library while support for legacy multi-byte encodings of Chinese and Japanese languages such as BIG5 or ISO-2022-JP is provided by the optional dependency encoding_rs.

In general, this library abides by the Postel’s law or Robustness Principle which states that an implementation must be conservative in its sending behavior and liberal in its receiving behavior. This means that mail-parser will make a best effort to parse non-conformant e-mail messages as long as these do not deviate too much from the standard.

Unlike other e-mail parsing libraries that return nested representations of the different MIME parts in a message, this library conforms to RFC 8621, Section 4.1.4 and provides a more human-friendly representation of the message contents consisting of just text body parts, html body parts and attachments. Additionally, conversion to/from HTML and plain text inline body parts is done automatically when the alternative version is missing.

Performance and memory safety were two important factors while designing mail-parser:

Zero-copy: Practically all strings returned by this library are Cow<str> references to the input raw message.
High performance Base64 decoding based on Chromium’s decoder (the fastest non-SIMD decoder).
Fast parsing of message header fields, character set names and HTML entities using perfect hashing.
Written in 100% safe Rust with no external dependencies.
Every function in the library has been fuzzed and meticulously tested with MIRI.
Thoroughly battle-tested with millions of real-world e-mail messages dating from 1995 until today.

The library conforms to all internet messaging RFCs:

And supports 41 different character set encodings:

UTF-8
UTF-16, UTF-16BE, UTF-16LE
UTF-7
US-ASCII
ISO-8859–1
ISO-8859–2
ISO-8859–3
ISO-8859–4
ISO-8859–5
ISO-8859–6
ISO-8859–7
ISO-8859–8
ISO-8859–9
ISO-8859–10
ISO-8859–13
ISO-8859–14
ISO-8859–15
ISO-8859–16
CP1250
CP1251
CP1252
CP1253
CP1254
CP1255
CP1256
CP1257
CP1258
KOI8-R
KOI8_U
MACINTOSH
IBM850
TIS-620
SHIFT_JIS
BIG5
EUC-JP
EUC-KR
GB18030
GBK
ISO-2022-JP
WINDOWS-874
IBM-866

Using the library is straightforward:

    let input = concat!(
            "From: Art Vandelay <[email protected]> (Vandelay Industries)\n",
            "To: \"Colleagues\": \"James Smythe\" <[email protected]>; Friends:\n",
            "    [email protected], =?UTF-8?Q?John_Sm=C3=AEth?= <[email protected]>;\n",
            "Date: Sat, 20 Nov 2021 14:22:01 -0800\n",
            "Subject: Why not both importing AND exporting? =?utf-8?b?4pi6?=\n",
            "Content-Type: multipart/mixed; boundary=\"festivus\";\n\n",
            "--festivus\n",
            "Content-Type: text/html; charset=\"us-ascii\"\n",
            "Content-Transfer-Encoding: base64\n\n",
            "PGh0bWw+PHA+SSB3YXMgdGhpbmtpbmcgYWJvdXQgcXVpdHRpbmcgdGhlICZsZHF1bztle\n",
            "HBvcnRpbmcmcmRxdW87IHRvIGZvY3VzIGp1c3Qgb24gdGhlICZsZHF1bztpbXBvcnRpbm\n",
            "cmcmRxdW87LDwvcD48cD5idXQgdGhlbiBJIHRob3VnaHQsIHdoeSBub3QgZG8gYm90aD8\n",
            "gJiN4MjYzQTs8L3A+PC9odG1sPg==\n",
            "--festivus\n",
            "Content-Type: message/rfc822\n\n",
            "From: \"Cosmo Kramer\" <[email protected]>\n",
            "Subject: Exporting my book about coffee tables\n",
            "Content-Type: multipart/mixed; boundary=\"giddyup\";\n\n",
            "--giddyup\n",
            "Content-Type: text/plain; charset=\"utf-16\"\n",
            "Content-Transfer-Encoding: quoted-printable\n\n",
            "=FF=FE=0C!5=D8\"=DD5=D8)=DD5=D8-=DD =005=D8*=DD5=D8\"=DD =005=D8\"=\n",
            "=DD5=D85=DD5=D8-=DD5=D8,=DD5=D8/=DD5=D81=DD =005=D8*=DD5=D86=DD =\n",
            "=005=D8=1F=DD5=D8,=DD5=D8,=DD5=D8(=DD =005=D8-=DD5=D8)=DD5=D8\"=\n",
            "=DD5=D8=1E=DD5=D80=DD5=D8\"=DD!=00\n",
            "--giddyup\n",
            "Content-Type: image/gif; name*1=\"about \"; name*0=\"Book \";\n",
            "              name*2*=utf-8''%e2%98%95 tables.gif\n",
            "Content-Transfer-Encoding: Base64\n",
            "Content-Disposition: attachment\n\n",
            "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\n",
            "--giddyup--\n",
            "--festivus--\n",
        )
        .as_bytes();
    
        let message = Message::parse(input);
    
        // Parses addresses (including comments), lists and groups
        assert_eq!(
            message.get_from(),
            &Address::Address(Addr {
                name: Some("Art Vandelay (Vandelay Industries)".into()),
                address: Some("[email protected]".into())
            })
        );
        assert_eq!(
            message.get_to(),
            &Address::GroupList(vec![
                Group {
                    name: Some("Colleagues".into()),
                    addresses: vec![Addr {
                        name: Some("James Smythe".into()),
                        address: Some("[email protected]".into())
                    }]
                },
                Group {
                    name: Some("Friends".into()),
                    addresses: vec![
                        Addr {
                            name: None,
                            address: Some("[email protected]".into())
                        },
                        Addr {
                            name: Some("John Smîth".into()),
                            address: Some("[email protected]".into())
                        }
                    ]
                }
            ])
        );
    
        assert_eq!(
            message.get_date().unwrap().to_iso8601(),
            "2021-11-20T14:22:01-08:00"
        );
    
        // RFC2047 support for encoded text in message readers
        assert_eq!(
            message.get_subject().unwrap(),
            "Why not both importing AND exporting? ☺"
        );
    
        // HTML and text body parts are returned conforming to RFC8621, Section 4.1.4 
        assert_eq!(
            message.get_html_body(0).unwrap().to_string(),
            concat!(
                "<html><p>I was thinking about quitting the &ldquo;exporting&rdquo; to ",
                "focus just on the &ldquo;importing&rdquo;,</p><p>but then I thought,",
                " why not do both? &#x263A;</p></html>"
            )
        );
    
        // HTML parts are converted to plain text (and viceversa) when missing
        assert_eq!(
            message.get_text_body(0).unwrap().to_string(),
            concat!(
                "I was thinking about quitting the “exporting” to focus just on the",
                " “importing”,\nbut then I thought, why not do both? ☺\n"
            )
        );
    
        // Supports nested messages as well as multipart/digest
        let nested_message = match message.get_attachment(0).unwrap() {
            MessagePart::Message(v) => v,
            _ => unreachable!(),
        };
    
        assert_eq!(
            nested_message.get_subject().unwrap(),
            "Exporting my book about coffee tables"
        );
    
        // Handles UTF-* as well as many legacy encodings
        assert_eq!(
            nested_message.get_text_body(0).unwrap().to_string(),
            "ℌ𝔢𝔩𝔭 𝔪𝔢 𝔢𝔵𝔭𝔬𝔯𝔱 𝔪𝔶 𝔟𝔬𝔬𝔨 𝔭𝔩𝔢𝔞𝔰𝔢!"
        );
        assert_eq!(
            nested_message.get_html_body(0).unwrap().to_string(),
            "<html><body>ℌ𝔢𝔩𝔭 𝔪𝔢 𝔢𝔵𝔭𝔬𝔯𝔱 𝔪𝔶 𝔟𝔬𝔬𝔨 𝔭𝔩𝔢𝔞𝔰𝔢!</body></html>"
        );
    
        let nested_attachment = match nested_message.get_attachment(0).unwrap() {
            MessagePart::Binary(v) => v,
            _ => unreachable!(),
        };
    
        assert_eq!(nested_attachment.len(), 42);
    
        // Full RFC2231 support for continuations and character sets
        assert_eq!(
            nested_attachment
                .get_header()
                .unwrap()
                .get_content_type()
                .unwrap()
                .get_attribute("name")
                .unwrap(),
            "Book about ☕ tables.gif"
        );
    
        // Integrates with Serde
        println!("{}", serde_json::to_string_pretty(&message).unwrap());
        println!("{}", serde_yaml::to_string(&message).unwrap());

The mail-parser library is available on Crates.io (https://crates.io/crates/mail-parser) and the documentation at https://docs.rs/mail-parser/.

Please follow us on Twitter for updates!