Summary

An email generated by a Python application can look perfectly normal in Gmail and iCloud and still arrive in Outlook with an empty body and no attachment. It isn't rejected, it isn't marked as spam, and SPF and DMARC pass. There are several possible reasons for this. One that is easy to miss, and which we ran into while developing an application, is the line endings.

In our case the MIME structure was fine. The message was serialised with EmailMessage.as_bytes(), which uses \n line endings under the default policy, and the resulting bytes were passed straight to smtplib's low-level data() method. Nothing in between converted them to the \r\n that SMTP expects. Some recipients coped with that. In our case, Outlook did not. Serialising with CRLF fixed it.

What the problem looked like

The message had an ordinary structure:

multipart/mixed
├── multipart/alternative
│   ├── text/plain
│   └── text/html
└── application/pdf

The same send looked like this at each recipient:

RecipientResultDKIM
BCC copy on the sending mail serverIntact, multipart/mixed—
iCloudIntact, body and PDF presentPASS
GmailIntact, body and PDF present—
OutlookEmpty body, no attachmentFAIL

The Outlook copy had this header:

Content-Type: text/plain

There was no multipart boundary, no charset and no Content-Transfer-Encoding. The Outlook copy was being presented as text/plain rather than the expected multipart message. The message was accepted and delivered to the inbox, with SPF and DMARC passing, so it didn't look like a delivery problem.

What we ruled out first

We started with the MIME structure, which is where most people would look. It seemed a reasonable suspect, so we simplified it. Nothing changed. We then tried the following, one at a time:

  • removing multipart/related and the inline image;
  • adding a name parameter to the PDF part;
  • changing the Content-Transfer-Encoding of the HTML part;
  • removing MIME-Version: 1.0 from the sub-parts.

None of them made any difference. A message with no attachment at all was affected too, so the PDF wasn't the cause either. And a PDF sent through the same mail server from an ordinary desktop email client reached Outlook without any trouble.

In hindsight, all of these changes had the same limitation. They altered the logical structure of the message, and the logical structure was never the problem.

Comparing the different copies

Two comparisons were more useful than any of the header changes.

The first was the copies themselves. The BCC copy held on the sending mail server was intact. So was the iCloud copy. Only the Outlook copy was broken. That told us the MIME structure had survived the journey from the application to the mail server, and that the problem only became visible on the Microsoft side. It didn't tell us why.

The second was DKIM. The same signature passed at iCloud and failed at Outlook. That is a strong sign that the two recipients were not verifying the same bytes, though it doesn't show where or how they came to differ. The DKIM failure was a clue, not the cause.

Both comparisons pointed away from the MIME structure and towards the bytes.

Looking at the actual SMTP bytes

The sending code was roughly this:

msg = EmailMessage()
# ... headers, body, attachment ...

raw = msg.as_bytes()

server.mail(sender)
server.rcpt(recipient)
server.data(raw)

We pointed the application at a test SMTP server and captured exactly what it received during the DATA phase:

Message typeCRLF (\r\n)Bare LF (\n)
HTML with PDF attachment0over 1,000
No attachment, text and link only0over 100

There wasn't a single CRLF anywhere in either message.

It helps to keep four things separate here, because they are easy to blur together:

  • MIME structure is the tree of parts, and it was correct.
  • MIME serialisation is how that tree is turned into text, and this is where \n came from.
  • SMTP DATA bytes are what actually goes to the server, and they contained only bare LFs.
  • Recipient-side parsing is what each provider makes of those bytes, and it varied.

Everything we had changed so far belonged to the first layer. The fault was in the second and third.

LF vs CRLF

A bare LF is a line feed that isn't immediately preceded by a carriage return. SMTP lines are meant to end in CRLF.

Some systems tolerate bare LFs. In our tests Gmail and iCloud displayed the message correctly, but what they do with bare LFs internally isn't something these tests can tell us.

Microsoft's position has changed over time. Microsoft 365 and Office 365 used to remove bare line feeds from messages so they could be delivered to older servers, but they no longer do so, in order to better support security standards such as DKIM.

That documentation is about messages that can't be relayed onwards, not about incoming mail. It doesn't describe what we saw, where an accepted message was silently shown as text/plain. That behaviour is our observation, not something Microsoft documents. Nor does it mean that Outlook always fails on bare LFs, or that every blank Outlook email has this cause.

What Python is doing

Two pieces of behaviour combine to produce the problem.

The first is the default policy. EmailMessage defaults to the default policy, an instance of EmailPolicy, which uses Python's standard \n line endings rather than the \r\n required for SMTP. email.policy.SMTP is the same apart from its line separator, which is \r\n. So msg.as_bytes() with no arguments gives you LF line endings.

The second is smtplib. For sendmail(), the documentation says that if the message is a string, lone \r and \n characters are converted to \r\n, but if it's a byte string it is sent unmodified.

Our code didn't use sendmail(). It used data(), which is one of smtplib's low-level methods. The documentation doesn't cover those in detail: it says they don't normally need to be called directly and refers you to the module source. So we can't point to the docs for data()'s behaviour. The evidence for what it did in our case is the captured byte stream.

We weren't the first to run into this. A user of the python-emails library reported that after it switched from as_string() to as_bytes(), messages were sent with LF instead of CRLF, because as_bytes() defaults to \n and smtplib leaves byte strings alone.

The same application also had a "send test email" feature that used send_message(). That method serialises the message with BytesGenerator using \r\n, then passes it to sendmail(). So before the fix, the application had two ways of serialising a message, and they produced different bytes. We didn't send a send_message() message to Outlook to confirm the result there.

The fix

The problem:

raw = msg.as_bytes()
server.data(raw)

The fix:

from email import policy

raw = msg.as_bytes(policy=policy.SMTP)
server.data(raw)

or, to keep the message's existing policy and change only the line separator:

raw = msg.as_bytes(policy=msg.policy.clone(linesep="\r\n"))

If you don't need low-level control over the SMTP conversation, this is simpler:

server.send_message(msg)

After the change, Outlook showed the body and the PDF attachment, and SPF, DKIM and DMARC all passed.

Testing for it

We added a test that checks the bytes the test SMTP server actually receives in the DATA phase:

def assert_crlf_only(raw: bytes):
    rest = raw.replace(b"\r\n", b"")
    assert rest.count(b"\n") == 0, "bare LF"
    assert rest.count(b"\r") == 0, "bare CR"

Testing at that boundary matters. Inspecting the EmailMessage object tells you about the structure, and it will look fine. Checking the output of as_bytes() is better, but it still misses whatever happens between serialisation and the socket. What the server receives is where the rest of the delivery chain starts, and no later step can be relied on to fix it.

It's also worth checking every sending path in the application, not just the main one. Ours had six, but they serialised the message in only two places, and only one of those was wrong. After the fix we measured the SMTP DATA from all six: no bare LFs and no bare CRs in any of them.

Lessons

If a Python-generated email is blank in Outlook but fine elsewhere, this is the order we would check things in now:

  1. Look at the MIME structure as received.
  2. Compare copies of the same message at different recipients.
  3. Treat DKIM results as a clue, not as the cause.
  4. If the structure already looks right, don't spend long changing MIME headers.
  5. Capture the actual SMTP DATA bytes.
  6. Count bare LFs and bare CRs.
  7. Check every sending path in the application.
  8. Make sure serialisation uses CRLF.
  9. Add a regression test at the SMTP DATA boundary.

A message that displays correctly in one mailbox isn't proof that it's well formed. Different receiving systems can handle the same malformed message differently, and the difference only shows up when you look at the bytes.

References