I experienced a strange Exchange 2013 email server issue in the production environment where I work. After a failed update of Exchange 2013 to CU5, the mail flow stopped working, and sending emails was stuck in the drafts folder.
Note: Though the guide is about Exchange 2013 server, the same solution is applicable for Exchange 2016, 2019, and any CU versions. Fundamentally things remain the same, and you can apply it in your environment if you encounter a similar issue in upgrading Exchange servers.
I recovered the Exchange application and user mailboxes, but the internal and external mail flow did not work after the failed CU upgrade. After two days of continuous research, trial and error, I made it work without setting up a new Exchange server.
The AD and Exchange Setup:
Here is the setup of the environment I faced the issue.
Domain controller and Exchange servers run on Windows 2012 R2 as Hyper-V virtual machines. The installed version was the first release of Exchange 2013, and mail flow was working fine till the update crashed.
The SP1 (CU4) or CU5 updates are recommended for customers using the first release two years ago. Also, there were some issues in authentication and outlook anywhere, which looked like a bug on the older version of Exchange 2013. The recommended fix was to update the latest CU5 release.
The Exchange server 2013 has a CU23 update as on updating this guide. It is highly recommended to patch your Exchange servers to the latest update.
How I Prepared Exchange CU Update?
Backup is always important. Better to keep several backup copies taken in different ways.
I took the backup of the domain controller and Exchange server by an Enterprise backup solution for virtual machines and Windows built-in backup (including system state and Information store). Also, I took the system state back of these two servers separately.
What Went Wrong in Exchange CU Upgrade?
The Exchange server had enough space in the C drive before starting the upgrade process. But I missed checking the space on storage LUN where the virtual machine VHD file is located.
The upgrade process started, prepared organization, removed existing Exchange 2013 installation, and copied new files. BOOM, the virtual machine went to pause mode with an ugly Windows blue screen error.
I made some space on the LUN and resumed the server. Unfortunately, it got restarted with a blank Window without new or old Exchange installation because the old setup was removed during the upgrade process.
How I Fixed Exchange 2013 Emails Stuck in Drafts After Failed CU/SP Upgrade
I’m not going to share all steps and attempts I tried to fix the issues, but here are the essential notes.
1) Windows server backup/restore for Exchange 2013 works fine. If you have taken the full Exchange 2013 backup as shown here on the Microsoft site, the restore will work fine.
2) Always keep the full and latest backup of the Domain controller (Active Directory) and Exchange server before the update.
3) A few DNS settings must be done to send and receive emails internally and externally for the first time when Exchange 2013 server is configured. I assume these changes are done initially. That’s why the mail flow worked fine till the update issue.
So, do an Exchange application restore after any issues created by failed CU or SP update. You do not need to change anything in Exchange server settings via the ECP control panel because the application restore would restore exact settings.
You can refer to our earlier guide to ensure these DNS and host file settings are there.
4) There is nothing you need to do in the server firewall or main hardware firewall settings (to allow or deny IPs and port forwarding) because you did not touch them here.
5) You do not need to change any settings in the Exchange server if your backup and restore are perfect.
The Real Culprit:
6) The culprit is domain controller Active Directory data, yes, that’s right.
During the Exchange 2013 CU5 update, the setup modified the AD schema on the domain controller to match the version it will install. In my case, it has already changed to the version of CU 5.
When I restored the Exchange server from an earlier backup, it was for the older schema version. Therefore some kind of authentication or compatibility issue existed with the new Exchange schema in AD and the old Exchange server version.
Here is the way to discover the Exchange server schema version in the domain controller.
You can visit the original link for more details. You should not change these numbers in your Active Directory by the ADSIEdit tool. It’s highly recommended NOT to touch and modify AD data manually.
The Solution Worked for Me
The solution is to either restore your AD data from backup (you have to think about how it will affect your other domain controllers, AD depended-applications, and clients) or upgrade the Exchange 2013 server to match the schema number.
In my case, I made enough space in the virtual machine’s LUN and reran the CU5 update. Without touching any settings in the Exchange server or ECP, mail flow started working, active sync, and Outlook came back online.
Here is the Recap of the Solution:
If the Exchange CU upgrade failed for some reason and you lost the Exchange setup on the server, follow the steps. I do not recommend changing anything in the domain controller or restoring the active directory.
- Restore the Exchange server from the latest backup. You will get the Exchange server now, but it may not work correctly in your environment because the AD schema has already been upgraded.
- Find out the reason why the earlier CU upgrade failed. In my case, it was a space issue for the Virtual Machine.
- Fix the issue that caused the CU upgrade to fail.
- Rerun the CU upgrade on the Exchange server.
After the upgrade, you can verify the server’s health and status by reading this great guide about updating Exchange 2013 to SP1.
I’m sure this post is helpful to someone who struggles with Exchange 2013/2016/2019 sending email issues after a failed Service Pack or CU (Cumulative Updates).