Recently I faced an email issue with Exchange 2013 server in the production environment. After a failed update of Exchange 2013 to CU5, the mail flow stopped working and sending emails stuck in the drafts folder. Even though I managed to recover Exchange application and user mailboxes, but mail flow of internal and external did not work. Tried to send emails from smartphones that also went to draft folder in the mailbox server where I can see them via Outlook web access. After two days of continues research, trial and error I managed to make it work without a new installation of Exchange server.
Though this actual scenario happened with Echange 2013, it is applicable for Exchange 2016 and 2019 version if you faced a similar issue at a similar place. If any Cumulative Updates (CU) go bad or fail during the installation, you may face these kinds of issues.
Domain controller and Exchange servers are running on Windows 2012 R2 as virtual machines of Hyper-V. The installed version was the first release of Exchange 2013 and mail flow was working fine till the update crash. The SP1 (CU4) or CU5 updates are recommended for customers who are using the first release which came 2 years ago. Also, there were some issues on authentication and outlook anywhere which looked like a bug on the older version of Exchange 2013 and recommended fix was to update the latest CU5 release.
Backup is always important. Even though we have enterprise backup solution, somehow it’s not working with Windows 2012 R2 virtual machines and Exchange 2013 application. So, I have used Windows 2012 R2 Windows server backup to take a full backup of Exchange 2013 server (including system state and Information store) and Domain controller. Also, I took the system state back of these two servers separately.
What Went Wrong?
The Exchange server had enough space inside in C drive before starting the upgrade process. But I missed checking the space on storage LUN where the virtual machine VHD file located.
The upgrade process started, prepared organization, removed existing Exchange 2013 installation and started copying new files. BOOM, the virtual machine went to pause mode with ugly Windows blue screen error (now it is changed in Windows 2012 and Windows 8).
I made some space on the LUN and resumed the server. Unfortunately, it got restarted with a blank Window without new or old Exchange installation, because the old installation was removed during the upgrade process.
How I Fixed Exchange 2013 Emails Stuck in Drafts After Failed CU or SP Update
I’m not going to share all steps and attempts I tried to fix the issues, but here are the important notes.
1) Windows server backup/restore for Exchange 2013 works fine. If you have taken full Exchange 2013 backup as shown here in Microsoft site, the restore will work fine.
2) Always keep the full and latest backup of Domain controller (Active Directory) and Exchange server before the update.
3) There are few DNS settings need to be done to send and receive emails internally and externally for the first time when Exchange 2013 server configured. I’m sure you would have done them initially that’s why the mail flow worked fine till the update issue.
So, if you do an Exchange application restore after any issues created by failed CU or SP update, you do not need to change anything in Exchange server settings via ECP control panel. Because the application restore would restore exact settings again.
You can refer our earlier guide to make sure these DNS and hosts files settings are there.
4) There is nothing you need to do in server firewall or main hardware firewall settings (to allow or deny IPs and port forwards) because we have not touched them here.
5) Mostly you do not need to change any settings in Exchange server if your backup and restore are perfect.
The Real Culprit
6) So, the culprit is domain controller Active Directory data, yes, that’s right.
During the Exchange 2013 CU5 update, it modified the AD schema on the domain controller to match with the version it’s going to install. In my case, it already changed to the version of CU 5.
When I restored the Exchange server from an earlier backup, it was for the older schema version. Therefore some kind of authentication or compatibility issue was there with the new Exchange schema in AD and old Exchange server version.
Here is the way to find out the Exchange server schema version in domain controller.
Here are server version and schema numbers. You can visit the original link for more updates. You should not change these numbers in your active directory by ADSIEdit tool. It’s highly recommended to NOT to touch and modify AD data manually.
The solution is either restore your AD data from backup (you have to think about how it will affect your other domain controllers and clients) or update the Exchange 2013 server again to match the schema number.
In my case, I made enough space in the virtual machine’s LUN and ran the CU5 update again. Without touching any settings in Exchange server or ECP, mail flow started working, active sync and Outlook anywhere all came back online.
You can verify the server health and status after the upgrade by reading this great guide about how to update Exchange 2013 to SP1.
I’m sure this post is helpful to someone who struggles with Exchange 2013/2016/2019 problem in sending emails after a failed service pack or CU (Cumulative Updates).