Migration Event Review – May 2017

Prepared by Marden Paul
17 May 2017 Revised 26 May

Executive Summary

Primary Objective

Migration of student and alumni accounts in Office 365 from US-based facilities to Canadian data centres with the establishment of a UofT tenant in Canada.

Expected Duration of the Outage

The migration commenced on Friday, 28 April at 6:00PM and was expected to conclude with services restored in the newly established Canadian tenant at 6:00PM Monday, 1 May.

Actual Duration of the Outage

UTmail+ was available again, less the backlog of email cached during the outage, by about 4:00AM, Thursday, 4 May.

Primary Cause

The number of accounts being migrated significantly affected the duration of the outage. Tenant-to-Tenant migrations between different data centre regions is a relatively new innovation from Microsoft and was only available to Canadian clients as of May 2016. Office 365 migrations are complex and require the synchronization of data across multiple interconnected services within Office 365. Sequence and timing are critical with the multiple interdependencies of the service components.

In the production environment, migration processes are subject to the competing resources within the tenant, the capacity forest, the layers of Office 365 and Azure Directory Services. Synchronization issues within Office 365 accounted for significant drag on the time to complete. The synchronization delay resulted in the inability to clear all remnants of the @mail and @alum references from the US tenant during the process – a necessity for breaking the connection with the US tenant prior to establishing accounts on the Canadian tenant.

Tenant size, movement between the US and Canadian-based data centres, and propagation and synchronization of data across multiple Office 365 components ultimately accounted for the extended time required to perform the migration.

Secondary Effects

Actions taken to speed the migration process included re-provisioning of licenses to some accounts that were apparently lacking following the return to operation. The licenses were eventually provisioned but were held up by the delays in data transfer and synchronisation. Remediation prior to their provisioning led to secondary complications post-restoration. These complications these were all rectified in the following days.

Current State

Office 365 in the Canadian tenant is fully operational. Planning for the Faculty and Staff migration are underway. The few remaining migration clean-up issues are defined on the Known Issues page at https://email.utoronto.ca.

Lessons Learned

  • Plan for Greater Outage Duration — For a unique event such as this tenant-to-tenant migration, additional outage time should have been secured and promoted. Had the migration completed sooner there would have been a realized benefit, but additional mitigations might have been taken by individuals to work around the outage had they expected additional time without email.
  • Expand Communications and Communicator Resources During Event – Because the event extended beyond the expected duration, a stated update schedule, e.g., “Updates will occur every 2 hours until event conclusion” would have provided additional assurance to students and some level of certainty as to when the next update would occur.
  • Timing and Alternatives – There was speculation/conversation about the timing of the outage. Internally, the project team and leadership recognised that this was the optimal window to perform the migration, but this was not fully shared with the affected communities. More communication about the rationale behind the migration and its timing would have alleviated some concerns.

O365 Tenant Migration to Canada Issues Status

Issue Description Status and Action
Missing Folders in Office 365 Reports that mail copied to some folders in inboxes is incomplete. Resolved.
Inbound Messages Rejected Some messages are looping in the delivery cycle. This is related to the licensing problem reported. Resolved.
SPAM Mail Some user are reporting issues with spam mail. Resolved.
Office ProPlus desktop apps cannot be downloaded Link is not available to eligible UTmail+ clients Resolved.
Error Message when logging in A number of users were reporting a Microsoft license error message:”X-OWA-Error: Microsoft.Exchange.Clients.Owa2.Server.Core.
Forwarding Messages from Office 365 Account Users forwarding their Office 365 accounts were not seeing messages. Resolved. Some forwarding settings were not populated properly. Log into your Office 365 account at mail.utoronto.ca to re-enter the forwarding options. See the instructions on setting up forwarding.
No Access to UTmail+ Archive Users would like to know when they can access their old UTmail+ account contents. Resolved: This feature has been enabled and is available to all users. See the instructions on how to access the UTmail+ Archive.
Inbox Rules Not Transferred User settings for inbox rules were not carried over. Resolved: Users need to recreate their inbox rules. Instructions on creating rules are on the Help Desk Knowledge Base.
External Mail Delivery to Alumni Domain Reported issues with @alum.utoronto.ca accounts not receiving external email. Resolved.
Rich Client Access to UTmail+ Rich Clients (Outlook,iPhone, MacMail etc.) were unable to connect to mailboxes in Office 365. Resolved: Timing issue for DNS changes to propagate to services through the web.

If you continue to have a problem with your account, please contact the Help Desk: 

Information Commons Help Desk
Phone: (416) 978-HELP (4357)
Email: help.desk[at]utoronto[dot]ca