DoIT will conduct three forum sessions to address the UNCP email system at Moore Hall Auditorium. All faculty and staff are invited to attend to hear updates on how the email issue occurred, the recovery efforts involved, and future steps that will be taken to prevent a similar occurrence.
Monday, October 27: 9 a.m. and 4 p.m.
Tuesday, October 28: 3 p.m.
Faculty and staff may get a message indicating the delays in email delivery, but all NEW messages are delivered in a timely manner. The backlogged messages are being delivered.
Faculty and staff can now access their old email account via Outlook, and campus users should review their email to verify that email folders and messages are intact for anything October 13 and prior. Please contact the DoIT Help Desk at 521.6260 or email@example.com if you find that data is missing for this date range.
Please check your student Webmail account at webmail.uncp.edu. Currently, email messages are being delivered more quickly to Webmail than BraveMail. Although email messages are being delivered to both Webmail and BraveMail when addressed to firstname.lastname@example.org, there is a time delay with the Microsoft server hosting BraveMail in delivery of messages. All messages will be delivered, but please check your Webmail account to ensure that you are receiving all your messages.
When you send out a mass email to students, such as through a distribution list, you may receive an error message "Delivery has failed to these recipients or distribution lists:" which lists all the students' email addresses. This does NOT mean that your email did not get delivered. It is successfully reaching all recipients EXCEPT for any student whose email has a problem (such as Inbox is full - aka "exceeds quota"). The list at the very end of the message indicates which student(s) did not receive your message, as well as the reason why the message did not get delivered.
If you have not activated your new email account, please follow the links below:
Students: Please activate a new student email, BraveMail.
Faculty and Staff (non-Exchange users): Please activate an Outlook Exchange email account now.
» Monday, October 27, 2008
8:00 a.m.: The backlogged messages continue being delivered, and 66,000 messages have been delivered to BraveMail.
» Sunday, October 26, 2008
8:00 p.m.: The Microsoft BraveMail delivery issues were resolved.
» Thursday, October 23, 2008
5:00 p.m.: Webmail is fully functioning. Email messages continue been delivered.
11:20 a.m.: The backlogged messages continue to be delivered.
8:00 a.m.: Webmail access was turned on.
» Wednesday, October 22, 2008
10:00 p.m.: Webmail access was turned off to devote all the server resources to email delivery.
5:00 p.m.: The listserv is back online. Backlogged listserv messages have all been delivered.
11:00 a.m.: The old Webmail server was turned on so that those who had not converted to Exchange prior to October 14 are now able to access the stored messages.
10:30 a.m.: About 25 percent of the backlogged messages were delivered.
8:00 a.m.: The backlogged messages continue to be delivered.
» Tuesday, October 21, 2008
4:30 p.m.: The backlogged messages started to be delivered to users' mailboxes.
3:30 p.m.: Client Services staff reported that a small group test for email functionlities was successful.
11:15 a.m.: The backlogged messages continue to be sent to Exchange and BraveMail accounts. Data transfer to Exchange has gone smoothly, but a large number of data to BraveMail has been rejected. It indicates that Microsoft may have a restriction on the amount of data transfer in a given time period from UNCP, which is a common practice against spam. Microsoft is assisting us with this issue.
7:33 a.m.: Tom Jackson continues his all-night effort to return to normal e-mail services. At this point, 40,000 of the potential 500,000 backlog messages have been sent out to be processed.
» Monday, October 20, 2008
5:00 p.m.: The messages continue to be delivered to users' mailboxes. Because the size of the backlog is huge, it will take a quite some time to finish this process.
4:30 p.m.: The backlogged email data is now started to be delivered to new email accounts.
4:20 p.m.: The index reconciliation process is complete.
3:45 p.m.: The email data since last Tuesday in the queue is ready to be delivered. The index reconciliation process continues.
2:50 p.m.: Inconsistent mailbox indices were found on the servers. DoIT is reconciling the indices.
2:40 p.m.: A small group of staff is testing email functionalities before delivering email data to all email users on campus.
12:00 p.m.: Technicians are checking whether email addresses are spelled correctly to avoid further problems. Because the old Webmail server was down, DoIT staff had to type in approximately 400 email addresses.
8:00 a.m.: The older Webmail server has been restored over the weekend. DoIT is now working to transfer the archived email data before last Tuesday to the new Exchange server. Email before the Tuesday should be delivered to your new mail boxes later today.
» Sunday, October 19, 2008
9:00 p.m.: Technicians worked throughout the weekend and restored the old server. Email should be flowing both from inside and outside of the UNCP system.
» Friday, October 17, 2008
5:00 p.m.: DoIT operations staff will continue to work over the weekend to fully restore webmail services.
11:30 a.m.: Students continue setting up BraveMail, and faculty and staff started setting up Outlook Exchange. For faculty and staff email data in the queue starts to be delivered.
7:00 a.m.: The old email server finished indexing the mailboxes. The users' verification process starts. New Exchange accounts have been created for all faculty and staff.
» Thursday, October 16, 2008
4:30 p.m.: More and more students are setting up BraveMail accounts.
1:30 p.m.: The initial verification process has been finished. DoIT discovers that all sub-folders also need to be rebuilt separately.
11:50 a.m.: 11,000 email accounts were verified. The verification process is in progress without any errors. This morning, 700 students set up BraveMail.
8:30 a.m.: DoIT starts setting up a new student email system, BraveMail.
7:50 a.m.: The email data verification process continues. Over 40 percent of approximately 15,000 email accounts has been verified.
» Wednesday, October 15, 2008
10:30 p.m.: The restoration process has been finished. The email data verification process is started.
9:20 p.m.: Email data has been restored from tape backup and the restoration process is in the final stages. Once complete, DoIT will begin verification of email integrity.
7:02 p.m.: Eighty-eight percent of email from the tape backup has been restored. DoIT will begin verifying the integrity of the email and continues to estimate that email will be available by 8 a.m. tomorrow.
4:45 p.m.: Eighty percent of the data was restored. DoIT estimates that the restoring process will be complete by 8:30 p.m., and email will be available by 8 a.m. tomorrow.
2:20 p.m.: Seventy-five percent of the data was restored.
12:45 p.m.: Sixty-one percent of the data was restored.
11:00 a.m.: Thirty-five percent of the data was restored.
8:00 a.m.: Twenty percent of the data was restored.
5:52 a.m.: Email is being restored. DoIT re-evaluates that email will be available to all users later today.
3:30 a.m.: The email restoring process has started.
» Tuesday, October 14, 2008
11:15 p.m.: The rebuilding drive process continues.
9:15 p.m.: Outlook Exchange email system has been restored. Exchange users can now send and receive email.
8:50 p.m.: The drive was rebuilt 20 percent so far. The process will be finished around 1 a.m. We will begin to restore the data.
7:40 p.m.: We replaced the defected drive with one spare drive. We are also trying to rebuild another drive.
6:20 p.m.: We determined that the source of the problem was hardware related. A new hardware was ordered and being shipped. In the meantime, we are looking at alternatives for incoming email data storage to avoid data loss.
5:00 p.m.: The server room staff reported that a serious hardware failure on our older campus email server. Email is not available for those who have not been moved to Outlook Exchange. Exchange email users are able to send out email. However, even Exchange users are not able to receive email because all the incoming email from outside is currently being routed through the old email server.
A portion of incoming messages are likely to be queued and delivered at a later time.
Updated: Monday, October 27, 2008
© The University of North Carolina at Pembroke
PO Box 1510 Pembroke, NC 28372-1510 • 910.521.6000