|You are here: HOME > NEWS|
Crash! Your e-mail was lost for 15 days
By Reilley Thayer, Staff Writer, and Tiffany Schmidt, Layout Editor
After 15 days and nights of intense repair and reconstruction work, the Division of Information Technology (DoIT) had the ocean of vital messages flowing again at 9 a.m. on Oct. 28.
Altogether, some 766,000 messages were captured and held until repairs were complete.
When the flow resumed, messages hit mailboxes like "bam, bam, bam," according to Chief Information Officer (CIO) Dr. Robert Orr.
UNCP had been using Webmail for eight years and this was the first time a full restore from backup had been necessary.
As a result of the outage, students now are using their new BraveMail system and faculty are hooked up to the Outlook Exchange server for the first time.
On Oct. 14, one of the hard drives in the UNCP hardware system failed and a replacement drive was delivered.
While installing the replacement drive, a second drive failed at 4:01 p.m.
The server that failed, "Brainy," was manufactured by Sun Microsystems and installed in 2000. The system has 16 hard drives that work simultaneously. At least 14 have to be working and online for the system to operate.
The rooms housing the servers are in the Oxendine Science Building and Lumbee Hall.
E-mails sent between Oct. 13 and Oct. 14 were lost. All incoming e-mails sent after Oct. 14 were stored and then redirected to the new e-mail programs, BraveMail and Outlook Exchange.
Dr. Orr, who is associate vice chancellor for information resources, already had plans underway to convert the student e-mail accounts to Microsoft Windows Live and the faculty and staff accounts to the new Microsoft Outlook Exchange because the equipment was very old.
"This positions us to move forward," Dr. Orr said, concerning the failure. "We are very close to where we need to be."
By 10:30 p.m. Oct. 15, the restoration process was finished and e-mail verification processes started.
While restoring the data from the tape backup, another hard drive out of the 16-drive array began reporting errors.
Rather than risk further problems, technicians plugged in a replacement drive from their pool of spares.
This did not stop the restore process, but slowed the operation until the new drive was initialized.
Messages that entered and left the university were operating on a normal delivery schedule by Oct. 21.
However, the flow of messages was slow because, on average, Webmail accounts handle 80,000-90,000 messages per day and they had about 766,000 messages in queue.
Webmail access was turned off at 10 p.m., Oct. 22, in order to devote all server resources to delivering messages.
DoIT announced at 8 p.m., Oct. 26, that the Microsoft BraveMail issues had been resolved.
On Oct. 27, 66,000 messages had been delivered as of 8 a.m. to BraveMail accounts.
To prevent another failure, the new infrastructure contains 50 to 60 hard drives per server rather than a mere 16.
Dr. Orr explained that this is not the first failure. One or another hard drive in the system usually fails once every three to four weeks, but the set of servers are built to carry on and compensate for the failed mechanism. The hard drives are totally enclosed mechanisms that wear out over time.
The university already had decided to outsource the campus's messaging systems and Microsoft provided the student accounts for free.
The UNC System CIO will visit UNCP in the near future to examine how DoIT managed the e-mail crisis and the new e-mail server systems, Dr. Orr reported.
"You need to walk away knowing that [DoIT] people know what they are doing and provide a reliable service," Dr. Orr said.
DoIT plans to keep Webmail running alongside BraveMail and Outlook Exchange into the spring semester. Keeping the Webmail accounts active will allow the accounts to run parallel for a better transition and permit users to visit their old accounts to retrieve messages there.
Eventually Webmail will be obsolete, so DoIT urges students to change their primary e-mail to username@BraveMail.uncp.edu.
During the problem period, the UNCP faculty sometimes received error messages when sending mass e-mails to students.
Now most all messages have been sent to students except for those that have full inboxes and other ordinary issues with their accounts.
Lecturer Soo Goh said he had difficulties communicating with his students because of the Webmail issues.
"Important information is being delayed. I rely heavily on e-mail for communication," Goh said.
"The e-mail system was really archaic and needed to be changed."
DoIT said that all messages will be delivered, although users should check their Webmail accounts for now to guarantee that they are received.
All new messages are being delivered, while some users are receiving messages informing them of delays on delivery for backlogged messages.
Faculty can contact the DoIT Helpdesk at 910-521-6260 for help with transferring any folders, contacts, etc. to their new Microsoft Office Outlook Web Access (OWA) accounts.
Freshman Morgan Beatty described complications she faced as a student when messages concerning classroom assignments could not be delivered.
"My teacher had sent out an e-mail to the entire class to say that she had changed the reading for the next class, but we didn't get it because the system was down," Beatty said. "It was confusing for a couple days."
Almost all of the university's 6,000 students have been signed up and are using their new BraveMail accounts, as well as their old Webmail accounts, according to Dr. Orr.
He said that prior to this outage DoIT had worked closely with the Student Government Association, forming focus groups to evaluate BraveMail and get student feedback.
Some of the new features available on BraveMail include shared messaging, photo applications, an event planner, message tracking, out of office assistant, mobile devices and a calendar.
Students will be able to keep their BraveMail accounts after graduation. After six months of inactivity, messages will be archived.
Accounts that have not been accessed within one year will be deactivated until the next time a user logs in. The account will be reactivated, but no old messages will have been saved.
Dr. Orr encourages students to contact the DoIT Helpdesk to let them know how they are doing with the new e-mail accounts. If necessary, DoIT will contact Microsoft for additional assistance.
The DoIT Helpdesk also asked users to review their accounts to see if there is an issue with e-mail messages not being intact from before Oct. 13.
For the time being, if sent to an firstname.lastname@example.org address (without BraveMail in the address), e-mail messages are being delivered to both Webmail and BraveMail.
Students can check their old Webmail account at webmail.uncp.edu.
More information is available through the DoIT Web page at http://www.uncp.edu/doit/.