E-Mail Server issues:
What is happening with our email server?
The non-technical answer is simply that our e-mail server is overwhelmed right now and because of this, it is shutting itself down. We are restarting it each time this happens, but it is occurring as frequently as every hour. When these stops occur, you lose the connection to the email server and Outlook stops working.
How do I get access to email again?
To reconnect, Outlook users on campus must close and reopen Outlook. If you receive a login prompt please replace the user name rcc\username with your e-mail address and click the save credentials check-box.
Outlook Web Access users will log in to https://owa.roguecc.edu/ if their account has not been migrated. If your account has been migrated you will receive an error stating that your rcc\username account was not found. If you receive this error please log on to https://outlook.office365.com/. You will need to use your e-mail address as your user name for your login.
What is being done about this?
To help reduce the stress on the server and lengthen the time period between shut downs, we have turned off the SPAM (junk email) filter. You may have noticed an increase in unwanted email.
When will this be fixed?
At this point over 95% of full-time faculty/staff/exempt employees have been migrated to our new email service that is being hosted online (otherwise known as "the cloud"). Once all full-time employees have been moved, we will be proceeding with part-time employees. The goal is to have all users in the cloud by Friday, March 14th (at the latest). This migration will resolve our current email issues.
What if I want to know more?
For those with specific questions regarding our response and procedures we have provided the following timeline and FAQ:
- Friday, February 28th - Late Friday night the database suffered an out-of-memory error
- Saturday, March 1st - The problem was discovered and troubleshooting began between RCC's IT Network Services, our Microsoft Cloud Migration Partner and Microsoft Support
Sunday, March 2nd - Troubleshooting continued with tenuous database
connection regained by Sunday evening.
- "Crash dump data" was compiled and sent to Microsoft with a standard 24-48 hour turn-around on response.
- Initial determination was that several issues were contributing to memory allocation issues that over time would overwhelm the database and server.
- The only way to fix the issue in the short term was to actively monitor server logs, "dismount" (turn off) the database, and then re-mount it when the errors begin again.
- This scenario repeated roughly every 15 minutes so a schedule of 24-hour coverage was arranged to manage it.
- Monday, March 3rd - Troubleshooting continued between IT and the migration partner on the database while actively monitoring the logs and initiating database dismounts/mounts at 15 minute intervals.
- Tuesday, March 4th - Troubleshooting continued with some issues fixed with timeframe between dismount/mounts increasing to an average of 45 minutes. Unfortunately, one of these fixes was to disable on-site anti-SPAM filtering. Unavoidably SPAM is now coming into all on-site mailboxes.
- Wednesday, March 5th - We received response from Microsoft and further fixes were implemented or scheduled for after-hours. The timeframe between dismount/mounts increased to 1 -1.5 hrs.
- Thursday, March 6th - At 7:45am there was a dismount issue that led to no e-mail connectivity for the first hour of the work-day. That issue was corrected and the timeframe between dismount/mounts increased to 1.5 - 3hrs.
- Friday, March 7th - Smaller corrections processed that allowed for greater uptime of the exchange database.
- Weekend of March 8th/9th - Continued corrections while migrations continued. Part-time faculty migration initiated.
- Monday, March 10th - Continued migration of part-time faculty.
- Tuesday, March 11th - over 90% of staff migrated, working on lists for student worker accounts.
- Wednesday, March 12th - Staff migrated, continuing work on lists for student worker accounts.
- Thursday, March 13th - Student worker accounts migrated, working on temp, generic, and collegenow accounts.
- Friday, March 14th - Collegenow accounts migrated, completing temp and generic accounts.
Q - Who is affected by this problem?
A - All users with e-mail accounts still hosted on our local e-mail
server, Mercury1 are affected.
Q - What symptoms will I see?
A - Mailflow within your Inbox will be slow throughout the day and you will experience a complete lack of connectivity every 1-3hrs that will last up to 10 minutes. You will also see SPAM in your inbox.
Q - When will I be moved to the Cloud?
A - At this point our full-time employees, part-time employees, and student workers have been migrated. We are currenty proceeding with temp, collegenow, and generic accounts. The goal is to get all users in the Cloud by Friday, March 14th.
Q - What caused this to happen?
A - There were several factors that contributed to this issue:
The database and the server itself is old (created in 2008) and was due to
be retired over a year ago. The Cloud selection and migration process took
much longer than anticipated
The database is over 1.2 Terabytes (over 1,200 Gigabytes) in size.
extremely large for an e-mail database
Users are utilizing the e-mail server for file storage by keeping large
numbers of attachments saved in folders instead of saving the attachment
locally and deleting the message
Users are not clearing out their folders, specifically their Inbox, Sent
Items, and Deleted Items folders
The College receives a high volume of SPAM, which has contributed to overtaxing the
The database was holding onto old mailbox search queries in hidden folders
which took up database space and resources
Q - How can we keep this from happening again?
A - Our migration to the cloud will build in automated database clean-up tasks, automated anti-SPAM, automated backups, and automated database divisions. As the database grows it will split out into multiple databases for easier management and maintenance.
What if I still have questions?
If we haven't answered your questions here, please review the information on our Office 365 Information page:
You can also contact the RWC Help Desk at 541-956-7042 or the RVC Help Desk at 541-245-7535 or via e-mail at firstname.lastname@example.org. A representative will get back to you as soon as they can.