Nuclear Weasels

Following my recent tweets regarding the number of servers were I work, I’ve had some feed back asking exactly how we have that many servers / services running here. I will now attempt to explain it as best I can.

The company I work for (like many other companies of its ilk) is a little strange, born out of academia it has always done things the SCI way whether that way conforms with industry best practice or not. Often in the past it has done things under the impression it would change best practice by doing the task in a new manner.

One such issue has been the companies IT systems.

The root causes of the issue are

1. We’re an educational establishment, we get cheap software.
2. We’re not for profit, so we tend to spend the excess on shiney new stuff.

What this lead to was a proliferation of servers over the past 8 years, every new project lead to us provisioning a new server.  At the time you never noticed the effect the growth was having on the IT department, for the few few years of their life servers generally don’t cause much fuss – its only when they get old they start missbehaving.

The cheap cost of software and its almost instantanious availability (Microsoft Licensing rocks!) meant that we have always been able to play with toys that would normally be outside our reach, for instance we have gone through 4 versions of exchange in the last 6 years who else would have the money to do this? I know several companies that are still standardised on Exchange 5.5 because it works and does the job well.

Along side core systems like Exchange we run several special services, such as Microsoft Content Management Server and SharePoint. Since I joined the company (3 years ago) we’ve gone through 3 versions of SharePont server alone.

Now, Evan asked how we can possibly have 40 servers. I will now try (as generically as possible) to explain it.

Internal Domain

Firstly we have 3 domain controllers supporting our internal domain, this follows a disasterous couple of weeks this year when we lost both of our previous domain controllers when their virtual hosts suffered issues. This prompted us to move the servers back to hardware, just to be sure that if hardware issues do strike again they don’t take down everything, that outage really highlighted what the loss of our internal Domain does to certain web services we run that authenticate against it.

We also run Exchange 2007, currently on a blade supported by an AX150i SAN. Along with another server which acts as a mail relay and anti spam Edge server. We are also running an Exchange 2003 internally for the next few months as we finalise our migration to 2007.

For our file storage needs we run two identical NAS devices, these are set to replicate data between the master and the slave to ensure if the main server goes down that we can switch to a current backup.

Count so far 8.

We also run an app server, this box runs software such as AutoCAD which need to be installed on a central server and deployed out to clients.

Our business system runs on another server, co-existing Navision with other software is a bad idea, we know this because we tried it.

Ansys Server we maintain one server dedicated to our engineering modling software, that box is bascially there to number crunch for us.

Although its virtualised, our print services count towards the totals.

Count so far 12.

After our foray in to Virtualisation we have 4 servers dedicated to this task, two run Virtuozzo Server, one runs Micrsoft Virtual Server 2005 and the latest run is running Hyper-V.

Count So far 16.

Perimeter Domain

Moving on to our perimeter domain we start again with domain controllers, this time two. One hardware and one virtualised box.

DNS server. We host a fair number of DNS records, after several years of issues running DNS with another service we decided to split it off to its own virtual instance.

Exchange 2003. We maintain a second Exchange server on the perimeter domain, this is for use by our clients and the mailing systems attached to our websites. We have tried other mail servers for this role but only Exchange Server has given us proven reliability.

Database Servers. We are in the process of migrating from SQL Server 2000 to SQL Server 2005 at our DBA’s request we (willingly!) avoided an in place upgrade of our server and provisioned a second box to do the job. Until the migration is complete we’re running two SQL servers.

Count so far 21.

IIS. We run a server dedicated to IIS which at the present time hosts around 40 sites of various ages.

Microsoft Content Management Server. A few years ago we invested time in MCMS we now maintain a server running 5 MCMS sites which we will be looking to maintain for the next 3 to 5 years.

SharePoint 2003. We have clients running extranets on this software, its another box.

SharePoint 2007. Installed last year we are now using SharePoint 2007 for our new sites and extranets.

Issue Tracker. Due to security issues with our issue tracking software it exists in its own virtual instance.

Count so far 26.

Bookshop. We have a very old server running our bookshop, installed in the year 2000 and never touched again.

Ultraseek. Our enterprise search service supports three of our frontline servers.

Other Servers

Firewall. Kind of important, we have a box dedicated to routing / VPN / Firewall duties.

Development Servers. We are running 5 development servers of various ages to support the web developers.

Another App Server. We are running a second app server under virtualisation because it won’t run on our desktops. Another to add to the mix.

The CMP server. My baby, recently retasked from another less glamerous life it now runs the management software I have been clamouring for over the last 3 years. Micrsoft Systems Center Operations Manager 2007, Configuration Manager 2007 R2 and 3Com Network Director.

Final Count 36.

Wait. Our VOIP system that got forgotten.

Final Count 37.

Okay, so 37 is a little lower than I originally stated but that is only because I managed to remove a few virtual servers this week by being ruthless.

Is 37 too many for one person to manage? Probably, as you’ll see above I forgot one again. Time to print this entry and put it on my wall just incase I forget again.

~Dan