The weekend of September 29/30 we moved our Data Center from one suburb of Cleveland to another. As you install new servers or replace others it's normal to make certain everything is right before you bring them on line. A data center move is another animal altogether.
We have three High Availability clusters in our shop, and they've been working just great (or so it seemed - it's not common to practice failovers on production systems, is it?) When they all had to be shut down, packed up and installed in a new environment we held our breath.
Things went well, for the most part, but two of the clusters showed some strange behavior. On one, the "cluster" servers all came up on one physical server, and the applications had some connectivity problems. While troubleshooting I saw this and moved the cluster servers to their "normal" physical server, and everything then worked as it should.
Another physical server had what we think was a controller failure, as we lost both drives in a mirrored set (the C: drive, of course) and that server had to be rebuilt. When I moved the cluster server to another physical server, the applications using Windows Authentication worked fine, but those using SQL logins failed the authentication. It turned out that on the new physical server, SQL Server was now set to Windows Authentication only, where originally it was set to Mixed Mode.
I found that the mode setting is in the registry, and since I'd only run SQL 2000 on that physical server, the setting had never been touched. These clusters had originally been set up under SQL Server 2000, and I upgraded the servers that could be upgraded to SQL 2005 in April-May, 2006. When I first set them up I did extensive testing to ensure proper failover occured, but after the upgrade I assumed the failover would be successful. Note to self: Never Assume.
Apparently setup has changed for SQL Server 2005, and cluster setup doesn't do everything it did in SQL 2000. Not that this is a bad thing, but you have to be aware of that when doing your cluster planning. Geoff Hiten addressed this in his blog from last week: To Cluster or Not to Cluster, That SSIS the Question. SQL Server, Analysis Services and Reporting Services are all cluster-aware, but Integration Services, Service Broker, the SQL Server Client tools, the SQL Browser, etc. are NOT cluster aware, and they have to be installed and updated on each physical node. It's important for failover consistency to ensure that all the servers are on the same release level. It's also important to test the failover when changes occur to make sure you're not caught short when hardware failures occur.
Thanks, Geoff, for explaining some of the issues when I was struggling with them.