Post SP2-Hotfixes Fails To Install Database Services On Clustered Servers With System Dbs On Non-default Location

Hi,

We've had this issue for awhile, but I have not been able to find a resolution for this. We have 2 identical servers (hardware & software) set-up: IBM x3850-(88633RU) running Windows Server 2003 Enterprise x64 (5.2.3790 SP1 Build 3790 R2). Server A was running SP1 (Build 9.0.2047) on a 3-node active-passive-passive failover cluster. Server B was running SP2 + Hotfix KB933097 (Build 9.0.3152) on a 2-node active-passive failover cluster. Both these servers are heavily utilized, and when originally installed with these versions of SQL Server Enterprise 2005 experienced no issues during or post-install. The original install of SQL 2005 Enterprise on both these servers had the Cluster Group Selection "Data files" setting changed to "M:MICROSOFT SQL SERVER" so that all system dbs and binaries would be installed to that location. On Server B where SP2 had already been applied, the update was downloaded on May 1, 2007, well after the GDR issues which had been plaguing SP2 releases was supposed to have been resolved.

Our problems began the night when we tried to apply hotfix KB933097 on Server A, and cumulative update packages 1,2,3 on Servers A & B, as well as catch-up the same SP2 and hotfix packages on other non-clustered servers where default system db locations remained the same. On all servers other than Servers A and B, the installs proceeded smoothly and without issue.

On Server B, which was already running SP2 + Hotfix KB933097, we tried installing Cumulative Update Package 1 (Build 9.0.3161) on the primary node. The install proceeded until the following errors were encountered:

"[Microsoft][SQL Native Client][SQL Server]Cannot find the object 'dm_exec_query_resource_semaphores', because it does not exist or you do not have permission."

"[Microsoft][SQL Native Client][SQL Server]Cannot find the object 'dm_exec_query_memory_grants', because it does not exist or you do not have permission."

After this, the setup finished with the message "Product: Microsoft SQL Server 2005 (64-bit) - Update 'Hotfix 3161 for SQL Server Database Services 2005 (64-bit) ENU (KB935356)' could not be installed. Error code 1603. Additional information is available in the log file C:Program FilesMicrosoft SQL Server90Setup BootstrapLOGHotfixSQL9_Hotfix_KB935356_sqlrun_sql.msp.log."

We rebooted the server and tried several times to reinstall the hotfix as sa and as the database service account with full windows and sa credentials, each time with no success, and rebooting between retries. All other components were installed successfully. We were hesitant to try uninstalling the hotfix, since it was unable to be removed successfully in previous attempts. However, when I connected to Database Engine using Management Studio, I found that the version of SQL being reported was in fact Build 9.0.3161! Everything also appeared to be running properly.

I tried to to Google the answer, but could only find information as applied to SP1 when system dbs master and mssqlsystemresource were located on different locations (http://support.microsoft.com/default.aspx?scid=kb;EN-US;918695 ). I had already verified that all system dbs, including the ones mentioned, were originally and currently located together on M:Microsoft SQL ServerMSSQL.1MSSQLData.


Finally I contacted a fellow DBA who advised me to continue installing all the other hotfixes, and then compare all SQL files against other servers where the install succeeded to ensure all system files were updated correctly (matching file sizes and datetime stamps). Since we were depending on a fix in Cumulative Update Package 3 to fix a critical log-shipping security issue, I went ahead with the rest of the installs, getting the same error with each package, rebooting between each install.

We found after the final install that the mssqlsystemresource files in the M:Microsoft SQL ServerMSSQL.1MSSQLData

directory were in fact updated after each install with a new timestamp, and that all files in C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLBinn matched on both file size and timestamp. The log-shipping security issue we were looking to fix also appeared to be resolved by the last update.

On Server A, which was running SP1 (Build 9.0.2047), we installed the same SP2 package as we had on Server B with no issue. We rebooted and then tried to apply the same Hotfix KB933097 (Build 9.0.3152) that had been applied successfully on Server B and found to our surprise that the same errors were being generated here as well. We continued to apply the rest of the hotfixes so that both servers matched software updates. Comparing the above directories showed that files seemed to have been updated, even though database services appeared to be not installed.

It looks like we now need Cumulative Update 5 hotfix package in order to deal with WMI alerting issues, but I'm afraid to apply it until we have a known resolution for the past errors.


Since this issue was supposed to be resolved by SP2, I'm not sure why these post-SP2 hotfixes should fail. It seems to me to be an issue of the directory location of the system files which is in question.

I can post the hotfix log files as well, if anyone is interested in seeing them.

Any help would be greatly appreciated. As well, does anyone have any idea when SP3 is scheduled to come out? I really don't like applying hotfixes which are usually untested for general purposes, and it seems Microsoft is delaying this in favour of releasing SQL 2008.