No Witness, Primary Shutsdown, Bring Failover Online
May 6, 2006
First of all, Great webcast today. My question is, I have everything up and running and would like to know what to do when the machine my primary is on quits or has a some type of disaster. Do I need to manually run recovery on each db that was mirrored? I'm not currently running a witness.
Server : Windows server 2008 DB Server : SQL Server 2008 (SP1) Â Here are the series of events which happened.
1.) Event ID: 1135 Cluster node 'XYZ' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
2.) Event ID: 1049 Cluster IP address resource 'SQL IP Address 1 (XYZ)' cannot be brought online because a duplicate IP address '10.9.8.113' was detected on the network. Â Please ensure all IP addresses are unique.
3.) Event ID: 1069 Cluster resource 'SQL IP Address 1 (XYZ)' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
4.) Event ID: 1049 Cluster IP address resource 'Cluster IP Address' cannot be brought online because a duplicate IP address '10.9.8.112' was detected on the network. Â Please ensure all IP addresses are unique.
5.) Event ID: 1069 Cluster resource 'Cluster IP Address' in clustered service or application 'Cluster Group' failed.
6.) Event ID: 1066 Cluster disk resource 'Cluster Disk 25' indicates corruption for volume '?Volume{88552e6f-aea2-11df-9790-0026b92fffa7}'. Chkdsk is being run to repair problems. The disk will be unavailable until Chkdsk completes. Chkdsk output will be logged to file 'C:WindowsClusterReportsChkDsk_ResCluster Disk 25_Disk16Part1.log'. Chkdsk may also write information to the Application Event Log.
7.) Event ID : 1066 Cluster disk resource 'Cluster Disk 26' indicates corruption for volume '?Volume{88552e05-aea2-11df-9790-0026b92fffa7}'. Chkdsk is being run to repair problems. The disk will be unavailable until Chkdsk completes. Chkdsk output will be logged to file 'C:WindowsClusterReportsChkDsk_ResCluster Disk 26_Disk4Part1.log'. Chkdsk may also write information to the Application Event Log.
8.)Â Event ID: 1049 Â (Same message as point 2)
9.)Â Event ID: 1069 Â Â Â Â (Same message as point 3)
10.) Event ID : 1049 (same message as point 4)
11.) Event ID :1069Â (same message as point 5)
12.) Event ID :1205 The Cluster service failed to bring clustered service or application 'Cluster Group' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
13.) Event ID: 1069 Cluster resource 'Cluster Disk 17' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
14.) Event D : 1049 (same message as point 2)
15.) Event ID: 1069 Cluster resource 'SQL IP Address 1 (XYZ)' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
16.) Event ID : 1205 Â The Cluster service failed to bring clustered service or application 'SQL Server (MSSQLSERVER)' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application. Â first of all,I went through all the logs, and could not find the reason for fail-over initialization. There should be some thing logged why the failover happened? secondly after failover the service was not coming online due to duplicate IP address detection.
Later when we try  to manually bring the service online from cluster management it comes online successfully. I don't understand how would duplicate IP address get resolved when we start manually.
Lastly we see few errors related to physical disk resource between failover retries, is this could be the correlated to failover error ?
Hello there,I have a user who can bring offline a database but he cannot bring thedatabase back online.The error which pops up is:Error5011: User does not have permission to alter database dbnameALTER DATABASE statement failed.sp_dboption command failed.Any ideas?Thanks
One of our database size 90 Gb was backed up and restored on a standyby server with NoRecovery option .A Stored Procedure was written to do production Database transaction log Backup every 15 min and restore the Tlog File on Standby server.
Once the last transaction log file is restored on standby server the tlog file is deleted.
Now due to some reason the restore tlog job failed but the tlog file got deleted.
Now till we will not restore the last database backup we will not be able to start the restore tlog job.
Now the database is in restoring mode.Is there any methord to get that database online without any transactionlog.
Hi there, i would please like to know how to get a database online in T-SQL, I managed to get it offline in T-SQL, but know I can't get the database back online.
Please help.
If someone can please help me with the T-SQL command.
Or direct me to a link where I can get more information.
We have 2 data centers, site 1 and site 2. Â Site 1 is generally our primary, and site 2 is our Disaster Recovery (DR) site. Â I want to setup a SQL instance to have extremely high availability. Â Therefore I was looking at using DB mirroring, with synchronous data writing, high-safety and auto-failover. Â This requires the usage of a witness server.My problem with this setup, is that if the witness and principal are both at site 1, if site 1 goes away (power failure, asteroid impact, lol, anything else that would be a *true* DR scenario), then there is no failover to the mirror server at site 2. Â However, if you put the witness at site 2, anytime the WAN link between site 1 and site 2 has an issue, the DB will auto-failover to site 2. Â Is the reason for the recommendation of the witness being at the primary site because the thinking is that WAN link failure is likely more common than a *true* DR scenario that takes out all of site 1?
Is there anyway to backup the current online log after complete loss of the current correponding datafile?
Example:
(2) Logical Disk Volumes
Disk 1 (D:) contains pubs_data.mdf Disk 2 (E:) contains pubs_log.ldf
Disk 1 becomes corrupt and goes offline leaving the the database pubs in a suspect state. Is backup of the current online log pubs_log.ldf possible? If a backup of the log is not possible are there any other restoration methods that can be used to bring this database back online, rescuing as much information as possible from the current online log pubs_log.ldf.
We have our Production server having database on which Few DTS packages execute every night. Most of them have Bulk Insert stored procedures running.
SO we have to set Recovery Model of the database to simple for that period of time, otherwise it will blow up our logs.
Is there any way we can set up log shipping between our production and standby server, but pause it for some time, set recovery model of primary db to simple, execute DTS Bulk Insert Jobs, Bring it Back to Full recovery Model AND finally bring back Log SHipping.
It it possible, if yes how can we achieve this.
If not what could be another DR solution in this scenario.
After adding the Witness Server to the Mirror session, the Witness Connection state between the Mirror and Witness Connection is Disconnected and the state between Principal and Witness Connection is Connected.
The procedures defined in Books Online was used to setup Database Mirroring...when the Witness server was added to the Mirror session, only the alter database T-SQL statement was executed on the Principal server.
ALTER DATABASE <db_name> SET WITNESS = 'TCP://<servername>:<port>'
After executing the above statement, a few seconds later the state between Principal and Witness Connection changed to Connected and the state between Mirror and Witness Connection remains Disconnected.
The Mirror session is not using Certificates, every server is on the same domain, using the same domain login account, and all servers have SP2 installed running Enterprise Edition.
Any idea's why the state between Mirror and Witness Connection remains Disconnected?
y'day we faced situation one of the primary server went down and unable to failover the services to second node . by checking in logs we found
Cluster network 'Public' is partitioned. Some attached failover cluster nodes cannot communicate with each other over the network. The failover cluster was not able to determine the location of the failure. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
In my environment always on is there. Today I observed that primary server fail over to secondary server .now the secondary server acting as primary role.
Can I know when is fail over is happened and who did the fail over. Is there any script to find this?
I have a 2 node cluster with 2 standalone 2k14 instances having alwayson setup. As per client requirement we have created a client access point with a cname alias in dns to connect to secondary replica. Now, everytime whenerver the roles switch over one has to manually move this resource from the previous secondary node to the new secondary node. This is tedious, and should not be done manually either, so I am looking for a way to automate it so that as soon as the role switches over, the resource group after some time should also switch over to the current secondary.
We have SQL cluster installed on top of windows cluster on VM environment. Node1 and Node2 under Windows Failover Cluster. SQL instance is currently on node2 the instance is up and running, but SQL Cluster service remains online pending and it restarts the instance on every 5 minutes.
SQL Browser service are running successfully.TCP/IP ports are enabled and configured.If we start the SQL server agent it is on for seconds and stopped immediately  .Cluster Service is attempt to connect to the SQL service every few minutes (setting in SQL cluster resource) for the IsAlive check, if this fails then the SQL resource is restarted even if the instance was online. Hope this is what happening exactly.
[sqsrvres] ODBC Error: [08001] [Microsoft][SQL Server Native Client 11.0]SQL Server Network Interfaces: Error Locating Server/Instance Specified [xFFFFFFFF]. (268435455) 00001024.00053314::2015/10/30-19:57:50.772 ERR  [RES] SQL Server <SQL Server (SIMAH_COMMDB)>: [sqsrvres] ODBC Error: [HYT00] [Microsoft][SQL Server Native Client 11.0]Login timeout expired (0) 00001024.00053314::2015/10/30-19:57:50.772 ERR  [RES] SQL Server <SQL Server (SIMAH_COMMDB)>: [sqsrvres] ODBC Error: [08001] [Microsoft][SQL Server
Native Client 11.0]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books
Online. (268435455) 00001024.00053314::2015/10/30-19:57:50.772 INFOÂ [RES] SQL Server <SQL Server (SIMAH_COMMDB)>: [sqsrvres] Could not connect to SQL Server (rc -1
1. Once fail over to secondary replica, what will happen to connected session in primary node? can the session fail over to secondary seamlessly or need to re-login. what happen committed transactions which has not write to disk. 2. Assume I have always on cluster with three nodes, if primary fails, how second node make write/ read mode. 3. after fail over done to 2nd secondary node what mode in production(readonly or read write). 4. how to rollback to production primary ,will change data in secondary will get updated in primary.
We have set up Mirroring with a witness server and everything works fine when we failover from the SQL Management console.
However, if we failover when our Maccola client is connected, the client blows up - clearly because it can no longer connect to the database.
The ODBC DSN used by the Maccola client shows a checkbox for the 'select a failover server' but the checkbox is grayed out.
Also the summary of settings for the DSN at the end of the wizard reveals that the failover to server (y/N) option is set to N.
The default setting for this DSN is 'populate the remaining values by querying the server' but it doesn't appear to be getting the settings for failover from the server or any other interactive DSN settings either. The server is clearly set for mirroring.
Another suspicious item is that the DSN cannot connect to the server with SA permissions, even though the server is set to mixed security and we use the correct authentication.
Is it possible that the client MACHINE is not authenticating with the domain or sql server properly. We are logged into the client with the domain account that is the SQL admin account on the sql server box.
We should be able to interact with the sql server settings through the ODBC DSN on the client shoulnd't we?
1. In alwaysON fail over cluster, Once fail over to secondary replica, what will happen to connected session in primary node? can the session fail over to secondary seamlessly or need to re-login. what happen committed transactions which has not write to disk.
2. Assume I have always on cluster with three nodes, if primary fails, how second node make write/ read mode.
3. After fail over done to 2nd secondary node what mode in production(readonly or read write).
4. How to rollback to production primary ,will change data in secondary will get updated in primary.
Does anyone know where I can get SQL Server 2005 witness hosting? Our primary server is in Vermont and our DR site is in New Hampshire. I would like to set up a witness server somewhere off the east coast. I don't want to have to purchase a new server and pay big prices for colo.
Is there such a thing as witness hosting? Would someone be willing to host my witness and I can host yours?
We are trying to set up database mirroring. The mirroring works for the principal and mirrored databases (set up via Management Studio), but when we try to set up the witness database, we are getting the error message €œThe ALTER DATABASE command could not be sent to the remote server instance 'TCP://€¦ (Error: 1456)€?. The same occurs if I try to do €œALTER DATABASE xxx SET WITNESS =€? in TSQL. We are using the Standard edition for all three. We have tried using several different servers for the witness. All same user, same domain. Any suggestions? Thanks.
I have set up a SQL Express sytem to act as a witness....all are under the same admin account, all can ping and see each other. When I attempt to start a mirror session, it fails to issue an "ALTER Database" comment and the mirror setup is set to "High Performance".
This is my first time setting this up and it took over a day to do this as it is a relatively new concept, I am not DBA, just a S/W developer set up with the task of communicating with two DB instances which are mirrored. A couple of questions:
1) Is it normal to see the mirror instance in the following mode: (Mirror, Synchronized/Restoring)?
2) Any special steps required on SQL express besides enabling TCP/IP within the Configuration manager? It doesn't need the same instance of the same databaes on it, does it?
I have managed to set up mirroring on SQl 2005 however i have a come across a problem that i dont know how to resolve.
Im running the mirroring with 'High Safety with automatic failover.'
If i instagate a failover on the Principal Database, the mirror becomes the Principal, and vice versa ... great .. how ever .. i tried the golden test by making the principal server dissapear by yanking out the network connection, expecting the mirror to become the principal.....nope it just stays there in a disconnected / restoring state which isnt any good to man nor beast . I thought that the Witness should have dectected that the principal database had 'dissapeared' and would have made the mirror the Principal.
Any pointers on where i might have gone wrong in the setting up process, i used the mirror wizard
I removed mirroring on a configuration that is currrently working. Next, I moved the witness server to another IP and changed the FQDN (I patched the machine to another newtork, added a FQDN to our DNS server and reconfigured the machines IP information. I can ping the witness at its new address from the principal and the mirror).
Would changing the Witness's IP and FQDN prevent Database Mirroring from being setup after the certificate has been created and distributed to the principal and mirror instances under the old IP and FQDN? Should the endpoint be reconfigured and a new certificate be generated and distributed after an IP and FQDN change?
I start with 3 servers, in High Safety mode with witness. I disconnect the network cable for the witness. Then I remove the witness from the mirroring session:
ALTER DATABASE db1 SET WITNESS OFF
I reconnect the witness network cable. Now I cannot add the witness back to the mirroring session (either with TSQL or thru the GUI). Here is the error message:
Msg 1433, Level 16, State 4, Server SERVERA, Line 1 All three server instances did not remain interconnected for the duration of the ALTER DATABASE SET WITNESS command. There may be no witness associated with the database. Verify the status and when necessary repeat the command.
I have to stop mirroring (SET PARTNER OFF) and then reconfigure a new mirroring session to be able to include the witness.
If I don't disconnect the network cable, I am able to remove and re-add the witness to the session, no problem.
The problem: One server sees the witness, one does not. If the dbs are active on the one that doesn't see the witness and I shut that server down everything works as it should. However, if the dbs are active on the server that sees the witness and I shut that one down the mirror does not take over because it can't see the witness and form a quorum. It throws Event ID 1438 every 20 seconds complaing "The server instance Witness rejected configure request...The reason 1451, and state3, can be used for diagnotics by Microsoft...."
The scenario: I finished setting up a mirror in a WORKGROUP, by going through the GUI and then at the choosing not to start the mirror. That allowed me to modify the NETBIOS names and change them to IP addresses in order to get around the FQDN requirement that I can't fulfill since the servers are in a workgroup. They are in the same subnet and can run SQL queries against each other.
When I query sys.database_mirroring from the database servers I get this on the good server: mirroring_witness_name mirroring_witness_state_desc ---------------------------------------------------------- TCP://155.17.92.63:5022 CONNECTED TCP://155.17.92.63:5022 CONNECTED
And this on the bad server: mirroring_witness_name mirroring_witness_state_desc ----------------------------------------------------- TCP://155.17.92.63:5022 DISCONNECTED TCP://155.17.92.63:5022 DISCONNECTED
here's are the results from sys.database_mirroring_witnesses database_name principal_server_name mirror_server_name safety_level safety_level_desc -------------- -------------------------- ------------------------- ------------ ------------------ dbadmin TCP://pscdaltpscom20:5022 TCP://155.17.92.65:5022 2 FULL dbtest TCP://pscdaltpscom20:5022 TCP://155.17.92.65:5022 2 FULL
Oddly, the server that I created the mirror from registered itself with it's NETBIOS name even though I told it to use an IP address. However, that's the one that is working properly. I think this might be a red herring.
Does anyone have any ideas why my second server can't communicate with the mirror? Any clarifications I can make? I have been completely unable to find a step-by-step guide to creating mirror databases outside of a domain...so if someone knows of one please point me that direction.
Not sure where I am going wrong ... I am able to set-up the mirroring, but the witness set-up always fails. I am running all the 3 SQL servers (Enterprise edition) with same domain account, which is also a member of sysadmin locally within each SQL server.
When I try to add the witness on Principal using the following command:
ALTER DATABASE MyDB
SET WITNESS = 'TCP://WitnessServer:7022'
I have even tried the IP address with port number and fully qualifying domain (TCP://WitnessServer.xx.xx.com:7022) and I get the following message in error log.
2006-06-20 10:16:55.07 spid57 The ALTER DATABASE command could not be sent to the remote server instance 'TCP://WitnessServer:7022'. The database mirroring configuration was not changed. Verify that the server is connected, and try again.
All the 3 servers are configured to use TCP and Names pipes. I can connect, ping and telenet the Mirror Server from Principal. Any suggestions?
I'm running into a problem where I had mirroring working correctly (on server A ->B with C being the witness) then I set another machine(s) up to also use mirroring (X->Y with C also the witness). When I checked the A->B configuration today the witness was disconnected so I removed mirroring all together and attempted to set it up and I'm now getting the error:
TITLE: Microsoft SQL Server Management Studio ------------------------------
Alter failed for Database 'A'. (Microsoft.SqlServer.Smo)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.3042.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Alter+Database&LinkId=20476
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
------------------------------
The ALTER DATABASE command could not be sent to the remote server instance 'TCP://C:5022'. The database mirroring configuration was not changed. Verify that the server is connected, and try again. (Microsoft SQL Server, Error: 1456)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=09.00.3042&EvtSrc=MSSQLServer&EvtID=1456&LinkId=20476
One of my client asking mirroring between two server with witness.Is it possible witness and mirror in same server(with same instance)? OR Is it possible witness and primary in same server(with same instance) ? Best method to establishing mirroring with in two servers(with witness) ?