Recovery :: Failover Cluster With Mirroring With AlwaysOn?
Jun 30, 2015
we have to build high availability SQL 2012 cluster for VDI and we have two options. One option is to build a server cluster with combination of failover and mirroring and other option is to build failover cluster with AlwaysOn.We are not sure which option to chose. We have contacted Microsoft support to provide us some documents and instructions for failovermirroring combination but they have send us instructions for AlwaysOn option.
What would be best way to build high availability cluster for VDI? Also, since first option is very complicated.
1. In alwaysON fail over cluster, Once fail over to secondary replica, what will happen to connected session in primary node? can the session fail over to secondary seamlessly or need to re-login. what happen committed transactions which has not write to disk.
2. Assume I have always on cluster with three nodes, if primary fails, how second node make write/ read mode.
3. After fail over done to 2nd secondary node what mode in production(readonly or read write).
4. How to rollback to production primary ,will change data in secondary will get updated in primary.
We have a requirement to build SQL environment which will give us local high availability and disaster recovery to second site. We have two sites- Site A & Site B. We are planning to have two nodes at Site A and 2 nodes at Site B. All four nodes will be part of same Windows failover cluster. We will build two SQL Cluster, InstanceA will be clustered between the nodes at Site A Server and InstanceB will be clustered between the nodes at Site B, we will enable Always On Between the InstanceA and InstanceB and will be primary owner where data will be written on InstanceA and will be replicated to InstaceB. URL....Now we want we will have instanceC on the Site B and data will be writen from the application available on Site B, will be replicated to the instance on the Site A as replica.
I'm getting an error adding Replica to SQL AlwaysOn failover cluster in the new availability group wizard. When I enter the name of the target node (secondary replica) server and press connect, I get the following:
A network-related or instance-specific error occurred while establishing a connection to SQL Server.
The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 2) The system cannot fine the file specified
The SQL Browser service is up and running on the target. I am using an Azure VM for my SQL instance. This cluster spans geographies from our on-premise site to Azure via a VPN. This is a multi-subnet cluster. I'm attempting to create a new AG from the primary replica node and the target is a node on Azure called SSASNodeAz03.
[URL]
Full error:
Connect to Server Cannot connect to ssasnodeaz03
Additional information: A network-related or instance-specific error occurred while establishing a connection to SQL Server.
The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 2) The system cannot fine the file specified
1. Once fail over to secondary replica, what will happen to connected session in primary node? can the session fail over to secondary seamlessly or need to re-login. what happen committed transactions which has not write to disk. 2. Assume I have always on cluster with three nodes, if primary fails, how second node make write/ read mode. 3. after fail over done to 2nd secondary node what mode in production(readonly or read write). 4. how to rollback to production primary ,will change data in secondary will get updated in primary.
We are implementing a multi-site (Windows Server Failover Cluster) WSFC to enable Always On between our primary and DR site. We are not going to use SQL clustered instances. We are not planning to use shared disks. Each node is running a standalone instance of SQL 2012.
I have successfully configured a 3 node multi-site Windows failover cluster with no shared storage. For quorum, I have defined a File Share Witness (FSW). The FSW has voting rights and is in the DR site. The setup looks like this –
WSFC –
•Node A – Site #1 (voting right = 1) •Node B – Site #1 (voting right = 1) •Node C – Site #2 (voting right = 0) •FSW – Site #2 (voting right = 1)
Again - There are no shared disks in our setup. We are not going to use SQL clustered instance. We are going to use Always On with these 3 nodes.
SQL Always On –
•Node A – Site #1 (Primary Replica) •Node B – Site #1 (Readable Secondary) •Node C – Site #2 (Readable Secondary)
All the setup including the “availability group” works properly under this setup. However, a failover to site #2 under DR situation is not working and I know why but don’t know what needs to be done to fix the problem.
The following works fine –
•Automatic failover between nodes A and B (same site – site #1) •Forced failover to node C in site #2 provided at least one of the nodes in site #1 is up (non – DR situation) - this will ensure the cluster is up
The following is not working –
•Forced failover to node C in site #3 when both nodes in site #1 are lost (true DR situation) – This is because the cluster is not up at this point.
I know I have to bring the cluster up somehow and I have not been able to do so by restarting the cluster service.
I tried to run the command to start cluster service.
Question –
How can I FORCE the cluster to come up in Site #2 on node C when it has no voting rights?
I have always worked with even number of nodes and shared disks with traditional clustering. I am not sure what needs to be done in this scenario with 3 nodes and a FSW.
How can we find the cluster failover count in always on ?
As my AG is configured as synchronous mode , AG went offline and we manually restarted the AG service when we check the properties on AG role they r in default setting ?
My environment has a 4 node cluster , 2 in primary and 2 in sec dc. Storage is sperate for both.
Need to setup always on for 4 Instances there on the 2 nodes of the primary dc. Is there any restriction in setting up always on for multiple instances for a cluster.
I have had a serious issue with a production AlwaysOn cluster whereby the service did not successfully transition to the secondary node and I cannot find the root cause of the issue.
Some details: It is a 2 node cluster (same datacenter) with a shared disk quorum, Windows Server 2012, both are virtual machines running on VMWare vSphere 5.5. SQL Server version is 2012 Enterprise SP2 CU6
The failover occurred because of a network incident (a spanning tree recalculation caused a connection timeout between both nodes). Initial entries in the SQL Log look normal for this event, for example:
05/08/2015 11:18:06: A connection timeout has occurred on a previously established connection to availability replica 'FIN-IE-PA078' with id [6910F4A9-87E7-4836-BA79-0F41BE90266D]. Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role. 05/08/2015 11:18:06: AlwaysOn Availability Groups connection with secondary database terminated for primary database 'UserManagement' on the availability replica with Replica ID: {6910f4a9-87e7-4836-ba79-0f41be90266d}. This is an informational message only. No user action is required.
[code]....
My interpretation of this is that the cluster failover attempts failed, because the network condition still persisted. The network interruption lasted approximately 2 minutes, and I would have expected the cluster to come back online at this point, after the restart delay period as suggested in the last entry in the error log. However this did not happen.
I am using SQL 2012 SE with clustering on Windows server 2008 R2. Now I want migrate it to windows server 2012 with minimal down time. So I want to evict the passive node and add a new node with windows server 2012 and install sql server 2012 SE on the new passive node and perform a failover(make the node with 2012 OS as active) and then evict the new passive node and add another node with windows 2012 and then do the same thing?
I have a 3-nodes AlwaysOn cluster (Windows Server 2008 R2 SP1 + SQL Server 2012 RTM), Node Majority quorum, the quorum vote for each node is 1.
Today the AlwaysOn AG was suddenly down due to the cluster service on node 1 stopped and can't be started.
The error in eventlog is -
The cluster database could not be loaded. The file may be missing or corrupt. Automatic repair might be attempted. The Cluster Service service terminated unexpectedly. It has done this 2 time(s). The following corrective action will be taken in 120000 milliseconds: Restart the service.
The failover cluster database could not be unloaded. If restarting the cluster service does not fix the problem, please restart the machine.
The Cluster Service service terminated with service-specific error The system cannot find the file specified..
The error log in cluster log is -
0000156c.000008f8::2012/09/05-08:09:36.057 INFO [DM] Key RegistryMachineCluster.restored does not appear to be loaded (status STATUS_OBJECT_NAME_NOT_FOUND(c0000034)) 0000156c.000008f8::2012/09/05-08:09:36.057 WARN [DM] Node 1: Failed to unload restored hive from the registry with error STATUS_INVALID_PARAMETER(c000000d) 0000156c.000008f8::2012/09/05-08:09:36.057 INFO [DM] Node 1: loading local hive 0000156c.000008f8::2012/09/05-08:09:36.057 ERR [DM] Node 1: failed to unload cluster hive, error 2.
Now the cluster service can't be started on node 1, error code 2. Looks like the clusdb in C:windowscluster is missing or corrupted. How to restore the clusdb file? And how to prevent this happen again?
All nodes were well patched, AlwaysOn and cluster related hotfixes were all installed. [URL] .... doesn't wok.
I'm getting the following error when I go to create a cluster in the Failover Cluster Manager in Windows Server 2008.
"The address 10.10.10.111 is not valid for its associated network"
I'm following the instruction in the book for the 70-462 exam. There was a step that had me create a DNS A record for the address sql-cluster.contoso.com. The IP address was mapped to 10.10.10.111. I'm not sure if this is the culprit but its the only time I used that IP address in the setup.
Below are 2 screenshots. The first screenshot is the error. The second screenshot is my DNS console.
I saw following point on Technet article about RBS.The local FILESTREAM provider is supported only when it is used on local hard disk drives or an attached Internet Small Computer System Interface (iSCSI) device. You cannot use the local RBS FILESTREAM provider on remote storage devices such as network attached storage (NAS).It looks like that we cannot use FILESTREAM on Failover Cluster because to setup Failover Cluster we need to have NAS. But then the NAS is made available locally for Failover Cluster so FILESTREAM should work right?Found another article which talks about setting up FILESTREAM on Failover Cluster. URL...
The main objective is to have a third party program operate on a failover cluster. The OS is Windows Server 2012 Datacenter loaded on 2 nodes. A virtual node exists along with supporting disks. This client software uses a SQL Server database. SQL Server 2012 Enterprise is installed and operating in a failover environment. However the client software is not failing over. If the connection to node A is lost, SQL Server fails over to node B. But the client application does not.
What needs to occur in order to associate the client software with the failover cluster? This software has 6 services total installed. Some are referred to as servers - looks like to communicate between remote client computers and the database. What is the process to associate the client software with the failover?
I have a Windows 2008 R2 Always on Cluster with 3 nodes (two in the primary site and one in the DR site).
Primary Site: -Primary Site Server1 -Primary Site Server2
DR Site 1 (to be decommed): -DR Site Server1
Our company is planning on decommissioning the DR site. But before we do this, we want to add a 4th site to the cluster. Migrate the data...and then decommission the original DR Site.
Is it possible to have this configuration:
Primary Site: -Primary Site Server1 -Primary Site Server2
DR Site 1 (to be decommed): -DR Site Server1
DR Site 2 (NEW DR Site): -DR Site Server1
IF this is possible, do I simply add the new DR site to the existing cluster (same steps as adding the first DR node to the cluster when the cluster was originally configured? or are there special steps?
I want to install service pack 3 to my SQL Server 2012 Enterprise running on windows server 2008 R2 Enterprise fail over cluster, I read about the SP installation in technet, its mentioned that the passive node should be patched first and to do this the passive node should be removed from the cluster, I need to know whether I should completely remove the node from windows cluster or remove the node by using SQL Server installer and install the service pack and then add it back to the cluster, Can I do this by pausing the node in cluster and perform the service pack installation ?
We have 2 data centers, site 1 and site 2. Site 1 is generally our primary, and site 2 is our Disaster Recovery (DR) site. I want to setup a SQL instance to have extremely high availability. Therefore I was looking at using DB mirroring, with synchronous data writing, high-safety and auto-failover. This requires the usage of a witness server.My problem with this setup, is that if the witness and principal are both at site 1, if site 1 goes away (power failure, asteroid impact, lol, anything else that would be a *true* DR scenario), then there is no failover to the mirror server at site 2. However, if you put the witness at site 2, anytime the WAN link between site 1 and site 2 has an issue, the DB will auto-failover to site 2. Is the reason for the recommendation of the witness being at the primary site because the thinking is that WAN link failure is likely more common than a *true* DR scenario that takes out all of site 1?
Came across this scenario in AlwaysOn Availability Group (two node), file share witness times out and RHS terminate and cause the cluster node to reboot. File share witness is for continuous failover and if the resource is unavailable my expectation was that it should go offline and should not impact Server or Sql Server. But its rebooting the cluster node to rectify the issue.
Configuration Windows Server 2012 R2 (VMs) Sql Server 2012 SP2 Errors
A component on the server did not respond in a timely fashion. This caused the cluster resource 'File Share Witness' (resource type 'File Share Witness', DLL 'clusres2.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.
The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue.
I have getting issues when i am creating listener for always On . Error shown as below
Can not bring the Windows server fail over cluster (WSFC) resources online. (Error Code 5942). The WSFC service may not be running or may not be accessible in its currents states, or the WSFC resources may not be in a state that could accept the request.
For information about this error code see "system error code" in windows development documentation
The attempt to create network name and IP address for the listener is failed. The WSFC service may not be running or may not be accessible in its currents states or the value provide for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate network name and IP address with network administrator. (Microsoft SQL Server error 41066) ...
Server : Windows server 2008 DB Server : SQL Server 2008 (SP1)
Here are the series of events which happened.
1.) Event ID: 1135 Cluster node 'XYZ' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
2.) Event ID: 1049 Cluster IP address resource 'SQL IP Address 1 (XYZ)' cannot be brought online because a duplicate IP address '10.9.8.113' was detected on the network. Please ensure all IP addresses are unique.
3.) Event ID: 1069 Cluster resource 'SQL IP Address 1 (XYZ)' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
4.) Event ID: 1049 Cluster IP address resource 'Cluster IP Address' cannot be brought online because a duplicate IP address '10.9.8.112' was detected on the network. Please ensure all IP addresses are unique.
5.) Event ID: 1069 Cluster resource 'Cluster IP Address' in clustered service or application 'Cluster Group' failed.
6.) Event ID: 1066 Cluster disk resource 'Cluster Disk 25' indicates corruption for volume '?Volume{88552e6f-aea2-11df-9790-0026b92fffa7}'. Chkdsk is being run to repair problems. The disk will be unavailable until Chkdsk completes. Chkdsk output will be logged to file 'C:WindowsClusterReportsChkDsk_ResCluster Disk 25_Disk16Part1.log'. Chkdsk may also write information to the Application Event Log.
7.) Event ID : 1066 Cluster disk resource 'Cluster Disk 26' indicates corruption for volume '?Volume{88552e05-aea2-11df-9790-0026b92fffa7}'. Chkdsk is being run to repair problems. The disk will be unavailable until Chkdsk completes. Chkdsk output will be logged to file 'C:WindowsClusterReportsChkDsk_ResCluster Disk 26_Disk4Part1.log'. Chkdsk may also write information to the Application Event Log.
8.) Event ID: 1049 (Same message as point 2)
9.) Event ID: 1069 (Same message as point 3)
10.) Event ID : 1049 (same message as point 4)
11.) Event ID :1069 (same message as point 5)
12.) Event ID :1205 The Cluster service failed to bring clustered service or application 'Cluster Group' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
13.) Event ID: 1069 Cluster resource 'Cluster Disk 17' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
14.) Event D : 1049 (same message as point 2)
15.) Event ID: 1069 Cluster resource 'SQL IP Address 1 (XYZ)' in clustered service or application 'SQL Server (MSSQLSERVER)' failed.
16.) Event ID : 1205 The Cluster service failed to bring clustered service or application 'SQL Server (MSSQLSERVER)' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
first of all,I went through all the logs, and could not find the reason for fail-over initialization. There should be some thing logged why the failover happened? secondly after failover the service was not coming online due to duplicate IP address detection.
Later when we try to manually bring the service online from cluster management it comes online successfully. I don't understand how would duplicate IP address get resolved when we start manually.
Lastly we see few errors related to physical disk resource between failover retries, is this could be the correlated to failover error ?
I Config Ha-Alwayson on 2 test servers . In addition, was defined a listener for them.i can connect to them from the listener and in directly. I did manual Failover and it worked.However all connection to all servers (primary and secondary and listener) was breaked. I expected my connection To The listener, be stable. But How can I test the Auto failover mechanism? I run this scenario :
1- I filled all free space from the primary server else a bit. 2- And run on it a Huge Update to fill remain free space. 3- MeanWhile I Run an insert command into listener IP. (in a while Loop)
I expected :
>>> After run update or in middle of it , The primary server face to a problem. (Full Log file). And This was happened. >>> After I expected The Failover act and change Primary And Secondary.And My insert commands Continues without Break Or Continue On new server After some Seconds
But It didn't Happend.Both Of 2 Command are stoped !!!!! And auto failover didnt act. I tryed To create a manual fail on primary server . I Tried to Offline the main database in primary server.
Then
1- What is the meaning Of fail that Auto failover act about it ? 2- In which scenario I can Test It ?
Data synchronization and manual failover works fine. But, sometimes, the AlwaysOn cluster automatically fails over to Sync Commit Secondary on Primary data center. Here is the error message from Failover Cluster Manager->Cluster Events:
"Cluster has missed two consecutive heartbeats for the local endpoint xx.xx.xx.yy:~3343~ connected to remote endpoint xx.xx.xx.zz:~3343~"
"Cluster has lost the UDP connection from local endpoint xx.xx.xx.yy:~3343~ connected to remote endpoint xx.xx.xx.zz:~3343~"
I had our network engineer check all connections multiple times and he confirmed everything is fine. But he was also able to confirm (using monitoring tools) that right at the time of a failover, there is almost 2GB worth of traffic going from Primary Server to DR server. That happens every time. I had checked the times of all failovers and there is no job or process occuring that will produce 2GB worth of data. Also, this happens regardless of which server is primary.
Even though the failover works fine, this unexpected automatic failover due to missed heartbeats are occurring often (2-3 times a month).
Here is the list of errors from the Cluster Validation Report:
Under Network Section, I see the following error messages in Red:
Validate Network Communication
Network interfaces Server4 (DR) - SAN_Team and Server1 (Primary) - SAN_Team - VLAN 20 are on the same cluster network, yet address xx.xx.xx.pp is not reachable from xx.xx.xx.yy using UDP on port 3343.
Network interfaces Server4 (DR) - SAN_Team and Server2 (Secondary) - SAN_Team - VLAN 20 are on the same cluster network, yet address xx.xx.xx.qq is not reachable from xx.xx.xx.yy using UDP on port 3343.
I have a 3 node 2014 AlwaysOn setup. The primary and secondary are set for automatic failover. The third node, of course, is manual (until 2016). The 2 nodes with are automatic are sitting in one datacenter, the third is in another. If the first datacenter was to go down, I would manually have to failover to the third node? What's the normal process here for having two datacenters and ensuring the availability group is always available?
I'm looking for a solution to have cross data center automatic failover in the event of a data center loss for highly critical databases. I would like to have local HA and also automatic failover to the DR site. This does not seem possible with AlwaysOn.
Is my only option for automatic cross data center failover to build a node in one data center and a node in the other data center with a node/FS at a third data center in order to maintain quorum? I'd like to have local HA in the mix but that doesn't seem possible.What pattern for the highest data security and also availability?
If there is a history kept somewhere of failover events of a database in an AO group? I have 2 replicas with automatic failover and I'm looking for a history of failovers.
An automatic failover set exists. This set consists of a primary replica and a secondary replica (the automatic failover target) that are both configured for synchronous-commit mode and set to AUTOMATIC failover.Configured the both AG Group database automatic failover and synchronous-commit mode.But automatic Failover failed also Cluster service not started automatically at Node2. It got connected through AO Listerner after starting Node1. As below SQL Error log during shutdown Node1
Date,Source,Severity,Message 10/27/2015 10:44:20,spid37s,Unknown,AlwaysOn Availability Groups: Waiting for local Windows Server Failover Clustering node to come online. This is an informational message only. No user action is required. 10/27/2015 10:44:20,spid37s,Unknown,AlwaysOn Availability Groups: Local Windows Server Failover Clustering node started.
We are planning to change all IPs of PRODUCTION Failover Cluster Setup. In my cluster setup ... we have 2 Physical Nodes with windows-2008, Roles are MSDTC and SQL-2008R2.
IP change for:
1. Both Nodes(Physical) 2. MSDTC 3. SQL Server 4. windows Cluster
So Almost... All IPs are going to change.
Im DBA here, I need to take care of SQL cluster and MSDTC. But I haven't performed this activity before.So I'm worrying about Impacts and consequences of this change. steps how should I perform this activity.
What happens when an automatic failover occurs, in a two server AlwaysOn Availability Group configuration, where the secondary replica is configured as read-only?
Will it only allow read-only connections, or will it become read-write and can accept INSERT, UPDATES and DELETES when assigned the new role as Primary?
Is it correct that adding a third server/node, that just acts as passive and should be used for automatic failover, to support true HADR, would NOT need another license .. and that licenses would only be required for the previous Primary and Secondary (Read-Only) replicas?
Is there any single TSQL query which provides below info.When did my AlwaysOn Availability group failed over and from which node it failed to which new node(i.e. replica)?
We had to failover our primary db server for maintenance to our secondary replica. The primary was rebooted during maintenance. We failed back after the maintenance and one of the databases is not synchronizing.
I checked sys.dm_hadr_database_replica_states, and it is showing that it is INITIALIZING.
It has been in this state for more than 45 mins now. The last_sent_time, last_received_time, last_hardened_time and last-redone_time are all stuck with a time stamp 45 mins ago.
They haven't changed. How do i resume this database and bring it back in sync?
I tried suspending and resuming the data movement, but hasn't worked.
