SQL Server 2008 :: Partition Sliding Window Causing Loads Of Blocks
Oct 29, 2015
We have a massive database with an almost massive amount of traffic to and from it.
I've been requested to implement a sliding window partitioning with 2 partitions an active and passive 1,I managed to test this on a very small testbed last month.
I currently moved 97k table on to the partition function leaving me another 26 k to go
I'm using the following stored procedure to implement the sliding window
CREATE PROCEDURE [dbo].[ManageFactSlidingWindow](@pFunction nvarchar(max),@pSchema nvarchar(max),@FG nvarchar(max),@moveDays int)
/*****************************************************************************
PROCEDURE NAME: [ManageFactSlidingWindow]
AUTHOR: Arshad Ali
CREATED: 02/24/2013
DESCRIPTION: This stored procedure manages sliding window for the partitioned table
VERSION HISTORY:
DATE EMAIL Company DESCRIPTION
[Code] .....
When I try to move the partition even a single day I get loads of locks.
CREATE PARTITION FUNCTION [TimeTicksRangePFN](bigint) AS RANGE RIGHT FOR VALUES (633294720000000000, 633320640000000000, 633347424000000000, 633374208000000000, 633399264000000000)
These numbers happen to correspond to the dates 11/1/7, 12/1/7, 1/1/8, 2/1/8 and 3/1/8 in ticks respectively.
I began with a partition scheme as follows:
CREATE PARTITION SCHEME [TimeTicksRangePScheme] AS PARTITION [TimeTicksRangePFN] TO ([FG_xxx_EventArchive00001], [FG_xxx_EventArchive00002], [FG_xxx_EventArchive00003], [FG_xxx_EventArchive00004], [FG_xxx_EventArchive00005], [PRIMARY])
While running my “sliding window script� , which I hoped would 1) roll off the oldest partition of my EventArchive table and 2) add a new partition with a tick boundary that equates to 3/5/8, I get an error related to my switch out table's index, the same table's Filegroup and Primary.
After getting the error, I scripted the partition function as a create in mgt studio and got…
CREATE PARTITION FUNCTION [TimeTicksRangePFN](bigint) AS RANGE RIGHT FOR VALUES (633320640000000000, 633347424000000000, 633374208000000000, 633399264000000000, 633402720000000000)
...which looks like what I had intended cuz the last boundary is the tick representation of 3/5/8 and the oldest has rolled off
scripting the scheme produced...
CREATE PARTITION SCHEME [TimeTicksRangePScheme] AS PARTITION [TimeTicksRangePFN] TO ([FG_xxx_EventArchive00001], [FG_xxx_EventArchive00003], [FG_xxx_EventArchive00004], [FG_xxx_EventArchive00005], [PRIMARY], [FG_xxx_EventArchive00001])
which looks nothing like what I intended, I thought I’d end up with …00002,…00003,…00004,…00005,…00001,PRIMARY
the script steps that seem most relevant start at the 5th step as follows...
5) creates table [dbo].Switch on the switch out filegroup with columns, PK and indexes matching exactly those of [dbo].EventArchive
6) switches partition 1 of [dbo].EventArchive to [dbo].Switch
7) ALTER PARTITION FUNCTION TimeTicksRangePFN() MERGE RANGE (633294720000000000) --this was the oldest date corresponding to 11/1/7
8) truncates [dbo].Switch
9) drops all indexes on [dbo].Switch except a clustered index (IX_TimeTicks), leaves PK constraint alone
10) ships the new data whose values range from 3/1/8 to less than 3/5/8 to [dbo].Switch and deletes them from their source
11) recreates all non clustered indexes on [dbo].Switch
12)ALTER TABLE [dbo].[Switch] WITH CHECK ADD CONSTRAINT RangeCK CHECK ([TimeTicks] < the number of ticks represented by 3/5/8)
13)ALTER PARTITION SCHEME TimeTicksRangePScheme NEXT USED [FG_xxx_EventArchive00001] --fg isnt really hardcoded
14)ALTER PARTITION FUNCTION TimeTicksRangePFN() SPLIT RANGE (the number of ticks represented by 3/5/8)
15)ALTER TABLE [dbo].[Switch] SWITCH TO [dbo].[EventArchive] PARTITION 5
step 15 is the one that fails with message "ALTER TABLE SWITCH statement failed. index 'xxx.dbo.Switch.IX_TimeTicks' is in filegroup 'FG_xxx_EventArchive00001' and partition 5 of index 'xxx.dbo.EventArchive.IX_TimeTicks' is in filegroup 'PRIMARY'.
I have a partition function as follows: CREATE PARTITION FUNCTION [TimeTicksRangePFN](bigint) AS RANGE RIGHT FOR VALUES (633294720000000000, 633320640000000000, 633347424000000000, 633374208000000000, 633399264000000000)
These numbers happen to correspond to the dates 11/1/7, 12/1/7, 1/1/8, 2/1/8 and 3/1/8 in ticks respectively.
I have a partition scheme as follows: CREATE PARTITION SCHEME [TimeTicksRangePScheme] AS PARTITION [TimeTicksRangePFN] TO ([FG_xxx_EventArchive00001], [FG_xxx_EventArchive00002], [FG_xxx_EventArchive00003], [FG_xxx_EventArchive00004], [FG_xxx_EventArchive00005], [PRIMARY])
After running my €śsliding window script€? , which I wanted to switch out the lowest partition with, and add a new partition with a new tick boundary that equates to 3/5/8, I get an error saying that 1 of my switch out table€™s indexes is in filegroup 1 but partition 5€™s index of the same name is in PRIMARY. At this point the partition function looks like€¦
CREATE PARTITION FUNCTION [TimeTicksRangePFN](bigint) AS RANGE RIGHT FOR VALUES (633320640000000000, 633347424000000000, 633374208000000000, 633399264000000000, 633402720000000000) which looks like what I had intended cuz the last boundary is tick representation of 3/5/8 and the oldest has rolled off
and the scheme looks like€¦
CREATE PARTITION SCHEME [TimeTicksRangePScheme] AS PARTITION [TimeTicksRangePFN] TO ([FG_xxx_EventArchive00001], [FG_xxx_EventArchive00003], [FG_xxx_EventArchive00004], [FG_xxx_EventArchive00005], [PRIMARY], [FG_xxx_EventArchive00001]) which looks nothing like what I intended, I thought I€™d end up with €¦00002,€¦00003,€¦00004,€¦00005,€¦00001,PRIMARY
the relevant script steps are...
5) creates table [dbo].Switch on the switch out filegroup with columns, PK and indexes matching those of [dbo].EventArchive (allows default location for indexes) 6) switches partition 1 of [dbo].EventArchive to [dbo].Switch 7) ALTER PARTITION FUNCTION TimeTicksRangePFN() MERGE RANGE (633294720000000000) 8) truncates [dbo].Switch 9) drops all indexes on [dbo].Switch except a clustered index (IX_TimeTicks), leaves PK constraint alone 10) ships the new data whose values range from 3/1/8 to less than 3/5/8 to [dbo].Switch and deletes them from their source 11) recreates all non clustered indexes on [dbo].Switch 12)ALTER TABLE [dbo].[Switch] WITH CHECK ADD CONSTRAINT RangeCK CHECK ([TimeTicks] < the number of ticks represented by 3/5/8) 13)ALTER PARTITION SCHEME TimeTicksRangePScheme NEXT USED [FG_xxx_EventArchive00001] 14)ALTER PARTITION FUNCTION TimeTicksRangePFN() SPLIT RANGE (the number of ticks represented by 3/5/8) 15)ALTER TABLE [dbo].[Switch] SWITCH TO [dbo].[EventArchive] PARTITION 5
step 15 is the one that fails with message "ALTER TABLE SWITCH statement failed. index 'xxx.dbo.Switch.IX_TimeTicks' is in filegroup 'FG_xxx_EventArchive00001' and partition 5 of index 'xxx.dbo.EventArchive.IX_TimeTicks' is in filegroup 'PRIMARY'.
Kimberly Tripp describes a recipe for switching partitions in and out, thru the use of staging tables, when it comes time to "slide the window" on a partitioned table. She says that the clustered index (on staging) must be the same as that chosen for the partitioned table itself but she doesnt discuss whether or not all of the non clustered indexes need to be the same too once the ALTER TABLE Orders SWITCH PARTITION 1 TO OrdersOctober2002 and ALTER TABLE OrdersOctober2004 SWITCH TO Orders PARTITION 24 run. For the data being switched out, I wouldnt want to do anything extra. For the data being switched in, I'd like to understand if she is implying that all other indexes would be built automatically as a result of the 2nd ALTER statement?
Kimberly's article is at http://www.sqlskills.com/resources/Whitepapers/Partitioning%20in%20SQL%20Server%202005%20Beta%20II.htm#_Toc79339965
I was trying to clean up some conversation in Service Broker and caused alot of blocking that I seem to unable to kill. there was 1 conversation that I was not able to end, so I wanted to restart sql service, But I can't even restart the SQL service. I get the following in Event Viewer
Timeout occurred while waiting for latch: class 'SERVICE_BROKER_TRANSMISSION_INIT', id 00000001A2B03540, type 2, Task 0x0000000000C2EDA8 : 0, waittime 5400, flags 0xa, owning task 0x00000002DEBCA5C8. Continuing to wait.
(idAuxiliarPF(BIGINT+PK), pf(BIGINT+FK), Data(DateTime), Descr(NVARCHAR))that has aprox. 50k rows.
I need to create a partition of the data to join to another table, the query that i have:
SELECT ROW_NUMBER() OVER (PARTITION BY pf ORDER BY Data DESC, idAuxiliarPF DESC) AS RN, pf, Data, Descr FROM dbo.PFAuxiliar WHERE Data <= GETDATE()This query takes around 40 seconds to return the results
If i remove the Descr column, the query it takes no time.
SELECT ROW_NUMBER() OVER (PARTITION BY pf ORDER BY Data DESC, idAuxiliarPF DESC) AS RN, pf, Data FROM dbo.PFAuxiliar WHERE Data <= GETDATE()I have two indexes, Clustered (idAuxiliarPF), NONClustered(pf).
Can we create the Partition on Existing Table?e.g Create table t ( col1 number(10,0), Col2 Varchar(10)) ;After the table Creation can we alter the table to partition the table.
Our monitoring tool shows that our production system periodically experiencing large rate - up to 800 memory pages/sec. How to find out which particular queries, S.P., processes that initiate this?
we planning to create partitioning on existing tables. The partitioning is on date column, there should be one partition for each year.
Creating of new partitions should be automated, and also we dont have any plans of archiving old data, all we want is that new partition creation should be automated.
When you load the data into a new partition table, can it to done online without any downtime? because I have few tables that are around 250 gigs and more.
We have restored a database on new server without keeping replication settings. Now while creating the publication no tables are shown in new publication wizard window. In fact we have hundred of tables in database and included in replication on source server from where backup was taken.
What may be the reason that tables are not appearing?
A common partitioning scenario is when the partition column has the same value for every record in the partition, as opposed to a range of values. Am I the only person who wonders why there isn't an option to automatically partition a table based on the unique values of the partition column? Instead of defining a partition function with constants, you ought to be able to just give it the column and be done. This would be particularly valuable for tables partitioned on a weekly or monthly date; when new data is added it could simply create a new partition if one doesn't already exist.
currently need to re-create an aggregate table in a proc every night to aggregate purchases broken down by person/store in groups of 3, 6 12 etc months.finding the performance of it is very slow as it covers 500,000 million rows.The query looks like
SELECTCusID(int) , StoreID(int) , SUM(L3M) as Last3Months , SUM(L6M) as Last6Months , SUM(L9M) as Last9Months
[code]...
I need to make changes to this because it is using a BETWEEN on a datetime column. I was wondering though, if anyone else has made agg tables like this before an found a better way of doing them?
In MSDN, it says that it is recommended to use windows authentication to connect to SQL Server rather than use mixed authentication.
I create user deltasqluser on windows OS, and I specify in my webform ASP.NET script below :
protected System.Web.UI.WebControls.Label Label1; private string _connString = @"data source=deltasql2000;initial catalog=northwind;integrated security=false;user id=deltasqluser"; /* comment : I login to my windows as deltakoronx, and I want to every user (including me), connected to sql server through IIS, will be identified as deltasqluser not as user's login (impersonate) */ private void Page_Load(object sender, System.EventArgs e) { SqlConnection conn = new SqlConnection(_connString); SqlCommand cmd = new SqlCommand(); cmd.Connection = conn; cmd.CommandType = CommandType.Text; cmd.CommandText = "select suser_sname()";
at web.config, I add : <identity impersonate="false" userName="deltasqluser" password="" />
at IIS webApplication1's properties, tab "Directory Security", at "Authentication and access control" section, I checked "enable anonymous access" with user : DELTAIUSR_DELTA and checked "Integrated Windows Authentication",
at query analyzer, I login as "sa" and execute script below : exec sp_grantdbaccess 'deltasqluser','northwind'
when I run the ASP.NET script, error at conn.Open(); with error message : Login failed for user 'NT AUTHORITYNETWORK SERVICE'.
What should I do so that IIS login to SQL Server as user deltasqluser not as "NT AUTHORITYNETWORK SERVICE" ?
When I attempt to load a database from dump format across a network (100mb Ethernet) It takes forever. (15 hours for 16GB!) can anyone help me find a starting point to troubleshoot this?
Thanks!
-Chris
P.S. File Copies of the same size move at a rapid fashion, and I cannot find any bottlenecks in the network.
Dim cnn As ADODB.Connection Dim rst As ADODB.Recordset
Private Sub Form_Load()
Set cnn = New ADODB.Connection cnn.ConnectionString = "driver={SQL Server};" & "server=SCHS-SQL;uid=sa;pwd=sa;database=Library" cnn.Open
Call loadrst
End Sub
Public Sub loadrst() Set rst = New ADODB.Recordset Dim sql1 As String sql1 = "select * from Books order by srno" rst.Open sql1, cnn, adOpenDynamic, adLockOptimistic, adCmdText
If rst.EOF = True Then MsgBox ("No records are present") Command1.Enabled = False Else Call display Command1.Enabled = True End If
End Sub
This is the code i use basically to connect my vb6 application to sql server 2005. I had started out lately trying to use sql server instead of access. So far none of the program have given any problems as the databases has a max one of 120 records. But the one which this code connects to has about 5200 records. I had imported the tables from access into sql server. The size of the database was around 17.67mb so i shrank it and it became 4mb. But still it takes roughly 2 minutes for the user to see the records in the grid. Could you tell me what to do?
Hello,we use two instances of the ms sql server 2000 (Version: 8.00.760) on a4 processor windows 2000 (sp 4) machine.A dts-packet on one instance completly blocks the work on the otherinstance. Both instances have 2 dedicated processors. The twoprocessors of the blocked instance are in idle states.How is this possible? Has someone had the same behavior of the sqlserver and found a solution for this problem?Thanks in advance for answersJürgen Simonsen
Hello. I have a 32-bit SQL 2005 (SP1) server that is my current Production server. I also have a 64-bit SQL 2005 (SP1) server that will become my Production server. I have several SSIS packages that load/refresh data on a nightly basis to a few of my databases from DB2 (MVS). I have the packages setup on the current Production server (32-bit) and all is working well through the Microsoft OLE DB provider for DB2. However, on the 64-bit server, I am experiencing some issues with the SSIS packages failing due to large loads. Loads that are loading tables with 500K, or less, data seem to run without issue (through SQL Agent Jobs). But, larger table loads are failing.
I do have a linked server set up on the 64-bit server to the 32-bit server, for other processes. And because of this I have lightweight pooling turned off on the 64-bit server (because of distributed querying). Lightweight pooling is turned on on the 32-bit server. Could this be what is causing some of my issue? Since I don't have the lightweight pooling option turned on (on the 64-bit server), am I not getting the proper amount of through-put for my 8 dual core CPU server?
I'm having trouble getting off the ground with the Web application walkthrough "Walkthrough: Creating a Web Application Using Visual C# or Visual Basic" in VS.NET Pro 2002 [Academic] documentation. After a bit of fishing around, and consulting the MS Knowledge Base, I got the pubs database installed. I also got the connection to work well enough that the dataset would fill in the IDE.
The problem is that when I try to run the web form, either from the IDE debug menu, or by accessing the .aspx file on localhost using Firefox, I get the error: SELECT permission denied on object 'titles', database 'pubs', owner 'dbo'. showing in the browser. My understanding is that this page is running as ASPNET, and I did already carry out the recommended commands to enable access: C:>osql -E -S MY-MACHINE-NAMEVSDOTNET -Q "sp_grantlogin 'MY-MACHINE-NAMEASPNET'" C:>osql -E -S MY-MACHINE-NAMEVSDOTNET -d Pubs -Q "sp_grantdbaccess 'MY-MACHINE-NAMEASPNET'" both of which commands returned successfully. Any suggestions as to what else I should do to get the necessary permissions to actually display the data in my browser? Does the IIS user account need permission also?
Thanks for any insight into this vexing problem. I must say that along the way, I have had some fun exploring the osql comand-line tool. Using the -E switch, I have been able to run select and upgrade queries, but this is all pretty much fishing in the dark. I would like to get back to actually working with the walthroughs in the Visual Studio documentation.
I installed Sql Server 2000 on a Windows 2003 machine and everythingappeared to be fine. It has Sql Server SP4 and Windows 2003 SP1 installed.The problem occurs whenever I use Enterprise Manager or Query Analyzer formore than a few minutes. After a while, all internet connectivity on theserver machine is broken. I cannot connect to it through Enterprise Manager,and on the machine itself, nothing related to the internet works at all.There are no errors in the Windows log or the Sql log. Disabling andenabling the Ethernet adapter fixes connectivity, until the next time I useEnterprise Manager for a while. What is going on?The machine is a Dell PowerEdge SC420 with a BroadCom NetXtreme Gigabitadapter and Windows 2003 Enterprise Edition.
I am at turning point with MS and SQLv6.5. I really need to make a decision for a large scale growth of a project. I was hoping people could respond for their experiences with SQL 7, with some rough load numbers ( concurrent connections, size DB, transactions/sec or day). Also is anyone using this product with MTS.
Production numbers only please. So many people have worked with the Beta and are doing testing, but it doesn't really count until it is in production.
Thought I would post here in case anybody can give some information.
Here is the background information:
I have 2 tables (stores and sales) from the Pubs database in Sql Server 2000 copied down to a SQL Server CE database. There is no foreign key/primary key relationship between the 2 tables in the CE database.
Here are the update queries that cause the error:
UPDATE st SET st.zip = 66668 FROM stores st INNER JOIN sales sa ON st.stor_id = sa.stor_id AND st.stor_id = 6380
Update stores SET stores.zip = 55555 FROM sales, stores WHERE stores.stor_id = 6380 AND stores.stor_id = sales.stor_id
Here is the error message that is generated when I run the query (Param 0 and Param 1 change according to what column and line the FROM clause is in):
Error: 0x80040e14 DB_E_ERRORSINCOMMAND Native Error: (25501) Description: There was an error parsing the query. [Token line number,Token line offset,,Token in error,,] Interface defining error: IID_ICommand Param. 0: 2 Param. 1: 1 Param. 2: 0 Param. 3: FROM Param. 4: Param. 5:
I ran the 2 queries in SQL Query Analyzer in SQL Server 2000 and they worked just fine. I also created 2 new tables (stores1 and sales1) in SQL Server 2000 using the Select Into clause. The new tables were created from the sales table and stores table in the Pubs database. The new tables had no foreign key/primary key relationship.
I ran the queries again in Query Analyzer against the new tables and the queries produced no errors.
Is it requirement that the OLE DB provider should have implemented IRowetChange interface so that it can be used to configure OLE DB destination?
Is there any way to configure a destnation for bulk and faster load? Normal OLE DB destination via IRowsetChange does load one row at a time ( InsertRow() ).
I recently got a new computer and reinstalled Visual Studio 2005, SQL, and all the goodies. When I build a SSIS project with multiple packages and run it, VS loads all of the packages in the project into the IDE before execution. None of the packages are related and I only want the current package to be loaded. I don't remember it working this way before. I've looked through the package, project, solution, and VS options and can't find anything that might control this behavior.
Hi everybody, I have a mirroring set up for around 30 databases in a 64 bit - 2 powerfull servers scenario - wih latest sp2 + hotfix. Mirroring works perfectly most of the times but when the principal is getting busy during the busy period of the day all of a sudden the processor on the mirror jumps to 100% and stays there for a long time - there are no live databases on the mirror server - all the databases are in mirrored mode. The principal server processor is less than 5% all the time. The network load between the principal and the mirror is also in normal limits. The principal server is dedicated to SQL but the mirror has other things too but the other processes use up maximum 15-20% of the CPU and when I stop SQL Server in one of those moments when the CPU jumps to 100% everything comes back to normal so it's clear that the issue is SQL on the mirror. Any ideas would be greatly appreciated.
