Could You Tell What's Wrong When I Split Table To The Target Partition Table?
Jan 22, 2007
Could you tell what's wrong when I split table to the target partition table?
USE TEST
--ADD FILEGROUP---------------------------------------------------------------------
ALTER DATABASE TEST ADD FILEGROUP FG_01
ALTER DATABASE TEST ADD FILEGROUP FG_02
ALTER DATABASE TEST ADD FILEGROUP FG_03
--ADD FILE--------------------------------------------------------------------------
ALTER DATABASE TEST ADD FILE (NAME = DF_01,
FILENAME = 'D:TESTDF_01.ndf',
SIZE = 10MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10MB)
TO FILEGROUP FG_01
ALTER DATABASE TEST ADD FILE (NAME = DF_02,
FILENAME = 'D:TESTDF_02.ndf',
SIZE = 10MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10MB)
TO FILEGROUP FG_02
ALTER DATABASE TEST ADD FILE (NAME = DF_03,
FILENAME = 'D:TESTDF_03.ndf',
SIZE = 10MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10MB)
TO FILEGROUP FG_03
--CREATE PARTITION FUNCTION---------------------------------------------------------
CREATE PARTITION FUNCTION PF_HIS_HTTP_LOG(datetime)
AS
RANGE LEFT FOR VALUES (
'20070101 23:59:59.997',
'20070102 23:59:59.997')
--CREATE PARTITION SCHEME-----------------------------------------------------------
CREATE PARTITION SCHEME PS_HIS_HTTP_LOG
AS
PARTITION PF_HIS_HTTP_LOG TO
( FG_01, FG_02, [PRIMARY])
--INSERT DATA,PARTITION 1 20070101-------------------------------------------------
DECLARE @i int
SET @i = 1
WHILE @i <= 100
BEGIN
INSERT INTO HIS_HTTP_LOG VALUES(CAST(@i AS varchar(32)),'192.168.1.1',5,'202.103.1.57',
6,'www.sohu.com',11,CONVERT" target="_blank">http://sina.com.cn','','www.sohu.com',11,CONVERT(datetime,'20070101 13:25:26.100',121),GETDATE())
SET @i = @i +1
END
--INSERT DATA ,PARTITION 2 20070102-------------------------------------------------
SET @i = 1
WHILE @i <= 200
BEGIN
INSERT INTO HIS_HTTP_LOG VALUES(CAST(@i AS varchar(32)),'192.168.1.1',5,'202.103.1.57',
6,'www.sohu.com',11,CONVERT" target="_blank">http://sina.com.cn','','www.sohu.com',11,CONVERT(datetime,'20070102 11:25:26.100',121),GETDATE())
SET @i = @i +1
END
--INSERT DATA TO TMP_HTTP_LOG 20070103---------------------------------------------
--DECLARE @i int
SET @i = 1
WHILE @i <= 400
BEGIN
INSERT INTO TMP_HTTP_LOG VALUES(CAST(@i AS varchar(32)),'192.168.1.1',5,'202.103.1.57', 6,'www.sohu.com',11,CONVERT" target="_blank">http://sina.com.cn','','www.sohu.com',11,CONVERT(datetime,'20070103 09:25:26.100',121),GETDATE())
SET @i = @i +1
END
--ADD CONSTRAINT--------------------------------------------------------------------
ALTER TABLE TMP_HTTP_LOG
WITH CHECK
ADD CONSTRAINT CK001
CHECK (STARTIME >= '20070103 00:00:00.000'
AND STARTIME <= '20070103 23:59:59.997')
--SPLIT RANGE ,SWITCH DATA----------------------------------------------------------
ALTER PARTITION SCHEME PS_HIS_HTTP_LOG NEXT USED FG_03
ALTER PARTITION FUNCTION PF_HIS_HTTP_LOG() SPLIT RANGE ('20070103 23:59:59.997')
ALTER TABLE TMP_HTTP_LOG SWITCH TO HIS_HTTP_LOG PARTITION 3
--==========================================ï¼?======================================
Why is the error in step of“ALTER TABLE TMP_HTTP_LOG SWITCH TO HIS_HTTP_LOG PARTITION 3�
error infomation:
message_id 4972,level 16,severity 1
ALTER TABLE SWITCH statement failed. Check constraints or partition function of source table 'TEST.dbo.TMP_HTTP_LOG' allows values that are not allowed by check constraints or partition function on target table 'TEST.dbo.HIS_HTTP_LOG'.
Please tell me why ? check constraints ?
Thank you very much !
Hi... I was hoping if someone could share me some thoughts with the issue that I am having at the moment.
Problem: When I run the package in my local machine and update local SS DB/table - new records writes OK in the table. BUT when I changed my destination meaning write record into another physical SS DB/table there is no INSERT data occurs. AND SO when I move/copy over that same package into another server (e.g. server that do not write record earlier) and run it locally IT WORKS fine too.
What I am trying to do is very simple - Add new records in a SS table using SSIS . I only care for new rows and not even changed rows. Here is my logic - 1. Create Ole DB source to RemoteSERVER - using SELECT stmt 2. I have LoopUp component that will look for NEW records - Directs all rows that don't find match and redirect rows (error output). 3. Since I don't care for any rows that is matched in my lookup - I do nothing or I trash the rows 4. I send the error rows (NEW rows) into OleDB destination
RESULTS when I run the package locally and destination table is also local - WORKS FINE; But when I run the package locally and destination table is in another Sserver (remote) - now rows is written.
The package is run thru BIDS manually so there is no sucurity restrictions attached to it.
I am not sure what I am missing. And I do not see error in my package either. It is not failing.
I’ve got a situation where the columns in a table we’re grabbing from a source database keep changing as we need more information from that database. As new columns are added to the source table, I would like to dynamically look for those new columns and add them to our local database’s schema if new ones exist. We’re dropping and creating our target db table each time right now based on a pre-defined known schema, but what we really want is to drop and recreate it based on a dynamic schema, and then import all of the records from the source table to ours.It looks like a starting point might be EXEC sp_columns_rowset 'tablename' and then creating some kind of dynamic SQL statement based on that. However, I'm hoping someone might have a resource that already handles this that they might be able to steer me towards.Sincerely, Bryan Ax
I want to update @Stop.UserField with thevalue from @UpdateSource where @UpdateSource.HasPathway=@Stop.UserField...but I need to use the @FieldDescription table to determine how to map the columns.
A customer wants to implement table partitioning on a replicated table.
They want to hold 13 months of data in the table and roll off the earliest/oldest month to an identical archive table. The table has a date field and partitioning by month makes sense all around.
So SWITCH PARTITION is the obvious solution to this, except for the fact that the table is replicated (transactional w/no subscriber updates).
What are his architectural or practical solutions to using table partitioning and replication?
We've got a source file which owns three different values: 'A','B','M'.
Where 'A' stands for "New Rows" and 'B' for "Delete rows" and 'M' for 'Update rows'
Using Conditional Split task we can redirect each subset into a OLEDB Destination but we are wondering how can we do the same using only one OLEDB? We've got only one table.
Hello,I am using SQL 2005 and Cognos' Data Manager. It is an ETL tool fordata warehousing.I have a problem with time it takes to load new changes, and I amseeking advice on a better way to manage the data.I have a table that tracks student attendance and it contains about 13million records. On a daily basis, there are 5,000 - 20,000 inserts and10,000 - 50,000 updates.The daily data comes for two different text files from my operationsystem; current and historical (CLSFIL and CLSHIS).The data is loaded into a staging area from the operational system,where data cleansing and other fields are added to the table.The final step is delivering the table to my target database, which isused for reporting.Heres the situation: I find it takes 45 minutes to do a relationalupdate, where only the records that changed in the last day will beloaded. However, if I choose the native API load instead of aRelational Load, it can load all 13M records in 7 minutes. The table isheavly indexedAt some point, the API load will take more time than the relationalload, (the changes and new records will remain a constant, but the filewill continue to grow).I'm seeking another solution is more efficient. I'm considering twotables for history and current and creating a view for reporting via aunion.This a good idea? How can I make the view effeicent to use the whereclause? Looking to bounce around ideas.Other Ideas?Thanks in AdvanceRob(I maintain the key relationships in the tool, not the tables. I knowI have lots to learn and improvments)CREATE TABLE "dbo"."F_BI_Class_Attendance_Detail"("CLASS_ATTENDANCE_ID" VARCHAR(50) NULL,"CLASSES_OFFERED_ID" VARCHAR(26) NULL,"CLASS_CAMPUS_ID" VARCHAR(10) NULL,"STUDENT_ID" CHAR(20) NULL,"FULL_CLASS_ID" CHAR(15) NOT NULL,"SESSION_ID" CHAR(10) NULL,"SECTION_ID" VARCHAR(5) NULL,"MEET_DT" DATETIME NULL,"MEETING" SMALLINT NULL,"PRESENT" CHAR(2) NOT NULL,"SESSION_SKEY" BIGINT NULL,"STUDENT_SKEY" BIGINT NULL,"CLASS_CAMPUS_SKEY" BIGINT NULL,"CLASSES_OFFERED_SKEY" BIGINT NULL,"LOAD_DT" DATETIME NULL,"COMPUTED_DT" DATETIME NULL);
I am using SQL 2005 SP2. I have a table partitioned on date range. I am trying to SWITCH, MERGE and SPLIT partitions. My SWITCH and MERGE work great. When the SPLIT query is executed, an error 9002 is thrown....
"The transaction log for database is full. To find out why space in log cannot be resued, see log_reuse_wait_desc column in sys.databases."
Below are more details...
- All SWITCH, MERGE and SPLIT are executed in one TRANSACTION. - After SWITCH and MERGE, I execute a query set the partition scheme "NEXT USED [PRIMARY]". - Finally i execute SPLIT statement.
I am trying to implement a sliding window on a table in SQL Server 2005 but i am having some problem. I have two tables, "Letture" and "LettureStorico". The first one receives data on a few seconds basis, some thousands of rows each day. The second is the historical record and should store all the records till midnigh of two days before, that is, if today is November 21st, LettureStorico stores rows till November 19th 23.59:59.997.
At some time during morning of each day i want to run a stored procedures that takes the records older than midnight of two days before in "Letture" and switch them as a partition in "LettureStorico"
Here's what i do:
/*----------------------------------------------------------*/ CREATE PARTITION FUNCTION [partizioneLive](datetime) AS RANGE LEFT FOR VALUES (N'2006-11-15 00:00:00')
CREATE PARTITION FUNCTION [partizioneStorico](datetime) AS RANGE LEFT FOR VALUES (N'2006-11-15 00:00:00')
CREATE PARTITION SCHEME [schemapartizioneLive] AS PARTITION [partizioneLive] ALL TO ([PRIMARY]) ALTER PARTITION SCHEME [schemapartizioneLive] NEXT USED [PRIMARY];/*(1)*/
CREATE PARTITION SCHEME [schemapartizioneStorico] AS PARTITION [partizioneStorico] ALL TO ([PRIMARY]) ALTER PARTITION SCHEME [schemapartizioneStorico] NEXT USED [PRIMARY]; /*(1)*/
CREATE TABLE [dbo].[Letture]( [IdLettura] [bigint] IDENTITY(1,1) NOT NULL, [IdTag] [int] NOT NULL, [IdGatewayBox] [int] NOT NULL, [IsEntrata] [bit] NOT NULL, [Data] [datetime] NOT NULL, [IsRettifica] [bit] NOT NULL, CONSTRAINT [PK_Letture] PRIMARY KEY CLUSTERED ( [Data], [IdLettura] ASC )WITH (IGNORE_DUP_KEY = OFF) ON schemaPartizioneLive(data) ) ON schemaPartizioneLive(data)
ALTER TABLE [dbo].[Letture] WITH CHECK ADD CONSTRAINT [CK_Letture] CHECK (([Data]>='20061115 00:00'))
CREATE TABLE [dbo].[LettureStorico]( [IdLettura] [bigint] IDENTITY(1,1) NOT NULL, [IdTag] [int] NOT NULL, [IdGatewayBox] [int] NOT NULL, [IsEntrata] [bit] NOT NULL, [Data] [datetime] NOT NULL, [IsRettifica] [bit] NOT NULL, CONSTRAINT [PK_LettureStorico] PRIMARY KEY CLUSTERED ( [Data], [IdLettura] ASC )WITH (IGNORE_DUP_KEY = OFF) ON schemaPartizioneStorico(data) ) ON schemaPartizioneStorico(data)
ALTER TABLE [dbo].[LettureStorico] WITH CHECK ADD CONSTRAINT [CK_LettureStorico] CHECK (([Data]<'20061115 00:00'))
Every morning i run a stored procedure that, after dropping the check constraints (i'll recreate the at the end), does the following:
/*-----------------------------------------------------------*/ SET @NewBoundary = dateadd(dd,-1, @dateOfToday)
--this new partition contains the rows i want to switch ALTER PARTITION FUNCTION PartizioneLive() SPLIT RANGE (@NewBoundary) --this new partition is empty ALTER PARTITION FUNCTION PartizioneStorico() SPLIT RANGE (@NewBoundary)
--this works fine, rows are moved ALTER TABLE Letture SWITCH PARTITION 2 TO LettureStorico PARTITION 2
--these two merges lead to two tables partitioned in two partitions each ALTER PARTITION FUNCTION PartizioneLive() MERGE RANGE (@OldBoundaryLive) ALTER PARTITION FUNCTION PartizioneStorico() MERGE RANGE (@OldBoundaryStorico)
Till now, everything is working as expected. Now, when i try to run the same Stored Procedure " a day later" (NewBoundary moved on 1 day) i receive, when i do the "ALTER PARTITION FUNCTION PartizioneLive() SPLIT RANGE (@NewBoundary)" i receive a 7707 error message: "Msg 7707, Level 16, State 1, Line 1 The associated partition function 'PartizioneLive' generates more partitions than there are file groups mentioned in the scheme 'schemapartizioneLive'."
How is this possible if i used the "ALL TO [PRIMARY]" and specified which file to use next as in (1) ? Why all this succeeds the first time (when i have 3 partitions) but not the second (again i have just three partitions, i checked) ?
I am using DTS and VBScript in DataPump tasks in order to transfer large amounts of data from text files to an SQL database.
As the database uses a normalized schema, there is often the case of inserting multiple records in a destination table from various fields of the same record of the source text file.
For example, if the source record contains information about goods sold like date, customer, item code, item name and total amount, and does so for a maximum of 3 goods per sale (row), therefore has the structure:
I have tried using a datapump task and VBScript, and I guess it has to do with the DTSTransformStat_**** constants, but none of those I used seems to work
Please help me how to do the Horizontal table partition?? I have to split the table in to multiple sub tables with same columns and less rows and then I have to use each sub table.
I want to replicate a database to a subscriber that will be used as a readonly copy. The data has to be replicated as close to instantly as possible.To do this I set up a database export of objects and data to populate thesubscriber, then I set up transactional replication. To verify thatreplication is working successfully, I count the rows in each table, thereare 3 tables in total. For one of the tables, the replication completes butalmost immediately afterward, the table starts to shrink, and after severalhours the record count is zero. This isn't happening to the other twotables, and I can't figure out why.If you have no idea what might be causing this, perhaps you can suggestsome places to start looking. This is Win2k SP4 with SQL 2000 SP3.Thanks much.
We need to pull from a table that is named tablename_mmddyy and populate a table with the same format tablename_mmddyy. The date will be different every month so I want to be able to build the tablenames every month. Is there a way to do this in SSIS? Thank you.
I having a bit of confuse here. Can you please help me?
I have about 5000 records all ready in oen table. Everything that I query is related to that table one way or the other. Now i having 2000 - 3000 more records to store in the database. In term of relation database then I can store the new data in a different table so I can can query it. Most of my queries are searching.
So the question is is this better to store the data in another table or should store everything in the old table? Thanks a lot in advance for your help. I really do appreciate that.
I'm encountering a very peculiar situation when I'm trying to compare source and target data using conditional split. Following is the Data Flow and how I'm trying to achieve this.
Source Data : Col_A (PK)Â Â Â Â Â Col_2 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 1Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 100 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 8 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 500 Target Data : Col_A (PK)Â Â Â Â Â Col_2 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 1Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 100 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 3 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 700 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 8Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 500 Look-up Target on Col_A to check for existing records. Now we have four columns in Look-up match output: Col_A, Col_B, Lkp_Col_A (Target Col), Lkp_Col_B (Target Col).
Conditional Split: Compare Col_B with Lkp_Col_B
Update target if there is any change in the existing value of Col_B.When I'm running the package for every record in source, the conditional split fails and even when there is no change in Col_B, some of the records (Not all and quite randomly) get updated with the same value. If I run the package for few records, it works absolutely fine.
1 HIS_HTTP_LOG a partition table2 REL_HTTP_LOG not a partition table,the same structure of HIS_HTTP_LOGï¼›3 When HIS_HTTP_LOG doesn't exist any index the following executed succeed ALTER PARTITION SCHEME PS_HIS_HTTP_LOG NEXT USED [FG_03] ALTER PARTITION FUNCTION PF_HIS_HTTP_LOG() SPLIT RANGE ('20070331 23:59:59.997') ALTER TABLE TMP_HTTP_LOG SWITCH TO HIS_HTTP_LOG PARTITION 3 4 However when I added the index in HIS_HTTP_LOG and execute the step 3,It made error: a) CREATE INDEX IDX_HIS_HTTP_LOG_001 ON HIS_HTTP_LOG(USERID)ON PS_HIS_HTTP_LOG (STARTIME) b) ALTER PARTITION SCHEME PS_HIS_HTTP_LOG NEXT USED [FG_03] ALTER PARTITION FUNCTION PF_HIS_HTTP_LOG() SPLIT RANGE ('20070331 23:59:59.997') ALTER TABLE TMP_HTTP_LOG SWITCH TO HIS_HTTP_LOG PARTITION 3 ========================= Error messages================================================"ALTER TABLE SWITCH statement failed. There is no identical index in source table 'TMP_HTTP_LOG SWITCH ' for the index 'IDX_HIS_HTTP_LOG_001' in target table 'HIS_HTTP_LOG' ." When I added index in REL_HTTP_LOG ,it gave me the same error message Could you tell me how can I solve the problem !
How do you alter the table to use the new partition (I know ALTER TABLE is in there but BOL doesn't give a valid example with the move option)? I can create the partition but I want to apply it to an existing table with no partition? Thanks
Hi all, My question is about Indexs on partition where I have a table with say 5 partitions and I want to create index on partitions and not on the whole table. The objective is that if i create a table level index on a partition table and eventually if I drop one of the partition or add another partition, what will happen to the index? 1) Do I need to re-create the index for the partion which are left after deleting one partition? 2) If a partition is added do I need re-create the index for the whole table or just create the index for that particular new partition?
Let me know if there is any white paper or code available. I have gone through the white paper published "SQL Server 2005" Partitioned Tables and Indexes Author: Kimberly L. Tripp, Founder, SQLskills.com
I would like to partition a table having only items posted in the last 24 hours in one filegroup and all others in another. Is there a way that my partition function could contain something like "col1 >= dateadd(d,-1,getdate())" and have items move to the secondary filegroup as they age pst 24 hours?
Control Flow Load Data Flow Task to Copy Data Flow Task to Scrip Task
Data Flow under Load Data Flow Task has Flat File Source to Row Count1 to OLE DB Destination (ODS database) Variable name for Row Count1 is RowCount
Data Flow under Copy Data Flow Task has OLE DB Source to RowCount2 to OLE DB Destination (WIMS database) Variable name for Row Count2 is RowNumber
Data Flow under Script Task is code:
' Microsoft SQL Server Integration Services Script Task ' Write scripts using Microsoft Visual Basic ' The ScriptMain class is the entry point of the Script Task.
' The execution engine calls this method when the task executes. ' To access the object model, use the Dts object. Connections, variables, events, ' and logging features are available as static members of the Dts class. ' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure. ' ' To open Code and Text Editor Help, press F1. ' To open Object Browser, press Ctrl+Alt+J.
Public Sub Main() Dim varMyRowCount As Variable = Dts.Variables("RowCount") Dim varMyRowNumber As Variable = Dts.Variables("RowNumber") Dim varPackageName As Variable = Dts.Variables("PackageName") Dim varStartTime As Variable = Dts.Variables("StartTime") Dim varInstanceID As Variable = Dts.Variables("ExecutionInstanceGUID") Dim varMailMsgtext As Variable = Dts.Variables("MailMsgText") Dim PackageDuration As Long Dim Filenum As Integer Dim FilNam As String
'<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ' Event log needs '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dim sSource As String Dim sLog As String Dim sEventMessage As String Dim sMachine As String '<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
We want to verify flat file count should be same as the data load to WIMS database. Now if I run this job I get the error message: Error: Failed to lock variable "RowCount" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
My co-work said even I get balance from RowCount1 and Rowcount2 still need to have query from table in Wims database because for SQL server sometimes process counts not same as the data results load in table so need select count(*) from table. How and where I can put at the job.
My design is row count from flat file put one variable then row count from final table put the other variable then send e-mail to me show both variable for testing. For real job I need if both variables same ok else send me an e-mail.
So I have been trying to get mySQL query to work for a large database that I have. I have (lets say) two tables Table_One and Table_Two. Table_One has three columns: Type, Animal and TestID and Table_Two has 2 columns Test_Name and Test_ID. Example with values is below:
In Table_One all types come under one column and the values of all Types (Mammal, Fish, Bird, Reptile) come under another column (Animals). Table_One and Two can be linked by Test_ID
I am trying to create a table such as shown below:
This should be my final table. The approach I am currently using is to make multiple instances of Table_One and using joins to form this final table. So the column Bird, Reptile, Mammal and Fish all come from a different copy of Table_one.
For e.g
Select Test_Name AS 'Test_Name', Table_Bird.Animal AS 'Birds', Table_Mammal.Animal AS 'Mammal', Table_Reptile.Animal AS 'Reptile, Table_Fish.Animal AS 'Fish' From Table_One
[Code] .....
The problem with this query is it only works when all entries for Birds, Mammals, Reptiles and Fish have some value. If one field is empty as for Test_Two or Test_Three, it doesn't return that record. I used Or instead of And in the WHERE clause but that didn't work as well.
I have a message that is validated using a schema.
If I send the message that is not compliant with the schema, the message is not enqueued on the target queue but it is send back to the Initiator queue and the message_body contains the description of the Error.
On the Initiator queue I have activated a SP that is I used to end the conversation and log the error.
One of the information that I need to log is the original XML message that causes the validation error so I can check it.
It seems that in the moment I'm on this SP, activated when there is something on the Initiator queue I have no memory of the original message or at least I'm not able to find it.