Integration Services :: Implementing SCD Type 2 Using Merge - Having Both Changing And Historic Attribute
Sep 15, 2015
I am trying to implement Slowly Changing dimension transformation using Merge.Meaning both changing and historic attribute is in place. It seems we can use Update only once in Merge, in our scenario we have to update...When the historic attribute also have changed (To update the row as expired, IsCurrent=0)Also When changing attribute is changed. (Historic attribute is same). This case also we need to use Update. I am using CDC to do this. Updated OUTPUT is moving to a temporary table and using Execute SQL task to get updated.
When ever i tried to build a cube, i get stuck in this attribute relationship. either i shows a "yellow" icon in the hierarchy or "red" underline in the attribute column.I dont know how to rectify those errors.
what the meaning of the TextQualified attribute on a flat file connection is? I am importing delimited flat files where text columns are not qualified by quotes. If I use the Suggest Types button in the Flat File Connection Manager Editor, it sets TextQualified to True for all columns. Importing works fine than, but it also does with TextQualified = False.
I am loading from SQL Server 2008 to Access 2010 using SSIS. One of the columns in the table I am loading into is a Number datatype and Fieldsize is long integer. The values are being truncated, so I want to change the Fieldsize to DOUBLE. Â However, when I do that I receive the error below. What should I do? I would like not to change my Windows registry.
This error can be caused by one of the following:
The maximum number of columns allowed in a table or the maximum number of locks for a single file is exceeded.
The indexed property of a field is changed from Yes (Duplicates OK) to Yes (No Duplicates) when duplicate data exists in the table.
An expression is not specified in the Expression property of a calculated field.Â
If the maximum number of locks per file was exceeded, you can increase the number by editing a registry entry. However, this is not a recommended option.
If you use Registry Editor incorrectly, you could cause serious problems that require you to reinstall the operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor at your own risk.
Make a backup of the registry. Find the MaxLocksPerFile registry value by using the Windows Registry Editor, and then increase the value. The MaxLocksPerFile value is saved as part of the following key:
If the Indexed property of a field and duplicate data is located in the table, reset the Indexed property to the previous setting, or remove duplicate records from the table.
While I was loading to the same table a few days ago, I received a warning and the task took approx 9 hours. I am attaching the screen shot.
We've two OLE DB sources under DFT. TableA from one OLE DB source brings ID's as ( 1, 3, 5 ) and TableB from another OLE DB source brings ID's as ( 0, 3, 6 Â ). Now would I be able to use merge component to get all non-matching ID's from both tables A & B and store in the OLE DB destination as ( 0, 1, 5, 6 ) [ 1 & 5 from TabelA and 0 & 6 from TableB ]If no, what other option I've to make this req. doable?
I have a scenario where we have to handle dynamically changing source columns.
For example , some times in the source files the number of columns will be increased or decreased, new columns can be added in the middle or in the end of the source file.
I have a source table #source with columns 'source', 'patientcode' ,'patientdesc' Â and it has 4 records as below
source patientcode  patientdesc canada abc         patient1 canada efg         patient2 canada hij         patient3 canada klm        patient4
I have a target table and it has 2 records as below.
source prefix  tgt_patientcode  tgt_patientdesc canada cn     abc patient1 canada cn      efg patient2
Now, I want to merge the source data with target table -that means, if the records are already avaible in target, then ignore and if it does not available then INSERT.
This is the query i used but new records are not getting inserted.
MERGE #target T USING #source  S ON S.SOURCE=T.Source WHEN NOT MATCHED BY TARGET  THEN INSERT (  Source, Prefix ,tgt_patientcode ,tgt_patientdesc) VALUES ('Canada' , 'cn' , s.patientcode, s.patientcode);
I want the output as below
source prefix  tgt_patientcode  tgt_patientdesc canada  cn   abc          patient1 canada cn     efg          patient2 canada cn     hij        patient3 canada cn     klm patient4
DDL as below :
create table #target (source varchar(100),prefix varchar(2),tgt_patientcode varchar(100),tgt_patientdesc varchar(100)) insert into #target values ('canada','cn','abc','patient1') insert into #target values ('canada','cn','efg','patient2')
I am in between of creating a dynamic SSSIS package which will run for multiple zones having different source connection.My source is in Oracle.I am having 3 DFT with the 3 different source tables.I want to create a package with above DFT dynamically so that my single package can run for the entire zone with dynamically source connection change.I have created a Master table which stores the zone source connection string and zone name. I have 2 different connection.so if in future any new zones come so only newly zone details need to be add in master table without opening the package.
In the first image as can be seens i have 2 different data sources and then they are being joined using "Merge Inner Join". The "sort" is on BusinessEntityID column of Person table and "Sort1" is on "PersonID" of Customer table. The merge join of these 2 result in 19,119 rows.
On the other hand, if i use single data source and use a query with inner join on  tables  used in the first image (ie. 2 tables being used in 2 different data sources) as depicted in second image. Also,  since merge cannot operate without SortKey i have defined TerritoryID as sort key in the advanced editor. The number of rows i get after this is "10,274". My select query was :
SELECT P.BusinessEntityID, P.PersonType, P.Title, P.FirstName, P.MiddleName, P.LastName, P.Suffix, C.TerritoryID FROM stg.Person AS P INNER JOIN stg.Customer AS C ON C.CustomerID = P.BusinessEntityID ORDER BY C.TerritoryID;
According to me, it should have been the same as in first case i am using merge inner join and in second case i am using SELECT query with inner join. Upon drilling down i found that in the first case , my sort keys are BusinessEntityID  and PersonID, if i modify this to CustomerID  and BusinessEntityID as this is my join condition (in ithe inner join query shown above), i get the desired output. What i was wondering was, how  the sort order change the Join Condition?
I'm using a shared data source to connect an Oracle server in my packages. After changing the database user password in the shared data source, I noticed the package concerned would fail with the following description.
I build my SSIS package based on the above file.But now i receive files with different columns order let say
lastName,FirstNamr,Address l1,f1,a1 L2,f2,a2 or Address,FirstName,LastName a1,f1,l1 a2,f2,l2
every time i receive multiple files in different order and i have to remap all my mappings. These are just a few columns and i have like 20 columns and the order can potentially change any time. so every time i have build new packages remap them etc.
through normal c# code it pretty easy. I tried to add script here but the script also needs a source and mapping so there is also a mapping issue. Is there a better way to do this.
I am using SSIS in SQL Server Enterprise 2005. I have two OLE DB data sources from two disparate databases (IBM DB2 and Microsoft SQL Server), some columns from each of which are to be included in the merged output results. I have noted the various requirements in the forum postings with regard to sorting the OLE DB sources and specifying the output source columns as being sorted, as well as the requirement that the join fields in the two sources be close/exact matches. Yet, when I run this in VS, while the work area reflects the expected number of rows being input into the Merge Join transformation, no count is reflected as output from that transformation into the final destination table.Specifically, my two data sources (IBM DB2 and MS SQL) are configured as follows:
IBM DB2 contains an SQL statement that uses Cast operations to create the result columns.and an ORDER BY clause to ensure that the output is sorted by the desired two columns.. The OLE DB source property setting for IsSorted is set to true; the Output Columns folder column definitions for "key_ source_dtsy" and "key_source_dtrt" have their SortKeyPosition properties set to 1 and 2, respectively. Those field are both defined as data type DT_STR, with lengths of 4 and 2, respectively. Below is the Path metadata from the Data Flow Path editor from the path from this source:
IBM DB2 source"Name"Â "Data Type"Â "Precision"Â "Scale"Â "Length"Â "Code Page"Â "Sort Key Position"Â "Comparison Flags"Â "Source Component""ID_CODE"Â "DT_STR"Â "0"Â "0"Â "10"Â "1252"Â "0"Â ""Â "Source F0005 User Defined Codes""CODE_DESCR_1"Â "DT_STR"Â "0"Â "0"Â "30"Â "1252"Â "0"Â ""Â "Source F0005 User Defined Codes""CODE_DESCR_2"Â "DT_STR"Â "0"Â "0"Â "30"Â "1252"Â "0"Â ""Â "Source F0005 User Defined Codes""key_source_dtsy"Â "DT_STR"Â "0"Â "0"Â "4"Â "1252"Â "1"Â ""Â "Source F0005 User Defined Codes""key_source_dtrt"Â "DT_STR"Â "0"Â "0"Â "2"Â "1252"Â "2"Â ""Â "Source F0005
User Defined Codes:
MS SQL contains an SQL statement that takes the columns as they are in the MS SQL table (no Cast operations needed); it also uses an ORDER BY clause to ensure the output is sorted by the join columns. The OLE DB source property setting for IsSorted is set to true; the Output Columns folder columns for "key_source_dtsy" and "key_source_dtrt" have their SortKeyPosition properties set to 1 and 2, respectively. Those field are both defined as data type DT_STR, with lengths of 4 and 2, respectively. Below is the Path metadata from the Data Flow Path editor from the path from this source:
MS SQL source"Name"Â "Data Type"Â "Precision"Â "Scale"Â "Length"Â "Code Page"Â "Sort Key Position"Â "Comparison Flags"Â "Source Component""id_code_name"Â "DT_I2"Â "0"Â "0"Â "0"Â "0"Â "0"Â ""Â "Source CodeName in db dwVdFY""key_source_dtsy"Â "DT_STR"Â "0"Â "0"Â "4"Â "1252"Â "1"Â ""Â "Source CodeName in db dwVdFY""key_source_dtrt"Â "DT_STR"Â "0"Â "0"Â "2"Â "1252"Â "2"Â ""Â "Source CodeName in db dwVdFY"
The Merge Join transformation specifies an INNER JOIN using the columns named "key_source_dtsy" and "key_source_dtrt" from the respective data sources.I know there are alternative ways of accomplishing my intent (Lookup, port MS SQL table to IBM DB2 so join can occur in SELECT statement, etc.; however, I'd like to use this functionality and assume that it should work.Â
How to Implement scd type 1 and 2 functionality using DTS. Implemented the functionality using T-SQLS, but want to make use of DTS. Can we do this using DTS without using T-SQLs.
I have to perform several data checks before loading data into target table. For example I am having 1 flat file with below column
Id Name Age Int Varchar(100)Â Int
My requirement is to create  package, checks will be performed on each record, column of the files. Any records which failed the checks considered as error records and will be written to the exception table.
I've a text file which having a datetime column value like YYYY-MM-DD-HH.MM.SS.XXXXXX. I cannot convert this is to datetime format from text file using BCP utility.Â
Presence of hypen "-" between DD and HH, the SQL server does not accept this is as Datetime.
Is there any option to covert the date value in format file in BCP.
before running the Package , I want to change TYPE GUESS ROW=0 (if 8) in REGEDIT,So Update the TYPE GUESS ROW Iin Regedit by using SSIS in my Package as First Step So Which Task, I should use & Command line,
I have excel column with numeric and special character values , when I take that into SQL table using SSIS, the special character values enter as null value. the example column values are given bellow
1 2 2/1Â 1/2 1/2 means 1 or 2 ,
how can I read this values exactly into SQL table?
How do I pass a single column of values from a successful merge join to an EXECUTE SQL statement so it can be used with an "IN" criteria of the WHERE clause? Â Here's an example of my update statement with two random key values:
UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK IN ("XYZ123", "DEF890")
Is this even possible in SSIS, or am I better off using a loop and running the update EXECUTE SQL Statement for each individual key value, as in the following example?
UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK = "XYZ123" UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK = "DEF890"
SS2005 relational table with 20 fields in it, two key.
Flat file has 10 fields in it, the two keys, four I care about, and four I don't. The four I don't are fields in the table that get updated from another feed, and they duplicate that data.
If I set the four I don't care about as fixed attributes and the four I do care about as changing attributes, none of the changing attribute fields in the table get updated. If I set them all as changing attributes, the appropriate fields get updated. However, one of the fields is a "date added" field, and if I make that a changing dimension, that field will always get updated on every record no matter what because the package uses a GETDATE() to set that value.
I'm thinking about just replacing the SCD with a SQL statement, but then I'll have to import the file into a temp table to do the joined update, and I'd rather avoid that and use what SSIS can give me.
When you pass a complex type (the one represented by class) to a web service the BIDS UI allows you to enter values for every field of that type as constants. But what if you want to pass a variable? Once again the UI allows you to specify a variable for that complex type parameter. But how to make this variable in SSIS?I understand it should have the type of Object. But how to specify what the runtime type of this object is? And how to assign all fields to that object?
In my ssis 2012 package, I have a 'object' type variable with some table like records. I want to do some SQL operations like insert/update on the records in another table based on this 'Object' type variable records. Basically I want to use a MERGE statement with another physical table with the records in the 'Object' type variable.how to map/use the Object type variable in Execute sql task.I am not good in script task. How to utilize this Object variable in a Execute sql task? Â
As part of my package, i require a date (Only date, not DateTime) which is 10 months previous to get date.Eg: for today if the package executes, then i want 12/1/2014 , which i will use in my package as a filter like 'where date='?' where ? is a paramter which is is derived from the above logic
So, I have a project parameter @ppdate with value as  -10. I create a variable with DateTime (because there is NO date type for SSIS) and gives the expression as below
dateadd("Month",@ppdate, DATEADD("D",-(DAY(GETDATE()))+1,GETDATE())) , I am getting '7/1/2011 11:33:38 AM' which i don't want - i want only '12/1/2014'. How can i get it?
To get '12/01/2014', Â If i change the variable from DateTime to string, then i think i cant use the value in the filter condition like ''where date='?' because this does not accept string. Is this correct?
We have an issue in a SCD where a number of records may be presented that have changes to their attributes of EVERY type.
Example,
BusinessKey: xxxxxxxx
BuildingTypeId: 7
BusinessUnitHistoryId: 4019
BusinessUnitId: 4019
CurrencyId: 26
DevelopmentTypeId: 14
MarketId: 182
Name: abcdefgh
CurrencyId is a fixed attribute MarketId & BuildingTypeId and the BusinessUnitId & BusinessUnitHistoryIds are historical attributes Name is a changing attribute
The behaviour of the ETL seems to suggest that if fixed attribute changes are detected, these rows will error and therefore the changing & historical attributes will NOT be amended during the SCD transformation. Is this correct... as it seems to be what is happening.
Is there a way to change an image data type? I want to make a change to some deployed SQL 2008 SSIS deployed packages. I have a TSQL SELECT that searches the packages for a string. But I would like to be able to change a string. I have googled it but cannot find anything.
Probably not the right forum - pointers would be appreciated - but I'll give it a try anyway:
I'm in the process of designing a relational database to be used in a BI scenario - ie. dimension and fact tables. The data will eventually be used to feed cubes in Analysis services, however end users will probably be allowed to run reports aginst views of the relational database.
I'm currently looking at the employee dimensions and my first try would designate AGE as a SCD Type 2 attribute. As a result every employee gets at least one new record every year as AGE increases. Given that BIRTHDATE is specified should I drop AGE from the tables and recreate it as a computed attribute in database views and/or cubes?
persn_key  persn_id  address   is_active  updated_date 1         10       NYC     0       2015-11-04 14:19:54.817 2         10      Chicago    1      null
and Fact table like
fact_key  persn_key units_purchased 1         1       10
persn_key is the surrogate key between tables.
My question here is as the dimension has SCD type 2 on it and every time when there is a change the persn_key gets a new key value but the fact table still points to oldest key.how to update the surrogate key on fact table to the current key value? As per the requirement fact surrogate key must be pointing to current active record on the dimension.
I need to convert a a string column to integer. Before converting, I need to check if it has blank values then convert it to NULL. Someone told me that its easier to convert it to NULL before converting to integer.
I have a Slowly changing dimension that I am using to populate a dimension table. My problem is this when I run the package and any of the fields are marked as Historical Attributes it will add an additional row regardless of the fact that the incoming data and the data in the warehouse match exactly.
I've tried several things to fix this problem but so far none of them have worked. Some of the things I have tried that haven€™t worked are to match all the data types (which I have to do anyways) I've tried trimming the strings, I've also tried adding just one column
I am using a data conversion to convert them from varchar (the source datatype) to nvarchar(the warehouse datatype)
I'm at a dead end here and don't know where to go any help would be greatly appreciated.
Access 97 SR 2 Front EndSQL Server 7.00.1063 Back EndWhen I attempt to relink the tables in my Access app. I am suddenlygetting this weird error message = "ODBC--Call Failed", followed by"[Microsoft][ODBC SQL Server Driver]Restricted Data Type AttributeViolation(#0)". This problem did not manifest last time I tried a fewweeks ago, and so far as I know there have been no software changessuch as Service Packs etc.The error occurs when attempting to link a view. Here is the scriptfor the tables and the view siipl_DoesNotWork:==================================CREATE TABLE [dbo].[tblBOMEntries] ([fldID] [int] IDENTITY (1, 1) NOT NULL ,[fldCMMPartID] [int] NULL ,[fldSIPartNo] [varchar] (20) NULL ,[fldDescription] [varchar] (50) NULL ,[fldQtyAssy] [int] NULL ,[fldItemNo] [int] NULL ,[fldOnIPL] [bit] NOT NULL ,[fldRefDesig] [text] NULL ,[fldParentID] [int] NULL ,[fldBOMLevel] [int] NULL ,[sys_ts] [timestamp] NULL ,[fldObsDate] [varchar] (15) NULL ,[fldEffDate] [varchar] (15) NULL ,[fldItemRevNo] [int] NULL ,[fldModNo] [varchar] (10) NULL ,[fldNewBOMFlag] [smallint] NOT NULL) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]GOCREATE TABLE [dbo].[tblCMMIndex] ([fldCMMPartID] [int] IDENTITY (1, 1) NOT NULL ,[fldMainDescription] [varchar] (50) NULL ,[fldSIPartNo] [varchar] (50) NULL ,[fldBOMRevision] [varchar] (5) NULL ,[fldBOMAttached] [bit] NOT NULL ,[fldBOMDate] [datetime] NULL ,[fldCMMID] [int] NULL ,[sys_ts] [timestamp] NULL ,[fldLink] [int] NULL ,[fldNewBOMRevision] [varchar] (5) NULL ,[fldNewBOMAttached] [bit] NOT NULL ,[fldNewBOMDate] [datetime] NULL ,[fldSiteID] [int] NULL) ON [PRIMARY]GOSET QUOTED_IDENTIFIER OFFGOSET ANSI_NULLS ONGOsetuser N'siipl'GOCREATE VIEW siipl.DoesNotWorkASSELECT dbo.tblBOMEntries.*,dbo.tblCMMIndex.fldCMMPartID AS Expr1FROM dbo.tblCMMIndex INNER JOINdbo.tblBOMEntries ONdbo.tblCMMIndex.fldCMMPartID = dbo.tblBOMEntries.fldCMMPartID================================However, if I create a new view based on exactly the same SQL asDoesNotWork I don't get the same error message:CREATE VIEW dbo.DoesWorkASSELECT dbo.tblBOMEntries.*, dbo.tblCMMIndex.fldCMMPartID AS Expr1FROM dbo.tblCMMIndex INNER JOINdbo.tblBOMEntries ONdbo.tblCMMIndex.fldCMMPartID = dbo.tblBOMEntries.fldCMMPartIDThe only difference that I can deduce between siipl.DoesNotWork anddbo.DoesWork is the ownership. Can that be the problem? If so, why?TIAEdward
