SSIS Lookup Task - Best Practice Concerning Cachetype Property
Oct 26, 2006
SSIS data flow transformation - Lookup task - best practice concerning Cachetype
I would like to know if there's any best practice concerning the CacheType property for the the Lookup task. Default value is "Full", but if the SSIS package is working with at lot of data, i.e. +10 mill. records from the OLE DB source to be handled through a variated numbers of data flow transformation tasks, it must have an impact memory usage if the lookup table also is a large table, i.e. +8 mill. records? When should I consider turning the property value to "none"
View 1 Replies
ADVERTISEMENT
May 25, 2006
HI, I use a lookup transform on one of my dataflow. My data look a bit like this (the actual data is more complicated, kit is a combination of type 1 - description and type 2 - code):
ID Code Description
--- -------- ----------------
1 AAA PRODUCT1
2 AAA PRODUCT2
3 AAA PRODUCT3
4 BBB PRODUCT4
The problem is simple: I would like to insert ID 1 and update it with subsequent rows that have the same code.
If the lookup transform finds a match, the row is updated, else, the row is inserted (using the error path of the lookup) via an OLE_DB command. The lookup cachetype is set to none. My problem is all rows are inserted. But if I use a second lookup that gives me the ID using the Code column, the second lookup sees the inserted data.
My question is why the second lookup is able to find out the inserted data while the first one cannot? The SCD wizard cannot resolve this either. I resolved this by using an asynchronous script component and manage the logic in there. But, still, a lookup with no cache (roud trip to the DB every time) should be able to do the job.
Thank you,
Ccote
View 1 Replies
View Related
May 15, 2007
Hi,
1. If I have millions of rows to be compared, then which cache type is prefereed for lookup, Partial or no caching?
2. If I have lookup connected to Oracle Oledb, cache type as partial and SqlCommandParam as following
select * from (SELECT ORDER_ID, OPER_KEY, STEP_KEY, SUM(OCCUR_COUNT) AS OCCUR_COUNT FROM SFWID_OPER_DESC_EXPLD GROUP BY ORDER_ID, OPER_KEY, STEP_KEY) refTable where refTable.ORDER_ID = ? and refTable.OPER_KEY = ? and refTable.STEP_KEY = ?
then it doesn't allow me to add the parameters from Advance tab of Lookup transformation edition, and raises following error
"Provider cannot derive parameter information and SetParametInfo has not been called"
what am I missing?
View 5 Replies
View Related
Mar 24, 2007
Can the WaitForMe Task property be used for building eventing based tasks (that is, "tasks with no defined execution lifecycle"), as Haselden's IS book terms it?
The SSIS run-time would appear to never interrogate this property.
I tried setting a breakpoint in a custom task on the WaitForMe
property. The breakpoint was never hit at run-time. Design-time only. It seems that the run-time ignores this property.
The documentation states that "On completion of any task, the runtime examines tasks that are still running in
the container, and if any of these tasks have WaitForMe set to
false, those tasks will be canceled." Well, the tasks with WaitForMe set to false are in fact not cancelled (e.g. System::CancelEvent is not signalled).
The intention of the WaitForMe Task property is to allow for event based tasks to not prevent a package (or other container) from finishing if the event never happens.
Tasks like the WMI Event Watcher, Message QueueTask set to receive, and third party tasks like file watchers, or service broker queue receivers would seem to fall in this category.
View 8 Replies
View Related
Aug 6, 2014
We have a LKP table that we will use to decode incoming codes from multiple sales feeds to link to the relevant surrogate key for the ultimate dimension table.
The LKP table has a start date, end date and active row flag to track history of the codes.
This table in its entirety is required for the FULL history load but we will need a CURRENT version of the table containing only the ACTIVE rows for the DAILY incremental loading.
I am thinking of putting a standard view over the top of the LKP table to present only the ACTIVE rows rather than creating a 2nd table and reloading it each night. We don't have enterprise edition to use materialised views.
My question is, will the LKP cache populate as the SSIS packages begins, loading it with all the rows from the view, as it would if I used the 2nd physical table with only ACTIVE rows?
View 3 Replies
View Related
Jan 22, 2008
Hello,
I am having a hard time setting the executable path for an Execute Process Task in SSIS. I have a variable that is initialized at package statup which holds the path to an executable in Windows. When I set the property "Executable" Path in an expression, I get a warning that the path for the executable is not set. One workaround was to try and initialize the variable with a bogus path with the hopes that the "correct" value will be written on run-time. NO LUCK. I still get the error and I cannot run the package until I put a static path.
Does anyone have a clue as to what is going on??
Mike
View 1 Replies
View Related
Jul 6, 2007
HI,
I need to open a File through File connection manager and want to assign these file properties to SSIS precreated varibale or Newly created varibale. I want to show file properties in Propertygrid. Properties grid will conatin File Propeties Column and SSIS varibale Combobox column. The combo box will contain New variable field. When user select New Variable field, then a new SSIS varibale window will open and we can able create New variable and that Newly created variable should add to that property comboBox.
For Instance if we create a new varibale Name "Creationdate" by clicking on New Varible in ComboBox, then that CreationDate variable should add to Property ComboBox in PropertyGrid. After adding when we select that variable Name "Creationdate" then that selected file Creation date should assign to SSIS varibale "Creationdate" field.
Any Comments or sample will help me.
Nitin
View 2 Replies
View Related
Apr 17, 2008
Hi,
I developed a simple custom control flow component which has several read/write properties and one readonly property (lets call it ROP) whichs Get method simple returns the value of a private variable (VAR as string). In the Execute method the VAR has a value assigened. When I put the value of ROP or VAR into MsgBox I can see the correct value. However when I execute the component I can not see the value of the ROP in the property window. I see the property but its value is empty string. For example when I put a breakpoint to postexecute or check the property before click OK in a MsgBox I would expect that the property value would be updated in SSIS as well. Is there a way how to display correct values of custom tasks properties in property window?
Thanks for any hints.
View 3 Replies
View Related
Jul 24, 2007
say i have a customer.aspx that allows a user to enter in customer data.
on customer.aspx, i have dropdownSalesRep which allows the user to associate a sales rep with the customerbut some customers come to directly, and not thru a sales rep, so I want the user to be able to specify "none"
Is it best to have a dummy record in my SalesReps table called "none" with an ID of say "999", or is there some other better way to deal with this?
View 5 Replies
View Related
Feb 2, 2006
Hi,
I am pulling data from FoxPro tables into SQL 2005, and want to only pull new or changed rows. Accordingly each table in Fox has a column LastChangedDateTime, indicating the last time the row was updated, and I have a table in SQL which has one row per Fox table, listing the table name and the most recent data pulled into SQL.
In 2000 DTS I would have pulled the SQL datetime value into a package variable, then used a parameterized SQL statement with ".. WHERE LastChangedDateTime > ? " to select the rows I require.
In SSIS this approach does not seem to be possible, and the options are that I either use a variable for the entire SQL statement or, as the first SSIS tutorial suggests, use a lookup against the SQL table.
Gut feel is that the lookup will perform slower than creating the variable SQL and executing that (given that the source table is 13 million rows and rising, and I only want the last 100,000 or so from today).
What is considered best practice under these circumstances?
Also is it possible to write SSIS scripts in C# rather than VB.NET, as the syntax differences are driving me mad? ;-)
Thanks in advance,
Richard R
View 3 Replies
View Related
Jul 23, 2005
What is the best way to setup relationships between one lookup tableand many other tables. The tables did not have any lookup tablerelationships which I am adding. One lookup table is used for same datain several different places.To use one lookup tables with several tables, I had to disable "CascadeUpdate" and only have "enforce relationships for updates and inserts"checked.Any pros/cons?Thanks in advance.P
View 1 Replies
View Related
Jun 27, 2007
Hi All,
Actually this is in regard to SCD Type 2 Dimension, Scenario is like that I am moving Fact table from some old source and I have dimensionA description value in fact which I want to replace with appropriate id from Dimension Table and that Dimension table is SCD Type 2 based on StartDate and EndDate and Fact Table doesn't contains direct date value rather there is timeId in Fact so to update the value in Fact table I have to Join Time Dimension table and other Dimension Table to replace fact Description with proper Id.
Lets assume DimensionA Structure
id
Description
StartDate
EndDate
Fact Table
id
measure1
measure2
TimeId
Description
Time Dimension
TimeId
Date
Day
Hour ...
View 1 Replies
View Related
Jul 10, 2007
I want to watch a directory 24 hours a day to launch SSIS packages as files arrive. There are several options I have found to accomplish this:
Use the WMI Event Watcher task in SSIS to launch the packages as files arrive. Leave the package with the WMI Event Watcher task running all day long everyday
Create a Windows service that uses WMI to detect file arrival and launch packages
Schedule SQL Server Agent to run the package when a WMI event is raised
What is the best practice? Are there better approaches that I have not listed?
View 15 Replies
View Related
Mar 25, 2008
Hi All,
I need to send out email when error occurs in the package. Is it a good practice to put the send email task in the event handler? Then MaximumErrorCount is set to 1. But for some reason, some time I saw more than one email are sent out. Please advise. Thanks
View 6 Replies
View Related
May 21, 2008
What is equivalent to dynamic property task of dts2000 in SSIS 2005;
Thanks
View 1 Replies
View Related
Jul 5, 2007
I wrote a custom task following the outline on MSDN. I signed it and installed it into the Tasks folder and in the GAC.
When I go to an SSIS project and add my task, the properties window shows "Could not get value for property 'd61935d9-430b-4c93-9f3e-a29f720d8659'. Specified cast is not valid." (where the guid is different obviously) for many of the properties.
What have I done wrong?
Update: I know this isn't my code because I tried a simple task that just returns success and doesn't do anything. I get the exact same errors, so I must be installing it incorrectly.
View 3 Replies
View Related
Dec 13, 2006
Hi,
We are planning for a DW solution based upon SQL Server 2005 EE. We will use a 64 bit machine for the Database engine and Analysis Services. Should we have seperate server for SSIS or should we use the same 64 bit box? If we go for a sepaerate server for SSIS should we use 32 bit or 64 bit ?
The primary datasources will be a new developed .NET application and a financial system based upon SQL Server 2000.
Every good advice/experiance is greatly appriciated!
Larra
View 1 Replies
View Related
Apr 15, 2005
I am importing data from xls file to a db table with a dts. In the time of the dts creation I am using 'Transform Data Task Properties' GUI window to map incoming xls fields (source) to the table columns (destination).
Question: Is there any way to invoke the 'Transform Data Task Property' GUI window in dts runtime and use it to change the mapping dynamically in the run time?
Thanks, Vadim.
View 3 Replies
View Related
Jan 9, 2007
Hi,
In a custom task I have a property of type Hashtable. This property gets changed by a custom UI. After changing the value and closing the UI, the Hashtable is properly filled during investigation in the validate() method.
Now saving the package and inspecting the XML file of the package, there is only this entry:
<DTS:ObjectData><InnerObject><PromotionCodes Type="12" Value=""/></InnerObject></DTS:ObjectData>
So it seems, that the value is empty and there is no further data to the object. Trying to reopen the saved package, the opening fails with something like (Had to translate it my own from german):
Error loading '<PackageName>' : Error loading a task. The contact information for the task is ''. This problem occurs on an error during task loading.
There should be a out of the box possibility to save collections or do you have to implement Load/Save XML your own if you have complex datatypes as properties? As the package has the element of the property included (but empty), it seems it tries to save it.
Anyone a hint. Anybody occuring the same behavior?
Thanks T.
View 2 Replies
View Related
Feb 22, 2008
Hi,
I've looked at so many examples on the web but I can't get my dataflow with lookup to work correctly. It just keeps inserting the records instead of updating them because they already exist. I know it's something simple but I've been trying to figure it out for the last 4 hours and am over my head. In the lookup I have 5 columns as available input and of course the matching lookup columns and selected 4 with lookup operation replace and tried add as new. It just keeps adding them to my destination table when they already exists. I can't figure out what i am doing wrong?
thanks,
View 8 Replies
View Related
Jun 6, 2006
Is there a way to set the Lookup task to be not case sensitive? For example, the lookup table has id 1, value 'ABCD'. The value that I'm using to lookup is 'abcd'. I cannot make it to return the id 1 unless I convert my lookup value to be all upper case.
And I don't want to use Fuzzy Lookup.
Thanks,
Ash
View 4 Replies
View Related
Apr 2, 2008
Hello,
I am new to SSIS. I need to transfer data from SQL Server 2005 Operational Database to SQL Server 2005 Report Database. The upload needs to work every night. There are few master tables and remaining are transactions tables.
I am planning to create 2 packages one for master tables and other for transactions tables.
Is it the good approach?
Also few of transaction tables are heavy in terms of number of records. Will it better if i further break them in many packages?
I am using book "Microsoft SQL Server 2005 Integration Services Step by Step".
Can you suggest any other good book?
Regards,
Shailendra Shelar
View 7 Replies
View Related
Apr 4, 2008
I've created a Union All data flow task which unions string fields from a couple of queries. I found that I needed to increase the length of the string field in the queries. Increasing the length causes an error in my Union All task. I can't seem to find a way to change the property of the fields in the Union All task. Most of the other data flow tasks have an advanced editor where I can change the length or type of the fields on the input and or output. I can't seem to find this for the Union All task.
Is there any way short of deleting the Output column and recreating it? (I like the order of my original output columns)
Thanks,
Bill Webster
View 6 Replies
View Related
May 2, 2008
Hi Folks,
I have a task I wrote which does not always update the property value (as seen in the properties pane)
Basically, change something on the form, then update the task host property with:
this.taskHostValue.Properties["Duration"].SetValue(this.taskHostValue, Convert.ToInt32(spnDuration.Value));
Stepping through this, it does exactly what it is supposed to. Having a look at the property value, it confirms it has changed.
Reopening the UI and resetting all the controls returns the expected results.
The package however does not realise it has changed. There is no * next to the package name in the top tabs.
As long as the package thinks it is unchanged, SaveXML does not get called either so the tasks do not persist.
Changing the value on the properties pane works fine though.
The frustrating thing is this is slightly random. Slight in the sense that sometimes it works but most of the time it does not.
The sample code I used was the MS download IncrementTask (Which works BTW) so I can't see it as being a VS / SSIS bug but rather something I am / am not doing. 3 tasks I have written all behave the same. I have to "nudge" them before savign the package.
Any ideas what the problem might be?
TIA
Cheers,
Crispin
View 3 Replies
View Related
Mar 18, 2008
hello guys,
I am having trouble with using disable property in the expression for data flow task. Here is the issue as explained below-
lets say i have 3 tables TableA, TableB, TableC from which i need to export data. So i create a table (TableList) where I save these table names and a unique id to these tables. e.g.
TableList will have-
TableName TableID
TableA 1
TableB 2
TableC 3
in the ssis package select these tableNames & Ids from tableList in Execute SQL Task. And assign the result set to a variable object (@TableList.
Then i use For Each Loop Container (For Each ADO Enumerator) , to loop through these tablesnames & iDs
Inside this loop container, i define three data flow tasks one for each table. So i have DataFlowTaskA (For TableA), DataFlowTaskB(For TableB), DataFlowTaskC (For TableC).
Now for a given table selected in the iteration, only the corresponding DataFlow Task should be exeuted. e.g. For the 1st iteration, if TableA is selected then only DataFlowTaskA should be executed and DataFlowTaskB& C should be skipped.
In order to achieve this, I am using a 3 variable @FlagA, @FlagB, @FlagC (type Boolean) one for each Table. and use the value of these flags for the "Disable" property of the data flow task (so @FlagA will be used for Disable property in the Expression for Data FlowTaskA, and so on..)
SotThe First Step inside the Loop, I use Script Task. (Input for the script task: read variable is @TableID and Read/Write varaibles are these 3 flags)
In this script task, I initialize these flags to true or false appropriately. So this is what i do
If (Dts.Variables("TableID").Value.ToString = "1") Then
Dts.Variables("@FlagA").Value = False
Else
Dts.Variables("@FlagA").Value = True
End If
If (Dts.Variables("TableID").Value.ToString = "2") Then
Dts.Variables("@FlagB").Value = False
Else
Dts.Variables("@FlagB").Value = True
End If
So in the 1st iteration, (if TableA comes) @FlagA=False and B&C will be True.
So the Disable property for DataFlowTask will be set false and for others it will be set to True. Thus, only DataFlowTaskA will be executed.
And this action should be repeated for each input table. this is the logic.
However only for the 1st iteration(say TableA is selected) it behaves as above. i.e. DataFlowTaskA is executed and DataFlowTaskB & C are skipped. But in the 2nd iteration(say TableB is selected) , it again executes DataFlowTaskA and doesnt exeute B & C (where it should have executed B & skipped A&C).
I do set daelay validation to true for all these but it still it doesnt working as expected. Even I checked the values for all the flags for each iteration and they seem to get the correct values. But somehow Diable propery in the expression not behaving as it should.
Am i missing anything. Do i need to set any other property to make this work.
I apprecite any help.
Thanks
View 3 Replies
View Related
Apr 3, 2006
Hi guys,
The default Engine Thread property of a data flow task is set to 5, is this the best setting? what if I would like to run complex data flow tasks on multi-processor machines, should I increase the engine thread? If so, then what is the recommended Engine Thread number for running complex data flow tasks in a multi processor system?
Even if i am running simple data flow tasks on a multi processor machine, should I change the engine thread?
Thanks!
Kervy
View 3 Replies
View Related
Mar 17, 2008
Hi All,
I have the following issue, i want to make dynamic the Argument property of the Execute Process Task but i couldn't able to do so, i have posted several times , hopefully i will get a solution today. I am using the following Argument to execute (winzip) program.
Argument: /e C:FTPFile1.zip C:FTPUnzipped
This Argument works but since File1.zip gets change every time, i mean it will be File2.zip, thus i have to change it manually on the Execute Process Task every time, however i want it to change by itself.
To make it clear, the /e mean winzip usage for extract file, the C:FTPFile1.zip is where the zipped file is, and the C:FTPUnzipped is where i dumped the unzipped files. as you see the file name will change every time like File1.zip, File2.zip, File3.zip, etc... Thus i want to make it daynamic, for example if i execute today File1.zip, then i have to execute tomorrow File2.zip automatically, and then File3.zip etc, that means the source file is changing every time, how do i make it then to be Dynamic.
Please help, i need your input badly.
Thank you in advance,
View 14 Replies
View Related
Mar 2, 2006
I have an SSIS package that unpivots data - each row from the source becomes 20-24 rows in the destination db. In the dataflow it looks up some type-data IDs before inserting into the destination table. The whole process flies through the data at an incredible rate, however I find that the final commit on data insertion takes a very long time, and two of the lookup tasks remain yellow throughout. It appears that there may be a contention issue because the final table has FKs to those lookup tables.
In T-SQL it is possible to do SELECT..... WITH (NOLOCK), but when I modified the SQL statement of the lookup task by adding this clause, it caused an error "Incorrect syntax near the keywork 'with'."
Is there any way around this problem? THNX!
PhilSky
View 2 Replies
View Related
Mar 7, 2007
I'm looking for a way to dynamically set the CacheType property of a Lookup transformation via a configuration variable. Is this possible? The CacheType does not appear to be a selectable property when choosing which properties to export to a configuration file. Can this property be manipulated through a script task inside the package?
View 10 Replies
View Related
Sep 19, 2007
Hi,
I am facing a very strange issue with the Send Mail Task:
I am fetching the recipient from a Recordset and storing it in a variable (MailList).
Now, I modify the Send Mail Task to include an Expression which updates the "ToLine" property with the email address in the "MailList" variable.
Moreover, I am not hard-coding the "To" property in the "Mail" tab of "Send Mail Task Editor" and leaving it blank. Now, it does not allow me to execute the package and gives me a "Package Validation Error" saying: No recipient is specified.
Any solutions?
Thanks in advance.
Regards,
B@ns
View 5 Replies
View Related
Apr 8, 2008
Hi, I'm using the lookup task to perform updates on some records and it seems to be working correctly but isn't making all updates. I have a case where there are 543 rows in my source connection, there are 436 of them that do not match the date column of the 543 in my destination table. It only updated 224 of them when it should have updated all 436? Does anyone know why it would only update this many instead of all? I also had another that only updated 18 out of 19 instead of all 19.
thanks,
View 4 Replies
View Related
Jul 17, 2006
This doesn't seem possible but I'll ask anyway...
Can I build a lookup task where the lookup query is based on a variable, rather than hardcoding the SQL staement?
View 3 Replies
View Related
Aug 2, 2005
I€™m trying to make a Lookup transformation with a relation on two string columns and a between dates condition set by parameter:
View 7 Replies
View Related
