Hello every one,
I have a simple dataflow from source to target.
My source is raw file and target is oracle table.
The problem I€™m facing in SSIS is its talking more than 10 minutes to load almost 30-40k records, I do not have any transformation activity this is just a simple dump from source to target.
Please do inform me of any setting to be talk care for fasten the process.
Thank you
Dear Friends,
I need to execute a SQL query, inside a dataflow (not in controlFlow) and need the records returned to continue the dataflow... In my case I cant use lookup and OLE DB COmmand and nothing else...
I need to execute a query and need the records for dataflow... with OLE DB command I cant see the fields returned... :-(
How can I do it? Using a script? Can I use a Script Component? That receive 2 parameters for input and give me the fields returned from query as output?
Thanks!!
found it
View 3 Replies View RelatedHey,
I have MS SQL database.
I have procedure:
code:--------------------------------------------------------------------------------
CREATE PROCEDURE dbo.Reg_DropTable
@ModuleId varchar(10)
AS
declare @TableName varchar, @kiek numeric
set @TableName = 'reg_'+@ModuleId
begin
DROP TABLE @TableName <- HERE I GOT ERROR
end
GO
--------------------------------------------------------------------------------
I got error when using variable with tables names.
How to do this?
Ps. Number is send to this function and it must drop table with name Reg_[That number]
Hi,
I'm wondering if it is possible to make an update of a specifics rows in a database table using dataflow tasks
thanks
When i use tablename or viewname variable in datasource component and data determination component ,
how can i manage the output columns and the input columns?
Yes,i can use default value init the columns ,but when variable value changed,error occurs.
Thanks a lot!
Hi Pals,
Here is my scenario in my ETL process, I have one DataFlow task.
Assuming that i have 10 clean records in my source database and i need to load all the 10 recs into my target table.
IS there any means of cross checking the no of rows from source table and number of rows loaded into my target table.
Any suggestions are greatly appreciated.
Thanks & Regards.
Hi all,
In my DataFlow i set the "OLEDB Source" which is a table in my Extract Server and need to do some transformations and stage the table which will be a Dimension in the staging DB,
Q1-Now i need only 3 columns from the Source table, which transformation do i need to use to just extract the the 3 columns?
Q2- Two Columns of 3,which i will need to transform as it is-no changes at all and One of the column which has values like "BOSTON...."
(I have a vague idea of what i need to do,need something solid suggestions/advices to kickoff,plan is to use this city column with a Replace function (as one of the forum member's Spirit1 adviced..thanks..!!))to take out the dots and then need to write a condition if BOSTON then Assign Code "BOS" which will be City_Code and this "City_Code" will have to be looked in City_Dimension to get the "City_Key_Number" for "Boston" and lastly the City_Code and City Key Number both have to be transformed to the destination Dimension.
Any ideas /suggestions will be appreciated.
Thanks in advance...!!
ravi
Greetings!
I am attempting to implement the following case statement BEFORE getting the data in to my destination table but I don't know how to create an expression for it.
In the mapping section of my OLE DB destination component I can only do mapping but I can't actually manipulate the data before it gets to the destination table.
What do I have to do to implement :
case
when SOPD.PRICE_TOP_NUMBER is NULL then -8
else SOPD.PRICE_TOP_NUMBER
end AS price_top_number,
before it goes to the destination column?!
Thanks for your help in advance.
I know the idea was to seperate workflow and dataflow, but I have come across a scenario where it would be useful for a branch of a dataflow to wait until another branch has finished.
I have some transactional data which records events for the start and end of a session. I want to build a list of unique sessions with the start and end date. I currently have the list of events sorted by time, followed by a conditional split for the start and end events. I can then insert all of the start events and would like to wait until all of the starts are inserted before updating them with their relevant end times.
Is this achievable?
Does anyone else think it would be a good idea to be able to set precendence across multiple branches of a data flow?
Does anyone have a better solution?
I know this is the wrong forum, but is there a way to model this against the transactional data in SSAS, I will move this question to the SSAS forum if anyone can think this would work!
Philip Coupar
This error seem to be very silly.did anyone come across this error.
I have been transferring data from textfile to a table using oledb destination.
The number of records in the text file are 2,091,650
Its was running just fine couple of days ago when the incoming data was little small then this...(arround 300,000).Now it seem to have a problem.
Here is the flow
1.File System task ->I copy the file to different location
2.Execute sql task->truncate tables
3.DataFlow task->I check for only the error files in this data flow.and all valid rows i transfer to a different text file.
4.Dataflow->filesource i connect to new text file created earlier.Here i convert fields to repective datatype and i insert if new or update record.
I dont know whats going on...
When i run my package it runs through the first three perfectly fine.When it comes to fourth step it sits there.....it dosent go to the tasks within this dataflow at all...and begining i have flat files source...
What could be the reason...
when i look at progress tab...i was able to look at the progress of other tasks but when it comes to this task it shows start>>>>>time and it sits there...
I am transfering data from a textfile to sql server.I use a data flow task for trasfering my text files.
Here is what i do.
1.Add text file source
What i want to achieve here is if the text file countains the column name in the first row i should delete them and if it does not contain column name in the first row just transfer it.
how can this be achieved???
2.add one more column to my text file which should contain the status(insert or update).
how can this be done??
3.before transfering data ot destination i want to know if the record exists if exists i just want to update it instead of insert.and if new record i want to insert it .and the status in the above new column need to change.
please help...
Hi all,
I have encountered a SQL 2005 deadlock issue while executing dataflow in a SSIS package. The deadlock happens when I have indexed two columns. If I don't have index, deadlock does not happen.
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
The above causes the rest of the dataflow execution to be terminated.
What my dataflow does is to extract data from 14 flat files and then insert the records into a single table (no primary key, but with two columns indexed).
Can anyone please advise how I can avoid deadlock with indexes in a table?
Thank you and much appreciated!
Greetings!
I am painfully learning SSIS and just when I think I have a better understanding of how things work and go down the path that I should take, I hit a wall. Frankly, discouraging and demoralizing.
I am performing a Select of rows that are then passed on to a ForEachLoop with an ADO Enumerator with a FullResultSet.
Variable mappings for the columns used are:
Variable IndexAccountNumber 0
AccountName 1
AccountAddress 2
CountryCode 3
A ScriptTask in the foreach loop allows me to confirm that I am processing rows.
A DataFlow in the ForEachLoop has in it a DERIVED COLUMN item that allows me to assign the four variable fields above to their respective DERIVED COLUMN.
An attempt to use a Flat File Destination to process the derived columns is being made. As I monitor the execution of the package I notice the following message.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "Flat File Destination" (382)" wrote 0 rows.
Can anyone PLEASE tell me why is it that I am able to see the data being processed through the script task? And yet not see any rows being written to the Flat file. Why?
Any information you may provide would be immensely appreciated.
Thank you and God Bless.
HI, I have a lookup that find a specific row. If that row does not exist, I need to create a new one. If it exists, I need to raise an error and trap it with the "on_error" event of the dataflow in order to log it. Is there a way to raise an error and specify the error message from the dataflow? I was thinking using the conditional split and verify if the value returned from the lookup (by set the error config to ignore the error) is not or not. But how can I raise an error when the value is not null?
Thank you,
Ccote
Hi:
I am getting the following error when I start debugging my Package, I am not sure what this is related to, but basically, input (datatype is a int, and its mapped to a column which is also int), so I am not sure whats happening here. The input column is actually a derived column, and its set as a 4 byte un-signed int, please advice on where should I start looking to troubleshoot this issue. This loanapplicationid is actually a user variable that is utilized by other tasks in my control flow as well:
Error: 0xC020901C at InsertApplicationCL5, OLE DB Destination [16]: There was an error with input column "LoanApplicationID" (1161) on input "OLE DB Destination Input" (29). The column status returned was: "The value violated the integrity constraints for the column.".
The point is, i want to calculate the max id of a table using Aggregate Transformation, then insert some rows with a OLEDB Command and finally , with another OLEDB Command select those rows with id >(max_id) calculated before.
How can i get a value that was calculated before? Can i store it in a variable?
Many Thanks!
Hello
Trying to figure out a clever solution for splitting multivalued columns out into n-columns.
For that I've build a custom component in SSIS using
ms-help://MS.VSCC.v80/MS.VSIPCC.v80/MS.SQLSVR.v9.en/dtsref9/html/4dc0f631-8fd6-4007-b573-ca67f58ca068.htm as an example.
I need to be able to add columns to the OutputCollection in designtime, but the designer returns an error:
Error at Data Flow Task [Uppercase [5910]]: The component "Uppercase" (5910)
does not allow setting output column datatype properties.
How do I enable the designer to accept designtime changes in the
columncollection?
Kind regards
Is it possible to set up at dataflow with for example an excel source that points to a sheet with varying number of colums.
In the file there is some information about number of columns. But the question is here how you use that information to define the columns in the dataflow ?
HI, I know that we can get package informations from msdb.dbo.sysdtspackages90 once packages are published into SQL Server. Is it possible to retreive informations for dataflows also from msdb tables?
Thank you,
Ccote
Hello all,
I got a text file with two columns. and I need to generate a integer key automatically with the row number (or any distinct number, I thought row number will be OK). and when I make the data flow task to import this text file into a raw file I need to get the unique rownumber as Id.
How can I make this in the data flow tak??
regards,
I was transfering more that 100,000 records from flat file to sql table
It took about 1 hour.Is this the way it is?????i used oledb command.
As the data passes by i got to insert to several table.Like i insert some of incoming data to one table then get the key from that table and insert rest of the data with the key field from previous table to another table.
In this case i felt OLedb would be best as we can use query.
I cannot use oledb destination as it has only error output(to insert some of incoming data and i want to have a look up to get the key but oledb des has only error output)
i cannot use sql destination as the database is sql server 2000.It dosent let me.
How can i increase the performance????Please let me know
I created a dataflow that transferred about 1 million records from a SQL database on one server to a differend SQL database on the same server. The processing took about 30 minutes. I used the Fast Load option.
I then created a "Execute SQL Task" and wrote a "SELECT * INTO TABLE" and this processing took about 30 - 60 seconds.
Can someone tell me why creating a Data Flow Tak would take so much longer or give differences between the two options above? Can someone give some pointers on how to make a Data Flow task more efficient?
Thanks.
I was wondering what has worked for all of you in regards to using a sproc as a source within a dataflow. I have had limited success doing this, but maybe there is a workaround I'm unaware of. Basically, using a SQL command in an OLE DB Source, I run an EXEC statement that returns a resultset from a stored procedure. I've noticed that depending on how the sproc is structured, I will either get metadata info in the columns tab of the OLE DB Source or not. Without this metadata of course I can not link it with a destination, since the destination believes that no data is being returned, even if this is not the case. This all seems to depend on the "final" select statement in the sproc being at the very top of the sproc. If it is not at the top, the columns tab will not be populated. Has anyone else had similar issues? Is there a workaround other than populating a temp table outside of the dataflow?
View 14 Replies View RelatedI am trying to recode a SQL2K DDQ into an SSIS Dataflow. I have no issue recoding it in SSIS except I am not sure that I am doing it correct way.
My DDQ has source,destination, update/delete/insert statements and few lookups. I have used activex script for the transformation because I have atleasst 10 If conditions. For every "if condition" the destination columns are popuated with different lookups/source columns and constant values.
Now When I start doing it with SSIS I have to use at least 10 Conditional split. and then at least one lookup,one OLEDBCommand and one OLEDB Destination for each of them . that brings my count of DF objects to 30-40
It makes my data flow to complex with two many objects. Earleir I could do whole of this in one sngle DDQ. It makes me think if I am doing it the correct way. should I be using Activex Script Task to these kind of activity.
Any advice would be appreciated.
Cheers,
siaj
I written a SSIS package to import a table from one database to another database. I used dataflow task with oledb source and oledb destination with fastload. For 2 million records its taking 5 min . The same import using DTS I am getting in 2 mins. DTS package is more faster than SSIS package ?. any reasons why SSIS is taking more time?
View 4 Replies View RelatedNeed help regarding ssis dataflow task
I need to create a ssis package. I want to import the data from a flat file to a table.
Lets say, the table has 5 columns -- col1, col2, col3, col4 , col5.(Assume that all columns can be NULLABLE) The datafile contains the data related to only three columns say col1, col2, col3. So when I use dataflow task to import the data from the file to the table, I will only get three columns, col1, col2, col3. Columns col4, col5 will be NULL.
However, I want to populate columns col4, col5 with some values which are stored in the variable.
IS there any way to do this??
Any help would be appreciated.
Thanks
Does anyone have a nice script that can "Propercase" the values of a column in the dataflow. I was thinking of a Transform Script Task, with a Input and Output Column, which can be transformed.
With Propercase i think:
STEVEN SPIELBERG becomming Steven Spielberg
Maybe also with some exceptions soo that "DELIVERY EU" stays "Delivery EU"
I have issue where based up a value in a column i need to do some processing of the previous and current row. The dataflow is also already sorted. I tried creating a Script Data Flow Transformation to do this but it isn't working right and the debugging of it sucks. Would anyone know of the best way to do this? or some helpful pointers? I tried "firing" information to help debug but doesn't help when the error message i get back is a stack overflow message.
An example of what I'm trying to do is process the sorted incoming rows for each person. Each person can have multiple rows. Based upon a "status" column in each row do some different processing on the previous or current row. Some Psuedo code:
if prev.PersonID = current.PersonID
if status = 1
change prev.PersonDate to today + 60 days
if status = 2
change current.PersonDate to prev.PersonDate
change prev.PersonDate to today + 1 day
else
send rows to output
Any comments or suggestions or helpful advice/critique would be MUCH appreciated!
I want to be able to loop through a view and execute a dataflow task for each record. I would like to pass the value of a column to the dataflow task to be used as a parameter in a data reader.
How can I do this?
I may have misunderstood how Lookupu works because it's not doing what I want.
From the OLTP datasource I have a long list of revenue items (from a SQL server database). I want to assign these to specific accounts as they are transferred into our accounting system. I have another table with a list of words to search for and which account they belong to.
For example if the OLTP source might be
Description - Amount
"Sales of cars"- "$20,000"
"Motorcycle sales" - "$15,000"
"Bike rentals" - "$2,000"
The account lookup table is like
Wordsearch - Account
"sale" - "ACCT_SAL"
"rental" - "ACCT_RENT"
So by looking up whether "wordsearch" is found in "desription" I should get an output of
ACCT_SAL - $20,000
ACCT_SAL - $15,000
ACCT_RENT - $2,000
Back in DTS I did this with an array and "If Instr" using VBScript in the Data Transformation Task. I'm sure there must be something in SSIS to do this - it should be something like a Fuzzy Lookup ?, but I'm drifting toward Script Component. Anyone got any ideas for SSIS
When I drop a new component onto the design surface, it appears with dotted lines around it, as it is selected.
But, the F2 key (the hotkey for rename) does not work.
I have to click on some other component, and then back on the new component, and then the F2 key works.
I remember reporting this bug back in the beta cycle, but it is still present even in the release -- I actually think Ã?'m using the SP1 version of 2005, but see version info below to be sure:
Remote terminal services broke my copy buffer again, as is so often does, so pasting in the version info failed -- falling back to manual typing -- SSIS 9.00.2047.00.
It just occurs to me that it might not be a bad idea to post again the problems in SSIS that I found back in the beta cycle which do not seem to have been fixed, to increase the chance that someone will notice them.
(During the beta cycle, I spent hours trying to get bugs posted into the beta bug system, but the beta web bug site had so many problems that I could never get it to work, unfortunately, so I fell back to just posting my bugs and hoping some developers would notice them. Kirk had a thread for bugs and RFEs at one point, where I posted some of the important ones.)
Here is another fairly bad one:
The dialog that pops up to show fields in the dataflow which have been orphaned, after some fields have been removed earlier in the dataflow, and which allows the user to rename them or delete them, has a very bad display problem:
It shows the box names before the field names, so if the box names are descriptive, the field names are far off-screeen, and it is very difficult to get them to display on screen.
So the user starts off with all the items in the drop-down being indistinguishable, as they all start with the box names, and the important part of the strings are far off-screen at the right -- the actual field names.
I don't recall if I actually ever posted this one; I may not have.