Data Flow Is Locking Up

Jan 10, 2006

I have a situation where we periodically maintain an updated version of some data. I select the fresh data and match it on an alternate key to the existing data, determine if its a new record or changed record, perform some transformations, and then end with either adding (OLEDB Dest) or updating (OLEDB Command) the table.  It's basically like using a SlowlyChangingDimension but much faster.  Anyway, on one of the more complicated tables the data flow is getting locked up between the initial source select and the destination (because the table is referenced in both).  I've fiddled with the isolation levels and transactions and destination table lock but nothing will prevent the lockup.

Is there 1) a better way to structure a data flow when you are pulling data from a table you will ultimately be using as a destination to prevent this lockup  (without using the SCD) or 2) something that can be done to prevent the data flow lockup??

View 5 Replies


ADVERTISEMENT

Data Flow OLE DB Command And Table Locking

Jan 7, 2008



Hi all,

I'm just testing an SSIS package and am having issues with dealing with locked records.

my situation is as follows:

my source table is oracle, my destination table is in SQL server. my data flow is a very simple update with a lookup transformation and then two OLEDB commands for update and insert.

On each of the OLEDB commands I have set the "command timeout" to 5 seconds (just for testing purposes). also each OLE DB command has a failure path that outputs to a flat file. I'm expecting that if the destination table/records is/are locked then after 5 seconds the record will be output to the flat file.

so to test this I begin a transaction on the destination table and don't commit it. then I start the SSIS job. it doesn't appear to even get to the OLE DB commands. it appears to stop at the beginning of the data flow task. the output window shows this:

"Information: 0x40043007 at Import from Phoenix, DTS.Pipeline: Pre-Execute phase is beginning."

but it just hangs there indefinately. the progress tab tells me that it get's from the validating stage and past the prepare for execute stage but hangs on pre-execute - 0 percent.

I've put the command timeout = 5 on everything that I can find. I've mucked arround with all the possible "validateExternalMetadata" properties even though I only guessed that it may be the cause. is there anything that I'm missing? where should I look next?

(yes it does work perfectly when there is no transaction locking the target table)

Cheers,
Andrew

View 21 Replies View Related

Reuse Existing Data Flow Components In A Custom Data Flow Component

Aug 29, 2007

Hello,

Is it possible to use existing data flow components (Merge Join, aggregation,...) in a custom data flow component?

Thanks,

Yoann

View 15 Replies View Related

How To Pass Parameter Froon Control Flow To Data Flow

Feb 14, 2006

Hi, All,

I need to pass a parameter from control flow to data flow. The data flow will use this parameter to get data from a Oracle source.

I have an Execute SQL task in control flow to assign value to the Parameter, next step is a data flow which will need take a parameter in the SQL statement to query the Oracle source,

The SQL Looks like this:

select * from ccst_acctsys_account

where to_char(LAST_MODIFIED_DATE, 'YYYYMMDD') >?

THe problem is the OLE DB source Edit doesn€™t have anything for mapping parameter.

Thanks in Advance





View 2 Replies View Related

HELP: How Do I Pass Variables From Control Flow To Data Flow

Mar 9, 2007

I have an Execute SQL Task that returns a Full Rowset from a SQL Server table and assigns it to a variable objRecs. I connect that to a foreach container with an ADO enumerator using objRecs variable and Rows in first table mode. I defined variables and mapped them to the columns.

I tested this by placing a Script task inside the foreach container and displaying the variables in a messagebox.

Now, for each row, I want to write a record to an MS Access table and then update a column back in the original SQL Server table where I retreived data in the Execute SQL task (i have the primary key). If I drop a Data Flow Task inside my foreach container, how do I pass the variables as input to an OLE DB Destination on the Data Flow?

Also, how would I update the original source table where source.id = objRects.id?

Thank you for your assistance. I have spent the day trying to figure this out (and thought it would be simple), but I am just not getting SSIS. Sorry if this has been covered.

Thanks,

Steve

View 17 Replies View Related

Handle Tasks In Control Flow Tab From Data Flow Tab

Jan 17, 2008

Dear All!
My package has a Data Flow Task. In Data Flow Task, I use a Script Component and a OLE BD Destination to transform data from txt file to database.
Within Data Flow Task, I want to call File System Task to move file to a folder or any Task of "Control Flow" Tab. So, Does SSIS support this task? Please show me if any
Thanks

View 3 Replies View Related

SSIS Variables Between Data Flow And Control Flow... How To????

May 17, 2007

Hi everyone,

Primary platform is 64 bit cluster.

How to move information allocated in SSIS variables from Data Flow to Control Flow layers??

We've got a SSIS package which load a value into a variable inside a Data Flow. Going back to Control Flow how could we retrive that value again????

Thanks in advance and regards,

View 4 Replies View Related

Is There A Way To Set A Variable In A Data Flow From A SQL Statement (like In Control Flow)

Jan 12, 2006

I'm currently setting variables at the package level with an ExecuteSQL task.  This works fine.  However, I'm now starting to think about restartability midway through a package.  It would be nice to have the variable(s) needed in a data flow set within the data flow so that I only have to restart that task. 

Is there a way to do that using an SQL statement as the source of the value in a data flow? 

OR, when using checkpoints will it save variable settings so that they are available when the package is restarted?  This would make my issue a moot point.

View 2 Replies View Related

Please Advise: Big Control Flow Or Big Data Flow

Jul 22, 2007

Hi all! I recently started working with SSIS and one of the things that is puzzling me the most is what's the best way to go:



A small control flow, with large data flow tasks
A control flow with more, but smaller, data flow tasksAny help will be greatly appreciated.
Thanks,
Ricardo

View 7 Replies View Related

Lookup Task Data Flow Transformation Causes Data Flow Task To Hang?

Dec 28, 2007

Hi,
I'm trying to implement an incremental data pull (Oracle to SQL) based on Andy's blog:
http://sqlblog.com/blogs/andy_leonard/archive/2007/07/09/ssis-design-pattern-incremental-loads.aspx

My development machine is decent: 1.86 GHz, Intel core 2 CPU, 3 GB of RAM.
However it seems the data flow task gets hung whenever I test the package against the ~6 million row source, as can be seen from these screenshots. I have no memory limitations on the lookup transformation. After the rows have been cached nothing happens. Memory for the dtsdebug process hovers around 1.8 GB and it uses 1-6 percent of CPU resources continuously. I am not using fast load to insert new records into my sql target table. (I am right clicking Sequence Container 3 and executing this container NOT the entire package in the screenshots)

http://i248.photobucket.com/albums/gg168/boston_sql92/1.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/2.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/3.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/4.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/5.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/6.jpg


The same package works fine against a similar test table with 150k rows.
http://i248.photobucket.com/albums/gg168/boston_sql92/7.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/8.jpg

The weird thing is it only takes 24 minutes for a full refresh of the entire source table from Oracle to the SQL target table.
Any hints,advice would be appreciated.

View 18 Replies View Related

Sampling Data Set Via Integration Services Data Flow For Data Mining Models Without Saving Training And Test Data Set?

Nov 24, 2006

Hi, all here,

Thank you very much for your kind attention.

I am wondering if it is possible to use SSIS to sample data set to training set and test set directly to my data mining models without saving them somewhere as occupying too much space? Really need guidance for that.

Thank you very much in advance for any help.

With best regards,

Yours sincerely,

View 5 Replies View Related

Need To Save A Value From Data Flow A To Use In Data Flow B

Mar 20, 2007

Good morning, all,

I am working on importing an Excel workbook, saved as multiple CSV flat files, that has both group level data and related detail row on the same sheet. I have been able to import the group data into a table. As part of the Data Flow task, I want to be able to save the key value for the group, which I will use when I insert the detail rows.

My Data Flow has the following components: The flat file with the data, which goes to a derived column transformation to strip out extraneous dashes, which leads to the OLEDB Destination component.

I want to save the value as a package level variable, so that I can reference it in another dataflow.

Is this possible, and if so, at what point do I save the value?

Thanks,
Kathryn

View 1 Replies View Related

Extract Outlook Contacts Data(on Public Directory) Directly In Data Flow

Jun 13, 2006

Hi everyone,

I have to extract, dayly a list of contacts on a exchange server in a table on our EDW on sql server 2005. Is it possible to get the information directly from a dataflow or i will have to developpe a script task ?

Need help desperatly !!!

View 3 Replies View Related

Data Flow Stuck In Phase The Final Commit Data Insertion Has Started

Jun 19, 2007

Hello,



I have noticed that for one of my data-flows, the process is really long during the phase "the final commit data insertion has started".

To be accurate, the process is fast until it reaches this phase. It happens often when I load millions of lines.



The extraction is done from a database SQL Server 2005 to a database SQL Server 2005, on the same server (with the SQL Server native provider).

I used a SQL Server destination but I have tried with an OLE DB destination and it is the same situation.



Why the process could be so long during this phase?

There is a way to optimised my package to avoid that?



Any idea is welcome.



Thanks.

Guillaume

View 6 Replies View Related

Data Flow Task Error To Extract Data From Sql Server To Excel

Mar 28, 2008

Hi All,

I want to export data from SQL Server2005 to an Excel spreadsheet thru "Data Flow Task". I am using OLE DB for SQL Server for the source connection and a Connection To Excel as my destination source. The Excel spreadsheet (2003) exists and has the first row with column names. I don't have any warnings before trying to execute.

The SQL datable fileds are
i) ID - Int

ii) RefID
iii) txtRemarks - nvarchar(MAX)
iv) ddlWaterLevel - nvarchar(50)

While executing the tasks, I got the error
Error: 0xC0202025 at Data Flow Task, Excel Destination [427]: Cannot create an OLE DB accessor. Verify that the column metadata is valid.
Error: 0xC004701A at Data Flow Task, DTS.Pipeline: component "Excel Destination" (427) failed the pre-execute phase and returned error code 0xC0202025.


After analysing I found in the DataFlow --> Excel destination --> Advanced Editor for Excel Destination, the default data type for txtRemarks shows as "Unicode string [DT_WSTR]". But this is supposed to be "Unicode text stream [DT_NTEXT]". Even if I change the data type in the design time, It doesn't accept.

Please do help me out.

thanks
Sanra

View 4 Replies View Related

SQL 2012 :: SSIS Data Flow Items Tab Missing For Adding Data Source / Destination

Apr 3, 2014

I need to see inside a SSIS 2012 project a new SSIS installed component, but in the SSDT 2010 I cannot see the SSIS Data Flow Items tab for adding data source/data destination respect to the choose toolbox items pane.

View 4 Replies View Related

Do We Have To Alawys Use Slowly Changing Dimensions (SCD) Component In The Data Flow For The Loading Of Table Data?

Feb 28, 2008

Hi, all experts here,
Do we always have to use SCD component for the loading of data into data warehouse to handle changes of rows?
I am looking forward to hearing from you and thank you very much in advance for your help.
With best regards,

View 4 Replies View Related

How Do I Call A Stored Procedure To Insert Data In SQL Server In SSIS Data Flow Task

Jan 29, 2008



I need to call a stored procedure to insert data into a table in SQL Server from SSIS data flow task.
I am currently trying to use OLe Db Destination, but I am not sure how to map inputs to OLE DB Destination to my stored procedure insert.
Thanks

View 6 Replies View Related

Error Writing Data To Same Destination In Single Data Flow

May 12, 2006

I am getting the following error running a data flow that splits the input data into multiple streams and writes the results of each stream to the same destination table:

"This operation conflicts with another pending operation on this transaction. The operation failed."

The flow starts with a single source table with one row per student and multiple scores for that student. It does a few lookups and then splits the stream (using Multicast) in several layers, ultimately generating 25 destinations (one for each score to be recorded), all going to the same table (like a fact table). This all is running under a transaction at the package level, which is distributed to a separate machine.

Apparently, I cannot have all of these streams inserting data into the same table at one time. I don't understand why not. In an OLTP system, many transactions are inserting records into the same table at once. Why can't I do that within the same transaction?

I suppose I can use a UnionAll to join them back together before writing to a single destination, but that seems like an unnecessary waste and clutters the flow. Can anyone offer a different solution or a reason why this fails in the first place?

Thanks in advance.

View 3 Replies View Related

Adding A New Data Column (not Derived) Midway Thru A Data Flow

Jan 5, 2006

I need to know what a table's max row Identity is part way thru a data flow.  I can't get it at the beginning of the data flow.  I need to either (1) add it to the data buffer part way thru or (2) set it into a package variable and then reference the var in a script component.

I've not found a way to add a database column to the data buffer without doing a lookup for each row (too slow and not appropriate here) or some goofy oledb source and then merge join into the data buffer on a contrived join.

I've read questions about referencing package vars in scripts but I can't get that to work.  DTS.Variables("varname").Value isn't recognised when I code it up.

Anyone have an idea or solution for either one of these?  If you're gonna explain the script code, please include the entire snipet including the INCLUDEs, etc.

View 8 Replies View Related

Locking Of Data In Multiuser Environment

Apr 22, 2008



Hello,

We are using SQL Server Express 2005 for our application.
I have to lock the data in multi user access.
If user1 access the particular row in employee table, it should lock for the User1.
If user2 tries to access it should prompt the message that the User1 has accessing the particular row. User2 should be able to data in the read only mode. Untill user1 close that row, it should not allow the user2 to make any changes.
Help me how I can achieve this.

View 1 Replies View Related

Set The Data Source Of Data Flow From External Application (C#)

Jan 11, 2006

I am new to SSIS programming, so bear with me if my question seems naive to you gurus. I have a situation that needs to set the data source for a data flow from external .NET application ('external' means that the application will run on different process than the SSIS). I am trying to set the data source on which the data flow works from my C# application in a DataSet format. Ideal solution is not to save the DataSet to any file on harddisk (I know that will work, but has the overhead of writing, reading and managing the temp file). What I want to achive is that the business logic of picking data for SSIS Data Flow to process is controlled inside my C# application, the Data Flow just does what it does best - Transformation. Have any of you successfully done this before?. Thanks!

View 1 Replies View Related

Data Unexpectedly Changing In Data Flow Task

Oct 3, 2007

I have a fairly simple data flow task that loads data from one table (OLE DB Source) into another table (OLE DB Destination). The data type for one of the pairs of columns is nVarChar(120) and it contains version information that looks like a decimal. When I run the export, the destination has a trailing zero added after the decimal point as if it were a numeric column which invalidates our comparisons (string 1.0 is not the same as string 1.00). There is no cast or convert done to this column, it is a straight copy. Any ideas what could be causing this or how to fix?

View 6 Replies View Related

Data Flow: Converting Data In Multiple Columns

Sep 28, 2006

Hi,

I'm just wondering what's the best approach in Data Flow to convert the following input file format:

Date, Code1, Value1, Code2, Value2

1-Jan-2006, abc1, 20.00, xyz3, 35.00

2-Jan-2006, abc1, 30.00, xyz5, 6.30

into the following output format (to be loaded into a SQL DB):

Date, Code, Value

1-Jan-2006, abc1, 20.00

1-Jan-2006, xyz3, 35.00

2-Jan-2006, abc1, 30.00

2-Jan-2006, xyz5, 6.30

I'm quite new to SSIS, so, I would appreciate detailed steps if possible. Thanks.

View 3 Replies View Related

Exporting Data From A Merge Join From One Data Flow To Another

Mar 1, 2006

Hi,

Does anyone know if it is possible to point data that underwent the "merge join" transformation (in one data flow) to the following data flow? I don't want to recreate all that merging, sorting and calling the same sources again in the following data flow if the data that I am using exists in the previous data flow. The merged data is simply too big to export to an excel file, so does anyone have any ideas? Thanks!

View 8 Replies View Related

One Data Flow Task And Multiple Data Flows

Jul 26, 2007

I have a data flow task which has around 5 data flows (like the 2nd diagram shown here). These 5 simple flows with just a row count transformation in between. Now, I want to fail the entire task immediately even if one of the data flows failed. Right now if one flow fails the remaining flows fails after a long time, not immediately. How can I make it fails immediately.

The other I would like to do is Can I place these 5 data flows in a transaction, so that if one data flow fails, others data flows also roll backs? ( I assume its not possible)

Thanks

View 1 Replies View Related

Data Reader Source In Data Flow Problem

Jul 18, 2006

hi all,

i have a package in ssis that needs to deliver data from outside servers with odbc connection. i have desined the package with dataflow object that includes inside a datareader source. the data reader source connect via ado.net odbc connection to the ouside servers and makes a query like: select * from x where y=? and then i pass the data to my sql server. my question is like the following:

how do i config the datasource reader or the dataflow so it will recognize an input value to my above query? i.e for example:

select * from x where y=5 (5 is a global variable that i have inside the package). i did not see anywhere where can i do it.

please help,

tomer

View 11 Replies View Related

Data Access :: Transactions And Table Locking

Sep 3, 2015

Multi user ASP.Net website, SQL Server backend.When using transactions to insert multiple rows into a table and then commiting them once you're happy that everything looks good, the table(s) gets locked before commiting.Basically, I want to perform updates and inserts under a transaction but still allow other users to carry on using the site and potentially the tables that are being used with the transaction.

Question: Is there a way to stop the table lock occurring and for SQL to just do a record lock?, thus avoiding blocking other users?I know that a possibility is to use Snapshot replication, but am a little worried about turning it on due to the small team we have for testing.

View 10 Replies View Related

PROBLEM: Tables Are Locking Up When Attempting To Save Data

Jul 14, 2003

Over the weekend, one of our out-of-house programmers ran an update to our three main tables. I know these are kind of broadstrokes, but basically he compared the data and updated certain fields when it met certain conditions (lots of rules basically). The three tables are one-to-one and contain a little over a million records. The comparison file contained around 400k records.

The scripts made it through 250k records from the comparison file before he had to stop it for the weekend.

When I came in to test the data yesterday - I was met with problems on my front end application - it would lock up on the write back to the database. I went into EP and experienced the same thing after making any changes to a record, it would just lock up. This only appears to be a problem on the 2 bigger tables of the 3. I currently have 12 gigs or so free on that box and I have already shrunk the log and data files.

I tried removing and re-adding the indexes, but I am freezing up everytime I try to either change or delete the Clustered Index on the Primary Key. I don't know why, but I thought maybe that was my issue.

I know this is pretty broad, but even if someone could give me ideas as to why SQL would lock up like that when trying to just save the data, it would be most helpful.

NOTE: There were NO structure changes in the update process and my restored data from Friday works perfect.

If you need more info, just ask. Thanks in advance for the help.

Don
elitecobra2000@yahoo.com

View 14 Replies View Related

Pass Data Between Two Data Flow Tasks

Jul 16, 2007

Hi Guys,



I have yet another question. How can i pass data b/w 2 data flow tasks? I'm trying to get some data from one sql server and then I want to pass this whole bunch of data to another data flow task which is going to get some data from second sql server. I would probably combine them using union all and then save that as a transaction in a third sql server?

Information regarding how to pass the data with some detailed discussion would be fine with me.



thanks

Gemma

View 5 Replies View Related

Scrubbing Data In A Data Flow Task

Mar 1, 2007

I can only assume that I either did something very stupid, the examples are completely wrong, or I don't have the slightest clue what it is that I'm doing.

I'm trying to do one of those incredibly complicated things in SSIS that was a brain-dead, point click operation in DTS.  I want to strip characters out of an input stream from a flat file source before loading them into a table.  I've defined my data flow task.  I have my flat file source properly defined.  I have my SQL Server destination properly defined.  For the life of me, I can't figure out how to scrub the data that I want.

My input is a csv file that has 22 columns of data.  Column 22 contains data and if data was not present, there is a - inserted.  So, I want to strip any - from the input before it goes into the column.  So, I grabbed a derived column task and drug it out on the surface of the designer.  I then set the derived column name to Column 22, the derived column to 'Replace Column 22', and tried the following in the expression section:

REPLACE(Column 22, "-", "" )

REPLACE( (DT_WSTR, 50) Column 22, "-", "" )

Both of these threw some really nasty error messages when I clicked the OK button.  The really irritating part is that I exactly followed the examples in BOL.REPLACE(Product, "Bike","")REPLACE((DT_WSTR,8)DaysToManufacture,"6","5") I know I'm not dreaming this, but it absolutely refuses to accept my expression.

View 6 Replies View Related

Transfer Data Between Two Data Flow Task

Feb 1, 2007

Hi,

Can I transfer data between two dataflow.

Is it possible through anyway?

Thanks

Dharmbir

View 4 Replies View Related

Using An ODBC Data Source In A Data Flow

Oct 2, 2006

Okay, this should be really simple but I don't get it. How do I use an ODBC data source in an SSIS data flow task? When I look at Data Flow Sources I see the following options:

Pointer

DataReader Source

Excel Source

Flat File Source

OLE DB Source

Raw File Source

XML Source

Which one do I use if I need to get the data from a connection manager that is ODBC based? The IBM OLEDB driver for the AS400 doesn't work correctly so I HAVE to use an ODBC driver to connect to an AS400 data source.

Thanks in advance for any info.

View 1 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved