Bulk Insert In Foreach Loop Not Setting Source ConnectionString
Feb 22, 2008
Hello all,
I am running into an issue that seems to be collaborated here: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=188886&SiteID=1.
To summarize, I have a Foreach Loop that uses a Foreach File Enumerator. The loop writes the file names to a variable which is used in the Expressions property for a flat file's ConnectionString. The flat file is used as a source destination in a Bulk Insert Task that is inside the Foreach Loop. The flat file's connection string is not picking up the file name, resulting in this error: "The specified connection "test.txt" is either not valid, or points to an invalid object. To continue, specify a valid connection."
It doesn't work for a file connection manager or a flat file connection manager.
It does work if I replace the Bulk Insert Task with a Data Flow (flat file source -> ole db destination). It also works if I set the file's connection string manually in a script task.
It looks to me as though the Bulk Insert isn't calling whatever method in the connection manager that reevaluates the property expression. Am I missing something or is this a bug? I looked at the connect site, but couldn't find this particular scenario.
Hi Guys The problem is... When i try to bulk insert a single file its working fine. When i want to loop a set of files in a folder and use Foreach Loop and BulkInsert Task...its failing..
In the flat file connection When i specify usage type as existing file...its loading the same file "n" number of times where n is the number of files in the folder.
When i select usage type as existing folder i get error " Cannot bulk load because the file "....folder" could not be opened. Can someone help me out with this? I have sql 2005 as as separate instance...beside my 2000 which is default
I can't import from XML files using a foreach loop. I load an XML file with a generated XSD. When I map the file to the table it has no errors. If I now go back and change to a different XML file, I get an error:
"Error 1 Validation error. Data Flow Task: DTS.Pipeline: input column "COLUMNNAME" (129) has lineage ID 2115 that was not previously used in the Data Flow task. Package.dtsx 0 0"
This is for testing purposes. When I run the foreach loop it does not work. Ironically, I do the exact same thing in another foreach loop with a completely different XML and it works fine.
Help with Looping in a SSIS Package Scenario: We have a web app that lets our existing clients insert new locations into a table (Clients) in a SQL Server DB. This table has an Identity column as the PK. This table also has another ID field HBID that is the PK for another table (HLocation) in another Database system (Sybase). The HBID field is given a Default value of €˜1€™ during an insert operation in our (SQL Server) table, Clients. The Sybase database uses a Sequence table for inserting a PK into the table. Whenever our clients insert a new record in Clients (SQL Server), I need to generate a new HBID from the Sequence table in (Sybase), update the HBID in the Clients table (SQL Server) and then finally insert the record with the new HBID into table HLocation (Sybase). I have devised a SSIS package for this as following: Note all variables are scoped at the Package Level. Execute SQL Task #1 that drives a For Each Loop with the following properties: General: Connection Type: OLE DB Connection: SQL Server DB SQL SourceType: Direct Input SQLStatement: SELECT COID, HBID, Name, DateCreated FROM Client WHERE HBID = '1'
ResultSet: = Full Resultset
Result Set: Result Name = 0 VariableName = USER::rsClient Variable Type = Object
ForEachLoop with the follwing properties:
Collection: Foreach ADO Enumerator ADO Object Source Variable = USER::rsClient Enumeration Mode = Rows in first table Variable Mappings: Variable: Index: COID 0 HBID 1 Name 2 DateCreated 3
Inside of the ForEachLoop I have another Execute SQL Task to generate a new HBID from Sybase set up as following.
Execute SQL Task #2
Connection Type: OLE DB Connection: Sybase SQL SourceType: Direct Input SQLStatement: UPDATE Autoinc SET INC_LAST = (INC_LAST+1) WHERE INC_KEY ='HBID'; SELECT INC_LAST AS NewHBID FROM Autoinc WHERE INC_KEY ='HBID'
ResultSet = SingleRow
Result Set: Result Name = NewHBID VariableName = USER::NewHBID, VariableType = Int32
Also Inside the ForEachLoop is a Script Task that has all of my variables as ReadWrite = COID, HBID, Name, DateCreated and NewHBID. I concatenate the values in a string a pass the string into a Message Box to make sure they are looping correctly and they are.For example the results might look like the following:
12698, 1, John Doe Trucking, 10/1/2007, 14550 13974, 1, Joe Smith Trucking, 10/1/2007, 14551 10542, 1, Dave Jones Trucking, 10/1/2007, 14552 Etc.
The values 14550 -14552 are the new HBID being generated in the loop.
The problem is that when I try to Update the HBID in the Client table (SQL Server) with another Execute SQL Task I keep getting the same NewHBID number. In this case 14550 would be updated for every record instead of the next number in the loop.
I have set up Execute SQL Task #3 as follows:
General: Connection Type: OLE DB Connection: SQL Server DB SQL SourceType: Direct Input SQLStatement: UPDATE Client SET HBID = ? WHERE HBID = '1'(SELECT COID, HBID, Name, DateCreated FROM Client)
ResultSet: = Full ResultSet
Result Set: Result Name = 0 VariableName = USER::rsNewClient, VariableType = Object
Parameter Mapping: VariableName USER::NewHBID, Direction = INPUT, DataType = Long ParameterName = 0
I€™ve tried putting Execute SQL Task #3 inside of the ForEachLoop, connecting it to the output of the ForEachLoop. I€™ve tried setting up a dataflow with a Derived Column using the USER::NewHBID as the Expression.
I still get the same results, 14550 added to every row.
Can any one help or shed some light?
Any and all suggestion will be deeply appreciated!
I was trying to insert some row from one table to another of different database. I was using Execute SQL task along with Foreach loop container.
In my execute SQL task I am using this query
SET IDENTITY_INSERT dbo.Table1 ON
INSERT INTO dbo.Table1
SELECT * FROM DB2.dbo.Table2 WHERE TableKey = ?
When executed I get this error: failed with the following error: "Syntax error, permission violation, or other nonspecific error". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
While the same query when executed in Management Studio Its successful.
The properties I set
For Each Loop Editor Settings: 1) Collection: a) Enumerator Set to ForEach ADO Enumerator b) ADO Object Source Variable: User:bjectVariablename c) Checked Rows in the first table 2) Variable Mapping: New Int Variable2 and Index = 0 to set it to first colunm. 3) Expression: Left blank
Execute SQL Task Editor: 1) General: a) Timeout : 0 b) CodePage: 1252 c) Result Set: None d) SQLSourceType: Directinput e) SQL Statement: SET IDENTITY_INSERT dbo.Table1 ON INSERT INTO dbo.Table1 SELECT * FROM DB2.dbo.Table2 WHERE TableKey = ? f) BypassPrepare: False 2)Parameter Mapping: Variable Name : New Integer variable2 selected Direction: Input DataType: Long ParameterName: 0
Can somebody help me in this regards.
Reference: a) http://www.whiteknighttechnology.com/cs/blogs/brian_knight/archive/2006/03/03/126.aspx
OK, a new package, with a Foreach container enumerating CSV files in a directory.
I create the container pointing it at the directory and retrieving the fully qualified name, and create a variable (called 'CSVFiles') with a package scope, but no value.
Inside the container is a bulk insert task. The destination db/table is set, and the input flat file connection manager for the CSV files is defined with the connection string set to the variable created above.
As it iterates through the files, the variable is correctly set to the next file in the directory (I put a message box in the stream to display the file name/variable). It resembles 'C: empLocation1.csv'.
But when it gets to the bulk insert, I get this error message:
[Bulk Insert Task] Error: The specified connection "CSVFiles" is either not valid, or points to an invalid object. To continue, specify a valid connection.
What's going on here? Can I not use a bulk insert task in the container? Or some other parameter needs to be set?
How is foreach loop container - foreach ADO enumerator performace in SSIS package compares to use of cursors in stored procedures
Is there any articles comparing them
I understand a lot of factors can affect the performance, however what is expected performance for the foreach ADO enumerator loop for large dataset. What is Microsoft recommendation for that - recommended - not recommended (using large datasets - over million records)
I have no "Foreach File Enumerator" oprtion in the Enumerator Property of the Foreach Loop Component.
I have this enumerator in the c:Program FilesMicrosoft SQL Server90DTSForEachEnumerators folder.
Also I check it in the GAC - it does not here. I try to execute gacutil.exe -iF ForEachFileEnumerator.dll, but it is failed with "Failure adding assembly to the cache: The module was expected to contain an assembly manifest." Seems it is not managed enumerator.
Please help me.
Also information on how to regeister unmanaged enumerators are welcome!
I are using a BULK INSERT to insert the data from a ascii file to a sql table. The table has a ProductInstanceId column that exists in the tables but does not exist in the ascii DICast data. I am setting the ProductInstanceId to a Guid that will be used for Metrics. I would like to create the Guid in C++ and then set it somehow during the BULK INSERT DICastRaw1hr and DICastRaw6hr. I am calling the BULK INSERT from C++/ADO. I do not see how you can set a static data in the BULK INSERT for a column that exists in the table but does not the source data ... seems there should be a way to do this with the format file?
The other way to do this is with a TRIGGER. I have the TRIGGER below. Prior to the calling the BULK INSERT using ADO I will use ADO to ALTER the TRIGGER with the new Guid. When the BULK INSERT runs the ProductInstanceId will be populated with the new Guid.
ALTER TRIGGER DICastRaw1hrInsertGuid ON Alphanumericdata.dbo.DICastRaw1hr FOR INSERT AS UPDATE dbo.DICastRaw1hr SET ProductInstanceId = '4f9a44eb-092b-445b-a224-cc7cdd207092' WHERE modelrundatetime = (select max(modelrundatetime) from Alphanumericdata.dbo.DICastraw1hr(NOLOCK))
More Questions:
- The Trigger is slow. The Bulk Insert without the Trigger runs in about 10 sec ... with the Trigger in about 40 sec. I tried to use the sql code below in the TRigger but it was only doing the UPDATE on the last row. The TRIGGER must run after the BULK INSERT is complete. Now I am using the select (bad). Any comments ...
ALTER TRIGGER DICastRaw1hrInsertDate ON Alphanumericdata.dbo.DICastRaw1hr FOR INSERT AS DECLARE @ID as integer SELECT @ID = i.recordid from inserted i UPDATE dbo.DICastRaw1hr SET ProductInstanceId = '4f9a44eb-092b-445b-a224-cc7cdd207092' WHERE recordid = @ID
- I understand that I could set the Guid in the Default Value part of the table definition using the NEWID() function. I need the Guid to be the same for all the rows that are inserted during the BULK INSERT (all have the same modelrundatetime) ... how would I do this?
I'm importing a large csv file two different ways - one with Bulk Import Task and the other way with the Data Flow Task (flat file source -> OLE DB destination).
With the Bulk Import Task I'm putting all the csv rows in one column. With the Data Flow Task I'm mapping each csv value to it's own column in the SQL table.
I used two different flat file sources and got the following:
Hi: I have some query that takes quite a long time to process in the sql server and every time the page seems to time out. I wondor is there any Timeout setting that I can defined in the database ConnectionString in web.config file so that I can extend the "wait" time?
I need to execute about dozen packages from another package... how do I dynamically pass the dozen package names to the package and execute using foreach loop...?
idea is to store the names of packages in a text file and set the file connection property reading each package names from the text file... in this way I can just configure/edit the text file from time to time, the packages and the units that I want to execute...
I'm just learning SSIS and I've hit my first bump. I am doing a bulk import from a tab delimited text file to an empty sql table that has a Idendity column defined. How do I tell the bulk insert task to skip that column when inserting from the text file. If I remove the identity column it imports the data fine, but I want to create the indentity column in the table too.
The foreach loop below runs fine and it sends emails as expected but the SP does not update the Companies table. Any thoughts? How do I cause the SP to execute? PROCEDURE newdawn.EmailSentDate @CID intAS UPDATE Companies SET Companies.LastEmailDate = (GetDate()) WHERE tblCompanies.C_ID = @CID RETURN foreach (GridViewRow row in GridView3.Rows) { pstrTo = row.Cells[2].Text; CEmail = row.Cells[2].Text; CName = row.Cells[3].Text; CID = row.Cells[1].Text; CState = TextBox1.Text; CCity = row.Cells[5].Text; CCat = row.Cells[4].Text; CCID = row.Cells[0].Text; SqlConnection con = new SqlConnection(ConfigurationManager.ConnectionStrings["localhomeexpoConnectionString2"].ConnectionString); SqlCommand cmd = new SqlCommand("EmailSentDate", con); cmd.CommandType = CommandType.StoredProcedure; cmd.Parameters.AddWithValue("@CID", CCID); try { System.String mailServerName = "mail.domain.com"; REST of code sends email to email address found in each row ....it all works find but SP above does not fire.
Sorry that I open a new thread again but I didn€™t found any good tutorials about foreach loop container. I would like to read every row from one table and do something. I red something about sql task€¦ Can someone give a little instruction, please?
I using data flow task to import from flat file to database but i need to use Foreach Loop Container to import multiple files in specific folder and all will be insert in the same table
I'm having a problem where I'm using a Execute SQL Task to retrieve a dataset and storing that in an object variable. Then on success of that execute sql task I use a foreach loop task to go through the dataset and do 2 tasks inside the foreach loop. When I execute this package I have ~12 records in the dataset however when I get to the foreach loop in the 2nd iteration it keeps repeating it. It acts like it is stuck on the second record (tuple) and never goes on. I'm using an ForEach ADO Enumerator in the foreach. I've even set a breakpoint on each iteration and no task fails in side the foreach loop. I'm completely perplexed why it will iterate to the 2nd record but get stuck there in an endless loop. I've tried this on 2 different computers (with different data values) and the same thing happens. Does anyone have any suggestions?
I defined Foreach File Enumerator,there are more than 2 files needed to deal with,it donesn't work if i don't specify filename in flat file connection manager editor,however if i specify filename then doesn't foreach files
Trying to through a process together and using the "ForEach Loop" task. When configuring the task and using the ForEach Loop Editor I do not have the "ForEach File Enumerator" in the drop down. Anyone know whats going on or what I need to do to make this appear. I only see 5 selections: ADO, ADO.Net Schema Rowset, Variable, NodeList, and SMO Enumerators. I am also using the SQL 2005 eval with SP1.
Here is what I am attempting to get accomplished. I have an SSIS package that contains a Foreach loop container. This container executes a number of SQL tasks in order: SQL Task 1, SQL Task 2, SQL Task 3.
if the SQL task 1 succeeds it should flow on to SQL Task 2 and 3. This works fine when the SQL tasks do not fail...
In the event of any SQL Task failing control should flow to a send mail task to alert about the failure. Next the Foreach loop container should go to the next enumeration in the Foreach loop container and start the next new SQL task 1. So far I have been able to get the control to flow to the send mail task when a SQL Task fails. What does not work is when one SQL Task fails the entire Foreach loop fails and does not move to the next enumeration. It should only fail the package and move on.
I need to put a FTP task inside a Foreach Loop Container to upload data files to many different FTP servers. The container holds a variable with object data type that includes necessory FTP info (FTP server address, login, password and remote path). How do I configure FTP task so that I can pass the variables to it?
I€™m new to SSIS and have a question about foreach loop container. Basically, I have 4 data flows and each of them has a variable that need be pass on to the next step, each of the variables will call a same stored procedure. What should I use here? I guess I can use foreach loop container with the stored procedure inside of it, is that right? I tried all day yesterday and cannot make it work. The foreach loop container did not call the procedure at all. Could someone kindly tell me how to do it?
We discovered that the Foreach loop isn't recalculating the number of files found upon each new loop iteration, so if any new files are placed into the receive directory during the looping, they are not detected/processed. is this a bug or product intention?
Can anyone point me to an example of using a ForEach Loop Container to step through the rows in a recordset. I cannot seem to see how this is done from the help files.
I have a foreach loop that is processing files in a directory. One of the tasks is the execute sql task that calls a stored proc. If this stored proc returns a certain value then I need to exit the loop and not process any more files. But I have been unable to figure out how to make this happen.
I would like your help with the Foreach loop container. Boy, am I having issues with using it for looping through an ADO.NET dataset or what!?!? My control flow has a data flow task that is executing a Data Reader task (creating a .NET dataset for me). Now I go back to the control flow and add a Foreach loop container to loop through each record in my dataset. But which type of the Foreach loop container should I be using? I see an option to use Foreach ADO.NET schema enumerator but not sure how to configure it. Also I used the Foreach ADO option, but setting up variables to go through every column is an absolute pain in the you know what!!?!! I have about 200 columns and I want an easier way to refer to those columns in my transformation phase.
