Multi-threading And SSIS Package
Apr 11, 2008
I have an SSIS package that is running the same set of stored procedures on about 50 remote database servers and importing the data through data flow tasks into a local database.
To do this I have a db table set up with the remote server and db names and using variables, i am looping through each one and getting the data. Everything works wonderfully...however, I need to find a way to be able to run multiple calls at the same time. Is it possible to multithread SSIS so that I can run 1 server retrieval on 1 thread and another retrieval on another thread? Any other suggestions on how to optimally do this?
View 4 Replies
ADVERTISEMENT
Aug 23, 2006
I have a vb 2005 app that executes a SSIS package in threads.
The SSIS package imports a large (20mb) file, does many alternation to it, exports it to the database.
The SSIS execution is quite frequent and there for multi-threading is needed.
Here is the issue:
If i take out the threading everything works great. The second that i add threading, and try to execute a few instances of the SSIS it starts crashing. The thing is that it errors out in different task and is completely unpredictable.
My question:
1) Can SSIS be used in a multi-threading environment?
2) If yes, How do i do it?
3) Does anyone have any suggestion as to what i should try?
p.s. Timing is definitely an issue for me. I got to get this working, tested, approved by 8/27/2006
---------
here are the 3 errors that i'm getting when i turn multi-threading on
1)Thread "WorkThread0" has exited with error code 0xC0202009.
2)An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Transaction (Process ID 112) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
3)The ProcessInput method on component "(i cannot display the name of the task for security reasons" (448) failed with error code 0xC0202009. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
View 5 Replies
View Related
Jun 22, 2006
Apologies if this is not the right forum.
We have some c~ code which calls a stored proc to trnsfer data from one db to another.
We are using multi threading to allow a whole days data to be processed in parrallel.
However, we are experiencing problems. After a while we get the following message:
"An exception during changed data captured caused all processing threads to abort".
Can anyone tell me what posssible reasons would cause this to happen?
It is not an area in which I have a lot of expertise, the code was written by a third party.
Many thanks
View 1 Replies
View Related
Nov 8, 2007
We're moving in a 4-socket (Xeon Dual Core CPU's) server. What version and product of SQL Server should I be looking for to fully take advantage of the new server's 4 processors. I'm not sure I understand the licensing per CPU. Is there any advantage in waiting for SQL Server 2008 as it concerns multi-threading and what should I be looking to spend for a SQL server solution that utilizes all four processors. Someone has told me that the SQL Server standard edition comes with 4 cpu licenses. Is this correct?
Thanks, Rob Huylo
Rob21379@aol.com
View 11 Replies
View Related
Jan 28, 2004
hi,
I want to execute two user created stored procedures in a multithreaded manner in ms-sql server . Can some tell me how i can do that.
Thanks in advance.
Regards,
Manpreet
View 14 Replies
View Related
Nov 23, 2005
Hi,I have SQL SERVER 2000 SP4 Enterprise , Windows 2003Enterprise onXeon 4 Processors (now with multi-threading CPU) and I havequestions whichseem weirds to me (used to have the same config withoutMulti-Threading) as following:1. SQL Server see 8 CPUs. Is this because ofmulti-threading CPUs ?2. when use SP_WHO2 ACTIVE ,2.1 Result has user SPID , which blockitself(it is UPDATE and SELECTstatements, most often)2.2 Result has user SPID , show morethan one result rows withthe same processe and command3. when use Properties of SQL Server , it has anerror 15407 in log.many thanks in advanceNiponW
View 2 Replies
View Related
Aug 29, 2007
I have a question:
Is SQL Compact Edition safe to use in ASP.NET WebApps?
And second if so, should i use transation isolation level or row locking hints to assure data quality during iniserts?
E.g. i want to assure that there is only one record named user2@domain2 and thus i want to make sure that my code
will not fail in a mutli-theaded scenario and insert second one or get a key exception back: see sample
transaction or locking hit{
select from users ....
if no user found
insert into users
}
Thanks
Tim
View 5 Replies
View Related
Aug 30, 2006
Hey, I've a few jobs which call SSIS packages. If I run the SSIS package, it runs fine but if I try to run the job which calls this package, it fails. Can someone help me troubleshoot this issue? None of my jobs that call an SSIS package work. All of them fail.
Thank you
Tej
View 7 Replies
View Related
Jun 3, 2002
I have some threading questions, please explain how these work. Thanks!!
How do the threads get created (programmatically, network, or through sequel)?
If I have 10 people in an application, is a thread created for each user?
If I run a large process can I subdivide the process and run it through mulitple threads?
View 1 Replies
View Related
Jan 10, 2007
Hi,
In our project we have two SSIS package.
And there is a task (Execute SSIS package) in First package that calls the execution of second package.
I m continuously receiving an error "Failed to decrypt protected XML node "PackagePassword" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available."
As we are running first package by job, job runs successfully logging above error
The protection level of second package is set to "EncryptSensitiveWithUserKey"
Can anybody please suggest how to handle it?
View 4 Replies
View Related
Sep 29, 2005
Ello,
We have quite a few SQL Servers running on servers with hyper threaded enabled processors. We have always left hyper threading on, even on our very busy servers but i've recently begun to question this due to comments i have received.
All our servers run SQL 2000 SP3a as "support" for hyper threading was added in SP3a
So... What do you think of Hyper Threading any why?
View 5 Replies
View Related
Aug 9, 2006
Over some time now, I've been developing a fairly hefty stored procedure, that does a lot of computations, and fairly few table lookups.
When I look at the performance on my server (a dual Xeon HT) I can see that it only uses 1 out of 4 possible "cpus" to work on the calculations, while the three others idle out, and was wondering if I can somehow force it to use max available CPU power?
View 2 Replies
View Related
Nov 1, 2006
I have figured out that SSIS installs once for as many instances of SQL Server 2005 as there are on the machine. I also figured out that even if there is only a named instance on the machine, it will always install under the "Default" machine named instance.
I also figured out that you can change the instance viewed by modifying C:Program FilesMicrosoft SQL Server90DTSBinnMsDtsSrvr.ini.xml
What I am unclear about is that in SQL Server 2000, each Instance had an MSDB db which stored all of the jobs. In SQL Server 2005, each instance still has an MSDB DB but theSSIS also has a MSDB DB ?? or is this pointing to the MSDB for the instance specified in the xml file???
There is not much info out on this at all and would be good if the MSDB help explained this install once so it would all be clear!
THanks for any help... thank goodness for newsgroups!
View 6 Replies
View Related
Apr 29, 2008
All,
We are experiencing some problems when running the same package on multiple threads of a C# application concurrently. The package instances seem to share a Dts.Variables collection.
I have created a test application and package to reproduce the issue. The package is very simple. It has two variables - Instance (Int32) and Content (string). The test application runs two concurrent threads that run through 25 iterations running the package. Each thread loads the package, sets the instance, and then runs the package. The package then takes the instance, sticks it in the Content variable and then writes it to a file called "FileX" where X is the Instance. Thread 1 sets the instance from 1 to 25, thread 2 sets it from 26 to 50. The test execution results in about 15% of the files containing the incorrect "instance content". That is, I'll see "42" in the file named "File14", or something along those lines. This tells me that the variables are being overwritten.
I ran another test where I created a copy of the package, so the code is identical, but the VersionGUID is different. Running the thread test application always succeeds. There is no file name/content mismatch. I then ran another test where I manually edit the XML of the package to change the VersionGUID before the package is loaded (by calling LoadFromXml). Again this never fails.
Changing the VersionGUID may work for us, but it is a total hack. Has anyone else experienced this or have any other solutions? Any thoughts on manually updating the VersionGUID to get around this problem? I hate doing this, but we are getting desperate.
Thanks in advance,
Rob
View 2 Replies
View Related
Aug 7, 2006
I have two SSIS packages that I want to run in one SQL Agent job as two individual steps. The two packages run fine when they are in separate jobs. However, when I run the job conaining both SSIS packages (under the same proxy), the first SSIS package starts, but hangs in the middle.
I then tried setting the DelayValidation flag to True as suggested for a similar issue in another thread from this forum. After changing the DelayValidation flag to True for all containers and tasks on the second SSIS package, the first SSIS package ran completely through sucessfully, but the job continued executing for hours and the second SSIS package never started. I finally killed the job.
Any ideas as to what is the problem here? I have logged to the event viewer and see that the first package completes sucessfully. They run successfully in separate jobs, but I can not get them to run together within the same job without hanging.
Any help is appreciated,
Paulette
View 5 Replies
View Related
Mar 6, 2008
I have a SSIS job, one of the last steps it performs is to execute a SQL 2000 DTS package. This has to be done as a SQL 2000 DTS package as it is performing rebuilds of SQL 2000 Analysis Services dimensions and cubes. We've found that when the DTS fails the SSIS job is happily completing showing as a success, we would prefer to know it went wrong.
As far as I'm aware SSIS merely starts the DTS off and doesn't care about it's result. I've taken a look in to turning on the logging for the execute DTS package and thought that the ExecuteDTS80PackageTaskTaskResult would give me the answer I need...but is merely written to the log not available as an event-handler. It also looks like it is not safe to put a SQL task in as the next item to go look at the SQL 2000 system tables to look at the log for the DTS package as the SSIS documentation warns that the DTS package can continue to run after the execute DTS package task has ended.
Ideally I want any error raised within the DTS package to cascade up to be an error in the SSIS job, I can then handle it appropriately. I cannot find a way to do this. Is there a way?
If not, can anyone suggest how in the remainder of the SSIS tasks I can be sure that the DTS has completed before I start any other tasks that will check for the SQL 2000 log of its execution?
View 5 Replies
View Related
Feb 2, 2007
Hi,
I have developed an SSIS package for ETL purpose. I am invoking the SSIS package through .Net console application by referencing the ManagedDTS Assembly. I am able to execute the package in Sql Server 2005 Developer Edition and it runs fine till completion.
But when i try to execute the packahe in Sql Server 2005 Standard edition, by invoking the package through .Net console application the status of the package is failure.
Can any one help me how to over come this problem.
View 1 Replies
View Related
May 2, 2008
Hi All,
I am in the process of moving from a 32-bit SQL Server 2005 Enterprise (9.0.3054) to a 64-bit SQL Server 2005 Enterprise (9.0.3054 with 4 CPUs and 8GB of memory on Win 2003 SP2) and the process has been very frustrating to say the least. I am having a problem with packages that I created on my 64-bit SQL Server. I am importing a few tables from the 32-SQL Server into the 64-bit SQL Server using the Task --> Import to create the package.
Sometimes when I am creating a package I get the following error in a message box:
SQL Server Import and Export Wizard
The SSIS Runtime object could not be created. Verify that DTS.dll is available and registered. The wizard cannot continue and it will terminate.
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.Windows.Forms)
Other times when I run a package that has run successfully before I get the following error:
Faulting application dtexecui.exe, version 9.0.3042.0, stamp 45cd726d, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x025d23f0.
Other times I get this error message:
.NET Runtime version 2.0.50727.1433 - Fatal Execution Engine Error (79FFEE24) (80131506)
And still other times
The package appears to hang when running. By this I mean that the Package Execution Progress shows progress up to a point then it just stops. (The package takes about 17 seconds to run normally) CPU usage is at 1% and the package cannot be stopped.
I have deleted and re-created the package several times and I have also re-installed the service pack on the SQL Server (9.0.3054) but that did not help.
Does anyone have any other suggestions to try?
Thanks.
View 4 Replies
View Related
Feb 21, 2008
I would like to standardize SSIS development so that developers all start with the same basic template. I have set it up so it is an available template ( http://support.microsoft.com/kb/908018 ) but I would like it to be the default when a new project or package is created. Is this an option?
View 4 Replies
View Related
Jun 3, 2014
I would like to fetch the data flow component name while package is executing. Since system variable named [System::SourceName] only fetches name of the control flow tasks? Is there a way to capture them?
View 5 Replies
View Related
Dec 6, 2007
Hi there
We have a SSIS run which runs as follows
The master package has a configuration file, specifying the connect strings
The master package passes these connect-strings to the child packages in a variable
Both master package and child packages have connection managers, setup to use localhost. This is done deliberately to be able to test the packages on individual development pc€™s.
We do not want to change anything inside the packages when deploying to test, and from test to production. All differences will be in the config files (which are pretty fixed, they very seldom change). That way we can be sure that we can deploy to production without any changes at all.
The package is run from the file system, through a job-schedule.
We experience the following when running on a not default sql-server instance (called dkms5253uedw)
Case 1:
The master package starts by executing three sql-scripts (drop foreign key€™s, truncate tables, create foreign key€™s). This works fine.
The master package then executes the first child package. We then in the sysdtslog get:
Error - €ścannot connect to database xxx€?
Info - €śpackage is preparing to get connection string from parent €¦€?
The child package then executes OK, does all it€™s work, and finish. Because there has been an error, the master package then stops with an error.
Case 2:
When we run exactly the same, but with the connection strings in the config file pointing to the default instance (dkms5253), the everything works fine.
Case 3:
When we run exactly the same, again against the dkms5253uedw instance, but now with the exact same databases defined in the default instance, it also works perfect.
Case 4:
When we then stop the sql-server on the default instance, the package faults again, this time with
Error - €śtimeout when connect to database xxx€?
Info - €śpackage is preparing to get connection string from parent €¦€?
And the continues as in the first case
From all this we conclude, that the child package tries to connect to the database before it knows the connection string it gets passed in the variable from the master package. It therefore tries to connect to the default instance, and this only works if the default instance is running and has the same databases defined. As far as we can see, the child package does no work against the default instance (no logging etc.).
We have tried delayed validation in the packages and in the connection managers, but with the same results (error).
So we are desperately hoping that someone can help us solve this problem.
Thanx,
/Nils M - Copenhagen
View 3 Replies
View Related
Jul 10, 2015
If I have several stored procedures, all of them making inserts/updates to the same tables, what is the safest way to run them in sequence?
The context is an asp.net application; I have an aspx page where a click on a button (or more) will launch stored procedures in execution. I have encountered the unfortunate situation before, when stored proc #2 started long before stored proc #1 finished and this caused problems.Â
If I group all stroed procs in one that simply calls all of them in the needed sequence:
exec stproc1
exec stproc2
.....
exec stprocn
Will they execute in a single thread? Is there any guarantee that stproc2 will be executed when stproc1 finishes, and so on?
View 2 Replies
View Related
May 21, 2007
hi,
I am interested in Passing value from a child Package variable to the Parent package that calls it in ssis.
I am able to call the Child package using the execute package task and use Configurations to pass values from the parent variable to the child, but I am not able to pass the value from the child to the parent.
I have a variable called datasetId in both the parent and child. it gets computed in the child and needs to be passed to the parent...
Any suggestions?
Thanks for any help in advance..
smathew
View 8 Replies
View Related
Sep 10, 2007
Deployed Report having SSIS package as source do not work when Indirect Package configuration is used in ETL package. It seems ETL package when called/executed from Report manager does not recognize environment variable to pick up the dtsconfig file.
The Report works when Direct package configuration is used to same dtsconfig file.
What could be the reason? Any solution for this? This will cause our build/deployment to QA and Prod very difficult.
View 1 Replies
View Related
Jul 6, 2006
Dear All,
I now have two SSIS package, "TESTING" and "LOADING". The "TESTING" package have an execute package task that call the "LOADING" package. When I want to execute the TESTING package, how can I setup the connection string so that I can edit the password of the database connected by the "LOADING" package?
Regards,
Strike
View 8 Replies
View Related
Oct 13, 2007
I have two SSIS packages in a project, one calling the other. The parent package works fine in my local mechine. After they are deployed to the production, I schedeul jobs to run the packages in the SqlServer. The child package works fine if I run it alone, but the parent package could not find its child package if I run the parent package . As I checked, all xml config files and the connection string pointing to the child package were set correctly. It seems the parent package did not use the xml config file. Can someone help me? Thanks in advance.
View 9 Replies
View Related
Mar 25, 2008
I have successfully created a SSIS package which execute a DTS 2000 package and with no problem to execute the task. But I failed to schedule this package. I was not success in setting the logging. When running the package in command line:
dtexec file "C:Documents and SettingslyangMy DocumentsVisual Studio 2005ProjectsTraingDTSTraingDTSDTSTraining.dtsx"
Error: 2008-03-24 08:03:24.36
Code: 0xC0012024
Source: Execute DTS 2000 Package Task
Description: The task "Execute DTS 2000 Package Task" cannot run on this edit
ion of Integration Services. It requires a higher level edition.
End Error
Warning: 2008-03-24 08:03:24.38
Code: 0x80019002
Source: DTSTraining
Description: The Execution method succeeded, but the number of errors raised
(2) reached the maximum allowed (1); resulting in failure. This occurs when the
number of errors reaches the number specified in MaximumErrorCount. Change the M
aximumErrorCount or fix the errors.
End Warning
DTExec: The package execution returned DTSER_FAILURE (1).
Any help will be greatly appreciated.
(32 bit machine, standartd edition of SQL 2005)
View 7 Replies
View Related
Sep 8, 2006
Hi,
I am making use of the DtUtil tool to deploy my package to SQL Server.
Following is my configuration:
32-bit machine and 32-bit named instance of Yukon.
I have some package variables which need to be set in the code.
Previously I did it as follows:
Set the package variables in the code. For example:
pkgFile.Variables["User::DestinationServerName"].Value = <myvalue>
Deploy the package as follows:
applnObj.SaveToSqlServer(pkgFile, null,
destinationServer, null, null);
Here the package was successfully deployed and when i open those packages using BIDS, I am able to see that the variables are set to the values as doen in teh code.
Because of oen problem I am not using SaveToSQLServer method. So I switched to DTUtil tool.
Now I am doing it this way:
Set the package variables as before.
Deploy the package to SQL Server using DTUtil tool.
Now is the problem:
The package is successfully deployed. But the variables are not set to the value that I have specified in the code.
I also tried DTexec utility to set the package variable. Even that does n't work.
Can anyone help me out? Is there any alternate method to set package variables?
Thanks,
Sandhya
View 8 Replies
View Related
Dec 22, 2006
I've seen a few posts in the MSDN documentation (see links below) stating that the MergeSynchronizationAgent no longer requires the STA threading model in SQL Server 2005 SP1. However, I'm still receiving the following exception message in my synchronization code (where it attempts to access the SynchronizationAgent property):
The MergeSynchronizationAgent class must be instantiated on a Single-Threaded Apartment (STA) thread.
I have Service Pack 1 for SQL Server 2005 installed on both my server as well as my local client (the client is running SQL Server 2005 Express). How can I verify the correct files are there (e.g. are the some specific date/time values for the RMO/COM objects?
Here's the list of MSDN links mentioned above:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=326837&SiteID=1
http://msdn2.microsoft.com/en-us/library/ms146869.aspx
http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.replication.mergesynchronizationagent.aspx
http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.replication.mergepullsubscription.synchronizationagent.aspx
View 3 Replies
View Related
Oct 20, 2015
I want to maintain all configurations in Single table, what is the best way to approach it.
View 5 Replies
View Related
Jan 28, 2007
Hi All,
I have a problem in which I want the run an SSIS based on a file being dropped in a directory. I've tried the WMI event watcher as well as the File Task Watcher component. The problem that I'm seeing is that if the process is currently processing in the pipeline while another file dropped, the newly dropped file doesn't get picked up. Is there a way to create a FileWatcher Task in SSIS which will spawn a SSIS job and return immediately for watching files?
View 3 Replies
View Related
Dec 4, 2006
Hi all,
I met a problem when trying to pass values to a SQL statement through parameters. It's a data flow task. I used the OLE DB connection. My statement is like the statement below(the real statement is little complex):
Select * from myTable where mydate>? and mydate<?
I used the "set query parameter" dialogbox to bulid two parameters varStartTime and varEndTime, the values for the two parameters were set to "1/1/2005" and "12/30/2006" respectively. But when I click the "Parse query" button, I got errors
Parameter Information cannot be derived from SQL statements. Set parameter information before preparing command.
I have referred to the posts, but the problem still exists. Any help will be highly appreciated. Thanks a lot!
View 1 Replies
View Related
Mar 27, 2008
I have used the following useful article regarding exporting a multi-record file:
http://vsteamsystemcentral.com/cs21/blogs/steve_fibich/archive/2007/09/25/multi-record-formated-flat-file-with-ssis.aspx
I have created the 9 datasources, ordering each on a field commmon to all.
I have created the required derived columns headers and have merged all the record types into a file.
The resulting file looks fine, except for the odd blank line between record types. Any ideas regarding cause and what to do to fix?
Any help is most appreciated!
View 10 Replies
View Related
