SSIS Multi-threading
Aug 23, 2006
I have a vb 2005 app that executes a SSIS package in threads.
The SSIS package imports a large (20mb) file, does many alternation to it, exports it to the database.
The SSIS execution is quite frequent and there for multi-threading is needed.
Here is the issue:
If i take out the threading everything works great. The second that i add threading, and try to execute a few instances of the SSIS it starts crashing. The thing is that it errors out in different task and is completely unpredictable.
My question:
1) Can SSIS be used in a multi-threading environment?
2) If yes, How do i do it?
3) Does anyone have any suggestion as to what i should try?
p.s. Timing is definitely an issue for me. I got to get this working, tested, approved by 8/27/2006
---------
here are the 3 errors that i'm getting when i turn multi-threading on
1)Thread "WorkThread0" has exited with error code 0xC0202009.
2)An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Transaction (Process ID 112) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
3)The ProcessInput method on component "(i cannot display the name of the task for security reasons" (448) failed with error code 0xC0202009. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
View 5 Replies
ADVERTISEMENT
Apr 11, 2008
I have an SSIS package that is running the same set of stored procedures on about 50 remote database servers and importing the data through data flow tasks into a local database.
To do this I have a db table set up with the remote server and db names and using variables, i am looping through each one and getting the data. Everything works wonderfully...however, I need to find a way to be able to run multiple calls at the same time. Is it possible to multithread SSIS so that I can run 1 server retrieval on 1 thread and another retrieval on another thread? Any other suggestions on how to optimally do this?
View 4 Replies
View Related
Jun 22, 2006
Apologies if this is not the right forum.
We have some c~ code which calls a stored proc to trnsfer data from one db to another.
We are using multi threading to allow a whole days data to be processed in parrallel.
However, we are experiencing problems. After a while we get the following message:
"An exception during changed data captured caused all processing threads to abort".
Can anyone tell me what posssible reasons would cause this to happen?
It is not an area in which I have a lot of expertise, the code was written by a third party.
Many thanks
View 1 Replies
View Related
Nov 8, 2007
We're moving in a 4-socket (Xeon Dual Core CPU's) server. What version and product of SQL Server should I be looking for to fully take advantage of the new server's 4 processors. I'm not sure I understand the licensing per CPU. Is there any advantage in waiting for SQL Server 2008 as it concerns multi-threading and what should I be looking to spend for a SQL server solution that utilizes all four processors. Someone has told me that the SQL Server standard edition comes with 4 cpu licenses. Is this correct?
Thanks, Rob Huylo
Rob21379@aol.com
View 11 Replies
View Related
Jan 28, 2004
hi,
I want to execute two user created stored procedures in a multithreaded manner in ms-sql server . Can some tell me how i can do that.
Thanks in advance.
Regards,
Manpreet
View 14 Replies
View Related
Nov 23, 2005
Hi,I have SQL SERVER 2000 SP4 Enterprise , Windows 2003Enterprise onXeon 4 Processors (now with multi-threading CPU) and I havequestions whichseem weirds to me (used to have the same config withoutMulti-Threading) as following:1. SQL Server see 8 CPUs. Is this because ofmulti-threading CPUs ?2. when use SP_WHO2 ACTIVE ,2.1 Result has user SPID , which blockitself(it is UPDATE and SELECTstatements, most often)2.2 Result has user SPID , show morethan one result rows withthe same processe and command3. when use Properties of SQL Server , it has anerror 15407 in log.many thanks in advanceNiponW
View 2 Replies
View Related
Aug 29, 2007
I have a question:
Is SQL Compact Edition safe to use in ASP.NET WebApps?
And second if so, should i use transation isolation level or row locking hints to assure data quality during iniserts?
E.g. i want to assure that there is only one record named user2@domain2 and thus i want to make sure that my code
will not fail in a mutli-theaded scenario and insert second one or get a key exception back: see sample
transaction or locking hit{
select from users ....
if no user found
insert into users
}
Thanks
Tim
View 5 Replies
View Related
Jun 3, 2002
I have some threading questions, please explain how these work. Thanks!!
How do the threads get created (programmatically, network, or through sequel)?
If I have 10 people in an application, is a thread created for each user?
If I run a large process can I subdivide the process and run it through mulitple threads?
View 1 Replies
View Related
Sep 29, 2005
Ello,
We have quite a few SQL Servers running on servers with hyper threaded enabled processors. We have always left hyper threading on, even on our very busy servers but i've recently begun to question this due to comments i have received.
All our servers run SQL 2000 SP3a as "support" for hyper threading was added in SP3a
So... What do you think of Hyper Threading any why?
View 5 Replies
View Related
Aug 9, 2006
Over some time now, I've been developing a fairly hefty stored procedure, that does a lot of computations, and fairly few table lookups.
When I look at the performance on my server (a dual Xeon HT) I can see that it only uses 1 out of 4 possible "cpus" to work on the calculations, while the three others idle out, and was wondering if I can somehow force it to use max available CPU power?
View 2 Replies
View Related
Nov 1, 2006
I have figured out that SSIS installs once for as many instances of SQL Server 2005 as there are on the machine. I also figured out that even if there is only a named instance on the machine, it will always install under the "Default" machine named instance.
I also figured out that you can change the instance viewed by modifying C:Program FilesMicrosoft SQL Server90DTSBinnMsDtsSrvr.ini.xml
What I am unclear about is that in SQL Server 2000, each Instance had an MSDB db which stored all of the jobs. In SQL Server 2005, each instance still has an MSDB DB but theSSIS also has a MSDB DB ?? or is this pointing to the MSDB for the instance specified in the xml file???
There is not much info out on this at all and would be good if the MSDB help explained this install once so it would all be clear!
THanks for any help... thank goodness for newsgroups!
View 6 Replies
View Related
Apr 29, 2008
All,
We are experiencing some problems when running the same package on multiple threads of a C# application concurrently. The package instances seem to share a Dts.Variables collection.
I have created a test application and package to reproduce the issue. The package is very simple. It has two variables - Instance (Int32) and Content (string). The test application runs two concurrent threads that run through 25 iterations running the package. Each thread loads the package, sets the instance, and then runs the package. The package then takes the instance, sticks it in the Content variable and then writes it to a file called "FileX" where X is the Instance. Thread 1 sets the instance from 1 to 25, thread 2 sets it from 26 to 50. The test execution results in about 15% of the files containing the incorrect "instance content". That is, I'll see "42" in the file named "File14", or something along those lines. This tells me that the variables are being overwritten.
I ran another test where I created a copy of the package, so the code is identical, but the VersionGUID is different. Running the thread test application always succeeds. There is no file name/content mismatch. I then ran another test where I manually edit the XML of the package to change the VersionGUID before the package is loaded (by calling LoadFromXml). Again this never fails.
Changing the VersionGUID may work for us, but it is a total hack. Has anyone else experienced this or have any other solutions? Any thoughts on manually updating the VersionGUID to get around this problem? I hate doing this, but we are getting desperate.
Thanks in advance,
Rob
View 2 Replies
View Related
Aug 7, 2006
I have two SSIS packages that I want to run in one SQL Agent job as two individual steps. The two packages run fine when they are in separate jobs. However, when I run the job conaining both SSIS packages (under the same proxy), the first SSIS package starts, but hangs in the middle.
I then tried setting the DelayValidation flag to True as suggested for a similar issue in another thread from this forum. After changing the DelayValidation flag to True for all containers and tasks on the second SSIS package, the first SSIS package ran completely through sucessfully, but the job continued executing for hours and the second SSIS package never started. I finally killed the job.
Any ideas as to what is the problem here? I have logged to the event viewer and see that the first package completes sucessfully. They run successfully in separate jobs, but I can not get them to run together within the same job without hanging.
Any help is appreciated,
Paulette
View 5 Replies
View Related
Jul 10, 2015
If I have several stored procedures, all of them making inserts/updates to the same tables, what is the safest way to run them in sequence?
The context is an asp.net application; I have an aspx page where a click on a button (or more) will launch stored procedures in execution. I have encountered the unfortunate situation before, when stored proc #2 started long before stored proc #1 finished and this caused problems.
If I group all stroed procs in one that simply calls all of them in the needed sequence:
exec stproc1
exec stproc2
.....
exec stprocn
Will they execute in a single thread? Is there any guarantee that stproc2 will be executed when stproc1 finishes, and so on?
View 2 Replies
View Related
Dec 22, 2006
I've seen a few posts in the MSDN documentation (see links below) stating that the MergeSynchronizationAgent no longer requires the STA threading model in SQL Server 2005 SP1. However, I'm still receiving the following exception message in my synchronization code (where it attempts to access the SynchronizationAgent property):
The MergeSynchronizationAgent class must be instantiated on a Single-Threaded Apartment (STA) thread.
I have Service Pack 1 for SQL Server 2005 installed on both my server as well as my local client (the client is running SQL Server 2005 Express). How can I verify the correct files are there (e.g. are the some specific date/time values for the RMO/COM objects?
Here's the list of MSDN links mentioned above:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=326837&SiteID=1
http://msdn2.microsoft.com/en-us/library/ms146869.aspx
http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.replication.mergesynchronizationagent.aspx
http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.replication.mergepullsubscription.synchronizationagent.aspx
View 3 Replies
View Related
Oct 20, 2015
I want to maintain all configurations in Single table, what is the best way to approach it.
View 5 Replies
View Related
Jan 28, 2007
Hi All,
I have a problem in which I want the run an SSIS based on a file being dropped in a directory. I've tried the WMI event watcher as well as the File Task Watcher component. The problem that I'm seeing is that if the process is currently processing in the pipeline while another file dropped, the newly dropped file doesn't get picked up. Is there a way to create a FileWatcher Task in SSIS which will spawn a SSIS job and return immediately for watching files?
View 3 Replies
View Related
Dec 4, 2006
Hi all,
I met a problem when trying to pass values to a SQL statement through parameters. It's a data flow task. I used the OLE DB connection. My statement is like the statement below(the real statement is little complex):
Select * from myTable where mydate>? and mydate<?
I used the "set query parameter" dialogbox to bulid two parameters varStartTime and varEndTime, the values for the two parameters were set to "1/1/2005" and "12/30/2006" respectively. But when I click the "Parse query" button, I got errors
Parameter Information cannot be derived from SQL statements. Set parameter information before preparing command.
I have referred to the posts, but the problem still exists. Any help will be highly appreciated. Thanks a lot!
View 1 Replies
View Related
Mar 27, 2008
I have used the following useful article regarding exporting a multi-record file:
http://vsteamsystemcentral.com/cs21/blogs/steve_fibich/archive/2007/09/25/multi-record-formated-flat-file-with-ssis.aspx
I have created the 9 datasources, ordering each on a field commmon to all.
I have created the required derived columns headers and have merged all the record types into a file.
The resulting file looks fine, except for the odd blank line between record types. Any ideas regarding cause and what to do to fix?
Any help is most appreciated!
View 10 Replies
View Related
Oct 12, 2014
I have one scenario
Table
Col1. Col2
1. A,b,c,df,ghf
2. C,b
3. B
Output should be
Col1. Col2
1. A
1. B
1. C
1. Df
1. Ghf
2. C
2. B
3. B
View 9 Replies
View Related
Feb 21, 2008
I am using the following useful article regarding exporting a multi-record file:
http://vsteamsystemcentral.com/cs21/blogs/steve_fibich/archive/2007/09/25/multi-record-formated-flat-file-with-ssis.aspx
I have created the 2 datasources, ordering each on a field commmon to both.
I have created the two derived columns headers and am now moving on to the merge.
It is failing with the following error:
"the input is not sorted"
And whilst I definitely have an order by on the query, when I look at the metadata between the datasource and the derived column, the Sort Key Position items displays "0" for all my fields, I was expecting the sort field to have a "1" in this column. What am I missing?
Any help would be most appreciated!
View 7 Replies
View Related
Sep 23, 2014
Disaster Recovery Options based on the following criteria.
--Currently running SQL 2012 standard edition
--We have 18000 databases (same schema across databases)- majority of databases are less than 2gb-- across 64 instances approximately
--Recovery needs to happen within 1 hour (Not sure that this is realistic
-- We are building a new data center and building dr from the ground up.
What I have looked into is:
1. Transactional Replication: Too Much Data Not viable
2. AlwaysOn Availability Groups (Need enterprise) Again too many databases and would have to upgrade all instances
3. Log Shipping is a viable option and the only one I can come up with that would work right now. Might be a management nightmare but with this many databases probably all options with be a nightmare.
View 1 Replies
View Related
Aug 17, 2015
More often than not, I typically don't touch DTC on clusters anymore; however on a project where the vendor states that it's required. So a couple things here.
1) Do you really need DTC per instance or one for all?
2) Should DTC be in its own resource group or within the instance's group?
2a) If in it's own resource group, how do you tie an instance to an outside resource group? tmMappingSet right?
View 9 Replies
View Related
Feb 4, 2008
the stored procedure don't delete all the records
need help
Code Snippet
DECLARE @empid varchar(500)
set @empid ='55329429,58830803,309128726,55696314'
DELETE FROM [Table_1]
WHERE charindex(','+CONVERT(varchar,[empid])+',',','+@empid+',') > 0
UPDATE [empList]
SET StartDate = CONVERT(DATETIME, '1900-01-01 00:00:00', 102), val_ok = 0
WHERE charindex(','+CONVERT(varchar,[empid])+',',','+@empid+',') > 0
UPDATE [empList]
SET StartDate = CONVERT(DATETIME, '1900-01-01 00:00:00', 102), val_ok = 0
WHERE charindex(','+CONVERT(varchar,[empid])+',',','+@empid+',') > 0
TNX
View 2 Replies
View Related
Jul 20, 2005
Hello,I am trying to construct a query across 5 tables but primarily 3tables. Plan, Provider, ProviderLocation are the three primary tablesthe other tables are lookup tables for values the other tables.PlanID is the primary in Plan andPlanProviderProviderLocationLookups---------------------------------------------PlanIDProviderIDProviderIDLookupTypePlanNamePlanIDProviderStatusLookupKeyRegionIDLastName...LookupValue....FirstName...Given a PlanID I want all the Providers with a ProviderStatus = 0I can get the query to work just fine if there are records but what Iwant is if there are no records then I at least want one record withthe Plan information. Here is a sample of the Query:SELECT pln.PlanName, pln.PlanID, l3.LookupValue as Region,p.ProviderID, p.SSNEIN, pl.DisplayLocationOnPCP,pl.NoDisplayDate, pl.ProviderStatus, pl.InvalidDate,l1.LookupValue as ReasonMain, l2.LookupValue as ReasonSub,pl.InvalidDataFROM Plans plnINNER JOIN Lookups l3 ON l3.LookupType = 'REGN'AND pln.RegionID = l3.Lookupkeyleft outer JOIN Provider p ON pln.PlanID = p.PlanIDleft outer JOIN ProviderLocation pl ON p.ProviderID = pl.ProviderIDleft outer JOIN Lookups l1 ON l1.LookupType = 'PLRM'AND pl.ReasonMain = l1.LookupKeyleft outer JOIN Lookups l2 ON l2.LookupType = 'PLX1'AND pl.ReasonSub = l2.LookupkeyWHERE pln.PlanID = '123456789' AND pl.ProviderStatus = 0ORDER BY p.PlanID, p.ProviderID, pl.SiteLocationNumI know the problew the ProviderStatus on the Where clause is keepingany records from being returned but I'm not good enough at this toanother select.Can anybody give me some suggestions?ThanksDavid
View 5 Replies
View Related
Feb 26, 2008
I am attempting to create a multi-record file (as described in my last thread) and have found the following set of instructions very helpful:
http://vsteamsystemcentral.com/cs21/blogs/steve_fibich/archive/2007/09/25/multi-record-formated-flat-file-with-ssis.aspx
I have been able to create a sample file with two of my record types.
I now need to build on this further, because I have 9 record types in total that need to be extracted to a single flat file.
does anyone have any ideas how I might extend the example above to include more record types or know of another means of achieving this?
Thanks in advance for any help you might be able to provide.
View 3 Replies
View Related
Mar 27, 2007
I am new to Reporting Services and hope that what I am looking to do is within capabilities :-)
I have many identical schema databases residing on a number of data servers. These support individual clients accessing them via a web interface. What I need to be able to do is run reports across all of the databases. So the layout is:
Dataserver A
Database A1
Database A2
Database A3
Dataserver B
Database B1
Database B2
Dataserver C
Database C1
Database C2
Database C3
I would like to run a report that pulls table data from A1, A2, A3, B1, B2, C1, C2, C3
Now the actual number of servers is 7 and the number of databases is close to 1000. All servers are running SQL2005.
Is this something that Reporting Services is able to handle or do I need to look at some other solution?
Thanks,
Michael
View 5 Replies
View Related
Jul 20, 2005
Greetings,We are trying to set up a set of "Leading Practices" for ourdevelopers, as well as ourselves, and hope some gentle reader canrecommend some documentation in favor of what appears to be the rightposition to take.We do not allow third party applications to run on our SQL Servers. Wewant to include DTS Packages under the definition of third partyapplications, insisting instead that the developers save theirpackages as COM Formatted files into their source code control systemsand run them from their app servers. The devlopers would like to hearthis from someone besides ourselves.While strong recomendations to remove guest access to MSDB altogetherabound, I have been unable to find a straight forward discussion ofthe advantages of structured file storage and app server off load ofDTS packages.Can anyone suggest any articles, white papers, rants, etc attemptingto formulate a solution to the benefits of taking msdb away fromguest, with the advantages of running DTS from an App server orworkstation platform, with the packages protected in source codecontrol?Thank youJohn Pollinsjpollins @ eqt . com
View 2 Replies
View Related
Mar 22, 2001
Is it possible to create a stored procedure that can be used on multiple dBs??
For instance, I want to create a stored procedure and use it on DB1 and DB2.
Right now, I can create a stored procedure in DB1 but only I can run it.
I want to run the stored procedures on DB2 that I created in DB1..
I hope this makes sense..
Any insight into the workings of SPs would be most appreciated...
View 3 Replies
View Related
Mar 13, 2007
-- declare table declare @Table1 table(ID int, Value int )-- Insert sample data provided into tableinsert into @Table1select 1 , 2 union allselect 2 , 1 union allselect 7 , 2 -- declare table declare @Table2 table(ID int, Value int )-- Insert sample data provided into tableinsert into @Table2select 1 , 2 union allselect 5 , 1 union allselect 3 , 2 -- declare table declare @Table3 table(ID int, Value int )-- Insert sample data provided into tableinsert into @Table3select 1 , 2 union allselect 2 , 1 union allselect 3 , 2 -- declare table declare @Table4 table(ID int, Value int )-- Insert sample data provided into tableinsert into @Table4select 5 , 2 union allselect 2 , 1 union allselect 10 , 2 /*Is there anyway I can write one sql query which will give me the following result:IDs = All unique ID from all tablesSumOfAllValueFromAllTables =for the same ID, value from Table1+for the same ID, value from Table2 + so on ...IDs - SumOfAllValueFromAllTables===============================1 - 62 - 33 - 45 - 37 - 210 - 2*/--I have tried by the following way, but for more tables, it is becoming complicated, because I have 12 tables.--Is there any better way to do it? Thanks for the help.select coalesce(t1.ID,t2.ID) as IDs, coalesce(sum(t1.value),0) +coalesce(sum(t2.value),0) as SumOfAllValueFromAllTablesfrom @Table1 t1full join @Table2 t2 on t1.id=t2.idgroup by t1.ID,t2.IDorder by IDs
View 1 Replies
View Related
Jul 7, 2004
I have a search form that takes 5 inputs from textboxes or Drop Down Lists.
I built a query that works just fine in Analyizer, but the second i try to turn it into a sproc that takes parameters I can only get it to work with one parameter so far....
Here is the query that works just fine
Select ftr_location.loc_name, ftr_file.loc_id, ftr_file.file_date, ftr_file.file_type_id, ftr_file.acct_no,
ftr_file.cust_fname, ftr_file.cust_lname, ftr_status.status_name, ftr_file.destruction_date
FROM dbo.ftr_file
Inner Join ftr_location On ftr_location.loc_id = ftr_file.loc_id
Inner Join ftr_Status On ftr_status.status_id = ftr_file.status_id
Inner Join ftr_doc_types on ftr_doc_types.file_type_id = ftr_file.file_type_id
WHERE ftr_file.Loc_id = 18 and ftr_file.file_date= '12/4/2004'
GO
The sproc version that I cant get to work is:
CREATE PROCEDURE dbo.usp_ftr_searchfile_s
(
@ResultLocation int = null,
@ResultFileDate datetime = null
)
AS
DECLARE @SQLString VARCHAR(4000)
SET @SQLString = 'Select ftr_location.loc_name, ftr_file.loc_id, ftr_file.file_date, ftr_file.file_type_id, ftr_file.acct_no,'+
'ftr_file.cust_fname, ftr_file.cust_lname, ftr_status.status_name, ftr_file.destruction_date' +
' FROM dbo.ftr_file' +
' Inner Join ftr_location On ftr_location.loc_id = ftr_file.loc_id' +
' Inner Join ftr_Status On ftr_status.status_id = ftr_file.status_id' +
' Inner Join ftr_doc_types on ftr_doc_types.file_type_id = ftr_file.file_type_id' +
' WHERE 1=1 '
If @ResultLocation Is Not Null
Set @SQLString = @SQLString + 'And ftr_file.loc_id=' + cast @ResultLocation
If @ResultLocation Is Not Null
Set @SQLString = @SQLString + 'And ftr_file.file_date=' + cast @ResultFileDate
EXEC (@SQLString)
GO
Any feedback would be most helpful, as I can get it to work with one parameter but not when I add teh second in the mix.
View 1 Replies
View Related
Apr 12, 2005
I have a dataadapter on my asp.net page and am having issues passing values to the sql statement. See below:
SELECT tblTemp.SensorID, tblSensor.SensorNum, tblTemp.TempTime, tblTemp.TempVal FROM tblTemp
INNER JOIN tblSensor ON tblTemp.SensorID = tblSensor.SensorID
WHERE (tblTemp.TempTime BETWEEN @From AND @To)
AND (tblTemp.SensorID IN (@Sensor))
ORDER BY tblTemp.TempTime
@Sensor is populated in vb with the selected values from a checkboxlist control. I attempted to test this in the "Data Adapter Preview" window, but continually get an error. Is there a way to pass multiple values to a parameter?
Any help would be appreciated, thanks!
View 2 Replies
View Related
Jan 5, 2004
Hi All,
I am trying to update a huge table with about 70 million records and the table do not have a primary key.
I want to update a "FLAG" field in the table with value 'N' default value of flag fields for all records is NULL.
I want to update the whole table in chunks say 50,000 records a time...
like
update tablename (first 50000)
set flag = 'N' where flag = NULL.
Hope I am clear..:-)
thanks
hardlyworking..
View 7 Replies
View Related
