Dts.Variables Collection, Threading, And Contention
Apr 29, 2008
All,
We are experiencing some problems when running the same package on multiple threads of a C# application concurrently. The package instances seem to share a Dts.Variables collection.
I have created a test application and package to reproduce the issue. The package is very simple. It has two variables - Instance (Int32) and Content (string). The test application runs two concurrent threads that run through 25 iterations running the package. Each thread loads the package, sets the instance, and then runs the package. The package then takes the instance, sticks it in the Content variable and then writes it to a file called "FileX" where X is the Instance. Thread 1 sets the instance from 1 to 25, thread 2 sets it from 26 to 50. The test execution results in about 15% of the files containing the incorrect "instance content". That is, I'll see "42" in the file named "File14", or something along those lines. This tells me that the variables are being overwritten.
I ran another test where I created a copy of the package, so the code is identical, but the VersionGUID is different. Running the thread test application always succeeds. There is no file name/content mismatch. I then ran another test where I manually edit the XML of the package to change the VersionGUID before the package is loaded (by calling LoadFromXml). Again this never fails.
Changing the VersionGUID may work for us, but it is a total hack. Has anyone else experienced this or have any other solutions? Any thoughts on manually updating the VersionGUID to get around this problem? I hate doing this, but we are getting desperate.
Precedence Constraint Editor Values: Expression Operation: Expression and Constraint Value: Failure Expression: @GotRecs == 1 Radio button: Logical AND. All contraints must evaluate to True
Stored Procedure has an IF (that contains a SELECT) statement that checks for records if no records are returned processing stops. I SET the parameter after the BEGIN that follows the IF statement.
The step that contains the above code is step #3. I placed the constraint between step #3 and step #4 (Data Flow task) just Exports to a Flat File.
When I clicked the TEST button in the Precedence Constraint Editor I got the following msg: Error at Constraint 3: The variable €œGotRecs€? was not found in the variables collection. The variable might not exist in the correct scope.
Hello, I am trying to configure a Script Component as a data source. Although this should be a simple exercise, I am running into a problem.
My control flow contains a Foreach Loop with a file iterator. The Directory Expression of the Foreach Loop Editor is supplied by an expression mapped to a package level variable called inputdirectory. The FileNameRetrieval Expression is mapped to a package scoped variable called filename.
My data flow is encapsulated by the Foreach Loop, and contains a Script Component as Source for the Data Flow Source, and a Flat File for the Data Flow Destination. The contents of the Script Designer are as follows:
Code Block Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper Imports Microsoft.SqlServer.Dts.Runtime.Wrapper Imports System.IO Public Class ScriptMain
Inherits UserComponent Dim thisFileDate As Date Dim thisFileName As String Dim thisFilePath As String Public Overrides Sub CreateNewOutputRows()
thisFileName = ReadOnlyVariables("filename").Value.ToString() thisFilePath = ReadOnlyVariables("inputdirectory").Value.ToString() thisFileDate = File.GetCreationTime(thisFilePath & "" & thisFileName) FileSpecBuffer.AddRow() FileSpecBuffer.FileName = thisFileName FileSpecBuffer.FullPath = thisFilePath & "" & thisFileName FileSpecBuffer.CreateDate = thisFileDate End Sub End Class
When I debug the package, I get the following error:
The collection of variables locked for read access is not available at this point.
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponent.get_ReadOnlyVariables()
at ScriptComponent_67311120e6eb4162a3ea1f70847f04de.ScriptMain.CreateNewOutputRows()
at ScriptComponent_67311120e6eb4162a3ea1f70847f04de.UserComponent.PrimeOutput(Int32 Outputs, Int32[] OutputIDs, PipelineBuffer[] Buffers)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
My googlefu fails me at reconciling this. I have read various posts about changing the line ReadOnlyVariables("filename").Value to ReadWriteVariables("filename").Value, and handling the buffer assignment in PostExecute. The problem is that all examples shown are either for Script Component as Destination or Script Component as Transformation. I have tried playing with the Custom Properties in the Script Commponent set the ReadOnlyVariables and ReadWriteVariables, using a PreExecute method, a PostExecute method, all with different errors returning. I'm at a loss here. Could anybody provide me with a simple working example so that I can correctly populate my output buffer in the context of Script Component as Source?
I fully understand that I could just run a Script Task Using System.IO.Directory, and System.IO.File, but I really want to make this package work in the manner I've described. Any help would be appreciated.
Has anyone come across the problem of transactions not being able to write to the transaction log while a 'dump transaction' operation is in progress and therefore causing the transaction to timeout??
Is there any way of eliminating the timeouts/contention??
this is sanjeev, i have SSIS package, using my c# program i want to add one execute package task to this package's sequence container.
it is creating the new package with out any probelm. but when i opened the package and try to move the newly created exeute package task it is giving the following error.
the element cannot be found in a collection. this error happens when you try to retrieve an element from a collection on a container during the execution of the package
Hi - have 150,000,000 row table receiving BULK INSERTS all day - I have also found out my client likes to performan READ opertions simultaniously against this table with of course chronic contention issues.
Need to provide a quick-fix, will index views enable me to abstact the reporting data away from the BULK INSERT table and is there any connection issues between the base table and the Index View?
 I have used Extented event to monitor the occurances of TempDB contention on Production server . I found there are several entried logged in in 30 mints .Now I am trying to determin if Tempdb contention on PFS, GAM or SGAM page then I will decide if I need to increase the number of TempDB data files on Production server . Currently , There are 8 TempDB Data files configured on its separate Disks .There are Page_IDs I found in the extented events for Tempdb files -
Page_IDÂ =1 for PFS page Page_ID = 2 for GAM page Page_ID =3 for SGAM page
but I found the Below Page_IDs and I know there is a formula that you can use to identify if page is PFS,GAM or SGAM ? How should I use this formula and what should I look for to determine if page is PFS,GAM or SGAM ? Is there any threshold value for the duration of TempDB contention occured ?
We have quite a few SQL Servers running on servers with hyper threaded enabled processors. We have always left hyper threading on, even on our very busy servers but i've recently begun to question this due to comments i have received.
All our servers run SQL 2000 SP3a as "support" for hyper threading was added in SP3a
So... What do you think of Hyper Threading any why?
Over some time now, I've been developing a fairly hefty stored procedure, that does a lot of computations, and fairly few table lookups.
When I look at the performance on my server (a dual Xeon HT) I can see that it only uses 1 out of 4 possible "cpus" to work on the calculations, while the three others idle out, and was wondering if I can somehow force it to use max available CPU power?
I have a vb 2005 app that executes a SSIS package in threads. The SSIS package imports a large (20mb) file, does many alternation to it, exports it to the database. The SSIS execution is quite frequent and there for multi-threading is needed.
Here is the issue: If i take out the threading everything works great. The second that i add threading, and try to execute a few instances of the SSIS it starts crashing. The thing is that it errors out in different task and is completely unpredictable.
My question: 1) Can SSIS be used in a multi-threading environment? 2) If yes, How do i do it? 3) Does anyone have any suggestion as to what i should try?
p.s. Timing is definitely an issue for me. I got to get this working, tested, approved by 8/27/2006 --------- here are the 3 errors that i'm getting when i turn multi-threading on 1)Thread "WorkThread0" has exited with error code 0xC0202009.
2)An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Transaction (Process ID 112) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
3)The ProcessInput method on component "(i cannot display the name of the task for security reasons" (448) failed with error code 0xC0202009. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
We're moving in a 4-socket (Xeon Dual Core CPU's) server. What version and product of SQL Server should I be looking for to fully take advantage of the new server's 4 processors. I'm not sure I understand the licensing per CPU. Is there any advantage in waiting for SQL Server 2008 as it concerns multi-threading and what should I be looking to spend for a SQL server solution that utilizes all four processors. Someone has told me that the SQL Server standard edition comes with 4 cpu licenses. Is this correct?
I have an SSIS package that is running the same set of stored procedures on about 50 remote database servers and importing the data through data flow tasks into a local database.
To do this I have a db table set up with the remote server and db names and using variables, i am looping through each one and getting the data. Everything works wonderfully...however, I need to find a way to be able to run multiple calls at the same time. Is it possible to multithread SSIS so that I can run 1 server retrieval on 1 thread and another retrieval on another thread? Any other suggestions on how to optimally do this?
Hi,I have SQL SERVER 2000 SP4 Enterprise , Windows 2003Enterprise onXeon 4 Processors (now with multi-threading CPU) and I havequestions whichseem weirds to me (used to have the same config withoutMulti-Threading) as following:1. SQL Server see 8 CPUs. Is this because ofmulti-threading CPUs ?2. when use SP_WHO2 ACTIVE ,2.1 Result has user SPID , which blockitself(it is UPDATE and SELECTstatements, most often)2.2 Result has user SPID , show morethan one result rows withthe same processe and command3. when use Properties of SQL Server , it has anerror 15407 in log.many thanks in advanceNiponW
If I have several stored procedures, all of them making inserts/updates to the same tables, what is the safest way to run them in sequence?
The context is an asp.net application; I have an aspx page where a click on a button (or more) will launch stored procedures in execution. I have encountered the unfortunate situation before, when stored proc #2 started long before stored proc #1 finished and this caused problems.Â
If I group all stroed procs in one that simply calls all of them in the needed sequence:
exec stproc1 exec stproc2 ..... exec stprocn
Will they execute in a single thread? Is there any guarantee that stproc2 will be executed when stproc1 finishes, and so on?
Is SQL Compact Edition safe to use in ASP.NET WebApps?
And second if so, should i use transation isolation level or row locking hints to assure data quality during iniserts?
E.g. i want to assure that there is only one record named user2@domain2 and thus i want to make sure that my code will not fail in a mutli-theaded scenario and insert second one or get a key exception back: see sample
transaction or locking hit{ select from users .... if no user found insert into users }
I've seen a few posts in the MSDN documentation (see links below) stating that the MergeSynchronizationAgent no longer requires the STA threading model in SQL Server 2005 SP1. However, I'm still receiving the following exception message in my synchronization code (where it attempts to access the SynchronizationAgent property):
The MergeSynchronizationAgent class must be instantiated on a Single-Threaded Apartment (STA) thread.
I have Service Pack 1 for SQL Server 2005 installed on both my server as well as my local client (the client is running SQL Server 2005 Express). How can I verify the correct files are there (e.g. are the some specific date/time values for the RMO/COM objects?
I am not comfortable with DTS 2000 but I need to execute a encapsulated DTS 2000 package from a SSIS package. The real problem is when I need to pass SSIS variables to DTS 2000 package. The DTS 2000 package have 3 global variables that I can identify on " Execute DTS 2000 Package Task Editor - Inner Variables ". I believe the SSIS variables must be mapped on " Execute DTS 2000 Package Task Editor - OuterVariables ". How can I associate the SSIS variables(OuterVariables ) to "Inner Variables"? How can I do it? Much Thanks.
I recently migrated a database from server A to server B. The backup jobs I am trying to build on server B are failing because of the following error: -------------- [Microsoft SQL-DMO] Error 21776: [SQL-DMO]The name 'WinDat' was not found in the Databases collection. If the name is a qualified name, use [] to separate various parts of the name, and try again. -------------
How do I add this database to the databases collection so it will be recognized?
I can add two reportitem controls, ie reportitems!begbal.value + reportitems!deposits.value, without a problem. However, when I add the 3rd reportitem control to the expression, ie + reportitems!withdrawals.value, some really funky arithmetic occurs. All of these controls I am referring to are in the same group footer.
actuallu the earlier obviously worked but now i am fetching values from 2 colums and in collection we can pass 2 strings i did that.but then how to increment the collection c.will just post the code . query fetching 2 records can add only 1 NameValueCollection c = new NameValueCollection(); cmd2.CommandText="select Pname ,HoursWorked from TimeSheet1 ts , ProjResource pr,ProjectDetails pd where ts.rid = pr.rid and ts.pcode=pr.pcode and pd.Pcode = ts.Pcode and pr.Rid = '" + Ridtxt.Text + " '"; cmd2 .Connection = con1; con1.Open(); SqlDataReader dr = cmd2.ExecuteReader();
while (dr.Read()) { c.Add(dr["pname"].ToString(), dr["hoursworked"].ToString()); }
as earliar supose c# 12 asp 11 only c# getting added even though loop is while dr.read() { } actually how to increment key as in c both strings are occupied
hi evryone I have a question! in my ASP.NET page I use a SqlDataAdapter object for working on my database first I declare it as a global vaiable(SqlDataAdapter objDataAdapter;) then I construct it in my Page_Load procedure like this (objDataAdapter=new SqlDataAdapter(strSql,objConnention)) then when I want to use it in my other procedures an error tells me that your object does not exist for solving this error I should construct my objDataAdapter in evry procedure using it. but I think such variables should not be erased from Heap automatically(with Garbage Collection) because it has a reference to Stack. I mean the global variable... any opinion ? Thanks
I'm having trouble obtaining errors raised in a stored procedure via the ADO Errors collection after the second FETCH NEXT statement from within that stored procedure.
Consider the following table created in a SQL Server database:
This is a very simple table with one column, and three rows containing the values 1, 2 and 3.
Consider this stored procedure: CREATE PROCEDURE TestStoredProc as BEGIN set rowcount 0 Set NoCount ON
declare @TestInt int declare @ErrMsg char(7) declare TestCursor cursor forward_only for select * from TestTable
open TestCursor Fetch next from TestCursor into @TestInt
While @@fetch_status<>-1 Begin select @ErrMsg = 'Error ' + convert(char, @testint) raiserror(@ErrMsg, 16, 1) raiserror(@ErrMsg, 16, 1) Fetch next from TestCursor into @TestInt end
Close TestCursor DeAllocate TestCursor return END
This stored procedure simply defines a cursor on all rows in TestTable. For each row fetched from the cursor, the error message 'Error n' is raised twice, where n is the integer that had just been fetched from the cursor.
Finally, consider this VB code using ADO to execute the above stored procedure. After the stored procedure is executed, the code loops through the errors collection, and creates a message box for each error in the collection:
Private Sub Form_Load() Dim cn As Connection Dim cm As Command Dim oErr As Error
If I have N different xml document formats and I want to store those xml data files as typed xml in one table (one column), can I do it using xml schema collection (by adding schema file to schema collection for each document type and assigning xml schema collection to this column)?
Is this possible using xml schema collection? Or did I miss something about xml schema collection usage scenario?
I need to feed head office sql server with the data from regional servers. Servers are spread through all continents Data input done locally on Head office server as well and plus need to ship data from other servers. So clarify this - Head office server is not standby one. Mirroring is out of the picture, I think.. Initially, I thought ship a log every 15 min and restore on Head office server but is this going to create an issue for the local data processing?
