SSIS Behaves Differently In Different Environments Even Though The Code Is Same
Oct 9, 2007
Hi,
SSIS is behaving differently in different environments but the code is same.
One thing is nor working correctly that is I am converting a string data type column to float data type in data conversion. In our local environments the package is working fine but in production environments it is not working correclty. It is unable to convert the data it is throwing an error.
"The data value cannot be converted for reasons other than sign mismatch or data overflow"
Our Transactions/sec counter jumped quite a bit when we moved to SQL Server 2005. The move coincided with increased load so we didn't think anything of it until recently. Upon further review, the counter just seems too high.
There was an article in SQL Server magazine a few years ago by Brian Moran where he states, "Transactions/sec doesn't measure activity unless it's inside a transaction. Batch Requests/sec measures all batches you send to the server even if they don't participate in a transaction." He goes on to say that Transactions/sec will be skewed lower because it is a subset of Batch Requests/sec. (http://www.sqlmag.com/Article/ArticleID/26380/sql_server_26380.html)
The article was written for SQL Server 2000. We conducted tests in 2000 and found what he said to be right on the money. SELECT statements increased Batch Requests/sec, but not Transactions/sec. UPDATE/INSERT/DELETE statements increased both in lockstep. Makes perfect sense so far.
We conducted the same tests in 2005 and found a radically different story. While SELECT statements behaved the same, UPDATE/INSERT/DELETE statements showed Transactions/sec skyrocket 2-10x more than Batch Requests/sec for the duration of the statement. In other words, a single transaction submitted by our application fires off exponentially more transactions than the one we submitted. I was unable to pinpoint exactly what these "hidden" transactions were actually doing. Is this something that occurred in 2000 but simply wasn't reported? Or is it new behavior in 2005?
While trying to answer these questions we noticed a second strange behavior in 2005. When no queries are being executed the Transactions/sec counter still jumps every six seconds like clockwork. And these phantom transactions number in the thousands. We tried to use profiler to capture what SQL was being executed, but nothing shows up in any SQL Statement or Batch event. However, when we turned on the SQLTransaction event we found it, sort of. An object called GhostCleanupTask runs every six seconds causing thousands of transactions. We don't know exactly what it is doing, but we noticed that it ran consistently on some databases, but never on other databases. Both sets of databases are identical and in use.
So, all of this investigation leads me with three final questions.
1. What is behind all the extra transactions caught by perfmon when I submit a single transaction? 2. What is GhostCleanupTask and why does it take so many transactions? (And why does it only run on certain databases?) 3. If a potential customer asks for our Transactions/sec count, is it accurate to give them the big number, knowing that our application is only actually submitting a fraction of that? On the other hand, the system apparently is actually doing that many transactions. (For instance, on our production server during peak, Batch Requests/sec is about 4,000, while Transactions/sec hits 26,000.
I created a custom assembly using C# to transform some binary data into text, and in this assembly I used one win32 dll developed by our customer to help me to do the tranformation. The code I used to call the win32 dll is like below: [DllImport("tdasuie.dll", EntryPoint = "AlrtLogConditionToText", ExactSpelling = false, CharSet = CharSet.Auto, SetLastError = true)] private static extern UInt32 AlrtLogConditionToText(Byte[] pbCondition, StringBuilder pszText, UInt32 dwSize);
I defined a C# method to call the above win32 method and return a string. Then in the report, I called this C# method to get the correct string.
In the report designer, the C# method in the custom assembly can return the correct string in the preview window. But after I deployed the report into the report server, the textbox will only display "#error" in the report manager web page.
The following code does not function if I use SQLOLEDB if I omit the provide and default to ODBC OLE DB it works correctly. I am assume I am coding something wrong for a SQLOLEDB provide. Any help is greatly appricated.
VB Code
Public Function SqlExecuteResult(xSQL As String, sServer As String, sDatabase As String, sUserName As String, sPassword As String, sCaller As String, Optional bLog As Boolean = False) As Object
Dim oDB As Object Dim oRS As Object
Set oDB = CreateObject("adodb.connection") Set oRS = CreateObject("adodb.recordset")
Private Sub Form_Load() Dim rs As Object Set rs = SqlExecuteResult("exec NextEntry 'SentMessages'", "surecomp-bob", "pmsureus33", "sa", "", "") MsgBox rs.fields(0) End Sub
SQL proceedure
CREATE PROCEDURE NextEntry @CounterName Varchar(20) AS
begin declare @counter int select @counter = counter from counters where countername = @counterName select @counter = @counter + 1 update counters set counter = @counter where countername = @countername select counter from counters where countername = @counterName End GO
I want to store each SQL Server's (prod, test, etc) connection string in XML to make deployment & configuration easy. However, the connection string must be encrypted. Does anyone have any ideas or suggestions on how to accomplish both. I know how to encrypt it but I dont think SSIS can work with it.
TABLEONE: SEC TABLETWO: B_DATE and E_DATE (they are numbers – not dates); RATE (is a number)
I first calculate the difference between E_DATE and B_DATE and between a chosen date and B_DATE creating the new columns GG_CED_C and GG_MAT_C. These new columns are calculated perfectly (they make the difference between the two dates according to the 30/360 date count convention) and don’t give me any problems. They are all filled with numbers. I’m able to use these new columns to make other calculations. For example I succeed in calculating RATE/GG_CED_C* GG_MAT_C but if I try to calculate RATE/360*GG_MAT_C AS RATEO the result is that in some records I have my new column RATEO correctly calculated, while in some other records there is no value in the column.
FROM ( SEC, B_DATE, E_DATE, RATE, GG_CED_C, GG_MAT_C
(YEAR2 - YEAR1)*360 + (MONTH2 - MONTH1)*30 + (DAY2 - DAY1) AS GG_CED_C, (YEARRIL - YEAR1)*360 + (MONTHRIL - MONTH1)*30 + (DAYRIL - DAY1) AS GG_MAT_C, FROM ( SELECT A.SEC, , B_DATE, E_DATE, RATE, TRUNC(B_DATE/10000,0) AS YEAR1 , TRUNC(MOD(B_DATE,10000)/100,0) AS MONTH1 , CASE WHEN MOD(B_DATE,1000)= 229 THEN 30 WHEN MOD(B_DATE,1000)= 228 AND MOD(TRUNC(B_DATE/10000,0),4) > 0 THEN 30 ELSE MIN(MOD(B_DATE,100),30) END AS DAY1, TRUNC(E_DATE/10000,0) AS YEAR2 , TRUNC(MOD(E_DATE,10000)/100,0) AS MONTH2 , CASE WHEN MOD(E_DATE,1000)= 229 THEN 30 WHEN MOD(E_DATE,1000)= 228 AND MOD(TRUNC(E_DATE/10000,0),4) > 0 THEN 30 ELSE MIN(MOD(E_DATE,100),30) END AS DAY2, TRUNC(&DRIL/10000,0) AS YEARRIL , TRUNC(MOD(&DRIL,10000)/100,0) AS MONTHRIL , CASE WHEN MOD(&DRIL,1000)= 229 THEN 30 WHEN MOD(&DRIL,1000)= 228 AND MOD(TRUNC(&DRIL/10000,0),4) > 0 THEN 30 ELSE MIN(MOD(&DRIL,100),30) END AS DAYRIL,
FROM TABLEONE A, TABLETWO C
What's happening? After what I've seen in this query I'm wondering if SQL is reliable when it comes to calculations (after all its main duty is to query data and not to make calculations among them)(?????.
I'm finding that the standard components often just don't quite meet my needs, but would only need some fairly minor changes to save me and my team a lot of work (and produce more elegant solutions). So I was just wondering whether the source code was available for the standard components that come with SSIS, or if there is anyway to extend their functionality? Or do you just have to start form scratch?
Recently in an SSIS package I am getting the following error for a particular Data flow task.
Error: 2008-01-25 12:01:48.58
Code: 0xC0202009
Source: Import Datasynapse Data User Events Source [3017]
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x8000FFFF.
End Error
Error: 2008-01-25 12:01:48.73
Code: 0xC004701A
Source: Import Datasynapse Data DTS.Pipeline
Description: component "User Events Source" (3017) failed the pre-execute phase and returned error code 0xC0202009.
End Error
Our guess is when the data size of User Events table is more it throws this error. If we try to transfer small subset of data it succeeds. What could be reason for this error?
Since this is very urgent, immediate response would be very much appreciated.
Right now, we have the production and development databases and ASP pages on one server. This is causing a strain on the performance, of course. Well, the finally, the customer agreed to get 2 less powerful servers. This is what we are planning to do with the 2 extra servers.
Server 1 : Original Server (fast) We will only have the Production Database (NO IIS)
Server 2: New Machine23 (faster box than Server 2) Offload IIS away from the production Database so we can allow more memory for Database processing.
Server 3: New Machine 3 (slower box we just got) Move the entire development environment to one server (SQL Server Database + IIS with all the ASP pages)
Does this sound OK? or should we have IIS sitting on one box (holding pages for Development and production) and the other machine just holds the Development database.
I would like to hear other opinions/stories on different IIS & SQL Server configurations to maximize performance.
I created a PDA application with a database, which has a table with a uniqueidentifier field and primarykey.
While doing the bulk insert from dataset into sql mobile database, It is inserting the record but it is not inserting the id which was entered into the sql server 2005 database, instead the id by creating a new id and the code is as below.
conAdap = new SqlCeDataAdapter(strQuery, conSqlceConnection);
SqlCeCommandBuilder cmdBuilder = new SqlCeCommandBuilder(conAdap);
This has probably been covered in other posts. I have been working with SSIS for the past month and I am trying to follow best practices on various items. Having worked with a different ETL tool prior to this, I am wondering what is the best approach to use for Connections and File Paths.
What I would normally do with DataStage for this would be to assign a Job Variable (and eventually Sequence Variable) of the type: Path. So, if I was developing a job I would create SourceFilePath, ErrorFilePath, etc. I would use these variables in a FlatFile or Dataset Stage. For instance I would assign a filename for a source as: #SourceFilePath#SourceFile1.txt. During execution the job would load the variable and then the filename would be: C:MyDocumentsDatafilesSourceFile1.txt.
When its time to move to another environment, I don't have to worry about changing values for file connections because it is managed dynamically by a config file or whatever method.
What is the best practice that emulates this behaviour for SSIS? I've been thick and can't get my head around this. Any direction to blogs or user sites would be great. Examples, even better!
After migrating our SQL 2000 DTS packages to SSIS I wanted to know what changes I will have to make to my VB code. Here is the VB code I use to execute 2000 DTS package. Anybody with VB or similar situation.... what changes I will have to make so that I can execute my SSIS (sql 2005) package.
I have a function written in vb and want to know if there is a way to use it in an ssis package? Is there a task that I can paste this function in and execute it? Like the vbscript task or T-sql task? The function opens up 2 tables and updates table1 recordset with data from table2 if the (if statement) is correct and It uses 2 do loops. Any ideas?
I'm hoping someone can help me out - at least by pointing me to who I can ask, if not answering the question directly. I have some encrypted values in a SQL Server 2000 database that I unencrypt and use in a website that I just converted from .NET 1.1 to .NET 2. The data is pulled from the database using standard ADO with no changes between the .NET 1.1 version and the .NET 2 version - yet for some data entries, when the identical value is pulled by the .NET 2 code it is changed or shorted. For example: 1.1 code traces out a value pulled from the db as: ᒪ࢖淨�d�把���媑쬹�䜻ꖉ��� The same value pulled from the database by .NET 2 looks like this: ᒪ࢖淨�d�把媑쬹�䜻ꖉ Do you know why the database value would be interpreted diferently by .NET 2 than by .NET 1.1? How can I bring this in sync so that both 1.1 sites and 2 sites can use the same data?
I was wondering if anyone has a neat (preferably automated) method of creating small testing databases from large production instances. My requirement would be to copy the schema and a subset of configuration data from a production database into a test database. The subset of data would be a full copy of a subset of tables, rather than a subset of data within one or more tables. There is a mixture of SQL2000 and SQL2005 servers involved in this requirement. I'm familar with the scripting mechanisms of Enterprise Manager and Management studio and DTS packages, sufficent to perform a process like this manually, but want to productionise and schedule this process to be performed automatically. I'm sure this must be a commonly performed task, so I'm interested to know if anyone has a "best practice" for this requirement.
Another DBA on my team is trying to tell me that it is a widely accepted best practice to only have a single version of SQL server in the enterprise environment. This seems counter intuitive to me and I cannot find a confirming source for this assertion.
I would think that it largely depends on your environment but that once a new version of SQL server has been decided on for a given application, for whatever reasons, that new version should be used moving forward on all new development.
It also seems like that as other databases grow and hardware comes to its end of life that older databases be either migrated or ported to the new version as seems appropriate.
Is there any reason an enterprise should always be on X version and only X version?
Hi,It appears that binary_checksum can give the same checksum fordifferent strings, which is a bit worrying. (I guess the algorithm isthe problem in the context of a repeating pattern.)e.g.select binary_checksum('A'),binary_checksum('AAAAAAAAAAAAAAAAA'),binary_checksum('AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA A'),binary_checksum('AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAA')My question...Is this approach to generating checksums adequate for managing theobject scripts in the SQL Server to ensure that they haven't changed. Iguess that the probability of somebody making a change to a script andending up with the same checksum is almost negligible. Has anybody usedthis approach in an FDA validated production environment, i.e. 'no ifs,no buts'? Would it stand up to scrutiny?Any experiences, thoughts?RegardsLiam
We have multiple development environments on one machine for our VisualStudio .net projects.These are web applications so we have organized them by IP Address andHost Headers.When I installed Reporting Services, it installed into my default website (localhost) and that is fine.However, we want to be able to create our reports in our different .netprojects. The only projects that will allow me to create reports arethe projects using the default website.When I right click to create a new project, the Business Intelligenceprojects (reports) type is not there.Can I do this? If so, how do I configure it?
I've seach the threads before. a lot of discussion about C# to VSA.
So could anyone with experience on conversion from C# to SSIS/VSA give me a suggestion?
I 've C# code application which complete some ETL process. Now I need to convert all the ETL process to SSIS pacakge. the function of part of C# code is to get the result from stored procedure and then write the result to excel file, not row by row.
the C# code call Microsoft.Office.Interop.Excel as a object to write excel file.
But in VSA, I can't add Microsoft.Office.Interop.Excel as a reference so I can convert C# to VB.net regarding writing excel file.
There is an int filed in my table called "WeekNo" and when I use order by WeekNo Desc, I am getting the following result. 9 8 7 7 6 5 4 3 2 18 17 16 15 15 14 13 13 12 11 10 10 1
This does not seem right, can anyone comment why i am getting this result.
I've been trying to find an answer to the mystery of how date conversions differ between SQL server and Excel. In Excel the number 37711 is displayed as the date '3/31/2003'. The same number in SQL server yields '2003-04-02' (I used the following: select cast(37711 as datetime) ).
Any idea what is going on here and how I might resolve this problem?
Dependingon the printer, the report prints differently. Sometimes it's all messed up on certain printers on others it prints fine. I see the problem mostly with older HP printers..
I have SQL stored proc that calls a CLR function. This function does a "select ... for xml" statement, manipulates the XML a little, and returns the manipulated XML to the stored proc.
This all works fine when I call the stored proc from a query window, but when I have BizTalk call the stored proc, the CLR function fails. I have a feeling this may have to do with BizTalk using MSDTC , but I am not sure.
Here's a code snippet from where CLR function fails:
SqlConnection conn = new SqlConnection("Context Connection=true"); conn.Open();SqlCommand cmd = new SqlCommand("Select * From Items FOR XML AUTO",conn);XmlDocument xdoc = new XmlDocument();xdoc.Load(cmd.ExecuteXmlReader());
Under BizTalk, the last line fails with: System.InvalidOperationException "Invalid command sent to ExecuteXmlReader. The command must return an Xml result."
Now to see why XmlReader doesn't like the returned data, i changed the last 2 lines of that snippet to this: SqlDataReader dr = cmd.ExecuteReader();
dr.Read();Object obj = dr[0];
If i have a breakpoint after that last line, obj is of type string when i call the proc myself, but it is a byte[] under Biztalk. If i look at the bytes themselves, its close to the expected xml...but with some nontext bytes sprinkled around. I can't seem to cast or encode the byte array into anything useful.
Anyone have any idea what is going on here? Why would the same code return different types based on a) who is calling it, or b) the type of transaction used?
I'm looking for the minimum date of an entry into a history table. The table contains multiple entries for the customer and the item with an activation and deactivation date for each entry.
I could use the following:
select customerId, item, min(activationDate) from history group by customerId, item
or a sub query
select customerId, item, activationDate
from history h1
where activationDate=(select min(activationDate) from history h2 where h2.customerId=h1.customerId and h2.item=h1.item)
How are these two queries parsed differently by SQL.
Same RDL, 2 different servers. I run the report on my computer and export to PDF, it prints properly. When the customer runs the report on their server (SSRS 2K5 SP1, same as mine), they get it displayed differently. The columns on the report extend to the next page and the lines are thicker.
Is this a formatting issue on the customer's PC? It uses standard fonts (Tahoma, Sans-serif).
We are upgrading a production database server to new hardware. The server is currently running SQL Server 2000 Standard Edition. We are thinking about installing SQL Server 2000 Enterprise Edition, however that would mean the test server (2000 Standard) and production server (2000 Enterprise) have different edtions of SQL Server. How much of a risk does this present? Later in the year we would upgrade test to SQL Server 2000 EE, but for a couple of months the environments would be different.
If I have stored procedures that reference linked servers, is there any way to avoid changing the procedures when moving between environments (dev, qa, prod)?
Do i resort to dynamic sql? Use case statements to switch between environments?
I modified an existing package created by someone (who left the organization). Its a ftp bulk insert from flat files to the tables. The need was to updates 2 tables out of 9 with additional columns. I made the changes to the package and it runs successfully on my laptop.
After that I build the package and tested it on the development server, although the package runs successfully but the data is written to my DB tables in my laptop instead of development server DB tables!.I see "." in the place of the connection manager server name. The DB name is the same in all the three environment (local, development & production). There is a configuration table used in the package.
Is there a way to simulate hosting three environments on one Report Server. Due to resource limitations, I would like to set up three environments DEV, TEST, STAGE on one report server such that the data source for each environment would be different. Can I use a different folder structure for each and then deploy the reports by providing server url accordingly?
e.g. http://servername/DEV_Reports or should it be (http://servername/ReportServer/DEV_Reports)
Is it possible to config environments ( dev,tst,prd ) in  VS Tabular solution , like it is done in SSIS ?  i find Variable tab , but i dont find the way to set a connection (string) to be change dynamically between the environments.
