Run Times Of SSIS Package Not Making Sense....HELP!
Feb 27, 2007
We have an SSIS package that was created to migrate data in from a few production databases. The steps for the package are as follows...
backup databases on server 1 (prod database server) restore database to SSIS server (server 2) .
truncate worker tables in SSIS server's (server 2) Main DB database.
copy data from restored db tables to working db tables ( database to database)
Start Multiple threads (15 ) and run steps from here in parrallel
Combination of Data flow tasks and SQL scripts and Stored procedures used to flatten data out and combine data for reporting purposes.
The average run time is 8 hours.
the issue we are seeing is this, the package will fluctuate in run times from 4 hours to over 11 with no change in the data or the underlying SSIS package. We have looked for any changes or things that would effect this but have not found anything that changed...
Also, certain steps are running shorter while others double in time. there doesnt seem to be any rhyme or reason to this behaviour. The server is x64 12GB of RAM 2 dual core 3.2Ghz.
Please let me know if you need any more information or specifics...
the only thing I have seen so far that looks out of place is Tempdb has one of its files that is 20+GB.
I have a few questions related to using CONTAINSTABLE in a query that I hope someone can help with.
I am working on a project to add document search capabilities to my companies product using fulltext indexing. Part of this requirement is an ability to breakdown the component parts of of the search query and provide information on *why* documentX ranked higher than documentY. This is a bit convoluted, but taking this (very simple) example - the user wishes to search for 2 skills - "HTML" and/or "XML". The generated query looks a little like :-
Select DOC.DOC_ID, RANK1.RANK, RANK2.RANK, RANK3.RANK from DOCS DOC inner join CONTAINSTABLE(docs, doc, 'HTML AND XML') as RANK1 on RANK1.DOC_ID=DOC.DOC_ID inner join CONTAINSTABLE(docs, doc, 'HTML') as RANK2 on RANK2.DOC_ID=DOC.DOC_ID inner join CONTAINSTABLE(docs, doc, 'XML') as RANK3 on RANK3.DOC_ID=DOC.DOC_ID
This returns the "overall" rank, and a rank for the 2 component parts, so I can say this doc ranked XXX overall because it scored "rank1" for HTML and scored "rank2" for XML etc....
My question on this part is about the values for the "overall rank". If the query contained an OR it always seems to return the highest of the "rankX" values, and if it doesnt, it returns the lowest. e.g. for the example for java and word and excel and access - the overall ranking is 2 , java=36, word=2, excel=16 and access=36 for java and word or excel and access - the overall ranking is 16 , java=36, word=2, excel=16 and access=36 for (java and word) or (excel and access) - the overall ranking is 16 , java=36, word=2, excel=16 and access=36
So in the first example, regardless of what the other values are, the rank returned is always 2 (the score for "word"). My resultset has 100ish rows, all with a rank of < 5 for word, but all with ranks of 18-100 for the other 3 values - yet the "overall" rank always matched the "word" rank.....?? This doesnt feel right to me somehow, I would expect a different value as if the document ranked really highly for one value but low for the other, it doesnt feel right the value is clamped to the lowest? Or am I just understanding it wrong? If I use "freetexttable" the overall rank is a little more meaningful - but unfortunately I also need to use weighting, which brings me to my next question . . .
This question is about rankings returned from the ISABOUT function. In the following example, select * from documents as DOC inner join containstable(docs,doc,'project') as doc0 on DOC.DOC_ID=doc0."key" inner join containstable(docs,doc,'ISABOUT (project weight (1.0))') as doc1 on DOC.DOC_ID=doc1."key" inner join containstable(docs,doc,'ISABOUT (project weight (0.5))') as doc2 on DOC.DOC_ID=doc2."key" inner join containstable(docs,doc,'ISABOUT (project weight (0.1))') as doc3 on DOC.DOC_ID=doc3."key" inner join containstable(docs,doc,'ISABOUT (project weight (0.0))') as doc4 on DOC.DOC_ID=doc4."key" order by doc0.rank desc
The values I get from the doc1/2/3/4.RANK columns dont seem right. In this example,
doc0.rank = 133 doc1.rank = 150 doc2.rank = 330
doc3.rank = 924
doc4.rank = 0
These values dont make any sense to me, as the rank seems to go UP when the documentation on ISABOUT says it goes down (I think it says somewhere the calculated rank is multiplied by the weight?). Once again, is there something I missed or am I understanding it wrong?
Thanks in advance for any help into understanding the whys of this...
I created an SSIS Package and once complete there is 4 instances of the same data. I checked the source and it only has 1 instance. Why is it pulling the data 4 times? It will first show 3 instances and then another will show up before the package finishes running. Is there a bug in the program or is there a place I'm not aware of where it sets how many times the package should run.
SQL Server 2012VS 2010 SQL Server data toolsFTP Connection Manager - port 21, chunk size 1kb, passive mode=false, saved plain text pwDownloads 1 csv file to local directory on SQL Server box.This always works when run from the package. And always gives a timeout error when scheduled.
Message Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Started:Â 11:04:59 AM Error: 2014-03-27 11:05:31.12 Â Â Code: 0xC001602A Â Â [code]...
SQL Server Agent is set to Logon As my domain account, and in the package history it says "logged on as " my account.
This is probably a st00p1d n00b question, but here goes:
I have a DTS package that I got from someone else, but it needs to be edited. It was set as a local package, but when I double-click it in Enterprise Manager and edit it, then save and quit, I can go right back in and changes are gone, as if it was never edited.
Well, lo and behold, I got the bright idea to edit the package using Notepad, then bring the package back into Enterprise Manager. Now it says it can't do it because it's no longer a DTS Package. Fortunately, I have another unmodified copy, or I would have really thrown out the baby with the bathwater.
So my question is: What do I need to use to edit the package offline, then be able to bring it back into Enterprise Manager as a valid DTS Package?
I am working on a configuration database in SSIS. One of the modules in the package is giving me endless amounts of grief. The module is for some reason set up to add data onto the end of any data already stored in the table. Instead, I would like it to replace this data. I have tried an Execute SQL Task that should delete all of the rows in the table, but this isn't working. Is there a more efficient way to do this?
I created a SSIS project with some SSIS packages within my local machine. Once all development and testing stuff was finished I imported the same to SSIS package store within Integration services. Then I created another test folder within my local machine and copied all the packages along with the project .sln file to that test folder.
Now the problem, If I make any changes to the package within test folder it automatically saves the changes to my other folder. Does anybody have a reason why it is doing so.
Hey, I've a few jobs which call SSIS packages. If I run the SSIS package, it runs fine but if I try to run the job which calls this package, it fails. Can someone help me troubleshoot this issue? None of my jobs that call an SSIS package work. All of them fail.
We have a package that runs a webservice. The webservice connects to exchange for multiple mail accounts so it sometimes takes awhile to run.
If run from SQL Server Management Studio, it usually completes, but may occassionally time out.
If run from a job (i.e. we want this scheduled) then it NEVER completes successfully - althought there is very little information about what the error was. It seems to fail too quickly to be a time out error...
Any idea why this is happening? Or how to prevent it from timing out? Is there a way to initiate the webservice asynchronously from SSIS?
Hello, I am fairly new to SSIS and do not have a wealth of 'in-depth' knowledge of it and I am running into some problems.
I am part of a team that is using Agile techniques (Scrum based) to implement a reasonably complicated system. This system has the requirement for numerous file feeds and multiple transactional database instaces whos base data is driven from a master seed database.
We are getting to a point where we are starting to hook up alot of the infrastructure and I am running into problems. Especially with the continous evolution of the database. Every iteration, the SSIS packes break because of DB changes. This is a little scary to me because I am starting to see SSIS as adding multiple points of failure to the system. Agile is based on refactoring often and that includes the database for this project. Every database change either breaks or requires a change to the SSIS packages that moves data from file feeds and staging databases to the master seed database. This problem is even more pronounced when considering the processes that move data from the seed database to the transactional databases. It discourages changing the database which is a really bad thing because that can lead to code archtecture smells and adds fragility.
I have a mixed solution with a C# project and an SSIS project. Now matter how many times I set the C# project as the startup project, it seems that as soon as I make a change to the SSIS package, it decides that it's so unbelievably important that it HAS to be the startup project. Is there any way to disable this?
I am importing xml multiple times a day from a vendor. However when SSIS created the ID's for nested XML data it is not unique. So importing the first time and I get 3-4 records it looks fine. However subsequent imports all use the same ID's so it isn't unique, how do I go about changing this as I cant find anything about it.
I have packages that must generate log errors dynamically including time of execution and the name of the task.
I make it changing the properties of the connection inserting two expressions.
1.-I alter the File Usage Type to 1 to generate this files. 2.- I alter the connection string as: @[User::myvariable] +"constant_description"+ time description+ +".txt"
The time description is : (DT_STR,40,1252) DAY (GETDATE())+"-"+(DT_STR,40,1252) MONTH( GETDATE())+"-"+(DT_STR,40,1252) YEAR( GETDATE())+" + REPLACE( (DT_STR,10,1252) (DT_DBTIME) GETDATE(),":","_") But it is not the problem
In those packages I have one connector for all the tasks and in execution time it creates one file for each of the tasks.
The problem is the way I insert in the filename the task name.
I have a pre-execute event handler in each task that modifies a string variable( myvariable) appending the task name. When I execute de package it works great but when I only execute a task, the program do not enter in the event and do not put the task’s name.
How can I put that name without using that handler? There is another handler can I use to do it that happens before the system generates the new file name and after pre-execute? Anyone knows another way to do this kind of things?
And there is a task (Execute SSIS package) in First package that calls the execution of second package.
I m continuously receiving an error "Failed to decrypt protected XML node "PackagePassword" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available."
As we are running first package by job, job runs successfully logging above error
The protection level of second package is set to "EncryptSensitiveWithUserKey"
I have inherited a SQL 2005 server with a few small databases on it. There's a maintenance plan here that doesn't seem to make a lot of sense to me. Can anyone comment:
Every Sunday at 4:00 AM
1. Reorganize index on All user database Tables and Views - compact large objects. 2. Rebuild index on local server connection, All user databases, Tables and view, Original amount of free space. 3. Shrink database. All user databases. Limit 100MB.
I'm confused a little about item 3. Won't a shrink be kind of useless after all of the work that goes on in steps 1 and 2. When I ran this manually, the transaction logs jumped significantly.
I have a SSIS job, one of the last steps it performs is to execute a SQL 2000 DTS package. This has to be done as a SQL 2000 DTS package as it is performing rebuilds of SQL 2000 Analysis Services dimensions and cubes. We've found that when the DTS fails the SSIS job is happily completing showing as a success, we would prefer to know it went wrong.
As far as I'm aware SSIS merely starts the DTS off and doesn't care about it's result. I've taken a look in to turning on the logging for the execute DTS package and thought that the ExecuteDTS80PackageTaskTaskResult would give me the answer I need...but is merely written to the log not available as an event-handler. It also looks like it is not safe to put a SQL task in as the next item to go look at the SQL 2000 system tables to look at the log for the DTS package as the SSIS documentation warns that the DTS package can continue to run after the execute DTS package task has ended.
Ideally I want any error raised within the DTS package to cascade up to be an error in the SSIS job, I can then handle it appropriately. I cannot find a way to do this. Is there a way?
If not, can anyone suggest how in the remainder of the SSIS tasks I can be sure that the DTS has completed before I start any other tasks that will check for the SQL 2000 log of its execution?
I have developed an SSIS package for ETL purpose. I am invoking the SSIS package through .Net console application by referencing the ManagedDTS Assembly. I am able to execute the package in Sql Server 2005 Developer Edition and it runs fine till completion.
But when i try to execute the packahe in Sql Server 2005 Standard edition, by invoking the package through .Net console application the status of the package is failure.
Can any one help me how to over come this problem.
I am in the process of moving from a 32-bit SQL Server 2005 Enterprise (9.0.3054) to a 64-bit SQL Server 2005 Enterprise (9.0.3054 with 4 CPUs and 8GB of memory on Win 2003 SP2) and the process has been very frustrating to say the least. I am having a problem with packages that I created on my 64-bit SQL Server. I am importing a few tables from the 32-SQL Server into the 64-bit SQL Server using the Task --> Import to create the package.
Sometimes when I am creating a package I get the following error in a message box:
SQL Server Import and Export Wizard
The SSIS Runtime object could not be created. Verify that DTS.dll is available and registered. The wizard cannot continue and it will terminate.
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.Windows.Forms)
Other times when I run a package that has run successfully before I get the following error:
Faulting application dtexecui.exe, version 9.0.3042.0, stamp 45cd726d, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x025d23f0.
The package appears to hang when running. By this I mean that the Package Execution Progress shows progress up to a point then it just stops. (The package takes about 17 seconds to run normally) CPU usage is at 1% and the package cannot be stopped.
I have deleted and re-created the package several times and I have also re-installed the service pack on the SQL Server (9.0.3054) but that did not help.
I would like to standardize SSIS development so that developers all start with the same basic template. I have set it up so it is an available template ( http://support.microsoft.com/kb/908018 ) but I would like it to be the default when a new project or package is created. Is this an option?
I would like to fetch the data flow component name while package is executing. Since system variable named [System::SourceName] only fetches name of the control flow tasks? Is there a way to capture them?
The master package has a configuration file, specifying the connect strings The master package passes these connect-strings to the child packages in a variable Both master package and child packages have connection managers, setup to use localhost. This is done deliberately to be able to test the packages on individual development pc€™s. We do not want to change anything inside the packages when deploying to test, and from test to production. All differences will be in the config files (which are pretty fixed, they very seldom change). That way we can be sure that we can deploy to production without any changes at all.
The package is run from the file system, through a job-schedule.
We experience the following when running on a not default sql-server instance (called dkms5253uedw)
Case 1: The master package starts by executing three sql-scripts (drop foreign key€™s, truncate tables, create foreign key€™s). This works fine.
The master package then executes the first child package. We then in the sysdtslog get:
Error - €œcannot connect to database xxx€? Info - €œpackage is preparing to get connection string from parent €¦€?
The child package then executes OK, does all it€™s work, and finish. Because there has been an error, the master package then stops with an error.
Case 2: When we run exactly the same, but with the connection strings in the config file pointing to the default instance (dkms5253), the everything works fine.
Case 3: When we run exactly the same, again against the dkms5253uedw instance, but now with the exact same databases defined in the default instance, it also works perfect.
Case 4: When we then stop the sql-server on the default instance, the package faults again, this time with
Error - €œtimeout when connect to database xxx€? Info - €œpackage is preparing to get connection string from parent €¦€?
And the continues as in the first case
From all this we conclude, that the child package tries to connect to the database before it knows the connection string it gets passed in the variable from the master package. It therefore tries to connect to the default instance, and this only works if the default instance is running and has the same databases defined. As far as we can see, the child package does no work against the default instance (no logging etc.).
We have tried delayed validation in the packages and in the connection managers, but with the same results (error).
So we are desperately hoping that someone can help us solve this problem.
l've written a cursor to koop through a table and then insert the last 100 records into a table.Reason why l want the last 100 records is to monitor and log the last 100 trans avery hr or so.
-- Declare the variables to store the values returned by FETCH. SET ROWCOUNT 100 DECLARE @customer_No char(15), @loan_No char(12), @date_Issued datetime , @maturity_Date datetime , @status int
DECLARE loan_cursor CURSOR FOR SELECT customer_No, loan_No, date_Issued, maturity_Date, status
FROM loan
OPEN loan_cursor
-- Perform the first fetch and store the values in variables.
FETCH NEXT FROM loan_cursor INTO @customer_No, @loan_No, @date_Issued , @maturity_Date, @status,
-- Check @@FETCH_STATUS to see if there are any more rows to fetch.
WHILE @@FETCH_STATUS = 0 BEGIN
-- This is executed as long as the previous fetch succeeds. FETCH NEXT FROM loan_cursor INTO @customer_No , @loan_No , @date_Issued , @maturity_Date , @status END
CLOSE loan_cursor
DEALLOCATE loan_cursor;
insert into Loan1 (customer_No, loan_No, date_Issued , maturity_Date , status )
select @customer_No, @loan_No, @date_Issued, @maturity_Date, @status FROM loan ORDER BY date_Issued desc;
I need any one's advice/imput on this...PLEASE!My computer will now begin the process of taking all the MS Access (NativeJet Engines - x30 total departmetns) and put the tbles/BE on SQL Server 2005and the Ms Access FE on MS Sharepoint.This is the kicker, say 20 out of the 30 (ball park) was created by oneperson and that is their whole job function was to create/maintain a QAtracking system and more.The person who created the 20 out 30 only knows intermediate ms access andsome vba, but NOT sql or net conversions (visual studio - all the differentlanguages), so the IT director asks me (I develop in MS Access andintermediate in VBA and can create web sites using publisher, front page andHTML) he asks me and this other person if we want to take on the challenge ofhelping him and the other IT guy in the conversion process of all of thesedb's.What does this do the developers who developed and still maintain thesecurrent 30 ms access db's, well you guessed it, it now takes all that hardwork that those developers did and still do (they still add more forms,updates) and it NOW takes the databases owners away from them and grant itnow to the person (s) who will maintain SQL Server 2005 ( I hope will be aDBA)???Is this true, once all the databases are converted, the owners will no longerbe able to go behind the scenes in tables, queries,etc.... It will now be inthe hands of a DBA?You know the funny thing is the IT Director wasn't even sure if he was goingto hire a DBA, who in the heck will maintain all of those db's on the server?There is only one other guy and he certainly does not have the training orskills or TIME.MY POINT QUESTION IS:when these conversion take place like this at a company, most of the time thems access dbs that have now be put into sql will now take the ownership awayfrom the owner (they cannot develop no more, unless they are sql friendly/dba)and put all of that into one persons hand (DBA) to maintain and development??????--Message posted via SQLMonster.comhttp://www.sqlmonster.com/Uwe/Forum...eneral/200606/1
Thanks to several guys here, I now understand how SQL Server configuration option works... Pretty nifty stuff.
Now, I'm trying to see if I can configure the Server property of the Connection Manager that holds the information for where my configuration table is. I thought about this and tried it, but it doesn't work. Then it occurred to me, this may not make sense to try to do because it is like the question, "what came first? the egg or the chicken?"
I am interested in Passing value from a child Package variable to the Parent package that calls it in ssis.
I am able to call the Child package using the execute package task and use Configurations to pass values from the parent variable to the child, but I am not able to pass the value from the child to the parent.
I have a variable called datasetId in both the parent and child. it gets computed in the child and needs to be passed to the parent...
Deployed Report having SSIS package as source do not work when Indirect Package configuration is used in ETL package. It seems ETL package when called/executed from Report manager does not recognize environment variable to pick up the dtsconfig file.
The Report works when Direct package configuration is used to same dtsconfig file.
What could be the reason? Any solution for this? This will cause our build/deployment to QA and Prod very difficult.
I now have two SSIS package, "TESTING" and "LOADING". The "TESTING" package have an execute package task that call the "LOADING" package. When I want to execute the TESTING package, how can I setup the connection string so that I can edit the password of the database connected by the "LOADING" package?
I have two SSIS packages in a project, one calling the other. The parent package works fine in my local mechine. After they are deployed to the production, I schedeul jobs to run the packages in the SqlServer. The child package works fine if I run it alone, but the parent package could not find its child package if I run the parent package . As I checked, all xml config files and the connection string pointing to the child package were set correctly. It seems the parent package did not use the xml config file. Can someone help me? Thanks in advance.
I have successfully created a SSIS package which execute a DTS 2000 package and with no problem to execute the task. But I failed to schedule this package. I was not success in setting the logging. When running the package in command line:
dtexec file "C:Documents and SettingslyangMy DocumentsVisual Studio 2005ProjectsTraingDTSTraingDTSDTSTraining.dtsx"
Error: 2008-03-24 08:03:24.36 Code: 0xC0012024 Source: Execute DTS 2000 Package Task Description: The task "Execute DTS 2000 Package Task" cannot run on this edit ion of Integration Services. It requires a higher level edition. End Error Warning: 2008-03-24 08:03:24.38
Code: 0x80019002 Source: DTSTraining Description: The Execution method succeeded, but the number of errors raised (2) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the M aximumErrorCount or fix the errors. End Warning DTExec: The package execution returned DTSER_FAILURE (1).
I am making use of the DtUtil tool to deploy my package to SQL Server. Following is my configuration: 32-bit machine and 32-bit named instance of Yukon.
I have some package variables which need to be set in the code.
Previously I did it as follows:
Set the package variables in the code. For example:
Here the package was successfully deployed and when i open those packages using BIDS, I am able to see that the variables are set to the values as doen in teh code.
Because of oen problem I am not using SaveToSQLServer method. So I switched to DTUtil tool. Now I am doing it this way:
Set the package variables as before. Deploy the package to SQL Server using DTUtil tool.
Now is the problem: The package is successfully deployed. But the variables are not set to the value that I have specified in the code.
I also tried DTexec utility to set the package variable. Even that does n't work. Can anyone help me out? Is there any alternate method to set package variables?
How can I know my sql server 2005 express service pack? The same goes for SSMSE. I can't find these information in the about box.
What about intelli sense for queries in SSMSE? Is it there yet or should I wait for sql server 2008 for dev. (http://channel9.msdn.com/Showpost.aspx?postid=387069)? I haven't seen the video but does anyone know any details? When it will be available, is it part of the sql server 2008 CTP, will it be available in the Express edition?
I am new to SSIS. i am trying to port database from SQL SERVER 2000 to 2005. i am using "Transfer SQL Server Objects" for this. i am just trying to move one object for testing wether it works or not. and it is not working. i am getting this error.
[Transfer SQL Server Objects Task] Error: Execution failed with the following error: "ERROR : errorCode=-1071636471 description=An OLE DB error has occurred. Error code: 0x80040E37. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E37 Description: "Invalid object name 'dbo.consta_AE'.". helpFile=dtsmsg.rll helpContext=0 idofInterfaceWithError={8BDFE893-E9D8-4D23-9739-DA807BCDC2AC}".
Both DBs are on seprate machines. of you need any more info please let me know. that would be great help.
