Usage Of Dtutil And Dtexec In SQL Server 2005 SSIS
Nov 22, 2006
1) We are migrating data from SQL Server 2000 to 2005 DB.
2) We are developing SSIS packages in client, build, deploy and install and copying to SQL Server 2005 server to execute for migration.
3) For this we encrypt all packages with password, then decrypt all while executing in a batch file using dtexec. This works fine.
4) What does not work is executing a master execute task in which are individual execute tasks. This is so since the other packages are still encrypted.
5) We decrypted them separately first using dtutil using Decrypt and Copy/Move option but while executing error is thrown that these other packages are still protected by password.
6) Do we need to decrypt, then encrypt using default package protection level and then decrypt again.
7) How exactly to just execute a master package when packages are developed on client.
I have a serious problem with my SSIS Package while executing using 32-bit DTExec and 64-bit DTExec.
Here are the details:
Environment:
Windows Server 2003 64-bit (Build 3790: Service Pack 2) SSIS 32-bit & 64-bit installed SQL Server 2005 (Microsoft SQL Server 2005 - 9.00.1399.06 (X64) - RTM)
SSIS Package details (compiled in 64 bit)
Script tasks only Microsoft Visual Basic .NET (using TRY...CATCH block) PreCompileScriptIntoBinaryCode = TRUE Run64BitRunTime = TRUE
Execution
Batch file that uses DTExec to execute the Package.
SCENARIO I am trying to exeucte the above SSIS package using both 32-bit and 64-bit DTExec to make it failure by providing invalid connection string. Here are the details,
Wrong connection String using 32-bit Execution
While establishing the connection the error message has been nicely captured in to my Exception block and writes into the log file.
Wrong connection String using 64-bit Execution
While establishing the connection the error has not been catpured anywhere (although I have TRY CATCH block) and it haults there itself with the message "Process is terminated due to StackOverflowException". Later I found that the error is due to the connection string along with the unhandled exception.
Please suggest any one of the following my findings, also if you have any other advice would be very much appreciated.
1. Shall I go ahead and fix the issue by handling those unhandled errors? (e.g Appdomain, application). I tried several but still not working using 64-bit DTExec.
2. Shall I go ahead and use 32-bit DTExec to execute the package? If so, is there any other major issue...like performance or anyother bug?
P.S: We cannot apply any service pack for SQL Server 2005 at the moment. Sorry abt it. If you have any specific hotfix for DTExec (without affecting SQL Server) then we can decide.
Sorry for the lengthy one and Thanks very much for you help in advance .
I have stumbled on a problem with running a large number of SSIS packages in parallel, using the €œdtexec€? command from inside an SQL Server job.
I€™ve described the environment, the goal and the problem below. Sorry if it€™s a bit too long, but I tried to be as clear as possible.
The environment: Windows Server 2003 Enterprise x64 Edition, SQL Server 2005 32bit Enterprise Edition SP2.
The goal: We have a large number of text files that we€™re loading into a staging area of a data warehouse (based on SQL Server 2k5, as said above).
We have one €œmain€? SSIS package that takes a list of files to load from an XML file, loops through that list and for each file in the list starts an SSIS package by using €œdtexec€? command. The command is started asynchronously by using system.diagnostics.process.start() method. This means that a large number of SSIS packages are started in parallel. These packages perform the actual loading (with BULK insert).
I have successfully run the loading process from the command prompt (using the dtexec command to start the main package) a number of times.
In order to move the loading to a production environment and schedule it, we have set up an SQL Server Agent job. We€™ve created a proxy user with the necessary rights (the same user that runs the job from command prompt), created an the SQL Agent job (there is one step of type €œcmdexec€? that runs the €œmain€? SSIS package with the €œdtexec€? command).
If the input XML file for the main package contains a small number of files (for example 10), the SQL Server Agent job works fine €“ the SSIS packages are started in parallel and they finish work successfully.
The problem: When the number of the concurrently started SSIS packages gets too big, the packages start to fail. When a large number of SSIS package executions are already taking place, the new dtexec commands fail after 0 seconds of work with an empty error message.
Please bear in mind that the same loading still works perfectly from command prompt on the same server with the same user. It only fails when run from the SQL Agent Job.
I€™ve tried to understand the limit, when do the packages start to fail, and I believe that the threshold is 80 parallel executions (I understand that it might not be desirable to start so many SSIS packages at once, but I€™d like to do it despite this).
Additional information:
The dtexec utility provides an error message where the package variables are shown and the fact that the package ran 0 seconds, but the €œMessage€? is empty (€œMessage: €œ). Turning the logging on in all the packages does not provide an error message either, just a lot of run-time information. The try-catch block around the process.start() script in the main package€™s script task also does not reveal any errors. I€™ve increased the €œmax worker threads€? number for the cmdexec subsystem in the msdb.dbo.syssubsystems table to a safely high number and restarted the SQL Server, but this had no effect either.
The request:
Can anyone give ideas what could be the cause of the problem? If you have any ideas about how to further debug the problem, they are also very welcome. Thanks in advance!
I believe I have made a complete install of Sql Server 2005 on my dev. machine, including all client tools, and can run SSIS packages from the dev. environment.
However DtExec refuses to run any package and always throws the "insufficient version" error (as does the "RunPackage" component, probably for the same reason).
I have repeatedly attempted to find something more to install from the Sql Server 2005 eval release that will allow DTExec to run without the "insufficient version" error, but can find nothing else to install from the eval. The install wizard (and the "change" wizard) indicates everything is installed and will not build a list of any additional components to install.
Are we sure that DtExec will install and run from a sql server 2005 eval distribution?
Licenses are in the pipeline (not that pipleine) but we need to start testing our SSIS solution in the meantime.
p.s. Kudos to the dev. team, it really is an excellent tool!
Is there any way to measure bandwith usage during merge replication between sql server 2005 and sql server mobile 2005 running on a cradled wm5 mobile device.
Attaching the windows performance monitor to the network connection established over usb would work although I was wondering if there was something specific for this case integrated into Sql server 2005 / sql server mobile 2005 / Sql server management studio / third party tools that i could use ?
We have an application that we currently run on SQL Server 2000 that works by creating a DTS package that it then executes.
Due to performance reasons, we have been considering switching to 2005, for a few reasons. Can anyone confirm clarify the following?
1) SQL Server 2000 caps RAM usage at 2GB, whereas SQL Server 2005 is only limited by the OS - RAM usage is a big current issue for us, so if upgrading to 2005 would solve this it would help a lot. Can anyone confirm my understanding of this? 2) Would using the legacy DTS in SQL Server 2005 take advantage of this RAM difference, or is it running on the old 2000 engine and only able to use the 2GB?
I am doing a project on web usage mining of my universities server logs and im just wondering how i go about mining them in sql server 2005?
Do i mine them in one table? do i normalise the web log data? what algorithms will i use on them as im trying to get usage patterns from the users and also where most of the users come from.
I have six stored procedures that I have to run hourly during the day. When I schedule them into separate SQL jobs (they must run concurrently), the percent CPU of Sqlagent90.exe process goes way up, eventually pegging out the overal CPU usage on the server. When I run the six stored procedures concurrently from query windows in SQL Server Management Sudio, the overall CPU usage stays very low.
Is there something about the SQL Server Agent that inherently adds CPU overheard? Thanks in advance ....
we are working on a release 2.0 mobile solution right now. In our version 1.0 we did not have to worry about memory issues as our application was the only application running on our target devices (e.g. T-Mobile MDA Compact II Pocket PCs, WM2005). Now we need to share the available memory with others. As our application relies on its SQL Server 2005 Mobile Edition database we are wondering about memory usage of that server.
We know that a Pocket PC divides its memory into Storage and Program. If our application uses a 5 MB database and 1.5 MB for DLLs and it's exe-file. These files reside in the storage space when not loaded. When the application starts up it is loaded in the program memory. What happens to the 5 MB database file? Is is loaded into Program memory as well? Are only portions of that file loaded? Or is nothing loaded at all?
Does anyone have a deeper insight into that server an can answer my questions.
I have recently installed SQL Server 2005 (Developer Ed) + SP1 onto a VMWare based Windows 2003 + SP1 server. SQL Server works fine when connecting to it using Mangement Studio on Windows XP. However, I have noticed strange CPU usage on the server which seems to be caused by Management Studio (either directly or indirectly). When no-one is connecting to the server using Management Studio, the server happily ticks along with CPU usage around 1-5% range. However, as soon as someone connects to the SQL Server instance using Management Studio the CPU usage begin to go up and down constantly. The CPU usage ranges from 5-50% and it goes up and down (fairly regularly) every few seconds. It does this even when nothing is actually being done in Management Studio. The moment Management Studio is closed, the CPU usage goes back to normal. The processes on the server that appear to be causing the CPU spikes are services.exe and wmiprvse.exe. On a possibly connected note (though possibly not), the Security log in the server's Event Viewer shows that there are logins occuring every minute or so (most of the logins are from my account). Any ideas?
I developed my package in BI studio and tested runnning it through BI. Now when i have to deploy the package to main server it will be executed via batch job. This package requires an input file name too. Now when i am trying to run the package on my loacal box using following command on command prormpt i get error given below.
COMMAND USED BY ME: DTExec /FILE 'C:ProjectIntegrationServicesSFRIntegrationServicesinpkgSFRLocation.dtsx' /SET Package.Variables[InputParamFileName].Value;'C:DataDumpInputLOC00022'
ERROR MESSAGE: Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 1:29:54 PM Error: 2008-02-22 13:29:54.96 Code: 0xC0011007 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Unable to load the package as XML because of package does not ha ve a valid XML format. A specific XML parser error will be posted. End Error Error: 2008-02-22 13:29:54.96 Code: 0xC0011002 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Failed to open package file "'C:ProjectIntegrationServicesSFR IntegrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The speci fied path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage o r the XML file was specified and has an incorrect format. End Error Could not load package "'C:ProjectIntegrationServicesSFRIntegrationServices inpkgSFRLocation.dtsx'" because of error 0xC0011002. Description: Failed to open package file "'C:ProjectIntegrationServicesSFRInt egrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The specifie d path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of eit her providing an incorrect file name was specified when calling LoadPackage or t he XML file was specified and has an incorrect format. Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Started: 1:29:54 PM Finished: 1:29:54 PM Elapsed: 0.047 seconds
PLease let me know solution to creating batch job for executing this pkg with input param.
It looks like you can run an SSIS package via the DTEXEC command. You can also specifiy a configuraiton file flag on the command. Is there anyway to specify a configuraiton filter if you stored it into a table vs and xml file?
I've worked for awhile to get an SSIS package that connects to Oracle to succesfully run via DTExecUI from a 64 bit SQL Server. I completely understand that DTExecUI runs in 32 bit mode, but I need to initiate the package from a stored proc so I need to be able to utilize the 32 bit DTExec from either a xp_cmdshell call or from an sp_start_job (thanks to MatthewRouche for the suggestion). Both Ora92 and 10g were installed on the SQL Server.
Here's what I've tried unssuccessfully
1. SQL Job - Just calling the package from an SSIS step. I wouldn't expect this to run because it is 64 bit DTExec. 2. SQL Job - A job that calls the 32 bit version of dtexec from cmdexec. Here is 2 examples I have tried, the first is because I came across several postings saying that there is a problem with the "(x86)" for the Oracle driver.
\server_nameE$ProgramFilesLegacyMicrosoftSQLServerX8690DTSBinnDTExec.exe /DTS "PackageName" /SERVER "SERVER_NAME" /SET Package.Variables[PLT].Value;"8H" /CONFIGFILE "\SERVERConfigFile.dtsConfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW
\server_nameE$Program FilesMicrosoft SQL Server (x86)90DTSBinnDTExec.exe /DTS "PackageName" /SERVER "SERVER_NAME" /SET Package.Variables[PLT].Value;"8H" /CONFIGFILE "\SERVERConfigFile.dtsConfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW
Both of these give me the error below:
System.Exception: System.Data.OracleClient requires Oracle client software version 8.1.7 or greater.
I'm assuming that it is running in 32 bit mode because this was at the beginning of the job history log:
Executed as user: "XXX". ....3042.00 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved
3. I then tried to call the package from a batch file on a file server that had integration services installed on. Got the same error as above.
I'm obviously missing something if I can get it to work in DTExecUI but not DTExec 32-bit.
Any ideas? Below are some postings I have already visited several times:
Windows 64 bit and Oracle http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=358365&SiteID=1
Connecting to Oracle on 64-bit (x64) machine http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=159581&SiteID=1
One paragraph that stands out on this posting is:
Find the location of the application that is generating the error. Check the path to this location and see if it contains any parenthesis. If so, you must relocate the application to a directory without any parenthesis in the path. This didn€™t work for me, unless I was doing it wrong.
Error retrieving numeric data from Oracle 10g on 64 bit Itanium SSIS server http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1183296&SiteID=1
Connection Problems to Oracle Database http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1307460&SiteID=1
If this posting was complete, it looks like what I was looking for.
I developed my package in BI studio and tested runnning it through BI. Now when i have to deploy the package to main server it will be executed via batch job. This package requires an input file name too. Now when i am trying to run the package on my loacal box using following command on command prormpt i get error given below.
COMMAND USED BY ME: DTExec /FILE 'C:ProjectIntegrationServicesSFRIntegrationServicesinpkgSFRLocation.dtsx' /SET Package.Variables[InputParamFileName].Value;'C:DataDumpInputLOC00022'
ERROR MESSAGE: Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 1:29:54 PM Error: 2008-02-22 13:29:54.96 Code: 0xC0011007 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Unable to load the package as XML because of package does not ha ve a valid XML format. A specific XML parser error will be posted. End Error Error: 2008-02-22 13:29:54.96 Code: 0xC0011002 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Failed to open package file "'C:ProjectIntegrationServicesSFR IntegrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The speci fied path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage o r the XML file was specified and has an incorrect format. End Error Could not load package "'C:ProjectIntegrationServicesSFRIntegrationServices inpkgSFRLocation.dtsx'" because of error 0xC0011002. Description: Failed to open package file "'C:ProjectIntegrationServicesSFRInt egrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The specifie d path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of eit her providing an incorrect file name was specified when calling LoadPackage or t he XML file was specified and has an incorrect format. Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Started: 1:29:54 PM Finished: 1:29:54 PM Elapsed: 0.047 seconds
PLease let me know solution to creating batch job for executing this pkg with input param.
I'm running several SSIS-packages in stored procedure using dtexec. Actually, there is a software that is running the procedure, but that's not important here.
The problem: if some of the packages failes, the whole procedure does not fail and I don't know if all the packages are successfully completed.
Here is a sample list of ssis-packages from procedure:
Hello sql and .net gurus :-)I have a problem with my website www.eventguide.it. It's completly developed under .NET 2 and SQL Server 2005 Express. My problem is the folowing:The server is a Intel 3Ghz HT processor with 1GB Ram. No other page on the running system is a CPU consuming site. We optimized the SQL statements, the code, the caching and many other parts of the website (pooling on SQL access), but the SQL Server uses about 50% to100% of the CPU and about 400MB RAM all the time. The whole site seems to be very, very slow. In fact there are many of SQL operations on every page request, but we cache a lot of them in different ways (page output caching, application caching). So I don't understand we have so much performance problems. Any suggestions for optimised code in general? I read nearly all of the MS .NET performance papers - but real world experience is the missing part :-) It is better to cast the values of a SQL reader like thisDim String1 as String = Ctype(DataReader.item(0), String)Dim Integer1 as Intger = Ctype(DataReader.item(1), Integer)or like thisDim String1 as String = DataReader.item(0) Dim Integer1 as Intger = DataReader.item(1)Thanks a lot for your help!FOX
We made SSIS package in dev environment in windows 2008R2 and SQL server 2012. Same packages were placed on SAN disks in cluster environment and are invoked with security contexxt of admin user and wih dtexec utility ( we call this using an sp). rarely but procedure completed i see the log text file by SSIS package and find the return code 5 means package is unable to load. i canst found what are the exact reasons for return code 5.
In one of my SSIS package (which connect to DB2) i save the DB2 username and password in SSIS packge itself by using option "EncryptAllWithPassword". When I try to run the packge using dtexecui.exe it works perfectly fine. It prompts me for package password and then run fine.
Now I am trying to run it using command line dtxece.exe and i get fooolwoing error. Please suggest and help!!! DTExec.exe /FILE C:pkgDB2Load.dtsx /CONFIGFILE C:ConfigpkgConfigMaster.dtsConfig /Decrypt mypassword /CHECKPOINTING OFF /REPORTING EWCDI
Error Message: ---------------------- Description: An OLE DB error has occurred. Error code: 0x80040E21. An OLE DB record is available. Source: "IBM OLE DB Provider for DB2 Servers" Hresult: 0x80040E21 Description: " SQL30082N Attempt to establish connection fa iled with security reason "17" ("UNSUPPORTED FUNCTION"). SQLSTATE=08001 "
Our SSIS packages use the Web Service Task to call services to send email and write package failure data to a department wide database. These Web Service Calls are failing with HTTP 401 errors. It was caused by the passwords for the HTTP connections not being saved when the SSIS pkgs were saved to .dtsx files. I have tried saving the package with a password and the EncryptSensitiveWithPassword protection option. This password can be supplied when the package is called from another package or a package is executed in Visual Studio but cannot be supplied to DTexec to execute the package in a job. DTexec does have a /Password parameter but it is rejected if the package is loaded from a .dtsx file. This appears to be a bug in DTexec. It only accepts the /Password parameter when the package is loaded with the /SQL option. Specifying /Password and /File is not supported.
Is this a known bug? Are there any workarounds? Has anyone successfully called a Web Service from SSIS executed via DTexec?
I have a simple package in SSIS that reads approx 2.5million rows from SQL2k5 source and exports it into 4 Foxpro files. The way I do it is as below 1) use and oledb source to read data from SQL2k5 2) use a conditional split component to divide the source into 4 streams ( I do a recId % 4 = 0, 1, 2, 3 to give 4 streams) 3) use an OLEDB destination to export to the 4 prexisting foxpro tables.
This thing runs amazingly fast when I run(F5) it from Business Inteligence studio, but it is atleast 10 times slower when rightclick the package and execute using dtexec.exe
I can't figure out why! any clues??
I would elimiate the 64 bit /32 bit problems because export to foxpro only works in 32 bit mode and my package has been configured to run in 32 bit mode.
In my quest of writing a generic SSIS package programatically, I have another block.
When I execute the exe seperately it consumes more than half of the CPU, some times its more than 90%. Is there any way where I can say how much of CPU to be used?
Anyone aware of anything I can look at to improve performance of my SSIS packages? i basically have two master packages that call about 15 child package tasks each. All the package tasks are called sequentially and all they do is just load flat files into temporary tables and execute a simple stored proc. Each package tasks check to see if a file is there and if so it processes it else it just ends. The packages are scheduled to run every minute to check for files. If I watch the CPU during execution i can see the two instances of DTSEXEC.exe running and consuming about 25-30% of the CPU. The box that this runs on is a Quad processor with 4 GB of ram.
Any ideas on how to improve the performance of this? I have tried playing with the transaction level, but that didn't seem to do much performance wise.
I am facing 2 problems : PROBLEM 1 : We have a few packages that run pretty fast on a desktop server with 2 Gig RAM, Dual processor (approx 4-5 hours). But the same packages run very very slow on the another server containing 8 CPU and 12 Gig RAM (ran for 24 hours without completing).
PROBLEM 2 : The CPU% ranges from 40-80% and the PF usage is stagnant at 2GB on desktop server for the same package. But in the 8CPU server, the CPU % ranges from 0-10% but the PF Usage raises from 750 MB to 8 GB.
I am using a script to copy a number of packages (50+) from my desktop computer (file system) to a remote server (MSDB). The script works fine, but I can never get through the entire script without it hanging up. Some times only 3-4 packages get copied, sometimes a lot more. If I close the windows cmd window and restart at the point it hung, it continues, but usually hangs up again. If you repeat this 2 or 3 times, it eventually gets all of the packages copied. It does not always hang at the same point. I have tried doing the copies with "quiet" mode - overwriting the existing packages and I have tried first deleting the packages, then copying - doesn't make a difference. I have also tried moving the packages to a couple of different installations of SQL Server 2005 using the script - the remote one and the local one on my desktop - same thing on both.
If I have the script echo a message just after each copy, the message just after the last successful copy never appears (and I know the copy is successful, because I deleted all the packages first and have checked that the package is there after the script hangs).
i have to check whether SSIS package is present in MSDB or not.
I am using DTUTIL for this perpose . Command goes like this.
dtutil /SQL Package_for_testing /Exists
Error(0xC0014014) while checking for the existence of package "Package_for_testing" on SQL server
this is giving the error "The specified package does not exist. Description: The ExistsOnSQLServer method has encountered OLE DB error code 0x80 004005 (Login timeout expired). The SQL statement issued has failed. Source:"
But the package is there in MSDB when i checked manually.
I want to know what this error signify? and how i can solve it. With advance thanks Srinivasa Mahendrakar
I have a 64 bit Itanium SQL Cluster that I'm trying to set up. The database engine seems fine, but I'm trying to install some SSIS packages using dtUtil and I keep getting a connection error. The error is as follows
"Login timeout expired
An error has occured while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default seetings SQL Server does not allow remote connections.
Named Pipes Provider: Could not open a connection to SQL Server [2}. (Microsoft SQL Native Client)"
Also, in Management studio, I can make a connection to my SSIS Server, but when I go down to the MSDB folder to look at packages I get the same connection error.
I think I was able to deploy a package using the Deployment Manifest created by VS, but I'm not sure because I can't see the package in Mgt Studio.
It almost seems like it may be related to the cluster. I'm connected in to one of the Servers using Terminal Services. But I am trying to connect to the Cluster name in Mgt Studio.
