Cloud Of Confusion: Running SSIS Packages On 64 Bit SQL Server
Sep 25, 2007
It is clear to me that in order to be able to use certain SSIS components (for example the Excel jet provider) I must launch my packages using the 32bit DTEXEC located at Program Files (x86)Microsoft SQL Server90DTSBinn. However, when I do this it seems that there are other components of the package that no longer work as expected.
To test this I have created a simple package with two tasks (Run64BitRuntime is set to False):
1. Data flow task importing data from Excel
2. Execute SQL Task which does a simple select (select 1) from a Native OLE DB SQL data source (same SQL Server on which packages are stored). This task contains no input or output parameter.
When I try to execute the package using the 64bit DTEXEC, task 1 fails with the following error (as expected):
Code: 0xC0202009
Source: connection1 Connection manager "SourceConnectionExcel"
Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040154.
An OLE DB record is available. Source: "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered".
When I execute the package with the 32bit DTEXEC, task 2 fails with the following error
Code: 0xC002F210
Source: Execute SQL Task Execute SQL Task
Description: Executing the query "" failed with the following error: "Attempted to read or write protected memory. This is often an indication that other memory is corrupt.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
Now here is the confusing part: When I change task 2 to use the .Net provider instead of the OLE DB the package works fine. According to the MS documentation, both of these providers are supported on 32 and 64 bit so am I missing something? One more thing to note: before I was able to use the 32 bit DTEXEC I had to re-register it as described in this KB article: http://support.microsoft.com/kb/919224
I have stumbled on a problem with running a large number of SSIS packages in parallel, using the €œdtexec€? command from inside an SQL Server job.
I€™ve described the environment, the goal and the problem below. Sorry if it€™s a bit too long, but I tried to be as clear as possible.
The environment: Windows Server 2003 Enterprise x64 Edition, SQL Server 2005 32bit Enterprise Edition SP2.
The goal: We have a large number of text files that we€™re loading into a staging area of a data warehouse (based on SQL Server 2k5, as said above).
We have one €œmain€? SSIS package that takes a list of files to load from an XML file, loops through that list and for each file in the list starts an SSIS package by using €œdtexec€? command. The command is started asynchronously by using system.diagnostics.process.start() method. This means that a large number of SSIS packages are started in parallel. These packages perform the actual loading (with BULK insert).
I have successfully run the loading process from the command prompt (using the dtexec command to start the main package) a number of times.
In order to move the loading to a production environment and schedule it, we have set up an SQL Server Agent job. We€™ve created a proxy user with the necessary rights (the same user that runs the job from command prompt), created an the SQL Agent job (there is one step of type €œcmdexec€? that runs the €œmain€? SSIS package with the €œdtexec€? command).
If the input XML file for the main package contains a small number of files (for example 10), the SQL Server Agent job works fine €“ the SSIS packages are started in parallel and they finish work successfully.
The problem: When the number of the concurrently started SSIS packages gets too big, the packages start to fail. When a large number of SSIS package executions are already taking place, the new dtexec commands fail after 0 seconds of work with an empty error message.
Please bear in mind that the same loading still works perfectly from command prompt on the same server with the same user. It only fails when run from the SQL Agent Job.
I€™ve tried to understand the limit, when do the packages start to fail, and I believe that the threshold is 80 parallel executions (I understand that it might not be desirable to start so many SSIS packages at once, but I€™d like to do it despite this).
Additional information:
The dtexec utility provides an error message where the package variables are shown and the fact that the package ran 0 seconds, but the €œMessage€? is empty (€œMessage: €œ). Turning the logging on in all the packages does not provide an error message either, just a lot of run-time information. The try-catch block around the process.start() script in the main package€™s script task also does not reveal any errors. I€™ve increased the €œmax worker threads€? number for the cmdexec subsystem in the msdb.dbo.syssubsystems table to a safely high number and restarted the SQL Server, but this had no effect either.
The request:
Can anyone give ideas what could be the cause of the problem? If you have any ideas about how to further debug the problem, they are also very welcome. Thanks in advance!
Is there any way to check from SQL code if a specific SSIS stored package is running like a SQL system view or stored procedure? Management studio option works fine but not if you need to determine if a specific package already is running or not?
When I run an SSIS package from the MSDB in SQL Server Management Studio it never shows up under 'Running Packages'.
I have confirmed that the package is indeed running.
I've also tried to check the packages running on the server programatically by using the GetRunningPackages() method. This too returns nothing when clearly packages are running.
if I have some packages (.dtsx) that are depolyed to Server A and I have SQL Agent installed on Server B - if I run a run a package from SQL Agent, does it run on Server A or Server B?
Has anyone monitored the execution of SSIS packages with MOM? Are there extreme benefits over just utilizing the built in execution and event logs, as well as the Windows Event Viewer?
What is the recommended way to monitor SSIS execution?
I have a series of SSIS packages which populate 12 different databases. Which data source & target they use is controlled by values passed down from the SQL Agent Scheduler Job Step using "Set Values"
These values are passed to Variables which are used in the Expressions in the connection manager for the database to change the connection string and initial catalog.
e.g. REPLACE( @[User::ConString] , "Royalty", @[User::RoyDb] )
These jobs run successfully the majority of the time but each day I get a significant number of failures where one or more target or error trapping table cant be found. They are all using the same connection manager and most of the tables in the database get updated correctly but the job fails on trying to access one or two of the tables, with the following message in the On Error event:
-1071636248,0x,Opening a rowset for "[dbo].[RIGDUE_DAY]" failed. Check that the object exists in the database.
This happens both when I schedule the jobs in parallel with other jobs running the same packages & when I run the job by itself using the right click, start at step option. e.g.
I had one fail last night, I ran it by itself this morning, it failed, I ran it again, it succeeded. Nothing concerning the data it was transforming had changed.
I have applied the hotfix to service pack one concerning the paralel use of variables in a package as referred to in the following link.
We are using SQL Server 2005 Standard edition. We have the development studio installed on a developer€™s workstation. The SSIS package is accessing network folders using UNC and the tasks in packages are accessing SQL Server with a SQL Server id.
The SSIS package runs fine on the workstation. We deploy the SSIS package as a SSIS store on SQL Server. I receive Error 18456 Severity 14 State 8 error when I run the SSIS package as a SQL Agent job.
When I run the SSIS package from the Integration Services Management Studio, I get The AcquireConnection method call to the connection manager "dboSQLabm" failed with error code 0xC0202009.
I€™m sure this is a permission based problem since it runs fine on my workstation. I tried logon onto the SQL Server with my id (I€™m an admin to the server). I tried running the agent with my windows id. It looks like I€™m getting problem connecting to the SQL Server tables since the logs are properly written to the network drive.
In SQL DTS 2000, all we needed to do was give the SQL agent id permission to the network folder. I tried to search the forums, but none of the solutions seem to work for my situation.
I seem to be missing something with SQL 2005 SSIS and I am wondering how I would accomplish my taks. With SQL 2000 and DTS we designed our DTS packages so that we could load one file or database from start to finish. We then have controlling programs written in VB that will monitor directories, when a file shows up in a directory it loads the package, changes the necessary properties and executes it. These programs handle all of our error notifications and dealing with the files afterwards, making sure files are complete in the directory before the DTS runs, etc. With .NET and SQL 2005 I have learned that I can change the properties by using variables and expressions, which I think would handle all of our situation but I can not run them remotely. The documentation says that I could setup a scheduled job and then call sp_start_job from my program. That seems like it would require a lot of scheduled jobs and then I still need to get the parameters into the packages. The other option is to use a web service, that would require IIS on my SQL box which everywhere says I don't want. Am I using SSIS for something I am not supposed to be or how do I accomplish this in 2005? I have noticed that you can not even run packages through the SQL Server Management Studio. We have a number of packages that are built that are run on demand when needed. In SQL 2000 I could just click and execute the package. In 2005 is my only option to setup some sort of fake job in the job scheduler?
I need some help from you on a problem with SQL Server Agent that I believe many of you have experienced. When searching on the web I find some loose answers, but it would not solve my problems. So here is my story.
1. I have created two SSIS packages, package 1 and package 2. Both packages have a Execute SQL Task making a select sql statement to a table, say Customer in AdventureWorks DB. Package1 also makes a Execute Pacakge Task to Package 2. Both packages have protectionlevel property set to EncryptSensitiveWithUserKey
2. Then I created two package configuration files per package, one indirect xml configuration and one xml configuration file. The former I created an environment variable called pack1, which was specifying the path and file name of second config file. Likewise, I did for package2. Maybe I could have reused thie first configuration file, but I am not sure since package1 calls package2.
3. Now I build and deploy my packages. Then I run my manifest file, going through the deployment wizard setting the necceccary properties like server name, user, password, I have also create the same environment variables on the deployment machine. I ensure that the package1 and package2 can be run under integration services (file system). And both package works. So far so good
4. Now I want to create a SQL server agent job that runs package1 and package2. Here I found an article that decribed How to Schedule and Run a SSIS package ( DTS ) Job in SQL Server 2005 http://www.codeproject.com/useritems/Schedule__Run__SSIS__DTS.asp
5. I created a job executor account and then Create SQL proxy account and associate proxy account with job executor account as described. I created a new ssis job, specifying type ssis integration services, run as proxy, setting package source to SSIS package store, naming the server and package name (package2).
6. Now, I try running job1 which runs package2. ANd it works. I was aware if I needed to give file permission on users (read and execute) on my dtsx/dtsconfig/log files. So far so good
7. Now, I try to run job test which runs package1 (which also runs package2). But now the job fails :-(. The job runs for some seconds, but fails. I have to say that I have some script tasks with message boxes inside!!! hope this doesn't matter. I have created all neccessary accounts, proxies, credentials and so on. here is the view history
Date 17.10.2006 00:50:53 Log Job History (test)
Step ID 1 Server ABEO-GEIR Job Name test Step Name test Duration 00:00:07 Sql Severity 0 Sql Message ID 0 Operator Emailed Operator Net sent Operator Paged Retries Attempted 0
Message Executed as user: abeo-geir as_jobuser. The package execution failed. The step failed.
Continue: This error is well known for you. It has been mentioned in many postings. Well, how do I go from here. Setting logging on the job results in only this information
PackageStart,ABEO-GEIR,ABEO-GEIR as_jobuser,Package1,{3F5BD166-D24B-4838-B3F1-018B6C2C7523},{8E1B8E45-7F1D-4C24-852C-2694F993E6D2},17.10.2006 00:59:40,17.10.2006 00:59:40,0,0x,Beginning of package execution.
PackageEnd,ABEO-GEIR,ABEO-GEIR as_jobuser,Package1,{3F5BD166-D24B-4838-B3F1-018B6C2C7523},{8E1B8E45-7F1D-4C24-852C-2694F993E6D2},17.10.2006 00:59:41,17.10.2006 00:59:41,1,0x,End of package execution.
Well, I read some place that I could run the job as type operating system (cmdexec). Here i could not choose Run as, but had to select SQL SErver agent account. NOw I got this message
Date 17.10.2006 01:05:21 Log Job History (test)
Step ID 0 Server ABEO-GEIR Job Name test Step Name (Job outcome) Duration 00:00:00 Sql Severity 0 Sql Message ID 0 Operator Emailed Operator Net sent Operator Paged Retries Attempted 0
Message The job failed. The Job was invoked by User sa. The last step to run was step 1 (test).
As you see, I can't come any further. I NEED HELP FROM YOU. I AM STUCK. I know many of you have solved this and I hope you can guide me through thise. I don't know how many hours or days I have used on this.
I have created ssis package. It has been successfully running at UI level.
But when i am trying to execute it from the command prompt by using dtexec utility it is showing the following error messages.
Error: 2005-12-23 17:01:57.67 Code: 0xC00470FE Source: Data Flow Task DTS.Pipeline Description: The product level is insufficient for component "Flat File Source" (1). End Error Error: 2005-12-23 17:01:57.67 Code: 0xC00470FE Source: Data Flow Task DTS.Pipeline Description: The product level is insufficient for component "Script Component" (9). End Error
I have a SQL Scheduled Job in SQL 2005 SP2 that executes a few different SSIS packages. When I update the SSIS Packages, SQL Agent runs the job and shows success; however, the underlying packages are not executed. The packages are located on the server but we are using UNC paths in the jobs since much of the time they are started manually from a remote workstation and the jobs seem to look at the initiating workstation's file server rather than that of the SQL Server. Is this a known issue?
I'm running several SSIS-packages in stored procedure using dtexec. Actually, there is a software that is running the procedure, but that's not important here.
The problem: if some of the packages failes, the whole procedure does not fail and I don't know if all the packages are successfully completed.
Here is a sample list of ssis-packages from procedure:
We are in the process of upgrading to a new SQL 2012 server but we have many packages that load data from dbf files created with FoxPro into one of our databases. We have not converted the packages and run them with DTS but we get the following error:
Error: 2013-08-26 11:05:27.36 Code: 0xC0209303 Source: BenchmarkLoad Connection manager "OLEDB NPIONE.Investment.middleTierSQL" Description: The requested OLE DB provider SQLNCLI.1 is not registered. If the 64-bit driver is not installed, run the package in 32-bit mode. Error code: 0x00000000.
[Code] .....
I searched for OLEDB and ODBC drivers for SQL 2012 64 bit but cannot find any that is newer than the 2005 that we have. What can I do short of changing the source files to overcome this issue?
I have a ssis package that has multiple large lookups without memory restriction. When running the package manually from SSMS on the same server it runs on when running automatically under the job agent, the package errors out when the server memory gets depleted by the loading of the large lookup reference data. One of the messages I get is "An out-of-memory condition prevented the creation of the buffer object. "
Anyway, the package runs successfully when it runs automatically under the job agent.
I was curious as to why the above happens. Is that a bug or is the run time behavior different under these 2 environments by design.
We manage some SSIS servers, which has only SSIS and SSIS tools installed on them and not the sql server DB.
SSIS packages and configuration files are deployed on a NAS. We run the SSIS packages through DTEXEC by logging in to the server.
We want to allow developers to run their packages on their own on the server, but at the same time we dont want to give them physical access on the server i.e we do not want to add them into RDP users list on server properties. We want them to allow running their packages remotely on the server.
One way We could think of is by using powershell remoting and we are working on that. But is there any other way or any tool already present for the same.
I have created a big list of packages, some calling others. They all work fine from my computer using Visual Studio.
When I try to deploy them (building them with deployment turned on and running them either directly from Management Studio or as a job) I get the errors with the password of connection strings. From what I read so far its the encryption process that kills it.
I have tried to add a password to some packages, but it still didnt work (only when run directly on my computer in management studio after deploying to SQL Server, but not as a job).
I have tried to change ProtectionLevel to SecurityStorage, wouldnt let me save in Visual Studio (I understand it is ot allowed in VS because you are saving to File System, how the hell am I supposed to save it to anything else? why is it showing there if its not even valid?).
If anyone can please give me the steps to doing it properly, that would be awesome. I simply need to run the packages from SQL Server! thats all! I have no idea why it has to be soooo difficult :/
I've run into a problem with SSIS packages wherein tasks that write or copy files, or create or delete directories, quit execution without any hint of an error nor a failure message, when called from an ASP.NET 2.0 application running on any other machine than the one where the package was created from. By all indications it appeared to be an identity/permissions problem.
Our application involves a separate web server and database server. Both have SQL Server 2005 installed, but the application server originally only had Integration services. The packages are file system-deployed on the application server, and are called using Microsoft.SqlServer.Dts.Runtime methods. For all packages that involve file system tasks, the above problem occurs.
When the above packages are run using the command prompt (either DTEXEC or DTEXECUI) the packages execute just fine. This is expected since we are using an administrative account. However when a ShellExecute of the same command is called from ASP.NET, the same problem occurs.
I've tried giving administrative permissions to the ASPNET worker process user to no avail.
I have likewise attempted to use the SQL Server Agent job approach but that approach might not be acceptable for our clients since it means installing SQL Server 2005 Database services on the application server.
I have read the relevant threads in this forum, namely http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1044739&SiteID=1 and http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=927084&SiteID=1 but failed to find any solution appropriate for our set up.
I have a confusion with Bulk Insert task for several days. Here is what Books Online say, which i totally understand--- "The source file that the Bulk Insert task loads can be on the same server as the SQL Server database into which data is inserted, or on a remote server. If the file is on a remote server, you must specify the file name using the Universal Naming Convention (UNC) name in the path."
Now, If my Bulk Insert task (within a package) is on one server, say ETL ( so when i run the package, it will run from ETL server); but, My destination table where data will be inserted, is in another server, say PROD. So, My OLEDB connection will be pointing to a remote server .Now, If my text file is on ETL server, do i have to give UNC path?..Or If i have text file on the same server as my destination ( here PROD) do i Have to Give UNC naming?. I found in several places that Bulk insert Task runs on that server where its connections points to. My scnerio: I have multiple SSIS packages. Now These packages will be on ETL server, but all the databases will be hosted on PROD server. Now for my Bulk Insert , I asked them to keep all flat file on ETL server(where packages will be ). Looks like this will give me a problem later, since all my Destination will be on PROD server, during bulk insert , sql server will go and look for those files on PROD server rather than ETL server? Or i AM confused here?...... Say package.dtsx is On SER1which useses bulk insert tasks.. My OLEDB connection is SERV2 database B and Table T Now my flat file connection is D:myfilefile.txt
Now when i run this task where does these txt file will be pulled from. Does it pulls from local server (SER1) or does it goes to SERV2 looking for txt file.
I tried several combinations of moving files to different servers, yet i have no conclusions.
Is there a way using a stored procedure in a local database to add a record to a database executing in a cloud environment when both entities reside in different domains?
I have been trying to get a clear explanation on what I need to do in order to be able to execute SSIS packages which I have located on an app server. Is it that I need integration services installed on the app server itself and thats it? Or can someone tell me if there is any more to it, thanks folks!
With DTS you could copy and register some DLLs onto a non-SQL Server PC and run Packages. Is this still possible with SSIS? I imagine that, at the very least, you would have to copy dtexec.exe and the *,dtsx files to the other computer.
I have created my packages and i want them to place them on the server.Do i need to place the entire project of dts packages on the server or is there any option to place executables...if so please explain....
And to run these packages on the server do i need to set them as new job at sql server agent or is there any other way i need to run on the server.
I want then to run whenever the text file gets updated is it possible to set anything for my packages to run as and when the text file gets updated..
Is it possible to import packages in batch into MSDB (in lieu of right clicking and importing each one by one)? We have a lot of packages that are going through a lot of changes so a batch import would save a lot of time.
I installed SP2 to one of our servers to see what we'd need to do for our Production systems. The server in question is used for a log shipped copy of production, SSRS, SSIS and general duty stuff. I got errors in two areas during the installation -- 1 dealing with SSNS (don't use it yet on that server) and the second being the Client product.
After a reboot I went into a Business Studio solution I've been working on (and using) for several months. None of my packages are working--each having the following problem:
Unable to cast COM object of type 'Microsoft.SqlServer.Dts.Runtime.Wrapper.PackageNeutralClass' to interface type 'Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSContainer90'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{8BDFE892-E9D8-4D23-9739-DA807BCDC2AC}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))..
It seems to be in the designer as it is unable to display the the graphical version but it does let me switch to the code view.
What is the easiest way to troubleshoot and/or fix this?
Hi, I'm building a portal for supervisors can view their agents errors. We have a team that have been using Access to import data and QC it. I want to get these records into SQL but at the same time, the agents need to import their own work. I've create a DTS package that will import the data into the SQL table from a folder the agents have access to. I tried to create the strored proc to be run from the ASP page however it's not working and I'm having a hard time figuring out why. Here is the procCREATE PROCEDURE DTS AS exec master..xp_cmdshell 'dtsrun /KEMTSQL02 /ADJUSTMENTS /E'GODoes anyone know what I'm doing wrong? Any help would be great. Thanks!
on the server i am planning to run my packages at runtime using sql server agent...i remove the text files from exisiting location(for eg C to different location(D.when ever i receive a file at c:
I want my package to run.
How can i set this up in the sql server agent?????how will the sql server agent know that this package had received a file.
or is there anyway i can run my packages automatically other than sql server agent
I was recently tasked with creating an automated process to refresh SSIS packages from MSDB on one server to another and I decided to go the route of using Powershell, however this wasn't as straight forward as I originally imagined so I thought I would share my solution in case others encounter the same request. The below PowerShell code will create all Integration Services folders from the source MSDB (that contain SSIS packages) on the target instance of MSDB then copy all packages to the proper folder locations. The /QUIET option is used to automatically overwrite packages that already exist on the target so make sure you want to overwrite the current versions of your packages before executing.
I have added an email task to the ON Error Event of my SSIS package, so that I will always know when there are errors. However I would like the SQL Server job executing the package to succeed even if the package fails. What setting do I change in the SSIS packageto achieve this? MaximumErrorCount?
