I'd like to know if there's a way to control the execution of ETL packages, such that: Different packages, or at least packages that don't access the same table or database run asynchronously with respect to each other; e.g., two different packages run at the same time and If a package is called for execution more than once by different requests, force them to run synchronously, or one after the other.If this is possible, what resources would it require? Is this possible under, say, a dual or quad processor machine?
Has anyone monitored the execution of SSIS packages with MOM? Are there extreme benefits over just utilizing the built in execution and event logs, as well as the Windows Event Viewer?
What is the recommended way to monitor SSIS execution?
Our SSIS packages use the Web Service Task to call services to send email and write package failure data to a department wide database. These Web Service Calls are failing with HTTP 401 errors. It was caused by the passwords for the HTTP connections not being saved when the SSIS pkgs were saved to .dtsx files. I have tried saving the package with a password and the EncryptSensitiveWithPassword protection option. This password can be supplied when the package is called from another package or a package is executed in Visual Studio but cannot be supplied to DTexec to execute the package in a job. DTexec does have a /Password parameter but it is rejected if the package is loaded from a .dtsx file. This appears to be a bug in DTexec. It only accepts the /Password parameter when the package is loaded with the /SQL option. Specifying /Password and /File is not supported.
Is this a known bug? Are there any workarounds? Has anyone successfully called a Web Service from SSIS executed via DTexec?
I seem to be missing something with SQL 2005 SSIS and I am wondering how I would accomplish my taks. With SQL 2000 and DTS we designed our DTS packages so that we could load one file or database from start to finish. We then have controlling programs written in VB that will monitor directories, when a file shows up in a directory it loads the package, changes the necessary properties and executes it. These programs handle all of our error notifications and dealing with the files afterwards, making sure files are complete in the directory before the DTS runs, etc. With .NET and SQL 2005 I have learned that I can change the properties by using variables and expressions, which I think would handle all of our situation but I can not run them remotely. The documentation says that I could setup a scheduled job and then call sp_start_job from my program. That seems like it would require a lot of scheduled jobs and then I still need to get the parameters into the packages. The other option is to use a web service, that would require IIS on my SQL box which everywhere says I don't want. Am I using SSIS for something I am not supposed to be or how do I accomplish this in 2005? I have noticed that you can not even run packages through the SQL Server Management Studio. We have a number of packages that are built that are run on demand when needed. In SQL 2000 I could just click and execute the package. In 2005 is my only option to setup some sort of fake job in the job scheduler?
One of the step in sql agent job is to execute the SSIS package to copy files from remote shared location to local server where sql server is installed. The account on which SQL agent runs as has access to remote shared location. However SSIS package always fails with following error:
An error occurred with the following error message: "Access to the path '\sharedlocationBCKTST105092008.csv' is denied.".
We have tried using proxies but same issue come with proxy also. When using proxy, we created a proxy for a user who has access on that shared folder on Windows server(from which files are to be copied). If we login to SSIS server with the above user and execute SSIS package manually it works fine. However if we login with different user and run the job via SQL agent using proxy of the above user, then job fails throwing same above mentioned error
I am seeking a walkthrough for executing packages via a web service. All I have found so far are fragmented bits of information. This article is a good start but it leaves out some critical security setting information. http://msdn2.microsoft.com/en-us/library/ms403355.aspx#service.
Does anyone know of a good walkthrough that includes the security setup: Impersonation, Proxies, etc.?
I get hte following Error --------------------------------
"Dupaco Load MMAs"-------------"An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description: "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.". "
"Data Flow Task" -------"The AcquireConnection method call to the connection manager "Dupaco" failed with error code 0xC0202009. "
"Data Flow Task" ----------"component "OLE DB Destination" (119) failed validation and returned error code 0xC020801C. "
"One or more component failed validation. "
"There were errors during task validation. "
When i try to execute the same code on the server where all the package are hosted. It is working fine.
More Info:
In thepackage we are using one "Source Flat file connection" and one "Oledb Destination connection"
Please help me to solve this problem, as my client needs it very badly
I would like to call an SSIS package from a Service Broker Queue.
There is one way that I am aware of -
Using xp_cmdshell from within an activation stored procedure and using DTEXEC.
Is there a more elegant way of executing an SSIS package from within SSB?
Also, I am not interested in writing a .NET external activator to process my messages in the queue. I would like this operation to be strictly database oriented. Having said this, I am also trying to avoid triggers processing the messages in the queue.
Need some help... When we tried to run mulitple packages one after the other from a windows service, first one succeeds but later ones are throwing below error : "The script threw an exception: The element cannot be found in a collection. This error happens when you try to retrieve an element from a collection on a container during execution of the package and the element is not there. A deadlock was detected while trying to lock variable "System:ackageName, User::BusinessDate, User::Environment, User:ortfolioName" for read access. A lock could not be acquired after 16 attempts and timed out."
Later, we tried to create separate AppDomains for each package and execute via console application, but ended up with below error (The below expressions were defined in OnError Event) : "The result of the expression "@[User::ReportErrorFrom]" on property "FromLine" cannot be written to the property. The expression was evaluated, but cannot be set on the property. The result of the expression ""Error At :" + @[System:ourceName] + "" + "Error Description : "+ @[System::ErrorDescription] + "" " on property "MessageSource" cannot be written to the property. The expression was evaluated, but cannot be set on the property."
At last, we tried to span a separate process (System.Diagnostics.Process) for each package. this seems working but taking very long time: A package that normally takes 2 min, is taking 60 min.
We also tried creating an SSIS Package that executes mulitple packages. But only first package is getting executed, and second one is throwing below error (Here the variable it is trying to lock is of first package): "Failed to lock variable "UniqueInstrumentsQuery1" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
Please help us with some work around for this. Thanking you in advance,
I've run into a problem with SSIS packages wherein tasks that write or copy files, or create or delete directories, quit execution without any hint of an error nor a failure message, when called from an ASP.NET 2.0 application running on any other machine than the one where the package was created from. By all indications it appeared to be an identity/permissions problem.
Our application involves a separate web server and database server. Both have SQL Server 2005 installed, but the application server originally only had Integration services. The packages are file system-deployed on the application server, and are called using Microsoft.SqlServer.Dts.Runtime methods. For all packages that involve file system tasks, the above problem occurs.
When the above packages are run using the command prompt (either DTEXEC or DTEXECUI) the packages execute just fine. This is expected since we are using an administrative account. However when a ShellExecute of the same command is called from ASP.NET, the same problem occurs.
I've tried giving administrative permissions to the ASPNET worker process user to no avail.
I have likewise attempted to use the SQL Server Agent job approach but that approach might not be acceptable for our clients since it means installing SQL Server 2005 Database services on the application server.
I have read the relevant threads in this forum, namely http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1044739&SiteID=1 and http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=927084&SiteID=1 but failed to find any solution appropriate for our set up.
We manage some SSIS servers, which has only SSIS and SSIS tools installed on them and not the sql server DB.
SSIS packages and configuration files are deployed on a NAS. We run the SSIS packages through DTEXEC by logging in to the server.
We want to allow developers to run their packages on their own on the server, but at the same time we dont want to give them physical access on the server i.e we do not want to add them into RDP users list on server properties. We want them to allow running their packages remotely on the server.
One way We could think of is by using powershell remoting and we are working on that. But is there any other way or any tool already present for the same.
Hello all, I am struggling around defining a logging mechanism for my packages. I have 2 questions concerning that matter: I have used event handlers for my loggings (as defined here: http://blogs.conchango.com/jamiethomson/archive/2005/06/11/1593.aspx ), but the problem is with packages that failed validation. I cannot find log entry for these cases since no "onerror event" doesn't trigger (for instance when the table I'm loading to doesn't exsist).
And the second question: many of my packages are executed using execute process task (using dtexec command line). I am trying to capture the result of the execution as a log file by using the ">" in the command line in order to output the execution to a log file in the following format: dtexec /FILE "MyPackage.dtsx" > " MyPackageLog.log" This works fine when executed by myself but when using the Execute Process task (defined: Executable: DTExec.exe, Arguments: /FILE "MyPackage.dtsx" > " MyPackageLog.log") I get execution error€¦ Thanks, Liran
I am experiencing that my SISS packages are executing faster during development from within the Sql Business Intelligence Development Studio environment than when I execute them with dtexecui after I have deployed them as either File System Or Sql Server deployments.
After deployment the packages run about twice a long as they do from within Sql Business Intelligence Development Studio (i.e they reference the same data and are executed on the same machine).
In short the packages import data, during this import stage the data is massaged certain calculations are performed thereon where after this massaged data is used to perform further calculations.
A summary of tools used in SISS are Execute Sql Task, Flat File Source, Oledb Destination, Data Conversion, Aggregate, Conditional Split, Merge, For Each Ado Enumerators, Script Tasks, Derived Columns and Recordset Destinations.
In terms of time from within Sql Business Intelligence Development Studio when the packages are executed for 1 month's data the total combined time is about 10 minutes and when deployed as either File System Or Sql Server deployments they run for about 20 minutes. My problem is when they are executed for a years data instead of running for 120 minutes they run for at least 240 minutes or longer.
Is this normal or am I missing some settings when building the package for deployment?
Some time ago, I came across some posts that suggested that remote execution of SSIS packages was only supported if the package was wrapped by a SQL Server Job Agent job.
Is this correct?
Would the web service technique described below then not be supported?
This is my snippet of code. It works fine with remote DTSX but when they have got connections against Sql Server 2005, ending given the following error:
I am finishing a port of a project that was done with RS2000/AS2000/DTS2000. The cube process is triggered remotely by a Korn Shell script from Services for Unix that runs a DTS package:
I have finished porting the AS and RS parts to their 2005 equivalents and now I am trying to get the cube processing converted. I have built a SSIS package that processes the cube and I have deployed it to the SQL Server in MSDB. When I am logged on to the SQL Server box directly, I can execute the package via the Management Studio and from the command line via:
When I try to register the SQL Server in Integration Services from my client PC Management Studio it fails with "Access Denied". When I try to run the dtexec from my client PC it fails with "Access Denied".
I have walked through Kirk Haselden's instructions at:
(adding myself to Distributed COM Users, noting the correct DCom Config settings for MsDtsServer and restarting the Integration Services service) and the problem is still not resolved. I have posted to the DTS group on Technet and now I am posting here. This can not be run scheduled by SQL Agent as it needs to be triggered by other load/postload activities. Are there any other fixes for this problem that I can try?
Thanks,
Keehan
By the way, being able to have all the parts of this project in one dev environment is great. The previous application was developed in 3 separate locations which was pretty clunky.
I have created a SSIS package with Web Service task in it,which execute remote server web service
I got following error during Web service execution
Note:I am not using proxy settings,i also checked HTTP connection and it suceeded
Error:[Web Service Task] Error: An error occurred with the following error message: "Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebserviceTaskException: Could not execute the Web method. The error is: Object reference not set to an instance of an object.. at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebMethodInvokerProxy.InvokeMethod(DTSWebMethodInfo methodInfo, String serviceName, Object connection) at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTaskUtil.Invoke(DTSWebMethodInfo methodInfo, String serviceName, Object connection, VariableDispenser taskVariableDispenser) at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTask.executeThread()".
This is an ugly duck query but it seems to work fine. when I try to directly execute the remote stored procedure such as with
insert into query_log exec remoteServer.master.dbo.##up_fetchQuery
I get execution error
Server: Msg 2812, Level 16, State 62, Line 1 Could not find stored procedure '##up_xportQueryLog'. Database name 'master' ignored, referencing object in tempdb.
When I try
insert into query_log exec remoteServer.tempdb.dbo.##up_fetchQuery
I get
Server: Msg 2812, Level 16, State 62, Line 1 Could not find stored procedure '##up_xportQueryLog'. Database name 'tempdb' ignored, referencing object in tempdb. with
insert into query_log exec remoteServer..dbo.##up_fetchQuery
or
insert into query_log exec remoteServer...##up_fetchQuery
I get
Server: Msg 2812, Level 16, State 62, Line 1 Could not find stored procedure '##up_xportQueryLog'.
I guess the remote server has trouble resolving the name of the global temp stored procedure when its reference comes in as a remote stored procedure calls. Is there any way to directly call a global temp stored procedure from a remote server or do I need to stick with this goofy-looking work-around?
Is there a way, for example to script a DTS Package, so that it can be deleted and recreated at a later date if necessary? I have quiet alot of these, but few are used regularly. The msdb database is now up to 80 MB. However I don't want to delete them and have no way to recreate them.If I took a backup of msdb and then deleted the packages, would restoring msdb at a later date restore the packages?????
Scenarion: 1.- SSIS Package execute tasks on 2000 SQL Server Database 2.- Execution takes places using Business Intelligence Studio Question: 1.- How can I tracked that SQl 2000 tasks took place using a SSIS Package?
Hello, I am getting this error with a web service trying to fill a strongly-typed dataset stored in another project that the web service has connection to. I keep getting this error: An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified) I set it to allow TCP/IP only, TCP/IP and Named Pipes, and Named Pipes only remote connections allowed. But it still doesn't work. I don't understand what the problem may be. I stopped and started the database engine service every time (this is a SQL Express database attached to the server, by the way). Any insight would be helpful, Thanks.
I have troublesome problem with service broker. I move a lot of databases from one server to another servers (by backup and restore) but always have problem with databases where service broker is enabled. Bellow is my scenario:
1. backup database db1 (srv1) 2. restore db1 (srv2, srv3, srv4.....) 3. drop manually Services and Queues, only green filed (http://img122.imageshack.us/img122/549/clipqk5.jpg) (srv2, srv3, srv4.....) 4. alter database db1 set enable_broker (srv2, srv3, srv4.....)
After that SQL Server log on each server is full bellow errors, nobody so far can help me
Could you help, please
############ SRV2 03/03/2008 18:54:41,spid21s,Unknown,The query notification dialog on conversation handle '{E6BF547B-0AE6-DC11-A4BD-00E08127B819}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service 'SqlQueryNotificationService-53c1dc9b-7d22-4310-a5b6-18ac97ae4bf3' because it does not exist.</Description></Error>'. 03/03/2008 18:54:41,spid27s,Unknown,The query notification dialog on conversation handle '{AA68F7D8-F7E5-DC11-A4BD-00E08127B819}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service
SERV3 03/02/2008 00:32:25,spid21s,Unknown,The query notification dialog on conversation handle '{2BC70B27-F1E3-DC11-AB50-0017085CAA29}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service 'SqlQueryNotificationService-abc6a7ce-53f1-411e-85a7-50c09e2699e1' because it does not exist.</Description></Error>'. 03/02/2008 00:29:20,spid22s,Unknown,The query notification dialog on conversation handle '{DF112861-F9E3-DC11-AB50-0017085CAA29}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service
SRV4 03/02/2008 16:15:07,spid25s,Unknown,The query notification dialog on conversation handle '{4179E578-7DE4-DC11-9A69-00145E1B71FC}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service 'SqlQueryNotificationService-01e16746-e114-4e8c-b587-91e93768a9a8' because it does not exist.</Description></Error>'. 03/02/2008 16:15:07,spid20s,Unknown,The query notification dialog on conversation handle '{5079E578-7DE4-DC11-9A69-00145E1B71FC}.' closed due to the following error: '<?xml version="1.0"?><Error xmlns="http://schemas.microsoft.com/SQL/ServiceBroker/Error"><Code>-8490</Code><Description>Cannot find the remote service ############
I have SQL Server 2005 express installed on my machine, and I was wondering if there is any way to have an asp.net 2.0 web service hosted by a different company be able to send sql commands to the sql server on my machine.
Just set up my first Service Broker test. Wanted to send an Event Notification from one instance to another. Finally, the message arrives but profiler shows that the message is still being continously resent (and rejected as being duplicate). What am I doing wrong?
With best regards,
Artus
--
/* Local Event Source.sql */
USE master GO
CREATE DATABASE NotificationDB GO
ALTER DATABASE NotificationDB SET ENABLE_BROKER WITH ROLLBACK IMMEDIATE; GO
CREATE ENDPOINT BrokerEndpoint STATE = STARTED AS TCP ( LISTENER_PORT = 5554 ) FOR SERVICE_BROKER ( AUTHENTICATION = WINDOWS, ENCRYPTION = DISABLED ) GO
USE NotificationDB GO
CREATE EVENT NOTIFICATION TestEN ON DATABASE FOR CREATE_TABLE TO SERVICE 'ExpressService', 'AFEDD339-AD3D-4865-AF3C-299B0A0784C6' GO
CREATE ROUTE ExpressRoute WITH SERVICE_NAME = 'ExpressService' , BROKER_INSTANCE = 'AFEDD339-AD3D-4865-AF3C-299B0A0784C6', ADDRESS = 'TCP://localhost:5555' GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'Pa$$w0rd'; GO
ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY; GO
USE master GO
SELECT service_broker_guid FROM sys.databases WHERE database_id = DB_ID() GO
/* Remote Service.sql */
USE master GO
CREATE DATABASE Test GO
ALTER DATABASE Test SET ENABLE_BROKER WITH ROLLBACK IMMEDIATE; GO
CREATE ENDPOINT BrokerEndpoint STATE = STARTED AS TCP ( LISTENER_PORT = 5555 ) FOR SERVICE_BROKER ( AUTHENTICATION = WINDOWS, ENCRYPTION = DISABLED ) GO
USE Test GO
CREATE QUEUE ExpressQueue WITH STATUS = ON GO
CREATE SERVICE ExpressService ON QUEUE ExpressQueue ( [http://schemas.microsoft.com/SQL/Notifications/PostEventNotification] ); GO
GRANT SEND ON SERVICE::ExpressService TO [public] GO
CREATE ROUTE ExpressServiceRoute WITH SERVICE_NAME = 'ExpressService', BROKER_INSTANCE = 'AFEDD339-AD3D-4865-AF3C-299B0A0784C6', ADDRESS = 'LOCAL' GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'Pa$$w0rd'; GO
ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY; GO
SELECT service_broker_guid FROM sys.databases WHERE database_id = DB_ID() GO
I was trying to send a message on a different instance in the network using service broker. I have created the endpoint and route both the side. I could see that the messages are in the transmission_queue in the sender side with no error in the status.
What could be the reason.
I am doing the following: CREATE ROUTE inst02 WITH SERVICE_NAME = 'SERVICE2', ADDRESS = 'tcp://10.14.43.149:2341'
in the send script i am using this: BEGIN DIALOG CONVERSATION @dialog_handle FROM SERVICE [SERVICE1] TO SERVICE 'SERVICE2' ON CONTRACT [MainContract] WITH ENCRYPTION = OFF ;
in the sending side if i change the endpoint's authentication as windows kerberos , then i get thefollowing error in the transmission queue:
"Connection handshake failed. An OS call failed: (8009030e) 0x8009030e(No credentials are available in the security package). State 66."
I also have given access to the remote user on this endpoint(on the recever side) using this : GRANT CONNECT ON ENDPOINT::Endpoint_test to paras
I am executing the sending side send script using the same user wich has access to the remote endpoint.
Hi I have recently upgraded several of our SQL2005 servers to SP2a :) version 9.0.3050 and find that I can no longer script the Service Broker objects from a remote connection. The error I am getting is: Script failed for MessageType 'MyMessageType' (Microsoft.SqlServer.Smo) Additional Information Either the object or one of its properties is not supported on the target server version (Microsoft.SqlServer.Smo) Now I know that this worked pre SP2 using exactly the same setup. Has anyone else experienced this or know how to fix it?
If I have an asp page with a button, what are the means or methods to trigger the event to process a report, save into a report and store in a file server or any directory? All these is done backend without the need to display to the browser.
Using RS Web services or Delivery extension programming things like that?
I am replicating a couple of SQL2K databases via transactional rep and remote 2K distributor to 1 SQL2K5 database.
I want the distributor to email me when/if any of the agents fail etc. In order to set this up, I think that I need the SQL Server service to use a domain account (from error messages that I have been getting). It is currently using a local system account (it was a test box in a previous life).
I have tried this briefly, and found that it lost the publisher details in the replication monitor. Can anyone tell me what the implications for changing the SQL server service user and password are? Am I going to have to tear down and rebuild replication? The SQL Server Agent is fine - I am trying to get the two services in sync.