Best Practices Question - How Do You Execute Multiple Packages?
Jan 26, 2006
I have 200+ plus packages that need to be flexible in how they are run. For example, an end user may choose to run packages 1,2,3 and the next end user may choose to run packages 2,3,7, etc. Prior ro running a package, I set an "instance id" inside the group of packages so I can tie them all together in the logfile - I know that packages 1,2,3 were all run as group and that's distinct from packages 2,3,7 that were run in a differnt group.
Initially I embarked on a scenario where I had a queue table that loaded up the packages to be run and then had a little c# app that read the queue, generated the "instance id" and ran all the packages (either thru dtexec.exe or the Microsoft.SqlServer.DTS.Runtime). But now I wonder if using a master package that uses the Execute Package Task is the way to go. My 200+ packages are all independent and run based on a single config file and it seems as though going the parent package route will destroy some of that independence because I'll now be relying on parent package variables.
Any comments or suggestions?
View 2 Replies
ADVERTISEMENT
Mar 28, 2008
I have a schedule job that errors out on the first step when I attempt to run it. There are 3 packages that I created and I can run them all manually just fine.
I get this error when I try to run the job:
-----------------------------------------------
Date,Source,Severity,Step ID,Server,Job Name,Step Name,Notifications,Message,Duration,Sql Severity,
Sql Message ID,Operator Emailed,Operator Net sent,Operator Paged,Retries Attempted
03/28/2008 13:49:22,ST Customer Data,Error,0,Server01,ST Customer Data,(Job outcome),,The job failed.
The Job was invoked by User sa.
The last step to run was step 1 (config 1).
The job was requested to start at step 1 (config 1).,00:00:01,0,0,,,,0
03/28/2008 13:49:22,ST Customer Data,Error,1,Server01,ST Customer Data,config 1,,
Executed as user: Server01SYSTEM. ...9.00.3042.00 for 32-bit
Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 1:49:23 PM Error: 2008-03-28 13:49:23.23
Code: 0xC0016016 Source:
Description: Failed to decrypt protected XML node "DTS:Password" with
error 0x8009000B "Key not valid for use in specified state.".
You may not be authorized to access this information.
This error occurs when there is a cryptographic error.
Verify that the correct key is available. End Error
Error: 2008-03-28 13:49:23.57 Code: 0xC0202009
Source: Config 1 Customer Data Connection manager "SourceConnectionOLEDB"
Description: SSIS Error Code DTS_E_OLEDBERROR.
An OLE DB error has occurred. Error code: 0x80040E4D.
An OLE DB record is available.
Source: "Microsoft SQL Native Client" Hresult: 0x80040E4D
Description: "Communication link failure".
An OLE DB record is available.
Source: "Microsoft SQL Native Client"
Hresult: 0x80...
The package execution fa...
The step failed.,00:00:01,0,0,,,,0
-----------------------------------------------
Any ideas?
Thanks,
John
View 2 Replies
View Related
Oct 14, 2008
I'm pretty new to SSIS but I've managed to cobble together a number of individual packages to refresh SQL tables from a 3rd-party database.
Now, what I'd like to do is have a single package that I can use to invoke each of the individual ones. Since it will run on a quad, I'd like to invoke them such they'll run in parallel.
View 2 Replies
View Related
Aug 28, 2007
Need some help...
When we tried to run mulitple packages one after the other from a windows service, first one succeeds but later ones are throwing below error :
"The script threw an exception: The element cannot be found in a collection. This error happens when you try to retrieve an element from a collection on a container during execution of the package and the element is not there.
A deadlock was detected while trying to lock variable "System:ackageName, User::BusinessDate, User::Environment, User:ortfolioName" for read access. A lock could not be acquired after 16 attempts and timed out."
Later, we tried to create separate AppDomains for each package and execute via console application, but ended up with below error (The below expressions were defined in OnError Event) :
"The result of the expression "@[User::ReportErrorFrom]" on property "FromLine" cannot be written to the property. The expression was evaluated, but cannot be set on the property.
The result of the expression ""Error At :" + @[System:ourceName] + "" +
"Error Description : "+ @[System::ErrorDescription] + "" " on property "MessageSource" cannot be written to the property. The expression was evaluated, but cannot be set on the property."
At last, we tried to span a separate process (System.Diagnostics.Process) for each package. this seems working but taking very long time:
A package that normally takes 2 min, is taking 60 min.
We also tried creating an SSIS Package that executes mulitple packages. But only first package is getting executed, and second one is throwing below error (Here the variable it is trying to lock is of first package):
"Failed to lock variable "UniqueInstrumentsQuery1" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
Please help us with some work around for this. Thanking you in advance,
View 17 Replies
View Related
Sep 4, 2015
We are building a dataload application where parameters are store in a table. And there are multiple packages for each load.There is a column IsChecked column if it is 1 then only the child package should execute.Created a master package. In which i have taken execute SQL task in that storing a results in variable and based on the result the child package should execute. But In executesql task i selected result set as full result set. I am getting the below error.
[Execute SQL Task] Error: Executing the query "SELECT isnull(ID ,0) AS ID FROM DataLoadParameter..." failed with the following error: "The type of the value (DBNull) being assigned to variable "User::LoadValue" differs from the current variable type (Int32). Variables may not change type during execution. Variable types are strict, except for variables of type Object.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
View 3 Replies
View Related
Jun 1, 2006
Dear gurus,
I've got this one for Sql2k http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnsql2k/html/sql_busintbpwithdts.asp
Any help or link would be welcomed,
View 5 Replies
View Related
Apr 5, 2007
Hi All
I have a meaty job that currently runs as a set of sql scripts under sql agent and I'm transferring this job to run as a SSIS package. I have managed to set it up so that it runs as a sql job (cheers guys!), and I'm watching the job run now, maxing out the CPU usage as it tries to run 10 different tasks at the same time . great stuff!
However, I've been thinking about how to view the progress of this job/package (the job only has one step; "run SSIS package"). With the current job as single scripts, in SQL agent you can view which step of the job you are at.... is it possible to somehow view which package task you're at (ideally with nice shiny yellow green and red updates like in debugging), or do you have to output each step result into a log file?
Oh, and another question! If i have a checkpoint in a package, and the package falls over, when the job is rerun through SQL agent, will it start at the checkpoint, or run from the start?
Thanks!
michal
View 10 Replies
View Related
Apr 22, 2015
We manage some SSIS servers, which has only SSIS and SSIS tools installed on them and not the sql server DB.
SSIS packages and configuration files are deployed on a NAS. We run the SSIS packages through DTEXEC by logging in to the server.
We want to allow developers to run their packages on their own on the server, but at the same time we dont want to give them physical access on the server i.e we do not want to add them into RDP users list on server properties. We want them to allow running their packages remotely on the server.
One way We could think of is by using powershell remoting and we are working on that. But is there any other way or any tool already present for the same.
View 4 Replies
View Related
Apr 8, 2008
Hi All,
We're trying to run an EXE from SSIS through "Execute Process" task.
The EXE folder contains other DLLs as well.
The EXE interacts with the database and reporting services and sends some e-mails(max 500 a day) out to customers.
My question is:
Is it ok to run this kind of EXE on the production SQL box?
If not, why?
(People argue that running EXEs is not advisable on production boxes)
Q: Why did Microsoft introduce "Execute Process" task when we cannot run EXEs on the production box?
If somebody can educate whether it's ok to run such EXEs on prod SQLs.
In either case, some explanation is greatly appreciated.
Thanks,
Siva.
View 8 Replies
View Related
Jan 15, 2008
Hi everyone - I'm getting myself into a right muddle and am looking for advice.
How do people deal with connection string matters when taking a dataset defined in one assembly (and by default using the connection strings defined within that assembly in the settings files) and then using that assembly in an app (which also has a requirement to see the same physical database).
I'm not sure I've explained that terribly well but what I'm trying to avoid is duplicate copies of my database which so far seems to be the only way that I've managed to make stuff work. I'm not very experienced with SSCE data access (I'm a serices/sockets/ip person) and this may just be ignorance. Most of the examples seem to assume that the data is in the same place as the app whereas I'm trying to collate a whole series of functions into a helper assembly that I'd like to re-use for other things.
Ideally the dataset designer would provide an easy way of choosing from centralised connection strings - perhaps this is what the Dataset Project implies - but again the docs are mostly focused on SQL Server? Otherwise the best I've been able to do is make the connection properties public and try to update them that way or use a post-build action to copy the database from my datalayer project directory to that of my application |DataDirectory|
I suppose the question might be if you have a dataset containing multiple tableadapters that assumes one connection string, is there any easy way to keep such strings co-ordinated between projects without hardcoding them? With a server resource, the same non-specific connection string resolves to the same server (if that make sense) and this seems to be were I can't make the logical shift.
Does anyone have any thoughts and can they please point this SSCE noob in the right direction?
Cheers
View 2 Replies
View Related
Sep 12, 2007
I have couple of DTS Packages that I want to run on the server running SQL Server 2000, how do I go about running those? Any help will be greatly appreciated.
View 2 Replies
View Related
Jul 23, 2005
I have created a functioning DTS package inside Enterprise Manager andnow I want to be able to execute it outside of EM. The package importsdata from an SQL dbase to a Visual FoxPro dbase. I would like this DTSpackage to execute everytime an Insert(of certain critera) is made onthe SQL dbase. I am thinking that using an Insert Trigger on the SQLtable is the way to go about this. I am seeking advice as to... is thisthe best way to go about this.Also, and if so, in a trigger which route is the best to take...1. use dtsrun command line utility or2. setup a COM object to run the DTS packageI have tried both in SQL query analyzer and am having troubles with thesyntax. Could anyone possibly send the code for both ways. Thanks.Good Day to all,Brett
View 10 Replies
View Related
Sep 10, 2007
Does anybody have feedback on how to execute several packages using c#.
Thanks Sergio
View 1 Replies
View Related
May 22, 2001
Hi,
I'm using SQL 7.0 SP2 on NT SP6a and have a quick question.
I want to create a scheduled job to execute several DTS packages (all located on the same server). I can use Enterprise Manager to schedule an individual package but the command line of the step has something like:
DTSRun /~S 0x26CE410E3EFF5E8B738DC1C3527EEA1A /~N 0xCAAF8584582AA98C09F5CF132CB0BB5F296D1E9053DD6526 717A53AD4134C3463615FEF20330386927242B6CB7070036 /E
Is using the DTSRUN utility the only way to create several steps in a job to run DTS packages ?
Any help appreciated.
Thanks,
Tim
View 1 Replies
View Related
Sep 13, 2007
All,
Could someone please tell me what to do with it and what might be causing it?
The packages were fine yesterday.
Here is the error message:
[Script Component 1 1 [3941]] Error: System.Runtime.InteropServices.COMException (0x80040154):
Retrieving the COM class factory for component with CLSID {A138CF39-2CAE-42C2-ADB3-022658D79F2F}
failed due to the following error:
80040154. at Microsoft.VisualBasic.Vsa.VsaEngine.CreateEngine()
at Microsoft.VisualBasic.Vsa.VsaEngine.CheckEngine()
at Microsoft.VisualBasic.Vsa.VsaEngine.set_RootMoniker(String value)
at Microsoft.SqlServer.VSAHosting.Runtime.CreateVsaEngine()
at Microsoft.SqlServer.Dts.Pipeline.ScriptRuntime..ctor(String projectName,
String moniker, String language, Boolean showErrorUI)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.CreateUserComponent()
Thanks in Advance
View 1 Replies
View Related
Nov 8, 2007
Hi!
I was following this article http://www.codeproject.com/useritems/Execute_Package_by_C_.asp to execute list of SSIS packages via vb.net. I created a windows application in server A copied the exe file in server B. the SSIS package is in server B. I deployed the package and it runs file vis integration server. below are my codes. everytime i run the exe in server B. I can see the list of packages but when i hit execute button i get error msgs that says "failure" I am not sure how to get more error out of it. I am not sure whats causing this error.
----------------------------
Imports System.Data.SqlTypes
Imports System.Data.SqlClient
Imports Microsoft.SqlServer.Management.Smo
Imports Microsoft.SqlServer.Management.Smo.Agent
Imports Microsoft.SqlServer.Dts.Runtime
Imports Microsoft.SqlServer.Management.Common
Public Class Form1
Dim app As New Microsoft.SqlServer.Dts.Runtime.Application()
Dim pInfos As PackageInfos = app.GetPackageInfos("\", "SQLKEV", "", "")
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
For Each pInfo As PackageInfo In pInfos
If pInfo.Name <> "Maintenance Plans" Then
ComboBox1.Items.Add(pInfo.Name)
End If
Next
End Sub
Private Sub btnExec_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnExec.Click
Dim app As Microsoft.SqlServer.Dts.Runtime.Application
app = New Microsoft.SqlServer.Dts.Runtime.Application()
Dim pkgResults_Sql As DTSExecResult
Dim pkgIn As New Package()
Try
pkgIn = app.LoadFromSqlServer("" + ComboBox1.SelectedItem.ToString().Trim(), "SQLKEV", "", "", Nothing)
pkgIn.PackagePassword = "password21"
pkgResults_Sql = pkgIn.Execute()
MessageBox.Show(pkgResults_Sql.ToString())
Catch ex As Exception
MsgBox("this is an exception" + ex.ToString)
End Try
End Sub
End Class
----------------------------
please help
View 9 Replies
View Related
Jun 20, 2001
I have SQLServer 7, and have used Enterprise Manager to schedule local packages. The package that I'm trying to do is run a SQL script on a nightly basis. The problem that I'm having is that I need to be able to install the scheduled script customer box using an installer. I have access to execute command line programs in the installer. Do anyone know if you can schedule local packages (using SQL Scripts) from the command line? Or if a 3rd party application can do this. Any help or direction would be greatly appreciated. I've tried to use sp_add_job and sp_add_jobschedule, but haven't been able to get them to work.
Thanks
Brian
View 1 Replies
View Related
Apr 26, 2004
Hi, hoping someone may be able to shed some light on this.
I have a stored procedure that conditionally executes different DTS Packages. Everything seems to work, there's no errors at all, but the DTS Packages are never executed. Wondering if anyone has any ideas.
Here's the stored procedure....
CREATE PROCEDURE exportLists
AS
DECLARE @script VARCHAR(8000)
DECLARE @id INT
DECLARE @max INT
select regionid,count(distinct homephone) as phone
into #phone
from tbl_template
Where exportid is null
group by regionid
select officeid,listpath
into #office
from tbl_office
WHERE startdate <= GETDATE()
AND (enddate >= GETDATE() OR enddate IS NULL)
and listpath is not null
and listpath <> ''
Select case when phone > 2500 then '"exportLists" /A MDBFile="' + listpath + '" /A OfficeID ="' + convert(varchar(10),officeid) + '" '
Else '"resetTemplate" /A OfficeID="' + convert(varchar(10),officeid) + '" '
end as cmd
,Identity(int,1,1) as rowno
,phone
into #temp
from #office as O
inner Join #phone as p
on o.officeid=p.regionid
Set @max=@@rowcount
set @id = 1
While @id <=@max
Begin
Select @script='"Dtsrun /S "' + @@SERVERNAME + '" /E /N ' + cmd
+ case when phone > 2500
then '/A ExportID="' + convert(varchar(10),(Select coalesce(Max(listid),0) + 1 from tbl_sf3Lists)) + '" '
else ' ' end
, @id=@id+1
from #temp
Where rowno = @id
End
Return
GO
View 1 Replies
View Related
Feb 18, 2008
Hi
I have an SSIS package that wont run whne i try to run it in one go, but if i run it step by step it will succeed
as the reason its failing is in the 2nd step it references a table that doesn't exist (its created in the 1st step.
So at the start package is trying to validate, but it won't as the table is created in step 1.
So this is why it works if i execute this step by step, is there a way of turning off the pre run validation
on SSIS scripts ?
Thanks
View 1 Replies
View Related
Nov 30, 2006
HI, I have some parent parent packages that calls child packages. When I added a bunch of packages, I faced the buffer out of memory error. I then decided to set the child packages property ExecuteOutOfProcess to TRUE. I noticed that the execution time is longer now. Is this a good practice to set the ExecuteOutOfProcess to true? If so, is it normal that the execution time is longer?
Thank you,
Ccote
View 2 Replies
View Related
Dec 29, 1999
Please help,
I have a DTS package that consist of approximately 50 steps which pulls data from an as400 and populates a SQL 7.0 database. The package works perfect flawlessly when executed manually but won't run when scheduled.
The DTS package starts but hangs when it tries to connect to the as400, this is where the problem lies. The preceeding steps work fine but they execute against the SQL 7.0 database. I have client access installed on the server and configured. I checked and made sure the SQL server agent service was started and running also,I just can't figure why it works fine manually,but not when scheduled.
Any help would be welcome...
View 2 Replies
View Related
Aug 17, 2007
I am seeking a walkthrough for executing packages via a web service. All I have found so far are fragmented bits of information. This article is a good start but it leaves out some critical security setting information. http://msdn2.microsoft.com/en-us/library/ms403355.aspx#service.
Does anyone know of a good walkthrough that includes the security setup: Impersonation, Proxies, etc.?
View 6 Replies
View Related
Mar 28, 2007
Question is in the subject.
The reason I'm asking is because I want a workaround to a problem that a guy is having here:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1404349&SiteID=1
-Jamie
View 1 Replies
View Related
May 4, 2015
I have 12 packages to execute in order. I made a table in my DB where i mentioned the name of each package and his order in execution.
I want to create a master package that get the name and order from my DB table to execute all packages.
View 3 Replies
View Related
Oct 27, 2015
I have 100 packages all of them stored in "Integration services catalog". Is there a way to execute packages using script task in ssis 2012.
View 5 Replies
View Related
May 27, 2008
I've been executing a package and passing a parent variable to the child using package configurations but I'd now like to do this using a script task. The script task would then programmatically load the package and execute it.
How do I do this and still use the parent variable?
I've found examples of how to load a package but I haven't been able to find out how to I load it specifying the parent variables.
I think it's possible. MSDN shows the available methods but the example is for the base method.
http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.package.execute.aspx
Cheers,
Ben
View 1 Replies
View Related
Jun 15, 2006
I am trying to migrate our processing from command line based scripts and foxpro to SQL so I need to run the SSIS packages using dtexec. I copied the dtexec file and a few dll's that are missing to our production servers but i cant execute the packages. I dont want to install the full client tools (particularly managment/business inteligence studio) on our production servers due to the overhead and limited system disk space.
Can somebody tell me what the minimum install would be so I would be able to run SSIS packages using the dtexec or dtexecui tools? I would also like to install some of the other command line client tools like osql etc.
View 8 Replies
View Related
Jun 26, 2015
I have created for each container to call all the packages in a folder like below, also created a variable.
Then I add execute package task inside of foreach container and selected file system in a location and in connection called currently creating package name finally in connection properties i added variable in expression which i created and mapped into for each loop container. I referred below link
[URL] ....
All the packages are running but its not ending once all the packages executed its re run and continue the running process, how to stop once all the packages execute.
View 24 Replies
View Related
Sep 18, 2007
Hi all,
I have a problem while i create a proxy account.The situation is like this...There is a user who has an login in to the server.He has a stored procedure which calls some on the SSIS packages and XP_cmdshell...so this stored procedure basically load some data in to the tables .So for the login in order to execute the stored procedure as he is not a Sys admin I have created a proxy account in my account as Iam an SA and then in the proxies and in principals I selected his login name and this way I have created a credential and a proxy account.
Now the problem is if he logins with his id and password and try to execute the stored procedure it gives an error message
Server: Msg 15153, Level 16, State 1, Procedure xp_cmdshell, Line 1
The xp_cmdshell proxy account information cannot be retrieved or is invalid. Verify that the '##xp_cmdshell_proxy_account##' credential exists and contains valid information.
....so this mean the login is not able to see the proxy account.So what I did is I created a job and then in the job owner tab I have selected his login and then created a step with the type operating system (CmdExec) as I need to just execute the stored procedure and used the proxy account that I have created.
so I gave the command -- exec <stored procedure> --.
But this job fails and gives the error message as
[298] SQLServer Error: 536, Invalid length parameter passed to the SUBSTRING function. [SQLSTATE 42000]....
So now ....first My question is am I doing in a right way....if its right then why Iam not able to execute the stored procedure.
If there is any other way through which I can execute the stored procedure using a proxy account for the logins who are not sys admins....please do let me know.....
Thanks
Raja.V
View 2 Replies
View Related
Jan 2, 2002
I have several independant DTS packages that I would like to schedule and run as 1 job stream. (sql2000) I can schedule them individually, but I would prefer to have each one be a step of 1 big scheduled job.
Any ideas ?
View 2 Replies
View Related
Apr 24, 2008
I have a rather dumb question, it appears that we can have more than 1 package (.dtsx) in a solution. If I have multiple packages within the same solution, how do I invoke them from the main package in the solution? Thanks in Advance.
View 5 Replies
View Related
Jan 20, 2006
I'm having serious problems with the IDE for SSIS for projects that contain more than 5 packages. Especially if these packages call each other with a run package task thats configured with a file connection. Especially annoying are the 20+ "Document contains one or more extremely long lines of text.." messages that pop up during loading / validating. For my project with around 30 packages it takes me around 10 minutes to click through all these pop ups in addition to the long loading time.
Anyone got any tips on this specifically or how to improve performance in the IDE in general? As it is now, the product is a REAL pain to work with for large projects.
View 2 Replies
View Related
Jun 28, 2006
Greetings SSIS friends,
If I have more than one package within one project, how do I select which package I want to run first?
Thanks in advance.
View 7 Replies
View Related
