SSIS Logging Issues When Package Run From Job/Command Line
Feb 13, 2008
All,
I am logging all the tasks in my SSIS package to SQL Server. For each task I am logging atleast the Pre-Execute, PostExecute, OnError events. For Script tasks, I have custom logging and I am logging the ScriptTaskLogEntry event too.
When I run the package manually from BIDS, the logging works great ! But when I try to run the package from a job or from the command line, the number of events that is logged greatly reduces. For eg. when manually run, I get 104 records in the log table but when run from the command line I get 23 records only. Most of the custom logging messages from the Script Task do appear. Its the pre and post-execute events that are skipped. Any idea why ?
Here is command line from the job. I also use the same command line with "dtexec" from the command prompt.
I created a SSIS package that has a flat file manager whose connection string is a package variable, the reason is that there is a foreachfile container loop to loop thru CSV format files in a directory and load them to a sql table.
The package execution is working in the designer studio as well as in the management studio-- a copy was saved to a sql 2005 server where a sql job is deployed to run the package. However when I ran the job, it claims successful but doesn€™t do anything in reality. If I checked the box failing the package upon validation warming, the job will fail with the error: the command line parameters are invalid. The command line looks like below:
One thing that I think maybe wrong is on the data source tab of the job step GUI, the flat file manager€™s connection string is blank, compared to other connection managers having related values.
I see a weird behaviour when the package is executed as a SQL Server job and via command line. The package has environment variables configured. So when I run the package via a command line it picks up the user environment variables and gets executed fine with success. But when the package is scheduled as a job, it fails as it is not picking up environment variables. Does the SQL server has to be restarted to pick up the environment variables? I dont understand the reason of not picking up the environment variables when the package is run as a job. The job runs as the same user as the command line and the user variables are set for the environment variables. The logging is enabled on the package and I can see that it is logging as the same user even if its command line execution or executed as a job. Could you please anyone can explain why the package fails as a job and doesnt fail via command line. I am fighting with this for a long time.
Hi, I'm looking into the idea of building an enhanced version of dtexec.exe that builds in some extra logging features. My utility will execute packages using the Package.Execute() method.
Thing is, I'd still want to support all of the command-line options that dtexec supports. For example, my utility should accept "/set package.variables[myvariable].Value;myvalue" and pass it through to the executing package but I can't find a way of doing it using Package.Execute().
Am I missing something or is this just not possible?
I have a ForEachLoop parent pkg that reads thru the file system of dtsx to be executed.
I would like to pass CLI parms to the command itself (dtexec). I'm not looking at passing values to the child pkgs but rather additional command line values such as:
/CONFIGFILE "C:myconfig.dtsConfig.
In my expression builder for the PackageName I added the additional string but during the execution, the reference to my dtsconfig is ignored without any errors and the child pkg runs using the default values provided at design time.
Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 9:52:49 AM Warning: 2006-04-05 09:52:51.58 Code: 0x80012018 Source: Archive Data Description: The configuration entry, "Account_Number", has an incorrect form at because it does not begin with the package delimiter. Prepend "package" to t he package path. End Warning Warning: 2006-04-05 09:52:51.58 Code: 0x80012017 Source: Archive Data Description: The package path referenced an object that cannot be found: "Acc ount_Number". This occurs when an attempt is made to resolve a package path to a n object that cannot be found. End Warning DTExec: Could not set Account_Number value to '00001'. Started: 9:52:49 AM Finished: 9:52:51 AM Elapsed: 2.172 seconds
I'm trying to run have SQL Server Agent excute an SSIS package from the command line and I keep recieving an error message. I will assume that I have miss typed something. Can someone validate that the execute line looks workable?
In one of my SSIS package (which connect to DB2) i save the DB2 username and password in SSIS packge itself by using option "EncryptAllWithPassword". When I try to run the packge using dtexecui.exe it works perfectly fine. It prompts me for package password and then run fine.
Now I am trying to run it using command line dtxece.exe and i get fooolwoing error. Please suggest and help!!! DTExec.exe /FILE C:pkgDB2Load.dtsx /CONFIGFILE C:ConfigpkgConfigMaster.dtsConfig /Decrypt mypassword /CHECKPOINTING OFF /REPORTING EWCDI
Error Message: ---------------------- Description: An OLE DB error has occurred. Error code: 0x80040E21. An OLE DB record is available. Source: "IBM OLE DB Provider for DB2 Servers" Hresult: 0x80040E21 Description: " SQL30082N Attempt to establish connection fa iled with security reason "17" ("UNSUPPORTED FUNCTION"). SQLSTATE=08001 "
We have a package that has a connection called Load_DimItem.trc. We don't need this logging enabled for this package anymore. However, if I delete the connection, and delete the log provider (SSIS log provider for SQL profiler), I get errors when trying to close the package after debugging. I get: "Cannot detach from one or more processes. [3172] The object invoked has disconnected from its clients."
I have a Sql agent job which will exeute a SSIS package. While, one of my data sources is an excel file. I scriptted the sql agent job and recreate on a test server, it fails with The command line parameters are invalid.. The command line has the following for the excel data source, and that is the place fails the job. /CONNECTION UserDataExcel;"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\myserverUserDataUserData.xls;Extended Properties=""EXCEL 8.0;HDR=YES"";" I guess it has something to do with the double-quote inside, Please advise.
I am having the same problems as those in another post. SSIS package works fine when executed in BIDS and through execute package utility but it doesnt work when executed as a step in a job.
The other problem is that the logging also doesnt work when i try executing it as a job. So I have no clue about what to do without knowing what error it is. When I run the job it simply says the step has failed.
I have tried most of the solutions posted in other websites most of them to do with using proxies with credentials but havent hit a solution. I would love to get any input on what to do.
I am using the "SSIS Log Provider for SQL Server" to log events to a table for "OnError" and "OnPostExecute" events of a package. This works as expected and provides a nice clean output on the execution steps of the package.
I am curious as to why I do not see any detail for any/all tasks that fall under the "Data Flow" section of the package though. For instance, on my "Control Flow" tab, I added a "Data Flow" task that simply loads a few tables from a target to destination server. However, there is nothing shown in the logging output. Just that a Data Flow task was initiated. And when I'm configuring this logging under "SSIS-->Logging" in the checkbox area on the left, you cannot "drill into" data flow steps.
Is there a reason why there is no detailed logging for Data Flow tasks? Would getting to that require me to create a custom log provider?
I want to trim new line from string in sql server 2005...
I prepared one dtsx package and in that I want to trim new line from one of my column but i am unable to do that.
for example:
I want to retrive one column name DESCRIPTION and put that in text file but along with it I am getting blank space and new line also that I want to trim...
Hi, I decided to use the SQL Server log provider to store logging data of all my Integration Services packages. I also created some reports about this data for operating purposes. I have a problem occurs the name of the executing package is not always written to the log,but the name of the single task which failed. But that is not very useful information for operating, because I do not see any chance to get the name of the package by the information which is logged in the sysdtslog90 table in the database which I defined for SSIS Logging.
How do I configure the package to always log the package information into the table, too?
All, I am developing the data mart refresh task in SSIS. I wanted to call the package in command prompt. I need help. Can any one give any tips on it. Thanks and appricate in advance.
1. (calls an oracle package) - call lpaarchive.pibrdg.setlastbridgeruntime(sysdate);
2. (selects rows) - select timestamp ,
pitag ,
rtlmp from LPAARCHIVE.I_PIBRDG_BUS5MINRTLMPS
where timestamp <= (sysdate + 0.002777778)
order by TIMESTAMP ASC; I need to execute these 2 statements together. I tried using Execute SQL task to call the package and then an OLEDB source that calls a variable with the select statement. But it does not retrieve any rows. It seems like the result of calling the package is used by the select statement to give the final rows. Could anyone please help me resolve this issue.
How to execute ssis package from command prompt and also pass configuration file to it and set logging to ssis log provider for sql server. Writing all those options with cmd.
I have an SSIS package that runs fine through command pormpt although when I try to run it from a SQL Servr Agent Job CmdExec step it bombs out. Please help this has me stumped...the SSIS package uses an XML connection string so certain key settings such as connection strings and email info can be changed easily. Currently this is all on the same machine. I have not moved it beyond where I am developing.
On the command line I am using the following command...
Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 6:59:40 PM Progress: 2006-08-16 18:59:41.29 Source: Data Flow Task Validating: 0% complete End Progress Progress: 2006-08-16 18:59:41.29 Source: Data Flow Task Validating: 33% complete End Progress Progress: 2006-08-16 18:59:41.71 Source: Data Flow Task Validating: 66% complete End Progress Progress: 2006-08-16 18:59:41.73 Source: Data Flow Task Validating: 100% complete End Progress Progress: 2006-08-16 18:59:41.77 Source: Data Flow Task Validating: 0% complete End Progress Progress: 2006-08-16 18:59:41.77 Source: Data Flow Task Validating: 33% complete End Progress Progress: 2006-08-16 18:59:41.77 Source: Data Flow Task Validating: 66% complete End Progress Progress: 2006-08-16 18:59:41.77 Source: Data Flow Task Validating: 100% complete End Progress Progress: 2006-08-16 18:59:41.79 Source: Data Flow Task Prepare for Execute: 0% complete End Progress Progress: 2006-08-16 18:59:41.79 Source: Data Flow Task Prepare for Execute: 33% complete End Progress Progress: 2006-08-16 18:59:41.79 Source: Data Flow Task Prepare for Execute: 66% complete End Progress Progress: 2006-08-16 18:59:41.79 Source: Data Flow Task Prepare for Execute: 100% complete End Progress Progress: 2006-08-16 18:59:41.81 Source: Data Flow Task Pre-Execute: 0% complete End Progress Progress: 2006-08-16 18:59:41.84 Source: Data Flow Task Pre-Execute: 33% complete End Progress Progress: 2006-08-16 18:59:41.90 Source: Data Flow Task Pre-Execute: 66% complete End Progress Progress: 2006-08-16 18:59:41.90 Source: Data Flow Task Pre-Execute: 100% complete End Progress Progress: 2006-08-16 18:59:41.92 Source: Data Flow Task Post Execute: 0% complete End Progress Progress: 2006-08-16 18:59:41.92 Source: Data Flow Task Post Execute: 33% complete End Progress Progress: 2006-08-16 18:59:41.92 Source: Data Flow Task Post Execute: 66% complete End Progress Progress: 2006-08-16 18:59:41.92 Source: Data Flow Task Post Execute: 100% complete End Progress Progress: 2006-08-16 18:59:41.92 Source: Data Flow Task Cleanup: 0% complete End Progress Progress: 2006-08-16 18:59:41.93 Source: Data Flow Task Cleanup: 33% complete End Progress Progress: 2006-08-16 18:59:41.93 Source: Data Flow Task Cleanup: 66% complete End Progress Progress: 2006-08-16 18:59:41.93 Source: Data Flow Task Cleanup: 100% complete End Progress Progress: 2006-08-16 18:59:41.95 Source: Send Mail Task The SendMail task is initiated.: 0% complete End Progress Progress: 2006-08-16 18:59:42.09 Source: Send Mail Task The SendMail task is completed.: 100% complete End Progress DTExec: The package execution returned DTSER_SUCCESS (0). Started: 6:59:40 PM Finished: 6:59:42 PM Elapsed: 1.984 seconds
When I try to use the same command within SQL Server Agent Job using a CmdExec step I get the following error...
Description: The package is attempting to configure from the XML file "S:connectionscontacts.dtsConfig". End Info Warning: 2006-08-16 18:40:03.15 Code: 0x80012012 Source: contactsPackage Description: The configuration file name "S:connectionscontacts.dtsConfig" is not valid. Check the configuration file name. End Warning Warning: 2006-08-16 18:40:03.15 Code: 0x80012059 Source: contactsPackage Description: Failed to load at least one of the configuration entries for the package. Check configurations entries and previous warnings to see descriptions of which configuration failed. End Warning Info: 2006-08-16 18:40:03.20 ... Process Exit Code 1. The step failed.
I recently read the project real ETL design best practices whitepaper. I too, want to do custom logging as I do today, and also use SSIS logging. The paper recommended using the variable system::PackageExecutionId to tie the 2 logging methods together.
Using Query Analyzer, I can right click on an object and select "scriptobject to new window as create" and I get the text of the object'sdefinition (schema). Can I get same result from command line, i.e., fromosql, I can get text output for the definition of the object (something likedefncopy under Sybase)? If so, what is the command or store procedure name?Thanks in advance.
We already used Oracle Datasatage Server the following Query statement for Source and Lookup.here there is parameter maping in the SQl Statement . How can achive in SSIS the Folowing Querystatment?
Query 1: (source View Query) SELECT V_RDP_GOLD_PRICE.GDR_PRODUCT_ID, V_RDP_GOLD_PRICE.ASSET_TYPE, V_RDP_GOLD_PRICE.PREFERENCE_SEQ, V_RDP_GOLD_PRICE.RDP_PRICE_SOURCE, TO_CHAR(V_RDP_GOLD_PRICE.PRICE_DATE_TIME,'YYYY-MM-DD HH24:MI:SS'), TO_CHAR(V_RDP_GOLD_PRICE.REPORT_DATE,'YYYY-MM-DD HH24:MI:SS'), V_RDP_GOLD_PRICE.SOURCE_SYSTEM_ID FROM V_RDP_GOLD_PRICE V_RDP_GOLD_PRICE WHERE REPORT_DATE = (select max(report_date) from V_RDP_GOLD_PRICE where source_system_id = 'RM' )
Query 2: (look up )
SELECT GDR_PRODUCT_ID, TO_CHAR(MAX(PRICE_DATE_TIME),'YYYY-MM-DD HH24:MI:SS') , TO_CHAR(REPORT_DATE,'YYYY-MM-DD HH24:MI:SS') FROM V_RDP_GOLD_PRICE where GDR_PRODUCT_ID = :1 and report_date = TO_DATE(:2,'YYYY-MM-DD HH24:MI:SS') AND PRICE_DATE_TIME BETWEEN TO_DATE(:2,'YYYY-MM-DD HH24:MI:SS') - 7) AND TO_DATE(:2,'YYYY-MM-DD HH24:MI:SS') GROUP BY GDR_PRODUCT_ID, TO_CHAR(REPORT_DATE,'YYYY-MM-DD HH24:MI:SS')
please anyone give the sample control flow and how to pass the parameter?
hi I would appreciate if someone demonstrate how to automate ftp in a command line from within a batch file. I do want to move certain files from one server to another via ftp command line in an automatic fashion via running the batch periodically.
