Automate SSIS Package For Monthly Pull From SAP ECC 6.0
May 7, 2008
I want to automate SSIS package to pull data from SAP ECC 6.0 on monthly basis.
I am trying to use Microsoft .NET Data Provider for mySAP Business Suite which will call RFC_READ_TABLE function in SAP.
i.e.
Source
SAP Table: CE1LSC0
SAP Field: Budat (Calendar Date Field) Data Type : varchar Format: YYYYMMDD
Interface
Microsoft .NET Data Provider for mySAP Business Suite
Destination
SQL Table: TCE1LSC0
Right now I am pulling data manually from Import / Export task
SQL Script which i am to using to pull last month data:
Filter: To bring data on monthly basis ( If i run the package in May then i will like to pull data for month of April)
Code Snippet
select * from CE1LSC0
where CE1LSC0.BUDAT between 20080401 and 20080430
Now to automate
Call SAP Function Module: RFC_READ_TABLE requires following variables
@QUERY_TABLE
@DELIMITER
@OPTIONS
@FIELDS
Script:
Code SnippetEXEC rfc_name[{value | @variable [OUTPUT]}][,...n][@parameter = {value | @variable [OUTPUT]}][,...n] [;] SQL Stored Procedure to get variable: @LastMonthFirstDay , @ LastMonthLastDayCode SnippetCreate proc sp_GetDateasDECLARE @Today datetime, @FirstDay datetime , @LastDay datetime , @LastMonthFirstday int, @LastMonthLastDay intSET @Today = CAST(CONVERT(nvarchar(10), GETDATE(), 120) AS datetime) -- strips time offSET @LastDay = DATEADD(day, - DAY(@Today), @Today) -- last day of previous monthSET @FirstDay = DATEADD(month, -1, DATEADD(day, 1, @LastDay)) -- first day of previous monthset @LastMonthFirstday = Cast(Replace(Convert(varchar(10), @FirstDay, 120), '-', '') As int)--@FirstDay set @LastMonthLastDay = Cast(Replace(Convert(varchar(10), @LastDay, 120), '-', '') As int)--@LastDayselect @LastMonthFirstday as 'LastMonthFirstday', @LastMonthLastDay as 'LastMonthLastDay' My problem is that i am not able to provide my stored procedure variable to SAP RFC to automate my ssis package. Can anyone suggest something or advice how to automate my package... I have went through all the sap.net connector site and related documents but it of no use as it work with visual studio 2003 and I am using visual studio 2005 / MS SQL SERVER 2005....
I'm not at all comfortable with SSIS so please forgive me if I overload you all with information here:
I need to create a data table using SSIS which does not delete the previous days data. So far all the data tables we use to write reports in Visual Studio are constructed in SSIS as follows.
1 - Excecute SQL Task - DELETE FROM STOCK 2 - Data Flow Task 3 - Data Reader Source - SELECT * FROM ODBCDATASOURCE 4 - OLE DB Destination (Creates table STOCK)
The data tables which are created this way are stored in a data warehouse and scheduled to refresh once a day, which means that any data from yesterday is lost when the updates run. So, I tried to create a table which never has its previous days' data deleted by using just the last three steps above - and it worked great in Visual Studio, no problem at all. However, when I added this SSIS Package to the Update Job in SQL Server Management Studio, the job totally rejected the packed with the message: "The command line parameters are invalid. The step failed".
I thought I could work around this problem by asking the job step to excecute a simple SQL query to insert the data from table1 into table2 (and would thus negate the need for a SSIS Packege at all), but it threw me a curve ball with some message about not being able to use proxy accounts to run T-SQL Scripts.
If anyone knows how to create a SSIS package in which the data never expires please could you impart some wisdom my way. I only need to do this once for a specific report.
Please, when answering, bear in mind that I'm a simple fellow with little understanding of the inner workings of SQL Server and its various components, so please use short sentences and simple words.
Is there any way to automate SSIS deployment? (SSDT for SQL 2014)
I created a SSIS project and a TFS build definition, but while the build succeeds there's nothing in the Build drop folder.
Hence a question - how to automate the deployment?
Ideally i'm looking to something similar to DB projects where VS (or Release Management) get the files generated by the Build and deploy them to the target DB.Â
We are using SQL Server 2014 and SSDT-BI 2013. We have a reporting environment where business users create objects which need to be persisted for fiscal year reporting. Let's say for instance SQLSERVER1SRVR1 they create table objects like below in the reporting environment.
Accounting2014, Accounting2015 in AccountingDB;Â Sales2014, Sales2015 in SalesDB;Â Products2014, Products2015 in ProductsDB;Â Inventory2014, Inventory2015 in InventoryDB etc....
These tables are persisted for auditing in a different environment SQLSERVER2SRVR2 for finance & audit folks.We would want to automate this process using SSIS to create tables in corresponding database and load data. I tried using For Each Loop container but the catch is I could loop the source or destination but how do we loop on Source & Destination at the same time (i.e when source is in AccountingDB destination to be AccountingDB, source SalesDB then destination SalesDB so on etc....
Hey, I've a few jobs which call SSIS packages. If I run the SSIS package, it runs fine but if I try to run the job which calls this package, it fails. Can someone help me troubleshoot this issue? None of my jobs that call an SSIS package work. All of them fail.
I've been tasked to do some investigation at my workplace as to the best way to move forward with changing from our SQL 2000 DTS' to SQL 2005 SSIS packages.
Our main issue at the minute seems to be that we are using ODBC drivers to pull many of the raw tables which we then manipulate into our own SQL Server tables.
I've been looking about and am I right in saying there is an issue with this in SQL 2005?
Also, I've been looking for some Tutorials to help me on this, would anyone be able to push me in the right direction of any good tutorials that would help me in fixing these issues?
Anyone have any experience in connecting to a Basis database? I want to build an SSIS package to ETL data out of a Basis database and into a sql server 2005 db. I have downloaded the ODBC driver, but am not usre how to plug it in to SSIS to do teh extraction.
Also I am totally new to Basis...it is not an RDBMS at all but the literature says it supports SQL queries, so I am not sure about where to start.
i have the following stored proc which returns a resultset at the end, i have an SSIS package that calls this stored proc and outputs the result to a file. However, the package fails because it cannot pull the columns for the schema because of the return lines in the middle of the stored proc. If i remove the it works fine, pulls hte columns as normal. but if i leave them in ssis cannot get the columns. what can i do to get around this?
ALTER PROCEDURE [dbo].[uspCreateBrightPointFile]
AS
SET nocount ON
IF EXISTS (SELECT TOP 1 *
FROM brightpointfile)
TRUNCATE TABLE brightpointfile
-- Get the order information from the database where vendorconfirmationID = 0
INSERT INTO brightpointfile
SELECT o.orderid,
o.requestid,
'295193' AS araccountnumber,
o.orderdate,
'PRIORITY' AS shipmethod,
'Asurion Dobson' AS billname,
'PO Box 110808' AS billaddress1,
'Attn Account Receivable' AS billaddress2,
' ' AS billaddress3,
'Nashville' AS billcity,
'TN' AS billstate,
'37222' AS billzip,
c.fullname AS shipname,
Replicate(' ',100) AS shipaddress1,
Replicate(' ',100) AS shipaddress2,
' ' AS shipaddress3,
Replicate(' ',40) AS shipcity,
' ' AS shipstate,
Replicate(' ',10) AS shipzip,
'1' AS linenumber,
ve.sku AS itemcode,
o.quantity AS qty,
r.typeid,
r.customerid,
0 AS addressfound
FROM [order] o WITH (NoLock)
JOIN request r WITH (NoLock)
ON o.requestid = r.requestid
JOIN customer c WITH (NoLock)
ON r.customerid = c.customerid
JOIN (SELECT [subequipid],
[subid],
[clientequipid],
[serialno],
[statusid],
[startdate],
[enddate],
[createdate],
[createuserid]
FROM subequip s1 WITH (NoLock)
WHERE s1.statusid = 1
AND s1.startdate = (SELECT MAX(s2.startdate)
FROM subequip s2 WITH (NoLock)
WHERE s2.statusid = 1
AND s2.subid = s1.subid)) se
ON r.subid = se.subid
JOIN vendorequip ve WITH (NoLock)
ON se.clientequipid = ve.clientequipid
JOIN clientequip ce WITH (NoLock)
ON ce.clientequipid = se.clientequipid
WHERE vendorconfirmationid IS NULL -- order was never sent to CellStar
AND ve.typeid IN (1) -- only pull direct fulfillment (not store fulfillment)
AND ve.vendorid IN (2) -- only pull vendor = CellStar Insurance
AND o.orderdate > '2007-08-01' -- only pull orders after 08/01/2007
--IF @@ERROR <> 0
--RETURN 1
--First use the address types of 2
UPDATE brightpointfile WITH (ROWLOCK)
SET shipaddress1 = b.address,
shipaddress2 = b.address2,
shipcity = b.city,
shipstate = b.stateid,
shipzip = b.zipcode,
addressfound = 1
FROM customeraddress a WITH (NOLOCK),
address b WITH (NOLOCK)
WHERE a.customerid = brightpointfile.customerid
AND b.addressid = a.addressid
AND a.typeid = 2
--IF @@ERROR <> 0
--RETURN 2
--Update the rest where the address type is 1 and there is no type 2
UPDATE brightpointfile WITH (ROWLOCK)
SET shipaddress1 = b.address,
shipaddress2 = b.address2,
shipcity = b.city,
shipstate = b.stateid,
shipzip = b.zipcode
FROM customeraddress a WITH (NOLOCK),
address b WITH (NOLOCK)
WHERE a.customerid = brightpointfile.customerid
AND b.addressid = a.addressid
AND a.typeid = 1
AND brightpointfile.addressfound = 0
--IF @@ERROR <> 0
--RETURN 3
--Select all the records from the temp table, plus union in 2 extra records required by the client
I'm developing a Windows Mobile application, which is using RDA Pull for retrieving data from SQL Server 2005 database to PDA. Please, see the example:
Code Snippet
using (SqlCeEngine engine = new SqlCeEngine(connStr))
the sqlcesa30.dll cannot connect to SQL Server database.
In the sqlcesa30.log then I found following line:
Code Snippet
2007/04/17 10:43:31 Thread=1EE30 RSCB=16 Command=PULL Hr=80040E4D Login failed for user 'test'. 18456
The user 'test' is member of db_owner, db_datareader and public roles for the Demo database and in SQL Server Management Studio I'm able to login to the Demo database with using the 'test' users credentials and I'm able to run the select command on 'mytable'.
So, what's wrong? Why the sqlcesa30.dll process cannot login to the Demo database, and from another application with using the SAME connection string it works?
We have a custom web C#/SQL2K8R2 workflow application that I need to pull Oracle data into a varchar(max) field as an XML DOM document. I have no problem pulling the Oracle data using OLEDB, but I'm not sure how to create the XML DOM doc. Once I get it into the DOM doc, I then need to assign metadata about the XML DOM doc and insert it all into a staging table:
CREATE TABLE [stg].[EtlImports]( [EtlImportId] [int] IDENTITY(1,1) NOT NULL, [EtlSource] [varchar](50) NOT NULL, [EtlType] [varchar](50) NULL, [EtlDefn] [varchar](max) NULL, --Either a SQL statement or path to file on disk [EtlData] [varchar](max) NULL, --BLOB field to hold the XML data or FILESTREAM path to file on disk [EtlDateLanded] [datetime] NOT NULL, [EtlDateProcessed] [datetime] NULL, [EtlStatus] [varchar](50) NULL, [Comments] [varchar](4000) NULL ) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
I will have a separate SSIS package to pull the [XML/File] field and process the data into the workflow tables. Is there a wasy I can use the ADO Record-set Destination task to accomplish this, or do I have to create a custom C# script to create the XML DOM Doc?
I've got an SSIS solution file with project deployment model in VS 2013 and would like to deploy that to SSISDB on different environments.All these days I followed the regular way to create a project in SSISDB and deploy it to that. Now want to find out if i can automate this process and so got some questionsÂ
1. Can we automate the process of creating a project on SSISDB based on our SSIS project name? This will be like when we do a deployment it should check if the project exists or not on SSISDB based on our SSIS project name, if the project exists we just deploy the packages in the project and if the project does not exists in SSISDB it will create that project and deploy the packages.
2. Can we also automate the process of creating environments? In traditional way we manually create the environment variables under environment tab of SSISDB, but can we make that also as part of deployment? Like when we are releasing to Dev server we look if that particular Dev variable exists on that server, if it exists we just update the existing stuff and if it does not exists we just create it.
And there is a task (Execute SSIS package) in First package that calls the execution of second package.
I m continuously receiving an error "Failed to decrypt protected XML node "PackagePassword" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available."
As we are running first package by job, job runs successfully logging above error
The protection level of second package is set to "EncryptSensitiveWithUserKey"
I have a SSIS job, one of the last steps it performs is to execute a SQL 2000 DTS package. This has to be done as a SQL 2000 DTS package as it is performing rebuilds of SQL 2000 Analysis Services dimensions and cubes. We've found that when the DTS fails the SSIS job is happily completing showing as a success, we would prefer to know it went wrong.
As far as I'm aware SSIS merely starts the DTS off and doesn't care about it's result. I've taken a look in to turning on the logging for the execute DTS package and thought that the ExecuteDTS80PackageTaskTaskResult would give me the answer I need...but is merely written to the log not available as an event-handler. It also looks like it is not safe to put a SQL task in as the next item to go look at the SQL 2000 system tables to look at the log for the DTS package as the SSIS documentation warns that the DTS package can continue to run after the execute DTS package task has ended.
Ideally I want any error raised within the DTS package to cascade up to be an error in the SSIS job, I can then handle it appropriately. I cannot find a way to do this. Is there a way?
If not, can anyone suggest how in the remainder of the SSIS tasks I can be sure that the DTS has completed before I start any other tasks that will check for the SQL 2000 log of its execution?
I have developed an SSIS package for ETL purpose. I am invoking the SSIS package through .Net console application by referencing the ManagedDTS Assembly. I am able to execute the package in Sql Server 2005 Developer Edition and it runs fine till completion.
But when i try to execute the packahe in Sql Server 2005 Standard edition, by invoking the package through .Net console application the status of the package is failure.
Can any one help me how to over come this problem.
I am in the process of moving from a 32-bit SQL Server 2005 Enterprise (9.0.3054) to a 64-bit SQL Server 2005 Enterprise (9.0.3054 with 4 CPUs and 8GB of memory on Win 2003 SP2) and the process has been very frustrating to say the least. I am having a problem with packages that I created on my 64-bit SQL Server. I am importing a few tables from the 32-SQL Server into the 64-bit SQL Server using the Task --> Import to create the package.
Sometimes when I am creating a package I get the following error in a message box:
SQL Server Import and Export Wizard
The SSIS Runtime object could not be created. Verify that DTS.dll is available and registered. The wizard cannot continue and it will terminate.
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.Windows.Forms)
Other times when I run a package that has run successfully before I get the following error:
Faulting application dtexecui.exe, version 9.0.3042.0, stamp 45cd726d, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x025d23f0.
The package appears to hang when running. By this I mean that the Package Execution Progress shows progress up to a point then it just stops. (The package takes about 17 seconds to run normally) CPU usage is at 1% and the package cannot be stopped.
I have deleted and re-created the package several times and I have also re-installed the service pack on the SQL Server (9.0.3054) but that did not help.
I would like to standardize SSIS development so that developers all start with the same basic template. I have set it up so it is an available template ( http://support.microsoft.com/kb/908018 ) but I would like it to be the default when a new project or package is created. Is this an option?
I would like to fetch the data flow component name while package is executing. Since system variable named [System::SourceName] only fetches name of the control flow tasks? Is there a way to capture them?
The master package has a configuration file, specifying the connect strings The master package passes these connect-strings to the child packages in a variable Both master package and child packages have connection managers, setup to use localhost. This is done deliberately to be able to test the packages on individual development pc€™s. We do not want to change anything inside the packages when deploying to test, and from test to production. All differences will be in the config files (which are pretty fixed, they very seldom change). That way we can be sure that we can deploy to production without any changes at all.
The package is run from the file system, through a job-schedule.
We experience the following when running on a not default sql-server instance (called dkms5253uedw)
Case 1: The master package starts by executing three sql-scripts (drop foreign key€™s, truncate tables, create foreign key€™s). This works fine.
The master package then executes the first child package. We then in the sysdtslog get:
Error - €œcannot connect to database xxx€? Info - €œpackage is preparing to get connection string from parent €¦€?
The child package then executes OK, does all it€™s work, and finish. Because there has been an error, the master package then stops with an error.
Case 2: When we run exactly the same, but with the connection strings in the config file pointing to the default instance (dkms5253), the everything works fine.
Case 3: When we run exactly the same, again against the dkms5253uedw instance, but now with the exact same databases defined in the default instance, it also works perfect.
Case 4: When we then stop the sql-server on the default instance, the package faults again, this time with
Error - €œtimeout when connect to database xxx€? Info - €œpackage is preparing to get connection string from parent €¦€?
And the continues as in the first case
From all this we conclude, that the child package tries to connect to the database before it knows the connection string it gets passed in the variable from the master package. It therefore tries to connect to the default instance, and this only works if the default instance is running and has the same databases defined. As far as we can see, the child package does no work against the default instance (no logging etc.).
We have tried delayed validation in the packages and in the connection managers, but with the same results (error).
So we are desperately hoping that someone can help us solve this problem.
I am interested in Passing value from a child Package variable to the Parent package that calls it in ssis.
I am able to call the Child package using the execute package task and use Configurations to pass values from the parent variable to the child, but I am not able to pass the value from the child to the parent.
I have a variable called datasetId in both the parent and child. it gets computed in the child and needs to be passed to the parent...
Deployed Report having SSIS package as source do not work when Indirect Package configuration is used in ETL package. It seems ETL package when called/executed from Report manager does not recognize environment variable to pick up the dtsconfig file.
The Report works when Direct package configuration is used to same dtsconfig file.
What could be the reason? Any solution for this? This will cause our build/deployment to QA and Prod very difficult.
I now have two SSIS package, "TESTING" and "LOADING". The "TESTING" package have an execute package task that call the "LOADING" package. When I want to execute the TESTING package, how can I setup the connection string so that I can edit the password of the database connected by the "LOADING" package?
I have two SSIS packages in a project, one calling the other. The parent package works fine in my local mechine. After they are deployed to the production, I schedeul jobs to run the packages in the SqlServer. The child package works fine if I run it alone, but the parent package could not find its child package if I run the parent package . As I checked, all xml config files and the connection string pointing to the child package were set correctly. It seems the parent package did not use the xml config file. Can someone help me? Thanks in advance.
I have successfully created a SSIS package which execute a DTS 2000 package and with no problem to execute the task. But I failed to schedule this package. I was not success in setting the logging. When running the package in command line:
dtexec file "C:Documents and SettingslyangMy DocumentsVisual Studio 2005ProjectsTraingDTSTraingDTSDTSTraining.dtsx"
Error: 2008-03-24 08:03:24.36 Code: 0xC0012024 Source: Execute DTS 2000 Package Task Description: The task "Execute DTS 2000 Package Task" cannot run on this edit ion of Integration Services. It requires a higher level edition. End Error Warning: 2008-03-24 08:03:24.38
Code: 0x80019002 Source: DTSTraining Description: The Execution method succeeded, but the number of errors raised (2) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the M aximumErrorCount or fix the errors. End Warning DTExec: The package execution returned DTSER_FAILURE (1).
I am making use of the DtUtil tool to deploy my package to SQL Server. Following is my configuration: 32-bit machine and 32-bit named instance of Yukon.
I have some package variables which need to be set in the code.
Previously I did it as follows:
Set the package variables in the code. For example:
Here the package was successfully deployed and when i open those packages using BIDS, I am able to see that the variables are set to the values as doen in teh code.
Because of oen problem I am not using SaveToSQLServer method. So I switched to DTUtil tool. Now I am doing it this way:
Set the package variables as before. Deploy the package to SQL Server using DTUtil tool.
Now is the problem: The package is successfully deployed. But the variables are not set to the value that I have specified in the code.
I also tried DTexec utility to set the package variable. Even that does n't work. Can anyone help me out? Is there any alternate method to set package variables?
when i run the job using network service account credentials job is failing. But when i run the package individually, it is tasting success. when it runs as job, this is the error message i am getting
SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
I have changed to 32-Bit run time and ran the excel package, even then it is failing...i tried to use my credentials (i am admin on the box), even then it is failing...please suggest
I want to achieve the following in (SSIS/SSDT for SQL 2012) -Â
I have a generic SSIS package which simply sends out email notifications using SMTP email task (this package is within its own project, and has project level input parameters).
I need to be able to call this package in the Event handler section of every package (numbering in about less than 60) that we have. These packages are within their own respective projects.
I thought I could use the "execute package task", but it turns out , using this, I cannot call a package that is part of some other project. I also cannot call a package that is stored in the CATALOG. Is there any way I can do this ?
When I call the child package , I should be able to send in parameters like - error information and package name of the Parent package.
I have an SSIS package (TransAgentMaster) that I recently modified to include a call to a child package via the file system. The child package creates a text file. When I run the package in dev studio then the child package/text file is produced.
I then imported the TransAgentMaster as a stored packagesfilesystem package into SQL SSIS and executed the package. The child package produced the text file.
I then ran the SQL Server Agent to see if the child package would work and it did not generate the text file. Thus after updating a SSIS package importing the package into SSIS the job that calls the package will not call the child package. Please not that the TransAgentMaster package calls 7 children packages €¦ just not my new one.
Any thoughts why the agent will not run the child newly crated childe package?
Dear Friends, I store several configurations in the main database of my SSIS packages. I need to get the servername from a xml or txt file in order to get those configurations stored in my database. How you think is the better way to do that? Using a FlatFileSource to read the file and a script to save the value into a SSIS variable? Using the package configuration I cant do that... maybe I dont know, but I can save the SSIS variale in the configuration file, but what I need is to do the inverse, read the configuration file and save the value in the SSIS variable. How the best way you suggest?! Regards!! Thanks.
Scenarion: 1.- SSIS Package execute tasks on 2000 SQL Server Database 2.- Execution takes places using Business Intelligence Studio Question: 1.- How can I tracked that SQl 2000 tasks took place using a SSIS Package?
