We have a SSIS package which loads the data from csv files to DB. It only loads the new entries ie if the row already exists in the tables than it doesn't insert it. For this we load the CSV to temp tables for respective schemas and than those are compared with base tables of respective schemas and inserted new rows. For this we use Merge statement.
View 1 RepliesI have created mergse statement using SCD2.where I am inserting the data if my BBxkey is not matching with target and updating the rows if the bbxkey is matching and rowchecksum is different.
Working of Store procedure
There are 2 scenario covered in this procedure on the basis of that ETL happening.
There are 2 columns deriving from source table at run time, one is BBxkey which is nothing but a combination of one or more column or part of column and another column is a Rowchecksum column which is nothing but a Hashvalue of all the column of the tables for a row.
Merge case 1:-WHEN NOT MATCH THEN INSERT
If source BBxkey is not there in Archive table that means if BBxKey is null then those records are new and it will directly inserted into Archive table.
Merge case 2:-WHEN MATCH THEN UPDATE
If Source.BBxkey=Target.BBxkey && Source.Rowchecksum<>Target.Rowchecksum then this means source records are available in Archive table but data has been changed, in this case it will update the old record with latestversion 0 and insert the new record with latestversion 1.
my sp failing when source having more than 1 same bbxkey.
error [Execute SQL Task] Error: Executing the query "EXEC dbo.ETL_STAGE_ARCHIVE ?" failed with the following error: "The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows.".
Sample store procedure
DECLARE @Merge_Out TABLE (Action_Taken varchar(8),
TimeIn datetime,
BBXKey varchar(100),
RowChecksum nvarchar(4000),Col001 varchar(8000),Col002 varchar(8000),
Col003 varchar(8000),Col004 varchar(8000),Col005 varchar(8000),
[code].....
How Can I avoid such failure of my sp.
I want to handle those case where S.bbxkey=T.bbxkey && s.rowchecksum=t.rowchecksum
In a t-sql 2012 merge statement that is listed below, the insert statement on the merge statement listed below is not working. The update statement works though.
Merge test.dbo.LockCombination AS LKC1
USING
(select LKC.lockID,LKC.seq,A.lockCombo1,A.schoolnumber
from
[Inputtb] A
JOIN test.dbo.School SCH ON A.schoolnumber = SCH.type
[Code] ....
Thus would you tell me what I need to do to make the insert statement work on the merge statement listed above?
I am working on to convert my static Store procedure to Dynamic.
I have created a Store procedure with Merge statement which is inserting new record and updating existing record.
This SP I will use in SSIS Insted of Data Flow Task I will run in Execute SQL Task.
Now my biggest problem is I dont know how to convert static code toi dynamic
Below is my Store procedure code.
As you can see my Source Query
I have a filemaster table as shown below which consist of Input filename,Source table ,Destination table and BBX expression.
Input_FilenameSourceTableName DestinationTableName BBxKeyDerExpr
CCTFB ImportBBxCctfb ArchiveBBxCctfb SUBSTRING(Col001,1,6)
CEMXR ImportBBxCemxr ArchiveBBxCemxr SUBSTRING(Col001,1,10)
In my source query I want to change the value of Source table ,Destination table and BBX expression dynamically on the basis of input file.
Purpose of making dynamic is that I have created separate sp for all the input, my clients want to have sungle dynamic sp which will execute on the basis of input file.this input file name I wil get fromm variable which i have created in SSIS Package.
Lets consider @File_name is the variable in package which store the file name
if file name is CCTFB then my query should take the Source table ,Destination table and BBX expression value from file master table.
Like that I have 100 of source query and evry query have diffrent number of columns. How can I change the column number in uodate and insert statement dynamically on run time.
CAST(SUBSTRING(Col001,1,6) + SUBSTRING(Col002,1,10) AS varchar(100)) :-It creates a key for comparing, this value i can take it from filemaster
HASHBYTES('MD5', CAST(CHECKSUM(Col001, Col002,Col003,Col004) AS varchar(max))) -here numberv of column need to be changed .
(SUBSTRING(SOURCE.Col001,1,6) + SUBSTRING(SOURCE.Col002,1,10)) this condition also i can take it from file master.
[Code] ....
I am able to get inserted and updated rowcount, but not able to get the matching records count.
I have created a Dynamic Merge statement SCD2 Store procedure , which insert the records if no matches and if bbxkey matches from source table to destination table thne it updates old record as lateteverion 0 and insert new record with latest version 1.
I am getting below error when I ahve more than 1 bbxkey in my source table. How can I ignore this.
BBXkey is nothing but I am deriving by combining 2 columns.
Msg 8672, Level 16, State 1, Line 6
The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows.
I would like to fetch the data flow component name while package is executing. Since system variable named [System::SourceName] only fetches name of the control flow tasks? Is there a way to capture them?
View 5 Replies View RelatedI inherited an SSIS package that is rather simple. It grabs data from a SQL Query and then loads it into a SQL table. The first step of this process TRUNCATES the destination table and then reloads for the current year. This table has over a million rows and the DB SOURCE that we are pulling from is not in our domain, so one can imagine how long this takes.
This process is working fine, (given the 45 minutes it takes to repopulate data in the DESTINATION table), but what I really need is a way to load only the rows that are NEW and UPDATED. I would also need functionality to DELETE the rows that have been removed (sounds like a MERGE, right?).I tried using MERGE and MERGE JOIN transformations but these transformations seem to be different from the T-SQL MERGE statement. MERGE seems like a slow UNION and MERGE JOIN only seems to work with SELECTS.
I have a table which is updated daily using a MERGE statement. As records are insert, updated and deleted, I am saving the OUTPUT from the MERGE statement into a history table with a timestamp and action$ column appended to the record.
Using this history table, I'd like to rebuild the data based on specific past date. I was able to create a stored procedure that inspects each record in the history table and apply it to the data in a temp table. The stored procedure solution uses multiple queries to rebuild the data at a point in time. I was curious if there was an easier and more efficient solution using a table function.
Hi,
View 14 Replies View Related
Hi ,
I would like to get your advice in an issue that I am facing regarding merge replication.
The flow is like:
1) User enter his login name and password.
2) Host_name is set as the login name that the user enters.
3) Based on the HOST_NAME property after filtering user gets the records relevent only to him
(Differnet subscriber may use same mobile device).
4) He make some changes in the local sdf according to the business logic.
5) The application Syncs with server on the next login session.
My issue is when I am trying to run the application in emulator exactly at the sdf updating part the application terminates.
Error mess: Connection to Remote Device has been lost.....
If I execute the same sql insert query it works om the local sdf and master database.
Emulator used where Pocket PC 2003 SE and Windows Mobile 6 classic emulator.
I have installed 1) Visual Studio 2005 SP 1 2) net compact framework SP2.
When I comment all the code related to Sync ,application works witout any issue.
And the same application works in my "I mate kjam".
The code that I use is given below:
==========================================================================
public void Sync(string hostname, int newsub) // Parameters for HOSTNAME and flag to denote new user or existing //user.
{
SqlCeReplication repl = new SqlCeReplication();
repl.InternetUrl = @"http://231.000.000.211/Merge/sqlcesa30.dll";
repl.Publisher = @"SERVER";
repl.PublisherDatabase = @"Merge";
repl.PublisherSecurityMode = SecurityType.DBAuthentication;
repl.PublisherLogin = @"sa";
repl.PublisherPassword = @"password";
repl.Publication = @"Merge";
repl.Subscriber = @"MergeSub";
repl.HostName = hostname;
repl.SubscriberConnectionString = @"Data Source=""Program FilesMerge.sdf"";Max Database Size=128;Default Lock Escalation =100;";
try
{
if (newsub == 0)
{
if (File.Exists(@"Program FilesMerge.sdf"))
{
File.Delete(@"Program FilesMerge.sdf");
}
repl.AddSubscription(AddOption.CreateDatabase);
repl.Synchronize();
repl.Dispose();
MessageBox.Show("newuser sync Done");
}
else
{
if (!File.Exists(@"Program FilesMerge.sdf"))
{
repl.AddSubscription(AddOption.CreateDatabase);
repl.Synchronize();
repl.Dispose();
MessageBox.Show("FirstSync Done");
}
else
{
repl.Synchronize();
repl.Dispose();
MessageBox.Show("SameuserSync Done");
}
}
=================================================================================
Please advise......
We have SQL Server 2012 running on Windows 2008 Server. We would like to use a SSIS Package to generate a text file and then secure copy it to a vendor's ftp site. Would it be best to use an FTP Task or Execute Process Task (to call the batch file)? Would I need to install some software like winscp or does the Windows O/S has some secure copy or ftp programs that may be used?
View 4 Replies View RelatedLooking for steps to execute the package from CMD ?
View 1 Replies View RelatedWe had a scenario where we used to run the Process from front end thru application. on the back ground the the process call the SP & from there it calls the SSIS package then again to SP.
after SSIS package ran succesfully it will be updated on a table with sucess then call the SP & deletes the entry from the other table but in one scenario Package was success but the entry was not getting deleted as the process takes almost 2-3 hours loading 60 millions records. but the process was running in SQL 2008 but once we upgraded to SQL 2012 its not working for one application. its not returning any error as timeout also. we tried changing the server level setting for remote query time out also to 0 but no luck .
I have some "Execute T-SQL Statement Tasks" in a package. I would like to run this same package on another SQL Server without having to change it on the other server. Since the server name can be given when setting up the connection, I think if I leave the server name out then the package could run on any server? Is my assumption correct?
View 10 Replies View RelatedI have a ssis package in the DW environment. How can I run this package in the production environment by creating the job and run remotely.
View 7 Replies View RelatedWe are running SQL Server 2012 SP1 64-Bit EE on Windows Server 2008 R2 SP1. I have a SSIS Package which connects to a FTP Site and downloads a file. Then it truncates a table and loads the file data into a table. This package works okay when executed from within VSS and SSMS (In SSISDB, right click on the package and execute). However, when I execute it as a Job it does not run and appears to be failing on the first task which is the FTP Task. SQL job step - Type: SSIS Package; Run as: SQL Server Agent Service Account (domain account called playuser); Authentication: Windows Authentication.In the All Executions Standard Report for the SSISDB Catalog, it only says: FTP Download File: Errors: There were errors during task validation.
Is this because my domain account does not have access to the FTP Site?Is this where I need to come up with a Proxy Account with Credentials?Do I need to set up a SQL Server Login (Proxy Account) with the same username being used in the FTP Batch file?
FTP Commands in a batch file:
username
password
cd omb
asc
get STRMASTER
quit
I am tasked with creating a batch file to execute SSIS packages.
View 3 Replies View RelatedI have a bat file that kicks off a master package, which kicks off about 300 child packages. My bat file is working correctly, however there seems to be issues executing packages in 64 bit mode.
So my question is...
Is there a way to specify in the bat file to execute packages in 32 bit mode and not 64 bit?
bat file:
"C:Program FilesMicrosoft SQL Server110DTSBinnDTExec.exe" /f "E:MasterPackage.dtsx"
I have been handed a datawarehouse environment to look at however I cannot find the ssis packages (.dtsx).
I see that the packages have been deployed to SSISDB but I can't find the files to load into Datatools.
Once the files have been deployed is it possible to remove the package files? Is there a way of retro'ing the deployed packages out of SQL so I can view them in Datatools?
OS: Windows 2012, standard, 64-bit
SQL: 2012 + SP1 + CU2 = 11.0.3339
We recently decided to "break apart" our BI environment. We used to have everything on one box, DB Engine, SSIS, SSAS & SSRS. Everything has been running fine, but we now have other projects using these services, so we decided to break them apart into their own boxes.
We now have DB Engine on one Server, SSIS & SSRS on another server and SSAS on yet another server, so we now have three boxes that replaced one box. All are Windows 2012, standard, 64-bit with SQL Server 2012 + SP1 + CU2.
Since some of our SSIS packages have to access external resources, we used a domain account for it's service account. The DB Engine and SSAS box are using the default service accounts when installed. I can execute the packages fine on the SSIS server, I can even execute them via SQL Agent jobs on the SSIS box (we did install a default instance of SQL on the SSIS box), however when I try to execute a package from my laptop, it fails with the ugly "Login failed for user 'NT AUTHORITYANONYMOUS LOGON'".
I immediately double checked my SPNs and they all looked correct for the SSIS server and the service account we are using (and we had no duplicates). I also double checked the User Rights Assignment in the Local policy editor and all the correct Rights have been assigned (Log on as a service, Bypass traverse checking, Impersonate a client after authentication).
I was assigned a project to make a backup of data from 3 lists in a sharepoint with the objective of using that data to do reports on it using Report Builder.
I was abble to do the downloads using the add in to do downloads Sharepoint List Source and Destination "plug in" in SSIS, now the issue is how to update the backup table only with updated and new rows.
Daniel
I have a really big stored proc that needs to be rolled out to various databases as part of db installs I run through SSIS.
The Stored proc is too long to run using Execute SQL Task. Is there another way that just running the create script manually.
I create a package that has a foreach loop container, inside the FE container the is a dataflow, script task and a file system task, on the outside of the FE loop I have a SQL execute SQL task. I had it working then it just stopped. Been looking to see why but can't see why, is there something I missed. I changed the onError script task event handler, to propagate = false, because the last file in the source directory is being written to until 11:59 pm and is locked, and I get error file is being used by another process. I am at a loss as to why it would just stop working.
View 4 Replies View RelatedWe require to convert a list of SPs in to SSIS packages. Most of the SPs do the below steps:
mainly our store procedure r to have compare the present date to past date , and comparing emp id between the files and also some joins. updating table r take place.
I am working on FTP TASK in SSIS Package. i have to get files from FTP that file names are like 20141110.txt. i want to download any particular date file from ftp. How to i set expression in Remote path?
View 3 Replies View RelatedCurrently I am running a SSIS package scheduled daily at 7 A.M. It expects two feed files from two different folders. The first step in my package will rename the input files in those folders to names which the package can understand. I have created two variables in my package to read the files with those file names. I used these variables in connection managers.
If any of these folders doesn't have input files when the package runs, the package will fail.how to make the package the run successfully even there is no input feed?
I am using a OLE DB source task and i want to setup a parameter in the select statement. Is this possible and how would this be done. I know you can pass a parameter in the where clause like:
select id, lname, fname, startDate, endDate from Employee where id = ?
How would i set startDate and endDate as parameters as i would like to be able to change these parameters when running the package. I would like to set dates in the select statement as parameters like how you would in the where clause.
Select id, lname, fname, ? startDate, ? endDate, from employee where id = ?
I would like to be able to change these dates without having to go in to the package and hard code it back in there and then deploy.
The package uses a OLE DB source to Flatfile Destination.
I have a data flow task in which I have a ADO NET source and OLE DB Destination. I have in the ADO NET source a sql command which pulls all the columns in a table. My requirement is to ignore a particular column,say column99. I opened advanced editor and deleted the mapping between the external and output columns for column99. I had also set the Error and Truncation to "Ignore Failure" for column99. I had also mapped the destination column to <Ignore> in OLD DB destination.
But this still throws the error-
Description: The ADO NET Source was unable to process the data. Field table-column99 missing an escape character for a quote.Unable to update PK WHERE clause.Error processing data batch.
How do I solve this?
I have a ssis package stored in file system and I need to modify this package.
View 7 Replies View RelatedUsing server 2012 on local machine, I created an SSIS package that will execute in integrated services and Visual Studio solution but will not work when creating a job. Other solutions work well except when exporting data. The program pulls data from query and exports into .csv file. The messages I get are -
Data flow task 1:error:destination- Stage.csv failed the pre-execute phase and returned code 0xC020200E
and
Data flow task 1:error:Cannot open the datafile "pathStage.csv".
Version-
Microsoft SQL Server Management Studio11.0.3128.0
Microsoft Analysis Services Client Tools11.0.3128.0
Microsoft Data Access Components (MDAC)6.1.7601.17514
Microsoft MSXML3.0 6.0
Microsoft Internet Explorer9.11.9600.17041
Microsoft .NET Framework4.0.30319.18444
Operating System6.1.7601
I have a package on the default instance which runs and completes successfully. When that package is moved to the same SQL server, but a different instance, running under the same service account, it fails. The error is below with some specific stuff removed:
Delete Data from Level 1:Error: Executing the query "-- Variable to capture FileID's
DECLARE @DeleteFil..." failed with the following error: "The DELETE permission was denied on the object '[name removed]', database '[]', schema '[]'.". Possible failure reasons:
Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.
This makes me think that the package under the non-default instance ends up running under a different security context.
I have requirement of sending table data or result of an query as an email with SSIS package.
View 6 Replies View RelatedI am trying to dynamically create the connection to a database within an SSIS package.
the requirement is to allow the user to pass through the database as a variable and that variable will dynamically create the connection string in the connection manager.
Is this possible, if so how?