The Best Data Transformation Tool To Update/Insert
Feb 9, 2007
Hello,
I have some data coming up from an SQL Server source, I have to add it to another SQL Table on the basis of what destination table already have.
For instance, if there is already an entry there for particular record (based on matching primary key value), then I just need to updated 3 columns, otherwise need to insert.
The problem is, I know how to do it by using script component, but I am wondering if there is a better tool that saves me writing code
I'm using DTS package, a tool to transfer data from a txt file to database(Bulk Insert).
The Bulk Insert task provides an efficient way to copy large amounts of data into a SQL Server table or view.It seems that the Bulk Insert task supports only OLE DB connections for the destination database. But I want to use sql server authentication as OLEDB connection requires windows authentication.
So can the bulk insert be done using SQLServer authentication ? if yes then please help me.
I have given the code snippet below.
Code Sample:
Dim oPackage As New DTS.Package2() Dim oConnection As DTS.Connection Dim oStep As DTS.Step2 Dim oTask As DTS.Task Dim oCustomTask As DTS.BulkInsertTask Try oConnection = oPackage.Connections.New("SQLOLEDB") oStep = oPackage.Steps.New oTask = oPackage.Tasks.New("DTSBulkInsertTask") oCustomTask = oTask.CustomTask With oConnection oConnection.Catalog = "pubs" oConnection.DataSource = "(local)" oConnection.ID = 1 oConnection.UseTrustedConnection = True oConnection.UserID = "Tony Patton" oConnection.Password = "Builder" End With oPackage.Connections.Add(oConnection) oConnection = Nothing With oStep .Name = "GenericPkgStep" .ExecuteInMainThread = True End With With oCustomTask .Name = "GenericPkgTask" .DataFile = "c:dtsauthors.txt" .ConnectionID = 1 .DestinationTableName = "pubs..authors" .FieldTerminator = "|" .RowTerminator = " " End With oStep.TaskName = oCustomTask.Name With oPackage .Steps.Add(oStep) .Tasks.Add(oTask) .FailOnError = True End With oPackage.Execute() Catch ex As Exception MsgBox("Error: " & CStr(Err.Number) & vbCrLf_ & Err.Description, vbExclamation, oPackage.Name) Finally oConnection = Nothing oCustomTask = Nothing oTask = Nothing oStep = Nothing If Not (oPackage Is Nothing) Then oPackage.UnInitialize() End If End Try
I have a table called drugs with a cost, date, and drugname field. I have a history table that I want to insert into anytime the fields are changed in the drugs table. I want to capture the old data along with the new cost. is that possible in a update trigger? What I have below doesn't like when it comes to setting the new cost
create trigger Updt_drugs on drugs for update as -- updates record with sql user and timestamp --created 1-12-02 tim cronin declare @newcost money begin set @newcost = select cost from inserted insert into drugsold ([oldcost],[cost],[cdate],[drug]) select [cost],@newcost,[cdate],[drug] from deleted dt end
I'd like to do the following thing with a data flow task
Get all the records from a source (for example customers from a textfile, flat file source) Then check for each record if the customer already exists in a table, for example with a customerID. If not, insert the record in the table (ole db destination), else copy the customer thats already in the table to another table (history table) and update the record with the customer from the textfile.
Is this possible?, and what kind of data flow transformation do I need?
I am new to integration services. I am trying to a build a data warehouse and need to be able to insert new data as well as update data that has changed. I am getting the data in a flat file and need to import it into SQL 2005. I saw some post on www.sqlis.com/default.aspx?311 but I did the example is for OLE DB component. I am not sure how to achieve this using a text file as source. Any help would be greatly appreicated.
Hi everyone..i m new to this field.. can anyone explain me with simple example onhow to insert,update,select data from the sqldatabase? i m using vwd 2005 express edition along with sql express edition. plz explain the simple example with code (C#) including how to pass connection strings etc.thank you.jack.
Im using a store proc in SQL server.I use one table to store my hockey statistics data of each player in the league.When i submit my form i would like to check if there is any sheet of that player that allready exist .. If so then i do update.. if it does not exist i do the insert statement.Here the way i figured it out ! One store proc to do the work of the insert or update. And one to check if the player exist in the statistics table. But i just cant find a way to return a good value from my second store proc. Does my logic make sence ? any ideas ?thank you<code>create procedure Hockey_Player_UpdateForwardStatistics @LeagueID int, @GamePlayerID int, @Games int as declare @PlayerID int @PlayerID = Hockey_CheckPlayerStatistics(@LeagueID, @GamePlayerID) if @PlayerID = 0begininsert into Hockey_PlayerStatistics( Games)values( @Games)end ELSE beginupdate Hockey_PlayerStatistics set Games = @Gameswhere LeagueID = @LeagueID and PlayerID = @PlayerIDend</code>
How do i update the stats of tables when we insert data into it. I believe Auto stats update happens only when 500+ 20% of the rows are changed for a table. Once we insert say some 1000 records in to a particular table the query time takes too long (more than 1 min). The same query executes faster once i manually update the stats.
From: JAGADISH KUMAR GEDELA [jgedela@miraclesoft.com] Sent: 10/10/2007 4:13:43 PM To: jgedela@miraclesoft.com [jgedela@miraclesoft.com] Subject: forum Hi all,
I need to Insert the XML File data into SQL SERVER 2005 db(table). For that I created the table with XML Native column (using typed xml) *********************************create table command************ CREATE TABLE XmlCatalog ( ID INT PRIMARY KEY, Document XML(CONTENT xyz)) *********************************** In order to Create the table with typed xml ,before that we have to create the xml schema which i mentioned below ************************************create schema command******** CREATE XML SCHEMA COLLECTION xyz AS 'Place xml schema file ’ ************************************ I created the xml schema file by using the xmlspy software.
--------------------------Insert command--------- INSERT into XmlCatalog VALUES (1,'copy xml file ‘) ------------------------------- I need to retrieve the xml data from the table ------------select query---------- SELECT Document.query (‘data (/X12//UserId)') AS USERID, Document.query (‘data (/X12/X12_Q1/header/ISA//ISA_Authorization_Information_Qualifier)') AS ISA_Authorization_Information from XmlCatalog. -----------------
I Need to update/insert/delete the xml data in the table
Can you please suggest the procedure to implement the above requirement(insert/update/delete)
How to update a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
How to DELETE a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
how to INSERT a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
update XmlCatalog1 set Document1.modify('delete /X12_U1_837/X12_Q1_837/header/ISA//ISA_Authorization_Information_Qualifier') where id=2
---------- The error which i am getting is XML Validation: Invalid content. Expected element(s):ISA_Authorization_Information_Qualifier where element 'ISA_Authorization_Information' was specified. Location: /*:X12_U1_837[1]/*:X12_Q1_837[1]/*:header[1]/*:ISA[1]/*:ISA_Authorization_Information[1]
hi. i'm trying to create a c# application which would insert, update and delete data from a database. could anyone pls point me to the right direction in which i should take? thanks in advance.
I have created two table with same data structure. I need realtime effects (i.e. data) on both tables - Table1 & Table2.
Following Points to Consider.
1. Both tables are in the same database.
2. Table1 is using for data entry & I wants the same data in the Table2.
3. If any row insert, update & delete occers on Table1, the same effect should be done on Table2.
4. I need real time data insert, update & delete on Table2.
I knew that using triggers it could be possible, I have successfully created a trigger for inserting new rows (using logical table "Inserted") in Table2 but not succeed for update & delete yet.
I want to understand how can I impletement this successfully without any ambiguity.
I have attached data structure for tables. Thanx...
Is it possible to allow a user to insert and update data in a table but prevent them from performing deletes against that same table? For auditing purposes I need to prevent the end users from being able to delete data.
Im using VS2008, visual basic and SQL Server express.
General questions please: Is it necessary, to issue some sort of command, to 'commit' data to a SQL express database after a INSERT or UPDATE sql command?
I'm getting strange behavior where the data is not refreshed unless i exit my app and re-enter. In other words, i can run a sql command , the data is apparantly saved (because i get no errors) then if i refresh a data set or do a sql select query the data that i expect to return is not there.
Im fairly new to SQL express (and SQL server generally) so i dont know if its my coding or i need to switch some 'feature' on/off or not.
I hope thats clear
Also, could someone point me to documentation that explains each parameter in the connection string
Hola!I'm currently building a site that uses an external database to store all the product details, and an internal database that will act as a cache so that we don't have to keep hitting the external database to retrieve the products every time a customer requests a list.What I need to do is retrieve all these products from External and insert them into Internal if they don't exist - if they do already exist then I have to update Internal with new prices, number in stock etc.I was wondering if there was a way to insert / update these products en-mass without looping through and building a new insert / update query for every product - there could be thousands at a time!Does anyone have any ideas or could you point me in the right direction?I'm thinking that because I need to check if the products exist in a different data store than the original source, I don't have a choice but to loop through them all.Cheers,G.
We are experiencing problems inserting or updating image fields fromone table to another in SQL Server.When we do this what ever size of file we insert is doubled in sizewhen it is inserted into the destination table.This happens in insert and update queries, and if we use DTS.Any help would be greatly appreciated
Am trying to find a way to insert/update/delete data in a SQL mobile database on a Windows CE 5.0 device FROM a desktop PC.
This situation is completely stand alone, no network (apart form device/desktop), no GPRS etc etc etc.
I've looked at RDA but i dont believe it fits my app. (pulling data from a 2005 server that doesnt exist doesnt really help me much, push can't be used without a pull which kills the idea.)
The goal is a UI on the desktop that can manipulate data in the SQL mobile Database.
I've tried all i can find/think off in relation to this but to no avail.
My latest attempt has been using the simplest method possible (using a VS wizard datasource to the devices DB and tryign to whack that on a form) but this just creates a "Path not found. Check the directory for the database [Path = Mobile Device/ce_swipe/TestDB.sdf".
Am using SSIS to integrate between two database. First one is insert data from SQL to Sybase. its working fine and insert simulatenously. Now need to update table from sybase to SQL with condition(where). How to do this task. Is there any possiblities to execute SSIS without using SQL agent, update simultaneously whenever insert new data in both database.
I have an SQL data source on my page and I select "Table". On the next screen I pick the fields I want to show. Then I click the "Advanced" button because I want to allow Inserts, updates and deletes. But its all greyed out abd I can't check this option. The UID in the connection string I am connecting under has the correct permissions in SQL server to do inserts, update and deletes too. Anyone know why it would be greyed out? The connectionstring property in the aspx code is dynamic but this shouldn't be the reason because I have used this before with success
Hi, I have an example situation that seems like it should have a super easy solution, but my jobs keep failing. Here we go. . .
I have a SQL Server 2005 table as my source in a data flow task. This table contains raw data. We'll call it FACT_Product_Raw - which contains a field called ProductType varchar(1) Let's say that ProductType contains values of "A" or "B" or "C" - or for that matter, some null and garbage values
I have a lookup table, LOV_Product_Types This table contains 3 fields that will transform my raw data table We'll call these fields ProdTypeID smallint, ProdTypeRaw varchar(1) and ProdType smallint It contains pairs such that A = 1, B = 2, and so on.
Here's what I want to do. I want to ADD a field to FACT_Product_Raw that contains the "looked up" value from LOV_Product_Types. Let's say that I want to add the ProdTypeID field to my _Raw table.
I have used the _Raw table as both my source and destination It blows up every time. Help. Thanks, David
I have an API which uses the MDS WCF methods to update and insert entity members into MDS. Its working as expected. But whenever a entity member is inserted or updated, the validation flag is set to "requires re-validation" (With "?" symbol). Is there a setting which has to be set to validate the record once inserted or updated by API? Or should it be validated explicitly once the insert or update happens?
Hi, I'm reasonably new to ASP.NET 2.0 I'm in my wizard_FinishButtonClick event, and from here, I want to take data from the form and some session variables and put it into my database via a stored procedure. I also want the stored procedure to return an output value. I do not need to perform a select or a delete. For the life of me, I can't find a single example online or in my reference books that tells me how to accomplish this task using a SqlDataSource control. I can find lots of examples on sqldatasources that have a select statements (I don't need one) and use insert and update sql statements instead of stored procedures (I use stored procedures). I desperately need the syntax to: a) create the SqlDataSource with the appropriate syntax for calling a stored procedure to update and/or insert (again, this design side of VS2005 won't let me configure this datasource without including a select statement...which I don't need). b) syntax on how to create the parameters that will be sent to the stored procedure for this sqldatasource (including output parameters). c) syntax on how to set the values for these parameters (again...coming from form controls and session variables) d) syntax on how to execute, in the code-behind, the stored procedure via the sqldatasource. If anybody has sample code or a link or two, I would be most appreciative. Thank you in advance for any help!
I am currently working with C and SQL Server 2012. My requirement is to Bulk fetch the records and Insert/Update the same in the other table with some business logic? How do i do this?
I have a master securities table which has 7 fields. As a part of the daily process I am uploading flat files into database tables. The flat files contains the master(static) security data as well as the analytics(transaction) data. I need to
1) separate the master (static) data from the flat files,
2) check whether that data is present in the master table, if not then insert that data into the master table
3) If data present then move that existing record to an history table and then update the main master table.
All the 7 fields need to be checked to uniquely identify a single record in the master table.
How can this be done? Whether we can us a combination of data flow items or write a sql procedure to do all this.
My vendor requires data to be sent in Excel format. Some of my tables have rows over 65,536 so I need to use Excel 2007 (Max of 1,048,576). Right now my data sits in SQL 2000. I am using MS SQL Enterprise Manager 8.0 to prepare the data. Is there some kind of add on or selection I am missing to use DTS to export from SQL to Excel 2007?Thanks in advance.
I Have a problem when copying data from one server to another in Management studio, I need to create and exact copy of the original because of primary key relationships,
Currently when I export the data the data will run through an insert type statement, which means that all PKs are reissued, rather than being duplicated from the original, How can I be sure that the data will be copied exactly how it is on one server to the other.
I need a recommendation on a data modeling tool that can be used with a data warehouse. My warehouse is running SQL 2012.
Here is my challenge: Most of the tables in the warehouse do not have primary keys and none of the tables have foreign keys on them. However, there are indexes and unique keys/indexes on the tables. I am looking for a tool that I can create virtual relationships on how the data is related, so it is visually easier for the ETL developers to write the code.
I have looked at both ER/Studio 11 and ERwin 9.6. Neither of them do it exactly the way I want it too. However, ER/Studio is pretty close.
If you have two synchronous transformation components and the input of the second is connected to the output of the first, does the first transformation process (loop through) all rows in the buffer before outputting these rows to the second transformation? Or does the first transformation output each individual row to the second transormation as soon as it has finished processing it?
If I have a table with 1 or more Nullable fields and I want to make sure that when an INSERT or UPDATE occurs and one or more of these fields are left to NULL either explicitly or implicitly is there I can set these to non-null values without interfering with the INSERT or UPDATE in as far as the other fields in the table?
EXAMPLE:
CREATE TABLE dbo.MYTABLE( ID NUMERIC(18,0) IDENTITY(1,1) NOT NULL, FirstName VARCHAR(50) NULL, LastName VARCHAR(50) NULL,
[Code] ....
If an INSERT looks like any of the following what can I do to change the NULL being assigned to DateAdded to a real date, preferable the value of GetDate() at the time of the insert? I've heard of INSTEAD of Triggers but I'm not trying tto over rise the entire INSERT or update just the on (maybe 2) fields that are being left as null or explicitly set to null. The same would apply for any UPDATE where DateModified is not specified or explicitly set to NULL. I would want to change it so that DateModified is not null on any UPDATE.
INSERT INTO dbo.MYTABLE( FirstName, LastName, DateAdded) VALUES('John','Smith',NULL)
INSERT INTO dbo.MYTABLE( FirstName, LastName) VALUES('John','Smith')
INSERT INTO dbo.MYTABLE( FirstName, LastName, DateAdded) SELECT FirstName, LastName, NULL FROM MYOTHERTABLE
I have a web form with a text field that needs to take in as much as the user decides to type and insert it into an nvarchar(max) field in the database behind. I've tried using the new .write() method in my update statement, but it cuts off the text after a while. Is there a way to insert/update in SQL 2005 this without resorting to Bulk Insert? It bloats the transaction log and turning the logging off requires a call to sp_dboptions (or a straight-up ALTER DATABASE), which I'd like to avoid if I can.
We are transferring data between AS/400 and SQL Server 7.0 using DTS. Some of these transfers may need to be very close to real time. It doesn't seem like a continuously running job is the best solution for that.
Do you know any tools or utilities that can help us to move the data?
