Mass Insert / Update External Data Into Internal SQL Database
Nov 15, 2006
Hola!
I'm currently building a site that uses an external database to store all the product details, and an internal database that will act as a cache so that we don't have to keep hitting the external database to retrieve the products every time a customer requests a list.
What I need to do is retrieve all these products from External and insert them into Internal if they don't exist - if they do already exist then I have to update Internal with new prices, number in stock etc.
I was wondering if there was a way to insert / update these products en-mass without looping through and building a new insert / update query for every product - there could be thousands at a time!
Does anyone have any ideas or could you point me in the right direction?
I'm thinking that because I need to check if the products exist in a different data store than the original source, I don't have a choice but to loop through them all.
Cheers,
G.
I am a webdesigner who at the moment does not know SQL (although, I plan on remedying that) so I am developing a page with a DB designer - he is doing the DB work, I am doing the look/feel, but he asked me the following questions, of which I cannot seem to find an answer and the guys who tend our server are useless - so I am hoping someone here can help me. In general what he needs to know is:
"what is the external address (either in domain format or IP) and the equivalent internal address so that we can access the msSQL running on your server. The internal address is needed for the webpages to talk to it, while the external address is needed for development of the pages in visual studio and the database tools." Also, he later sent me an email asking that when I get this info (from someone) that it would be usefull to get a sample string/query to get access to the DB. I am running SQL 2000 and have enterprise manager. Where can I find this information? or how do I figure this out????
Thank you for all the help -- Please let me know if you need any more info.
Hi everyone..i m new to this field.. can anyone explain me with simple example onhow to insert,update,select data from the sqldatabase? i m using vwd 2005 express edition along with sql express edition. plz explain the simple example with code (C#) including how to pass connection strings etc.thank you.jack.
From: JAGADISH KUMAR GEDELA [jgedela@miraclesoft.com] Sent: 10/10/2007 4:13:43 PM To: jgedela@miraclesoft.com [jgedela@miraclesoft.com] Subject: forum Hi all,
I need to Insert the XML File data into SQL SERVER 2005 db(table). For that I created the table with XML Native column (using typed xml) *********************************create table command************ CREATE TABLE XmlCatalog ( ID INT PRIMARY KEY, Document XML(CONTENT xyz)) *********************************** In order to Create the table with typed xml ,before that we have to create the xml schema which i mentioned below ************************************create schema command******** CREATE XML SCHEMA COLLECTION xyz AS 'Place xml schema file ’ ************************************ I created the xml schema file by using the xmlspy software.
--------------------------Insert command--------- INSERT into XmlCatalog VALUES (1,'copy xml file ‘) ------------------------------- I need to retrieve the xml data from the table ------------select query---------- SELECT Document.query (‘data (/X12//UserId)') AS USERID, Document.query (‘data (/X12/X12_Q1/header/ISA//ISA_Authorization_Information_Qualifier)') AS ISA_Authorization_Information from XmlCatalog. -----------------
I Need to update/insert/delete the xml data in the table
Can you please suggest the procedure to implement the above requirement(insert/update/delete)
hi. i'm trying to create a c# application which would insert, update and delete data from a database. could anyone pls point me to the right direction in which i should take? thanks in advance.
I have a Hits table that tracks the hits on the id of a Link table: Hits: int linkId (foreign Key to Link) datetime dateCreated varchar(50) ip We recently had to merge Links from different systems that are implemented similarly. As a result, all the linkIds are now wrong in the Hits table because the ids all changed. I managed to track down all the old ids and their associated new ids and have it in a table that I call joined_links joined_linksint oldId int newId
So, how do I do a mass update of these linkIds in the Hits table in SQL? I know I could do it in .NET, but I'd rather not write an app to do that runs thousands of update statements. There's gotta be a way to do it something like this:
UPDATE Hits h SET h.linkId = (SELECT newId FROM joined_links WHERE oldId=h.linkId) but obviously I don't have visibility of that linkId in the subselect... A Loop maybe?
This is a run of the mill application that moves orders from one table to another. There are 2 tables, Ordersummary & HstOrders. Ordersummary has the following columns... Identifier FollowupId OrderNumber OrderReference OrderReferenceOrigin ...... ...... HstOrders has the following columns... Identifier OrderNumber OrderReference OrderType ...... ...... The above two tables are bound by Identifier. After each month end, Orders are moved from Ordersummary to HstOrders.
Now my task is to update all rows in OrderSummary with the order details as seen in HSTOrders for ordertype = 'CREDIT'. OrderReferenceOrigin(in Ordersummary) should be updated with the value of Orderreference(from hstorders).
I have to update each row at a time & I need to write a cursor for mass updates where ordersummary.identifier = hstorders.identifier.
Can someone please help with in writing this update statement as I never wrote a cursor.
I'm fairly new to SQL Server and I'm just wondering if it's possible to Update Statistice for all indexes somehow? I'm looking at the Update Statistics command and it doesn't seem to be possible.
The situation we have is a reporting DB that basically has all it's tables truncated and remade every night by some DTS jobs that import from another datasource and change the data and build some denormalzed tables etc. Some of the large Insert operations go from taking 8 mins to taking several hours sometimes and updating the stats seems to fix the problem for a while. So I'd like to make sure the optimizer has all the latest stats for all tables.
Am using SSIS to integrate between two database. First one is insert data from SQL to Sybase. its working fine and insert simulatenously. Now need to update table from sybase to SQL with condition(where). How to do this task. Is there any possiblities to execute SSIS without using SQL agent, update simultaneously whenever insert new data in both database.
I have the following problem. I need to insert 100.000 records (50Kb each) in one operation from a VB program into a SQL Server 2005 database. All of these records will be ready for inserting at the same time. How to make this insert as one big transaction instead of 100.000 small ones?
If I have a trigger on a field in a table, and I update one record trigger fire properly. If I do a update to that same field on all records in the table the trigger does not fire. I the error in the trigger, or do I need to change my update statement?
Hi,I need to update a field in about 20 records on a table. The table hasan update trigger (which updates the [lastedited] field whenever arecord is updated). As a result I'm getting an error: "Subqueryreturned more than 1 value.", and the update fails.Is there a way in the stored procedure to handle this issue?thanks for your help.Paul
I am in the process of designing a web application for our application that will be available to the general public.Basically will be used to collect information for a case report. I need to know if there is a way from our internal database driven application to push certain data up to the website that the database may be behind a firewall so I cannot make a direct connection to the external database. I thought I heard about some sort of data sharing protocol. I then will be storing the users entered data in the website database and pulling it from the internal application. So basically I need a secure way of sending and receiving data between a server/client app on a LAN system to the Web site database that could be hosted outside the LAN. I hope this made sense.
I have a report that was designed using SQL Reporting Services that sits on a SQL reporting server. It's nothing too exciting, it is essentially a three page application with legal jumbo on pages 2 and 3 and applicant data in fields on page 1.
We use rectangles to force page breaks to page 2 and to page 3.
When running the report on the report server, it shows and prints fine.
When running the report from the QA website internally, it shows and prints just fine.
When running the report from the production website from a machine internally, it shows and prints just fine.
When running the report from outside of the company network, the report is jacked. It obliterates large chunks of text, crams text together, and creates blank pages.
I need help in determining where I even begin with trouble shooting this!
I have a web form with a text field that needs to take in as much as the user decides to type and insert it into an nvarchar(max) field in the database behind. I've tried using the new .write() method in my update statement, but it cuts off the text after a while. Is there a way to insert/update in SQL 2005 this without resorting to Bulk Insert? It bloats the transaction log and turning the logging off requires a call to sp_dboptions (or a straight-up ALTER DATABASE), which I'd like to avoid if I can.
I actually work in an organisation and we have to find a solution about the data consistancy in the database. our partners use to send details to the organisation and inserted directly in the database, so we want to create a new database as a buffer database to insert informations from the partners then make an update to the main database. is there a better solution instead of that?
I am using two database server. Both are sql server.
My task is :I want to insert data from server 1 table to server 2 table and update same when modified the existing data from server 1. is it possible to do the integration in single package.
I know to do insert and update seperately by ssis but need to do same with single task.
I'm having an issue with a CLR Stored procedure. Everything works great in a 32 bit environment, I have a CLR SP that updates an xml file stored on a local drive. When I execute the Stored proc it does go and update what I want it to in a 32 bit system. When I run the CLR SP on a 64 bit cluster, I seem to have give the "everyone" group write permissions to my G: drive (which is where the file is located that I'm updating). What security context is this SP running under? I thought it would be either under the SQL Service account (which is a domain user in the local administrators group) or what I'm logged in as when I run it from Management Studio (which is a domain admin, also in the local administrators group). If I have given the local administrators group "Full Control" access to the G: drive, why isn't this enough? Why do I have to give the Everyone group write access?
The security context seems odd to me, it seems like it's not running as either one of those 2 users I mentioned, because if it was, then it should be able to update the xml file.
Any help appreciated. Here's the error I'm getting:
Msg 6522, Level 16, State 1, Procedure usp_XMLWriter, Line 0
A .NET Framework error occurred during execution of user defined routine or aggregate 'usp_XMLWriter':
System.UnauthorizedAccessException: Access to the path 'g:ssisPackagesBuildCalendarandy.dtsConfig' is denied.
System.UnauthorizedAccessException:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
Hi,I have 3 short(ish) questions, If someone could help I'd be very grateful..This is the situation - I have a formview which contains a button, which on clicking should insert a row into one table, and update a column value of another. I will provide the code at the end. 1. On clicking, I get the error: System.Data.SqlClient.SqlException: Incorrect syntax near the keyword 'WHERE'.it doesnt say where, but the stack trace is:[SqlException (0x80131904): Incorrect syntax near the keyword 'WHERE'.] System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) +180 System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +68 System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +199 System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +2411 System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +147 System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) +1089 System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) +314 System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe) +413 System.Data.SqlClient.SqlCommand.ExecuteNonQuery() +115 System.Web.UI.WebControls.SqlDataSourceView.ExecuteDbCommand(DbCommand command, DataSourceOperation operation) +395 System.Web.UI.WebControls.SqlDataSourceView.ExecuteUpdate(IDictionary keys, IDictionary values, IDictionary oldValues) +643 System.Web.UI.DataSourceView.Update(IDictionary keys, IDictionary values, IDictionary oldValues, DataSourceViewOperationCallback callback) +78 System.Web.UI.WebControls.FormView.HandleUpdate(String commandArg, Boolean causesValidation) +1151 System.Web.UI.WebControls.FormView.HandleEvent(EventArgs e, Boolean causesValidation, String validationGroup) +429 System.Web.UI.WebControls.FormView.OnBubbleEvent(Object source, EventArgs e) +88 System.Web.UI.Control.RaiseBubbleEvent(Object source, EventArgs args) +35 System.Web.UI.WebControls.FormViewRow.OnBubbleEvent(Object source, EventArgs e) +109 System.Web.UI.Control.RaiseBubbleEvent(Object source, EventArgs args) +35 System.Web.UI.WebControls.Button.OnCommand(CommandEventArgs e) +86 System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +155 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +7 System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +11 System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +33 System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4886 2. When updating, at the moment it will update with a new quantity provided in the text box, but id like the quantity in the textbox subtracted from the current column value - how would this be done? 3. For inserting, i was wondering if i need a command in the button_click to activate the insert command, similar to the one i have one for the update. I have tried but i cant seem to get the syntax right... Code: private bool ExecuteUpdate(int quantity){ SqlConnection con = new SqlConnection(); con.ConnectionString = "Data Source=.\SQLEXPRESS;AttachDbFilename=|DataDirectory|\ASPNETDB.MDF;Integrated Security=True;User Instance=True"; con.Open(); SqlCommand command = new SqlCommand(); command.Connection = con; TextBox TextBox1 = (TextBox)FormView1.FindControl("TextBox1"); Label labname = (Label)FormView1.FindControl("Label3"); Label labid = (Label)FormView1.FindControl("Label13"); command.CommandText = "UPDATE Items SET Quantityavailable = @qty WHERE productID=@productID"; command.Parameters.Add("@qty", TextBox1.Text); command.Parameters.Add("@productID", labid.Text); command.ExecuteNonQuery(); con.Close(); return true;} private bool ExecuteInsert(String quantity) { SqlConnection con = new SqlConnection(); con.ConnectionString = "Data Source=.\SQLEXPRESS;AttachDbFilename=|DataDirectory|\ASPNETDB.MDF;Integrated Security=True;User Instance=True"; con.Open(); SqlCommand command = new SqlCommand(); command.Connection = con; TextBox TextBox1 = (TextBox)FormView1.FindControl("TextBox1"); Label labname = (Label)FormView1.FindControl("Label3"); Label labid = (Label)FormView1.FindControl("Label13"); command.CommandText = "INSERT INTO Transactions (Usersname)VALUES (@User)"+ "INSERT INTO Transactions (Itemid)VALUES (@productID)"+ "INSERT INTO Transactions (itemname)VALUES (@Itemsname)"+ "INSERT INTO Transactions (Date)VALUES (+DateTime.Now.ToString() +)"+ "INSERT INTO Transactions (Qty)VALUES (@qty)"+ command.Parameters.Add("@User", System.Web.HttpContext.Current.User.Identity.Name); command.Parameters.Add("@Itemsname", labname.Text); command.Parameters.Add("@productID", labid.Text); command.Parameters.Add("@qty", TextBox1.Text); command.ExecuteNonQuery(); con.Close(); return true; }protected void Button2_Click(object sender, EventArgs e){ TextBox TextBox1 = FormView1.FindControl("TextBox1") as TextBox; ExecuteUpdate(Int32.Parse(TextBox1.Text) );} Huge thanks for anyone who can help shed some light!! Thanks again,Jon
HiMy problem is as follows:I need to transmit data between two databases on the same server, but Ihave to use dynamic database names (they must be configurable). Forexample I need to achive sth like that:insert into [database1].[dbo].[table1](select columns from [table2])when database1 is not known at implementation stage.I know I can use EXEC @t_sql_code, but I wonder if there is any otherway? (OPENROWSET doesn't seem to suit my needs)Thanks in advanceAmfi
I have a table called drugs with a cost, date, and drugname field. I have a history table that I want to insert into anytime the fields are changed in the drugs table. I want to capture the old data along with the new cost. is that possible in a update trigger? What I have below doesn't like when it comes to setting the new cost
create trigger Updt_drugs on drugs for update as -- updates record with sql user and timestamp --created 1-12-02 tim cronin declare @newcost money begin set @newcost = select cost from inserted insert into drugsold ([oldcost],[cost],[cdate],[drug]) select [cost],@newcost,[cdate],[drug] from deleted dt end
I'd like to do the following thing with a data flow task
Get all the records from a source (for example customers from a textfile, flat file source) Then check for each record if the customer already exists in a table, for example with a customerID. If not, insert the record in the table (ole db destination), else copy the customer thats already in the table to another table (history table) and update the record with the customer from the textfile.
Is this possible?, and what kind of data flow transformation do I need?
I am new to integration services. I am trying to a build a data warehouse and need to be able to insert new data as well as update data that has changed. I am getting the data in a flat file and need to import it into SQL 2005. I saw some post on www.sqlis.com/default.aspx?311 but I did the example is for OLE DB component. I am not sure how to achieve this using a text file as source. Any help would be greatly appreicated.
Im using a store proc in SQL server.I use one table to store my hockey statistics data of each player in the league.When i submit my form i would like to check if there is any sheet of that player that allready exist .. If so then i do update.. if it does not exist i do the insert statement.Here the way i figured it out ! One store proc to do the work of the insert or update. And one to check if the player exist in the statistics table. But i just cant find a way to return a good value from my second store proc. Does my logic make sence ? any ideas ?thank you<code>create procedure Hockey_Player_UpdateForwardStatistics @LeagueID int, @GamePlayerID int, @Games int as declare @PlayerID int @PlayerID = Hockey_CheckPlayerStatistics(@LeagueID, @GamePlayerID) if @PlayerID = 0begininsert into Hockey_PlayerStatistics( Games)values( @Games)end ELSE beginupdate Hockey_PlayerStatistics set Games = @Gameswhere LeagueID = @LeagueID and PlayerID = @PlayerIDend</code>
How do i update the stats of tables when we insert data into it. I believe Auto stats update happens only when 500+ 20% of the rows are changed for a table. Once we insert say some 1000 records in to a particular table the query time takes too long (more than 1 min). The same query executes faster once i manually update the stats.
How to update a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
How to DELETE a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
how to INSERT a particular value in xml file which was loaded into sql server 2005 database which is of xml-type
update XmlCatalog1 set Document1.modify('delete /X12_U1_837/X12_Q1_837/header/ISA//ISA_Authorization_Information_Qualifier') where id=2
---------- The error which i am getting is XML Validation: Invalid content. Expected element(s):ISA_Authorization_Information_Qualifier where element 'ISA_Authorization_Information' was specified. Location: /*:X12_U1_837[1]/*:X12_Q1_837[1]/*:header[1]/*:ISA[1]/*:ISA_Authorization_Information[1]
Hello, I have some data coming up from an SQL Server source, I have to add it to another SQL Table on the basis of what destination table already have.
For instance, if there is already an entry there for particular record (based on matching primary key value), then I just need to updated 3 columns, otherwise need to insert.
The problem is, I know how to do it by using script component, but I am wondering if there is a better tool that saves me writing code
I am creating a website. i have completed the GUI. i have labels on the left side of the form and a text box for each of those labels on the right side of the form. i have the two dropdown boxes that you have helped me create and bind. now im stuck on how to start the code to get this all working. i know what i need it to do (pseudocode) but dont know how to write the code for it. these books and online sources, really dont say where to put things, they just give examples, not stating where to place it. I am using a SQL database called Manufacturers i would like to insert or update, but not delete records wtih this tool pixelsyndicate mentioned using the GUI to do the code for me, would love to know how to do this. any help would be awesome, or any good beginner resources would help too
I need to create some kind of log file or table that will record whenever an insert, update or delete is made to any table in a database. I have seen triggers that do this kind of thing on a table level. Can this be done with a trigger or a stored procedure on a database level? If so some kind of example or syntax would be great.
