Best Method To Move Blocks Of Records From One Database To Another
Feb 7, 2008
I wanted to set up a mechanism that would transfer blocks of records (a few dozen to in rare cases a few thousand), with slight modification, from one database to another. It's a sort of custom partial archiving process that would be triggered from a web-based admin application in plain old ASP (not .net alas). Records in the target db would be identical except:
-- the primary key in the source table, an identity field, would be just an integer in the target table
-- the target table has an extra field, an integer batch ID supplied by the web application that triggers the process
It's a simple, if not efficient matter to do it within the web application: query the source table, suck the records into memory, and insert them one by one into the target db. This will be an infrequent process which can be done at off-hours, so a bit of inefficiency is not the end of the world. But I wondered if there is a more sensible, orthodox approach:
-- Could this process be done, and done efficiently, as a stored procedure with the batch ID passed as a parameter?
-- Is there any way to do a bulk insert from a recordset or array in memory using plain ASP, ADO and SQL? And if so, is that better than inserting records one by one?
I realize that the ASP.NET tableadapter and dataset features might provide a good solution, but in the short run I can't rewrite the whole application. Advice on the best general approach from an ASP-ADO platform would be appreciated, and I will try to figure out the details.
I wanted to set up a mechanism that would transfer blocks of records (a few dozen to in rare cases a few thousand), with slight modification, from one database to another. It's a sort of custom partial archiving process that would be triggered from a web-based admin application. Records in the target db would be identical except:
-- the primary key in the source table, an identity field, would be just an integer in the target table -- the target table has an extra field, an integer batch ID supplied by the web application that triggers the process
It's a simple, if not efficient matter to do it within the web application: query the source table, suck the records into memory, and insert them one by one into the target db. This will be an infrequent process which can be done at off-hours, so a bit of inefficiency is not the end of the world. But I wondered if there is a more sensible, orthodox approach:
-- Could this process be done, and done efficiently, as a stored procedure with the batch ID passed as a parameter? -- Is there any way to do a bulk insert from a recordset or array in memory using ADO and SQL? And if so, is that better than inserting records one by one?
Advice on the best general approach would be appreciated, and I will try to figure out the details.
I have a Audit tables database. I want to archive the Audit Tables data to another new database in the same server and remove the data in the Actual database but continues with the table seed where it's the primary ID.
Example: I have a AuditProducts Table with AuditID as BIGINT and auto increment value. contains10,000 records in AuditTable Database.
I want to archive the tables to a new database: Audit200707
After archived, the AuditProducts Table in AuditTable database will remove all the records and continues with 10001 in the next record.
How can I done the whole process using T-SQL so that it can schedule and run for every month? BTW I am using SQL Server 2005 SP2.
Somebody tellme that with a format of my database disk with 64K blocks NTFS, i can have better performance, is that true ? there is any problem in SQL with this block size ?
Hi,We use a database with about 40 related tables. Some tables contain asmuch as 30.000 records. We use Access97 as an interface to thedatabase. Now recently we have the problem that when we want to inserta row in one specific table (alwasy the same) the database makesblocks.Details:- about 10% of the data was inserted using copying from Excel, beforethis action there was no problem, though there is no evidence thatthis causes the problem.- inserting rows via the Query Analyzer works fine, via Access causestrouble.- the tempdb lofile has grown to 48Mb.Has anyone ideas about what is going on and what I can do to solve theproblem?TAV,Jan Willems
This question is regarding a "helper app" I'm building to go along with my ASP.NET appplication. It will be inserting/updating records in the database as a nightly process. It is a Windows application built in VB.net.
I have a table which should always only have one of each type of record in it. This table on average will have between 100k and 500k records.
Which operation would be faster and less strain on the server?
a. Use a "if exists" and see if a record of this type already exists, if it does, update it, if not, insert the new one.
b. Unconditionally issue a delete for the record I'm about to insert, then insert the new one.
c. Create a trigger that will delete the old record if a new one is inserted?
Here is the sql code I am using. However this code adds duplicate records. It adds the record and exactly one extra record... any help would be greatly appreciated...
It's wrapped in a little ASP code:
sql2 = "INSERT INTO prodcampsignup " &_ "(Camp_ID,UserName,Password,CampName,Host,Director ,Contact,Address, " &_ "City,State,Zip,Country,Phone,AlternatePhone,Fax,C ontactEmail,AdminEmail, " &_ "URL,CampProfile, InsertTime)" &_ "SELECT * FROM tempcampsignup WHERE Camp_ID = " & lastid
heres my problem, since I migrated to SQL-Server 2005, I was able to use the Row_Number() Over Method to make my Custom Paging Stored Procedure better. But theres onte thing that is still bothering me, and its the fact the Im still using and old and classic Count instruction to find my total of Rows, which slow down a little my Stored Procedure. What I want to know is: Is there a way to use something more efficiant to count theBig Total of Rows without using the Count instruction??? heres my stored procedure: SELECT RowNum, morerecords, Ad_Id FROM (Select ROW_NUMBER() OVER (ORDER BY Ad_Id) AS RowNum, morerecords = (Select Count(Ad_Id) From Ads) FROM Ads) as testWHERE RowNum Between 11 AND 20 The green part is the problem, the fields morerecords is the one Im using to count all my records, but its a waste of performance to use that in a custom paging method (since it will check every records, normally, theres a ton of condition with a lot of inner join, but I simplified things in my exemple)...I hope I was clear enough in my explication, and that someone will be able to help me. Thank for your time.
So I know that each employee should have 2 Type 1's and 4 Type 2's. I hope that makes sense, I'm trying to change my data because ours is very proprietary.
I need to identify employees who do not have all their stages and list the stages they are missing. The final report should only have employees and the associated missing types and stages.
I do a count by employee to see how many types they have to identify the ones that don't have all the types and stages.
My count would look something like this:
EmployeeNumber Type Total 100, 1, 2 100, 2, 2 200, 1, 1 200 1, 2
So I know that employee 100 should have 2 more Type 2's and employee 200 should have 1 more Type 1 and 2 more Type 2's based on the required list.
The problem I'm having is taking that required list and joining to my list of employees with missing data and pulling from it the types and stages that are missing by employee. I thought I could get a list of the employees that are missing information and right join it to the required list where the missing records would be nulls. But, that doesn't work because some employees do have the required information and so I'm not getting any nulls returned.
If I run an UPDATE query on a table which is partitioned by the column I am updating - will the records be moved to another partition?
ie. I have a table where Historical bit column marks whether a particular record should go to Partition1 (=0) or Partition2 (=1). Now, I update a record in that table and change the Historical column value from 0 to 1. What happens with that record?
Hi-I have a sql server database, and am wring web apps to access it.I've created databases different ways, and ended up with different owners (eg dbo, nt authorityetwork services...)I also have connection strings using windows authentication, and some using a user name and password.I have read that using windows authentication is the best way to go, as far as security goes, but I have noticed some connectivity issues when I upload the site to the server, and test it remotely. What is the safest 'owner' of the database, and what's the safest way to connect?Thanks Dan
Hi,I am trying to write a method which needs to call a stored procedure and then needs to get the response of the stored procedure back to the variable i declared in the method. private string GetFromCode(string strWebVersionFromCode, string strWebVersionString) { //call stored procedure } strWebVersionFromCode = GetFromCode(strFromCode, "web_version"); // is the var which will store the response.how should I do this?Please assist.
guys,I have a project need to move more than 100,000 records from onedatabase table to another database table every week. Currently, usersinput date range from web UI, my store procedure will take those dateranges to INSERT records to a table in another database, then deletethe records, but it will take really long time to finish this action(up to 1 or 2 hours).My question is if there is some other way I should do to speed up theaction, I am thinking about use bcp to copy those records to datafileand then use bcp to insert it into SQL Server table. Is this the rightway to do it or should I consider other solution (then, what is thesolution.)Thanks a lot!
Hi, I just have a Dataset with my tables and thats it I have a grid view with several datas on it no problem to get the data or insert but as soon as I try to delete or update some records the local machine through the same error Unable to find nongeneric method... I've try to create an Update query into my table adapters but still not working with this one Also, try to remove the original_{0} and got the same error... Please help if anyone has a solution
Hi,I have a table which I need to store times in. What is the best way of doing this? The only way I can think of so far is to save the times as a datetime such as '1/1/1900 03:00 PM' then format the date as "hh:mm tt" so that the day isn't displayed. Is this the best method?Thanks,Curt.
Lets assume database A is production, B is copy. SQL Server 2005 sp2, SQL CE 3.5
Database A has a variety of transactions against it 24x7 Database B (the copy) is for reporting and as a source of merge replication for SQL CE instances Merge replication and reporting is used 24x7 as well
I have the following requirements: Maintain an up to date copy of the production database (need not be up to the minute, could be hourly, even daily update) Database B is read-only. The merge replication is NOT bi-directional.
Here is the caveat (which I think prohibits using some solutions to this problem): The production application accomplishes much of it's functionality with in-memory copies of records. I have no control over the production application. When it works against the database, it sort of does a 'withdrawal-deposit' scenario. (to the best of my knowledge it's not using SQL Server transactions) So, for every record it works with, a copy is made out of the database, changes are made in memory, a delete of the database record is done, then the record is re-inserted.
With this kind of behavior in db A, I'm not sure what it would do to log-shipping or transactional replication. I do know that I want to minimize the changes required at the SQL CE instances to keep the sync operation to a minimal cost.
hi, I have a access database. In this database there are 10 tables and this tables are related to each other. I want to move these tables to another database according to a field in XYZ table (vertical fragmentation).
for example there is name coloumn in the XYZ table and I want to take only joe lines and other tables related to that from database A and move it to database B
Hi, I already submitted this type of question before and i receive reply. But unfortunately i found out errors when performing on my system.
My problem regarding to this one:
Suppose i have two databases with same tables with different records and I would like to copy the records from one database to another data and vice-versa. So that both the tables contains same number of records inside the tables.
Example:
Database1 (EmployeeTable) contains 6 records. Database2 (EmployeeTable) contains 10 records.
It should copy only those records which is not present in each other database. No duplicate records.
Before i was recommend to use Primary key, if it is not present use index.
I have a table that is corrupted and want to remove and add a backup version of it. How can i remove this table and add it again preserving all the foregin key restraints, permissions, dependencies, etc? Simply exporting and importing does not work. I could painfully remove the table and then painfully reconnect it again, recreating all the foreign key restraints, etc, by hand; but there has to be an easier way! What is the How-to?
I have a process that restores a backup from a primary server to a backup server daily. When doing the restore, sometime it fails (for various reasons).
I have coded a job to Set offline, set online, an then do the restore:
RESTORE DATABASE [xxx] FROM DISK = N'D:Backup Stagingxxx.bak' WITH FILE = 1, NOUNLOAD, REPLACE, STATS = 10
Sometimes it fails to bring back online, other errors as well. Is there a reliable method of doing this?
I’m working with an msde database on my local machine, but need to take that database and move it to my remote host. This is problematic because I can not just do a back up and restore. I was wondering if there were any asp.net applications that would generate lots of sql statements to reproduce the database on my remote host. I had built a program to back up all the tables in that way, but, forgot that the membership stuff uses stored procedures, and I don't not know how to modify my program in order to back those up too. Any help will be much appreciated, thank you very much for reading my post
If you want to move all the databases to new server. Stop the SQL server service on existing server. Copy mdf and ldf files of all databases to same location iin new server. Also take the backup of master and msdb database from existing server. Then restore master and msdb on new server.
"
My question is how to move my database to a DIFFEREN LOCATION on a different server. My two servers are different file layout (not my choice though.)
Thanks a lot in advance.
Wuchun P.S. Should I use the DTS at all to perform this task?
I need your suggestion or experience about how to move a database from SQL 6.5 to SQL 7.0. The server where SQL 6.5 resides locates in different network than SQL 7.0. DTS does not work for the database move because of two server can not talk directly.
We need to do this task very soon in production environment. I tried to backup database to a disk (backup device in SQL 6.5) but I failed to restore it in SQL 7.0 because the backup file was not detected by restore utility in SQL 7.0.
Please tell me what I should do. Besides BCP, are there any other methods to move the database from SQL 6.5 to SQL 7.0 in diffferent networks.
I am new to sql server 7 and dont know how to 1.make a copy of a database
and 2 to move a database i know there are sys stored procedures but the syntax looks crptic and i dont know where to type them in if any one could give me any really clear examples of how to do both of the above i would be really grateful, thanks!
i've got to move 37 SQL Server databases over to new server. My manager tells me that i have to conduct some "check totals" to make sure that the databases contained exactly the same data from what it was when it got moved.
Is there some stored_procedure to do this ? to make sure that all the stored procedures, views, tables got move across ?
Hi, I'm new to SQL Server. How do I port the modifications of my table structures, store procedures, functions on the staging server to the procduction server without affect the data?
Can I make this modification to production database while user are still accessing the database or I have to bring down the database to make these changes?
Is there a way to move database from developement to production servers along with the logins and their associated roles as well as passwords. I did backup and restore and used sp_change_users_login to fix the logins
I mean is there a way to copy logins and roles Thanks
Can anyone tell me if it is possible to move the data directory of SQLServerfrom the application server where SQL runs to another system (the file andprint server) in the network?regards Joop
I'm needing to move a database from SQL 7 to SQL 2000. What is thebest way to accomplish this and keep all tables, data, permissions, etcin place throughout the move.Should I use BCP, Replication or something else?Thanks for any help.
I have a web project in VS.NET 2005 on my development computer that uses an ASP.NET database (ASPNETDB.MDF) to manage user roles. Now I would like to move that database to a test computer with IIS. After copying the web project and database I am able to read from it but when I try to update I get the following error: Failed to update database "C:INETPUBWWWROOTMyWebAppAPP_DATAASPNETDB.MDF" because the database is read-only.
I think this is because the user 'Network Service' that IIS uses does not have access to the database. So how do I give 'Network Service' access to the database using SQL commands?
I'm trying to move a database created on SQL Server 2005 Express Edition to the SQL 2000 Server (that's the version the company uses). I understand that this is done by making a script that, when executed, creates an exact copy of the database structure on the target server as well as copying the stored procedures. After that I somehow need to export the data from SQL Server 2005 Express to SQL Server 2000. How can I do that. There's no export function in Management Studio Express.