Migrating Large Amounts Of Data From SQL Server 7.0 To 2000
Jul 20, 2005
I'm in the process of migrating a lot of data (millions of rows, 4GB+
of data) from an older SQL Server 7.0 database to a new SQL Server
2000 machine.
Time is not of the essence; my main concern during the migration is
that when I copy in the new data, the new database isn't paralyzed by
the amount of bulk copying being one. For this reason, I'm splitting
the data into one-month chunks (the data's all timestamped and goes
back about 3 years), exporting as CSV, compressing the files, and then
importing them on the target server. The reason I'm using CSV is
because we may want to also copy this data to other non-SQL Server
systems later, and CSV is pretty universal. I'm also copying in this
format because the target server is remotely hosted and is not
accessible by any method except FTP and Remote Desktop -- no
database-to-database copying allowed for security reasons.
My questions:
1) Given all of this, what would be the least intrusive way to copy
over all this data? The target server has to remain running and be
relatively uninterrupted. One of the issues that goes hand-in-hand
with this is indexes: should I copy over all the data first and then
create indexes, or allow SQL Server to rebuild indexes as I go?
2) Another option is to make a SQL Server backup of the database from
the old server, upload it, mount it, and then copy over the data. I'm
worried that this would slow operations down to a crawl, though, which
is why I'm taking the piecemeal approach.
p.s. my email was incorrect in the last mail. Hi all, is there a sql 2k thread. Am interseted in finding out what the largest database size of a sqlserver database people have worked with. We have a 1.2 Terabyte db with about 150-200 million new rows being processed everyday. Would like to share some thoughts on this with other people who are working with this much data and what they are doing with it.
bhala ---------------------------------------- Please check us out at: http://www.bivision.org/bivision
HiI have a VB.net web page which generates a datatable of values (3 columns and on average about 1000-3000 rows).What is the best way to get this data table into an SQL Server? I can create a table on SQL Server no problem but I've found simply looping through the datatable and doing 1000-3000 insert statements is slow (a few seconds). I'd like to make this as streamlined as possible so was wondering is there is a native way to insert all records in a batch via ADO.net or something.Any ideas?ThanksEd
Hello,Currently we have a database, and it is our desire for it to be ableto store millions of records. The data in the table can be divided upby client, and it stores nothing but about 7 integers.| table || id | clientId | int1 | int2 | int 3 | ... |Right now, our benchmarks indicate a drastic increase in performanceif we divide the data into different tables. For example,table_clientA, table_clientB, table_clientC, despite the fact thetables contain the exact same columns. This however does not seem veryclean or elegant to me, and rather illogical since a database existsas a single file on the harddrive.| table_clientA || id | clientId | int1 | int2 | int 3 | ...| table_clientB || id | clientId | int1 | int2 | int 3 | ...| table_clientC || id | clientId | int1 | int2 | int 3 | ...Is there anyway to duplicate this increase in database performancegained by splitting the table, perhaps by using a certain type ofindex?Thanks,Jeff BrubakerSoftware Developer
I was wondering if any one could help me, I need to store large amounts of data in my database, at present I have it set to nvchar (8000), I've looked around and noticed you can use text which stores up to 2 million, but is slow in displaying the information.
Any ideas or points in the right directions would be great.
Does anyone have ideas on the best way to move large amounts of databetween tables? I am doing several simple insert/select statementsfrom a staging table to several holding tables, but because of thevolume it is taking an extraordinary amount of time. I consideredusing cursors but have read that may not be the best thing for thissituation. Any thoughts?--Posted using the http://www.dbforumz.com interface, at author's requestArticles individually checked for conformance to usenet standardsTopic URL: http://www.dbforumz.com/General-Dis...pict254055.htmlVisit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbforumz.com/eform.php?p=877392
We are looking to store a large amount of user data that will bechanged and accessed daily by a large number of people. We expectaround 6-8 million subscribers to our service with each record beingapproximately 2000-2500 bytes. The system needs to be running 24/7and therefore cannot be shut down. What is the best way to implementthis? We were thinking of setting up a cluster of servers to hold theinformation and another cluster to backup the information. Is thispractical?Also, what software is available out there that can distribute querycalls across different servers and to manage large amounts of queryrequests?Thank you in advance.Ben
I have a dataset with 300,000 records and I'm getting the following error with MS Reporting Services. "An error has occurred during report processing. Exception of type System.OutOfMemoryException was thrown. any help with this would be highly appreciated.
I need to be able to graph roughly about 150 employees/ supervisor and their monthly cell phone usage in minutes. I understand that I will need to group this on say one graph for every ten employees so it doesn't look messy and cluttered. I have read some threads here but they dont seem to work for me.
So again each supervisor has 100+ subordinates and I need to graph theie phone usage by month
I'm trying to move my current use of an sql 2000 db to sql 2005.
I need to update a table definition (to change a field to an Identity)
I'm getting a dialog box (in SQL server management studio) on save saying :
'xxxx' table
- Saving Definition Changes to tables with large amounts of data could take a considerable amount of time. While changes are being saved, table data will not be accessible.
I press 'Yes' to the dialog box.
After 35 seconds, I get another dialog box saying:
'xxxx' table
- Unable to modify table.
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Well, the server is responding and I can query that talbe and other, I can add/delete rows to other columns. I can modify other (smaller) tables.
this may seem like a simple question, but I have a report/lease agreement I need to put together and wanted to know the simpliest way to add large amounts of text. Basically its all the legal stuff most leases include in the amount of some 14 pages.
Should this be just one long string-- or does ssrs have another way to format this
I was wondering what is the fastest way to UPDATE lots of recods. I heard the fastest way to perform lots of inserts in to use SqlCeResultSet. Would this also be the fastest way to update already existing records? If so, is this the fastest way to do that:
1. Create a SqlCeCommand object. 2. Set the CommandText to select the datat I want to update 3. Call the command object's ExecuteResultSet method to create a SqlCeResultSet object 4. Call the result set object's Read method to advance to the next record 5. Use the result set object to update the values using the SqlCeResultSet.SetValue method and the Update method. 6. repeat steps 4 and 5
Also I was wondering do call the SqlCeResultSet.Update method once per row, or just once? Also would it be possible and faster to wrap all that in a transaction?
Would parameterized updates be faster? Any help will be appreciated.
I have been looking into mirroring a large amount of small databases approx 150 databases.
As I understand this won't be feasible because of the way mirroring threading works, http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=441900&SiteID=1
As I understand it for every database being mirrored sql will ping the mirror second, causing a network bottleneck?.
Also that the amount of threads generated for each mirrored database will cause also cause a bottleneck?
At the moment our database servers are under very little pressure and as an estimate use about 10% of the resources allocated to them such as CPU utilization, memory, disk IO and network. Our server hardware is Dual Quad core Xeons with 4 - 8 gig of memory and variety of 10k SCSCI raid configurations from raid 5 or 1,0 and sql 2005 32bit.
Ive done some calculations on the log file generation rate compared to network bandwidth there is more than enough network bandwidth.
Has anybody had any luck in mirroring many small databases?
My concerns is how much traffic is caused by the pinging of the mirror for each database?,
How many threads will the mirroring cause and what is the max amount of threads sql can handle?
How much memory will be consumed by each one of these mirroring threads?
I am running into a problem inserting large amounts of text into my table. Everything works well when I test with a few simple words but when I try to do a test with larger amounts of text (ie 35,000 characters) the appropriate field is left blank. The Insert still performs (all the other fields recieve their data, but the "Description" field is blank. I have tried this with both "text" and "ntext" datatypes. I am using a stored procedure with input parameters. As I mentioned, the query goes off flawlessly with small amounts of data (eg "Hi there!") but not with the larger amount.I check and the ntext field claims to be able to accept 1073741823 bytes of data. Is there some other thing I should consider with large amounts of text?
I know the standard Microsoft recommendation is to make the pagefile at least 1.5 to 3 times larger then the amount of physical memory. However, if you're talking about a server with lots of memory such as 16GB or 32GB, would following this rule be unnecessary. With SQL 2000 running on Windows 2000 Server or Windows Server 2003 I typically see pagefile usage no more then 12% for a 2GB pagefile. Anything over 15% means I need to look at other indicators to see if a memory bottleneck has developed. If I have 32GB of physical memory and make the pagefile only 1.5 x 32GB I have a 48GB pagefile. 10% of this is 4.8GB, which I would hope I never see consumed.
I'm trying to get a database from SQL2005 to SQL2000. I already recreated the database in SQL2005 and I'm just trying to get some data from just a few tables.
It doesn't appear the DTS or any export functionality is available in my version of SQL Express 2005...so how do I do this. I've tried installing everything I can find.
I've also tried BCP but I keep getting these errors:
SQLState = 08001, NativeError = 53 Error = [Microsoft][SQL Native Client]Named Pipes Provider: Could not open a connection to SQL Server [53]. SQLState = HYT00, NativeError = 0 Error = [Microsoft][SQL Native Client]Login timeout expired SQLState = 08001, NativeError = 53 Error = [Microsoft][SQL Native Client]An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections.
This is a LOCAL database and I have tried -T for trusted, I have also tried a username and password combination. The database is set to Allow Remote connections. Nothing is working.
I have a 2-node cluster environment & I am planning to replace it with new hardware. Currently I am using MSA1000 Disk arrays. Now I would like to move my storage to SAN.
What are the necessary pre-requisite and actions to do this. Any particular thing, I must take care in planning.
Hi,I have developed an application using VB 6 (SP 5), MS Access 2000 andCrystal Reports 9. I migrated from from access to SQl Server 2000.This wasn't a problem as i could import all the tables to SQL Server.Also, i could make VB6 talk to SQl Server.The problem arsies when i run my application. The sql syntax foraccess seems to be different than that for SQL Server. A simpleexample being: In access boolean datatype is true/false ,whereas inSQL Server the boolean equivalent is bit (numerical 1 or 0). Thesekind of issues are causing problems and most queries don't run.Would i need to go and change all the queries in accordance with SQlServer syntax ,which would be very time consuming or is there anyfunction which will convert the access datatype into its equivalentSQl Server datatype??Any input/thoughts/suggestions would be appreciated.ThanksJatin
Any suggestions on migrating Access97 tables to sql server 2000? The DTS seems cumbersome and the Access97 upsizing wizard add-in is only compatible up to sql server 7.0.
Hi I am trying to migrate my database in Oracle 8i to Sql Server 2000 but am unsure of how to carry out the migration. How do I do it and what do I have to consider. My OS is Windows 2003. Thanks in advance
Has anyone had experience of migrating from 2000 to 2005 on the same box , and maintaining the SERVERNAMEINSTANCE_NAME. What are some effective ways of migrating and retaining the same servername/instance name ?
The issue revolves around minimising the use of extra boxes , as I would like to do the migration on the same server and keep the name the same .
Jack Vamvas -------------------- Search IT jobs from multiple sources- http://www.ITjobfeed.com
I need to migrate my access database to SQL2000 Server. I have three tables in my access database. In each table have 3000-6000 records. So how to put all these records in to sql2000 server tables. The table structure in Access database and sql2000 database are same.
Here core work is just insert all these 3000 - 6000 records (rows) from Access database to sql2000 database
We are migrating from SQL Server 2000 to 2005. We currently use ADO.Net to make connection to sql server. I just want to know after migration to sql server 2005 do we have to make any changes in code in the way we make connection to use ADO.Net2 which is part of sql server 2005. ??
whenever i modified the connection of my current database...I'll always got an error,when asking for "server name"..it says [DBNETLIB...etc]Named Pipes etc error..my current database was SQLEXPRESS and i want to change it as an SQL OLEDB,2000,2005.. but as i said il always got an error..sometimes it says "SQL server does not exit..ODBC" etc...when im entering the "server name"..so i could only used Access or SQLEXPRESS server database.. what should i do with the error?? thanks for helping again..
Just a quick question, does anyone know any major issues with Migrating Sybase Databases to SQL Server? Just wondering if their any oddities I should be aware of? Thanks nixies
First off, sorry if my cross posting offends anyone. I'm posting thisin Access and SQL Server groups - not sure which one is appropriate.I have a relatively simple ASP.NET/VB.NET application that is nowhitting an Access 2000 database over an intranet. We have to migratethe database to SQL Server 7. My experience with ASP.NET is prettylimited and my experience with SQL Server is nonexistent.We have an MSDN subscription, so I went to the downloads section to getSQL Server 7. Guess what? SQL Server 6.5 is available, as is 2000 and2005, but no SQL Server 7. So my first question is, does anyone know ifit is available for MSDN subscribers?The next question is, does anyone know of a good resource that explainshow to make the transition from Access 2000 to SQL Server 7? I know thedata has to be migrated and the connection between the application andthe database modified, but am really not sure exactly what to first andthe correct way to go about it. The application is a simple productconfigurator. There's not a lot of data and it's not a very complexdatabase.Thanks in advance. If you'd like, please copy responses tolcifers(AT)yahoo.com (AT) = @TIA.Cheers.- Luther
When I try to migrate a database on a SQL Server 2000 server to a SQL Server 2005 server with the Copy Database Wizard of the SQL Server Management Studio, I'm confronted with the following problem;
Performing operation...
- Add log for package (Success) - Add task for transferring database objects (Success) - Create package (Success) - Start SQL Server Agent Job (Success) - Execute SQL Server Agent Job (Error) Messages * The job failed. Check the event log on the destination server for details. (Copy Database Wizard)
When I take a look at 'Event viewer' on the SQL 2005 server, the following error is displayed;
InnerException-->An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
I already enabled the MSSQLSERVER network configuration protocols (TCP/IP and Named Pipes ).
I'm migrating SQl Server Reporting Services from 2000 to 2005.The reports are generated normally in 2000 but it seems that it is taking more time in 2005 or sometimes it does not generate the report at all.Could you kindly suggest a solution?
I have successfully moved my data from a SQL Server 2000 hosting site to a SQL Server 2005 hosting site. I Made a backup of my database using Enterprise Manager (2000) and imported the database tables using SQL Server Management Studio (2005). I do not know how to move the 25 or so stored procedures that I have in SQL Server 2000. I have a very short amount of time to figure this out and am hoping that someone can give me a brief step by step answer on how to get this done. I would appreciate any information you can provide. Thank you!!
