Managing Large Database
Aug 11, 2007
I am developing an office automation software for a government department. To start with, I have decided to first automate the salary section of the department. Because of some issues like, TCO, easy support and maintenance, scalability etc., I have decided to use SQL Server 2005 database with VB 2005 front-end.
I have planned the layout for this first part and found that in only employee salary table, 40,000 records per month will be stored. That comes out to be around 480,000 records annually. This is for one table alone, excluding data in other tables.
Since each section of the department will be integrated into this later, this application will become the backbone the department. The employee service, leave, salary, allowance, deduction and other records shall be maintained.
For this type of mission-critical application, I want to get some queries cleared:
(1) What backup strategy should be followed? I want to have schedule and maual backups both. Is there any way to have a mirror image on another server?
(2) What considerations to keep in mind in the inital stage of database design?
(3) How to keep the design scalable and configurable to meet future needs?
View 6 Replies
ADVERTISEMENT
Jul 28, 2007
Hi,
Our database has very large number of objects. We have a naming convension by modules, subprojects etc. But for example when we need to open a specific table it still takes time to find it. If we could create custom folders under table folder or stored procedure folder it will be easier to find an object. We could create sub folders by module, subproject and classify our objects with these folders. Will the next version SQL Server 2008 support this kind of functionality?
View 8 Replies
View Related
Dec 7, 2006
Hello,
I haven written a Windows program in Visual Studio 2005. This program stores data in and retrieves data from a SQL Express database. The customer is starting to use the program and filling the database with data. However the program is still under construction (new functionality will be added) and so is the database (new tables, views etc. will be added).
Of course I could send the customer the new database, but than all data already entered will be lost.....
I'm thinking about using the SQL Management Studio Express and VS2005 to connect remotely to the database (a VPN connection is availabe) but worry about performance issues.... Or should I use some kind of replication / synchronization and how, because this is new to me.
What is the best way to solve this 'problem' ?
View 5 Replies
View Related
Jun 26, 2006
we have a group of developers which have created and asked us (DBAs) tocreate many objects in the databases including tables / storedprocedures / functions / etc.since our company is growing, however we have an increasing amount ofobjects that have either been abandoned or have several versions.in an effort to clean of the huge amount of clutter and anytime thatsomething simple like a stored proc needs to change, it is almostimpossible to predict exactly where we will see negative effects ofthis change.i am looking for a system (preferably without developing our own tool)that would keep track of history of database objects (in terms of whocreated it and what purpose it has) as well as link that to all thedevelopers/users we might need to notify of any changes to that object.Also, this should be linked to the application which rely on theobject.bottom line... every object in the database needs to have at least 1corresponding contact as well as the applications which us it.with this information, we can much more easily maintain objects in ourDBs.thx
View 3 Replies
View Related
Apr 23, 2007
guys,I have a project need to move more than 100,000 records from onedatabase table to another database table every week. Currently, usersinput date range from web UI, my store procedure will take those dateranges to INSERT records to a table in another database, then deletethe records, but it will take really long time to finish this action(up to 1 or 2 hours).My question is if there is some other way I should do to speed up theaction, I am thinking about use bcp to copy those records to datafileand then use bcp to insert it into SQL Server table. Is this the rightway to do it or should I consider other solution (then, what is thesolution.)Thanks a lot!
View 8 Replies
View Related
Jul 23, 2007
What happens if you have a website and the hard drive on your server is say 250GB. Then the database exceeds that and the database is 300GB in size.How would you spread your database into two different hard drives?Thanks
View 2 Replies
View Related
Apr 17, 2003
Hello,
I would like to know whether MS SQL Server 2000 Can handle Large Database size 28 gig, currently we are using Sybase but we want migrate to SQL Server 2000. Are there any performance issues?
Any help is appreciated.
Thanks
Sejal
View 1 Replies
View Related
Aug 17, 2006
We have many installations of our shopping cart database. One specific database is huge, now about 25 GIG, compared to the others that range from 20 to 75 MEG. The server this one resides on has three other instances of the same database that are normal size.
In a particular table in the large database there are 9700 rows taking 380MB, the same table in a normal db has 162000 rows and takes 6.MB. The tables are identical and the indexes are the same.
Any ideas out there?
View 6 Replies
View Related
Sep 12, 2006
I upgraded a database from SQL2000 to SQL2005 and it went pretty smooth. After the transition was made, I backed up the DB. The size of the database was as expected, about 5 GB. About 5 hours later, a maintenance plan executed a few optimization jobs in the following order: Reorganize Index, Rebuild Index, Shrink Database, Update Statistics. Soon after that, another job backed up the database and it was then 32 GB. During the time after the first backup, no one was really using the database.
I've been trying to track this down for several days. Does anyone have any ideas for me?
Thanks,
Bobby Crawford
View 1 Replies
View Related
Dec 4, 2006
Hello!
I have a database with one table inside. The table has six columns with the following datatypes:
col1 -->smallint (2byte)
col2 -->int (4byte)
col3 -->smallint (2byte)
col4 -->smallint (2byte)
col5 -->smallint (2byte)
col6 -->int (4byte)
I have insert 1.359.320 rows with data and the size of the sdf file is 40.116.224 byte.
According to my calculation:
1.359.320 * 2byte + 1.359.320 * 4byte + 1.359.320 * 2byte + 1.359.320 * 2byte +1.359.320 * 2byte + 1.359.320 * 4byte = 21.749.120 byte
I hope somebody can explain me, why the database is so large.
Thank you
Sascha
View 4 Replies
View Related
Oct 8, 2004
Importing data from an Access database, I cannot overcome the limit of 1,000 records.
In DTS, I "copy one or more tables", select tables, run, and cannot see my 1,052 entries.
Where can I set a max size of ~1,500 in my sql target base?
View 1 Replies
View Related
Jun 15, 2000
Hi Everybody:
We now host 200 G data in SQL Server 7.0 in MSCS. The data will be 1 terabyte at end of the year. The possible solution to host large data with good performance is to migrate data to Oracle or UDB in Unix platform. But we would like to try any possibility in Microsoft world before we decide to migrate data to the other databases.
I need your experience and recommendation about hosting large database in SQL Server with good web performance. Somebody mentions Microsoft tera server. Who has experience using tera server which is still in the beta release? I would like to hear some successful story to handle the large data and understand precisely hardware, software, performance moitoring requirement.
Thank you very much.
Rich
View 2 Replies
View Related
Dec 14, 2006
Hi to All
I have large database with tables and views on the server around 80 to 100 mb amount of data.when i retrieve the data from the server it is taking long time Ex: "select * from vwcases" its taking 10 to 12 minuts time.this is my database problem.If you know the solution please reply back to me as soon as possible.
thanking You.
View 4 Replies
View Related
Mar 2, 2005
I need to query a database with about 3 million records using a 'like' statement on various varchar columns. I've have a non-clustered index on the columns included in my query, but the performance is still pretty slow. Can anyone suggest ways to increase the speed of my queries? Does full-text indexing improve performance or does it just facilitate more advanced searches (which I don't need)? Would char instead of varchar datatypes be faster?
thanks!
View 3 Replies
View Related
Jan 7, 2005
Hi Guys,
I am looking for a large datawarehouse database implemented in SQL Server to use it to run some examples in my thesis. if someone knows where I can find a large datawarehouse database (e.g Sales database) please tell me.
I appreciate any help.
Thanks
View 1 Replies
View Related
May 17, 2004
Does anyone know of a good way to copy a large database or backup of a database (~10 GB) across long distances (transatlantic) in a minimal amount of time?
I have tried regular file copies, BCP, DTC, my own handwritten extraction utility, PCAnywhere, and several file splitters, all to no avail. To complicate things, the source is SQL 7 with a different code page from the destination, which is SQL 2K. The file copies always seem to get interrupted after several hours.
View 6 Replies
View Related
Jan 26, 2007
Hello all,
We have total 7 databases in the warehouse and these db are big. All the data files are approximately 300G now and they all growing more. Some are small db some are large db. (ranging from 20G to 50G of data files). Can someone please recommend me the best backup plan for this scenario. The regular backup through SQL 2005 takes too long.
Thanks for your help.
tocroi
View 1 Replies
View Related
Feb 24, 2006
I developed a vb app that imports csv data into an sql server db. The original text file is 36.5mb. The db after import is 230mb and the log file is 555mb. Is this normal?
View 8 Replies
View Related
Oct 23, 2006
Hi Experts
We are debating what is best:
1. To combine all the company's data in one large database, and use schemas and file groups to create logical and physical distribution on drives and namespaces
or
2. Distribute the data into smaller databases with related data - eg. products and product description in one db, Customers in another and orders and orderlines in a third db.
Just what are the pros and cons?
regards
Jens Chr
View 3 Replies
View Related
Jun 21, 2007
Hello, All:
I have many, many Access databases that are roughly 1.5GB-3GBs each and they have millions of records. Each MS Access Database file corresponds to one Database in SQL server. I'm trying to simply transform the data as it is in Access to MS SQL 2005.
I'm using the 64 bit version of Windows Server 2003 and the 64 bit version of SQL 2005. The server is running four dual core AMD Operton processors and has 8GB of RAM with a 1TB RAID 5 configuration. I think the hardware should be sufficient but the SQL Server Import and Export Wizard can't seem to handle the large number of tables/records. If I do one table at a time, it works well; however, it produces the following error message whenever I try to import the entire database:
Pre-execute (Error)
Messages
Error 0xc0202009: {5A5BF7AD-E86B-4316-AD43-1912358C56F4}: An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft JET Database Engine" Hresult: 0x80004005 Description: "Unspecified error".
(SQL Server Import and Export Wizard)
Error 0xc020801c: Data Flow Task: The AcquireConnection method call to the connection manager "SourceConnectionOLEDB" failed with error code 0xC0202009.
(SQL Server Import and Export Wizard)
Error 0xc004701a: Data Flow Task: component "Source 64 - District Corporal Punishment Class" (5743) failed the pre-execute phase and returned error code 0xC020801C.
(SQL Server Import and Export Wizard)
Any ideas would be much appreciated!
Thank you,
Cody
View 1 Replies
View Related
Feb 19, 2012
I have a distribution database that has grown over 50g. Distribution cleanup takes over 15 hours to run and there are always very few rows in msrepl_Commands. Usually only a few hundred. Everything is being replicated fine, but I fear a storm coming. why distribution would be so large with so few rows? Also, Ghost Cleanup has been running all week, so I'm thinking maybe the records marked for deletion aren't being deleted?
View 1 Replies
View Related
Oct 9, 2014
I've just inherited the job of looking after our SQL server (2012). The server works fine but I am a bit concerned about the backup strategy we have in place. The current backup strategy is a SQL script that runs every 2 nights and backs up to an ISCSI box elsewhere on the network. One of the databases on the server is currently at 303GB, which will take about a full working day to restore to another server if this one fails. I am just looking at other ways which could reduce the amount of time to restore this if this server was to fail. i heard so people talking about sql clusters and configuring replication on posts by others, which i will look into if its the way to go, but i guess I am just looking what others would do if they were in the same position with such a large database.
View 1 Replies
View Related
Mar 21, 2008
Hi i wanna delete all the records from an large database 200 -300 tables, because i want make some changes an start from scratch,but keep the structures of the database key , index etc, i tried to generate script but when i run to many errors , plz help 10x
View 11 Replies
View Related
Feb 5, 2007
hi
is there a datatype thatcan handel a very large string ( i dont want to save it in a file and save the file in the bd ) i want to save it as it is
thanx a lot
View 7 Replies
View Related
Oct 23, 2007
Hopefully I'm posting in the right area. There is a database that has grown to about
41-42 GB in size in about a 2 month period. The previous database had grown to about
22 GB before it was purged out. I'm running this on SQL 2000, and I've tried running all
the DBCC SHINKFILE and SHRINKDATABASE commands to no avail. In this case,
the MDF file is the one that has grown out of control as opposed to the log file (LDF file).
Does anyone have any suggestions on what could be done to control the size?
View 17 Replies
View Related
Apr 26, 2001
hello everyone,
I have a question about the under what circumstances will a large database with mostly free space not shrink?
thanks
View 2 Replies
View Related
Nov 13, 2000
Good morning one and all,
I need to transfer a database (contining one table) containing over 35 million records from one server to another. I have two options at present :
(a) Use DTS to do the transfer
(b) Copy the mdf file across and sp_attach_db it
Does any1 have a better idea, or does any1 know which of the two methods will be the quickest?
TIA
Gurmi
View 1 Replies
View Related
Dec 7, 2000
Here's my delema, I have a file that's 308 bytes wide by 5.7 million records. The record length is fixed and the position and width of the known within the record. When I run DTS I recieve this error Msg MS DTS flat file provide and Err Diesdription: error creating file mapping view: not enough storage is available to process this command. Then when I try to continue with the wizard, it will not allow me to separate the data into the format that I need. Is there any other way to import this file using DTS?
View 1 Replies
View Related
May 29, 2008
Hi guys,
I am asking this question on behalf of a friend. I have little knowledge of SQL 2005 but my friend is quite knowledgeable, although this is the first time he is dealing with large database for a client. So here's the story.
His client has a database containing 1.5 million books. Now he is setting up a website which will enable users to search books. Searching by ISBN is no problem as it only takes 1 seconds. The problem is, searching by Title takes more than 20seconds, which is unacceptable. My friend has only done smaller database and he just recently thought of implementing indexing and now looking for other ideas.
Each row contains book details such as Title, Author1, Author2, Author3, Publisher, Publication Date, ISBN, etc.
Can anyone who are more experienced in doing large database share with me some design ideas? His client is aiming for 8seconds or less.
Thanks in advance!
View 14 Replies
View Related
Jul 7, 2015
I have come across a database system which isn't designed to work optimally. It is fairly large (~400GB) and performance of loading and querying is degrading (improper data types, fragmented indexes, non unique clustering key and other problems). So, I have quite a task in front of me, but I am up for the challenge. I figure this is not a unique situation, many of us would have come across this before. I have done this before too, but only for smaller databases, some of the operations here I expect to take a couple of hours or more to complete (depending on load/infrastructure speed etc, I know).
My plan is thus:
+ Take a full backup of the database
+ Set the recovery model of the DB to simple
+ Drop non clustered indexes
+ Drop clustered indexes
+ Remove PKs (wrong data types, too large!)
+ Narrow data types (add new column, update column in batches to old value, rename new column to old column)
+ Add PKs, which will create clustered indexes automatically based on PK ID
+ Create non clustered indexes
+ Run a SHRINKDB (normal operations I would never do this, but this is a special case, ensure log file is truncated to a logical size especially after all those table modifications...)
+ Set the recovery model of the DB to Full
+ Ensure everything works OK or better
View 9 Replies
View Related
Sep 20, 2015
We have a situation that requires converting all INT columns from source database to BIGINT in destination.
By ‘copying’ term, I consider any operation like insert-select, merge, asynchronous replication or any other that will get all data synchronized until some moment in time, i.e. 2015-12-31 10:00 PST when downtime is planned. Before that moment it isn’t important if data in source and destination DB are in sync.
Copying process starts early enough to allow appliance of any technique/technology – there’s enough time to find best solution.
Requirements:
1. Destination database is superset of source, having all tables and all columns plus few more columns in few tables. Additional columns have default NULL or they are computed. INT columns in source are BIGINT in destination.
2. If source and destination are not in sync at the moment downtime begins, it is allowed to have few minutes to wait for synchronization. For instance, if transaction logs from last hour or so need to be shipped, that would be ok. If large table need to be synchronized by comparing row-by-row, it’s not ok.
3. Source database performance must not be compromised by any copying if it’s done before scheduled downtime. There are couple of options I have in mind involving MERGE command but its appliance might be limited by current DB design. Additionally, based on business logic, I cannot be absolutely sure if some row, created while ago, wasn’t changed meanwhile.
For some large tables, there is no indicator at all when any row has changed – I cannot rely on IDs nor there is column like ‘[date last changed] DATETIME’
I have been reading about various kinds of replication, i.e. log shipping but I’m not clear if any of these is applicable. Key points, if there is replication involved, are copying process must start from empty destination database plus synchronization must not compromise source DB performance So, how to do this using any kind of replication?
View 17 Replies
View Related
May 29, 2007
My question is two fold:
We have a database 65 GB in size and has grown over 12 years.
1) How can I upgrade to 2005 without downtime?
2) Our upgrades on SQL 2000 now can take upwards of 10 hours to add just a column and rebuild index tables?
Any way we can speed this up without detaching the database and going offline?
thanks,
Larry Sitka
View 6 Replies
View Related
Mar 9, 2007
I need to copy data from TableA to TableB (>5 millions rows). The two are in the same database.
What is the best way of doing this?
I was thinking about using a simple INSERT INTO ... SELECT statement. Is there a faster way to do it with SSIS?
Thanks
View 13 Replies
View Related
