We have a database that's growing pretty fast because of firewall logs. We need the data available via an asp.net application. I don't have great experience with SQL other than installing and doing some development as a back end, so i'm wondering if there's a general rule of thumb of database size, when you should start breaking it out into smaller segments? if so, what are some good practices?
View 3 RepliesWe're a very small company, I’m the “technical” director who has evolved enough skill in a wide variety of tasks (network setup, machine config, email systems, html, asp, database...) and then one day you notice that parts of the system are starting to get way more complex and troublesome than the layman knowledge you have can cope with... Well, I think I’ve got to that point and I need some outside help to get our system to the next level!
OK, some rough details to start with. We run a small but fast-growing vehicle tracking system that sends back a LOT of data via GPRS to our SQL 2000 Enterprise server hosted on a dedicated server in London. The physical machine is a P4 3.2Ghz Dual-core Dell rackmount with 2GB RAM and 2 x 76GB SCSI disks in a RAID 1 array. This is partitioned into a 15GB C: partition and a 51GB D: partition. The system paging file is set to be 1536MB and is on the C: partition. The server is used for everything we do... it runs Smartermail email server (only about 5 or 6 domains and a few users, hardly used at all), SQL server as mentioned, web server & the proxy software that receives incoming data from our tracking devices.
There are 9 or 10 active databases on the SQL server. 8 of them take up less than a gigabyte between them and are sparingly used. The main “active” database on the SQL server is the tracking system – and this is big... As our tracking devices send in data every 10 – 30 seconds, the database is hit with hundreds of thousands of events per day. On a weekday, some half a million rows of data are written to the main “events” table on the database. Over 7 days from 26th November to 2nd December, almost exactly 3 million rows of data were written to the events table. We undertake to hold 3 months or so of data “live” for our customers and I periodically archive data off. I’ve been too busy to archive recently and the database is holding data on the events table going back to July 1st. The physical .mdf file is just under 30GB on partition d: at present. The plan is to drop the active data stored to only 1 – 2 months, but this still leaves a 12GB .mdf file.
The worrying thing with this is that this is only 700 or so devices writing to us at present... we aim to have thousands out there soon! We are looking into how we can hugely improve system performance and look to the future. Our hosting company is recommending VMWare virtual servers and SAN storage, but I’m not entirely sure that is the best way forward.
Our non-tech MD thinks the way forward is to have one database per customer and can't understand when I tell him I think that's bad as it will create all the system tables and bits & pieces for EVERY customer if we do that, right? Also it would be a nightmare to add a new column to a table as I'd have to update every single version of the database too... I want to avoid this unless I'm missing something and this is actually the best way to go forward?
I've had someone mention horizontal partitioning to me? not sure what implications this has to coding and table naming? Or is it all one big database spread among separate servers?
Currently our server is drowning on disk access and it's only going to get worse... any suggestions or links to reading online that I can do would be great, thanks!
Carl
I have a database that seems to have grown out of control. I have tried deleting tables, but that has not really reduced the size. What could have caused the database to grow this big and what can I do to reduce it's size. I have backed up, truncated the logs, ran the shrink database command, all to no avail. Pleas help.
View 5 Replies View RelatedDoes anyone know at what point SQL Server 7.0 decides to grow the database when the autogrow option is set? Our site just went down for 45 minutes because the growing process was taking too long as compared to the data coming in, so the device filled up.
Ray? Craig? You guys seem to know all, so jobs.com appreciates your input...
Hi,The tempdb file on one of our servers grew very large and used allavailable disk space. This is SQL Server 2000 SP4. I have installedhotfix version 8.00.2187. I opened a profiler trace but can't still getto the root of the problem. Any help will be appreciated.Egbon*** Sent via Developersdex http://www.developersdex.com ***
View 1 Replies View RelatedHi,I am using visual studio 2005 with sql server 2005.I would like to know what is the way of coping with an increasing database?Any answer?
View 1 Replies View RelatedHi ALL
I have question about database automatically growing for SQL Server 7. It seems to me that SQL database automatically grow will ONLY happen when it's getting really full, maybe above 90% full. Even if you manually increase the DB size, actually the increased DB size will only keep very short time. And DB size will be back to the smaller size again. If you have any suggestions I will be really appreciated. Thanks
I have a database that is growing by 40 to 50 megs a day. I understand that the '_W' objects are statisical information for query performance and not indexes, but does anybody know how much disk space is actually used by these objects. I do have the 'auto create statistics' and 'auto update statistics' set on.
View 1 Replies View Relatedfor the first time in my long SQL DBA live I see such a behaviours. My tempdb database is growing every damn second since a this morning. Now it reached 30Gb, the log file is empty (217 Mb).
We use SQL 2000 Ent on Win 2000 Advance Server. Running Siebel Call Center (7.5 ver) with about 300 users.
Some users time to time obtain and hold a huge amount Exclusive locks on the tempdb extents
Where do I look for a leak?
Any ideas?
thx
Dim
Hi All.I'm currently maintaining 4 servers - 1 for public/customers and 3for backups, development, etc...I regularly backup the entire SQL database for our public server andrestore it on each of the other servers. Lately, however, the databasebackups have grown (in size) incredibly fast - they've gone from about200MB to 2+ GB in 2 months. (I wasn't entirely surprised by this atfirst since our client traffic has drastically increased as well.) Theweird thing, though, is that (on two of the backup servers) when Irestore the backup then use those servers to create a new completebackup, the new backup is only about 200-300 MB in size.My assumption is that there's some kind of setting buried deep insidethe sql configuration allowing it to compress or otherwise alterbackups. Does anyone have any ideas/thoughts as to what may be causingthis issue?We're using SQL Server 7 on Windows 2000 servers.Thanks in advance.GreggJoin Bytes!
View 2 Replies View RelatedHi All,
Database size is not increasing automatically ,however I have set it as unlimited growth. Any idea about this ?
thanks for in advance,
Sedat Duztas
Probil
I upgraded from SQL 6.5 to SQL 7 last month, and so far, everything's
been going fine.
However, I'm not using my old SQL 6.5 backup scripts, which, when the
backup was done, would dump the transaction log with TRUNCATE_ONLY, shrinking
the log size.
My SQL 7 server is set up with a Maintenance Plan which does everything,
including backup, but the log file seems to be growing and growing. I'm
up to 4.5 gigs now, for a database with a data file of 3.5 gigs.
How do I "dump transaction with TRUNCATE_ONLY" on a SQL 7 database?
Thanks,
Todd Wallace
I have merge replication setup up for 6 SQLCE Subscribers.
I have noticed that the MSmerge_tombstone table is growing
at a fast rate regardless of any changes to the data in
the database. It seems to be consistantly adding 50 rows
of data to the table every 2 minutes. As the table grows
it causes the SQLCE subscirbers to fail with the following
message:
ERROR: -2147467259
SQL Server Reconciler failed: Run
ERROR: -2147200925
: Failed to enumerate changes in the filtered
articles.
ERROR: 0
: {call sp_MSsetupbelongs
(?,?,?,?,?,0,?,?,1,?,?,?,?,?,?)}
ERROR: 0
: The merge process timed out while executing a
query. Reconfigure the QueryTimeout parameter and retry
the operation.
I'm sure that this is due to the size of the
MSmerge_tombstone.
Should the MSmerge_tombstone table grow at this rate?
36,000 rows every 24hrs!
I understand there is the sp_mergecleanupmetadata Stored
procedure but if i use this does that mean that because i
have to reinitialise all the subscribers, they are going
to have to pull down the whole subscription again.
I have since Changed a settings to make subscription
expiration date to 8 days instead of never expires but
we're still getting 50 rows added every 2 minutes
SQL SERVER 2000 SP3
Hope someone can shed some light on this for me.
Thanks.
.
I wonder if anyone could explain why when monitoring the transaction log size it doesn't appear to be growing! I'm using the following code to test image data types with logging.. I've got 'Truncate log on Checkpoint' switched off and 'Select into Bulk copy' also switched off.
Running the following code I would expect to see the transaction log grow and grow and grow... Monitoring it using perfmon indicates that it isn't in fact logging...
DECLARE @ptrval varbinary(16)
SELECT @ptrval = TEXTPTR(pr_info)
FROM pub_info pr INNER JOIN publishers p
ON p.pub_id = pr.pub_id
AND p.pub_name = 'New Moon Books'
declare @Loop int
select @Loop = 0
While @Loop <= 10000
BEGIN
WRITETEXT pub_info.pr_info @ptrval
with log
'New Moon Books (NMB) '
select @Loop = @Loop + 1
END
Hello,
Does 7.0 store db Growth rate information?
I am looking for information that tells me how fast a db is growing in MB and or percentages over a given period of time, ie weekly, monthly, yearly etc. Either in real numbers or estimates. Does 7.0 already store something like this or do I need to create some code for this?
Or does someone have something like this already coded that they would be willing to share?
Thank you in advance.
Troy
In SQL Server 7.0 sp1 (NT 4.0 sp5) I have a server that
has a tempdb database that continues to grow. This server
contains the database for SMS. Over the weekend, the tempdb
had grown so much that it filled up the drive (37GB). I have
shrunk it down to a much more reasonable size and put a limit
on how large it can grow. I'm noticing today that it is
beginning to grow again. Is there a way I can look at the
information that is in tempdb right now? I have to think
that there are open transactions for some reason that can't
commit. I know that tempdb gets cleared out when SQL Server
is restarted, but I can't be restarting it this often.
On Microsoft's website, I did find an article about SMS Y2K
queries using large amounts of Tempdb and failing to complete.
The solution they have in this article Q234912 is to install
SMS sp1 which is already installed.
I haven't been able to find any other useful information yet
on this problem. I would appreciate any help you can offer.
Thanks!
Toni
Greetings,
My database has a situation where my transaction log is growing out of control. However I have not been able to figure out where any memory leaks are occuring.
Is there a way to monitor the database in order to find out at when the tlog is growing. Or even better, what sql is being executed that is causing this unreasonable tlog growth?
Thanks a million,
Roger
Hi,
I'm getting this SQL server log...
The log file for database 'P5_Nextel' is full. Back up the transaction log for the database to free up some log space
What i'm doing is, i just resizing the space allocated, but the problem is my disk is now out of space. How can i prevent this kind of problem without adding a new disk? Is there any other way?
Please advise.
Thanks!
Why does a log (.ldf) file keep growing and growing and growing? Is this related to the fact that the scheduled Maintenance keeps failing due to exclusive access problems?
SQL Server 2000 Std.
Thanks!
Hi there,
My tempdb is growing from its normal size of 800MB to 2GB.
I've been shrinking it using dbcc shinkfile/ dbcc shrinkdatabase.
Everytime I run the command, immediately it says that the execution is completed and successful. However, when I checked the disk space, it remained the same, as though no shrinking is done. Can anyone help?
Also, it was published that restarting the SQL server can re-create tempdb from scratch. I've tried it too, the tempdb just wont go back to its normal allocation. My constraint is limited disk space, would appreciate any good samaritan to give me some help here.
Thanks in advance!
I am having problems with the transaction log growing. I set up a maintenance plan to backup every 15 minutes. The recovery model is set to full. It is set to auto grow by 10 percent with unrestricted file growth. The space allocated is 2mb. Should this be set higher? It seems whenever I unchecked the auto shrink, the log is growing larger. The command I use to check is:
SELECT *
FROM sysfiles
WHERE name LIKE '%LOG%'
GO
This is the log size and the log space used.
1.492187538.02356
This has grown 4 percent since yesterday. Are there any good practices to maintain these log files?Any help would be appreciated.
msr976
Hi,We have created a SQL server 2000 database. We observe that thetransaction log keeps growing over time. We are now about to run out ofspace. We have been periodically shrinking the database. Neverthelessthe size has increased. I would imagine that a transaction log can beeliminated if we stop the database. Can that be done? Is there a way tocompletely wipe off the transaction log?Thanks,Yash
View 4 Replies View RelatedHi,
my log files are growing like anything. One of my log file size is 20GB.
How i have to reduce the log file size.
If i run DBCC command is it come backs...
Pls tell me the way how i have to find the free space and reduce logsizes.
After taking backups also my log file sizes are not reducing.
Thanks!
Kavira
I have to admit, I'm usually using the MySql database, but in this particular case I have to use MSSQL 2000.
Over to my problem.
I'm building a web-based system (who isn't these days) in which the user types arbitrary information that is published when the user pushes the save button. Nothing new about that.
Here comes the tricky part, when the user wants to edit an existing item I copy all information in the database and sets the id of the 'edit-copy' to the negative value ( id 45 becomes id -45 for the edit-copy). This is also done on all items in other tables that is connected to the main item.
This way I get a copy that the user may edit without messing up the published information. When the user is done I either delete all the negative items (cancel) or delete the positive items and update the negative to become positive (save).
So far so good, allmost... my problem is that the transaction log grows tremendously.
Is there any other way to accomplish a safe edit that doesn't affect the transaction log as much as my current solution?
Could I be doing something wrong when updateing or deleteing my items?
I notice this morning that my tempdb grows very fast. I have 26GB in my
hardrive and all the space occupied by tempdb and finaly the qeury got failed due to 0 space in hardrive and there is no space to grow tempdb.
The select query supposed to bring about 40000 rows.
I ran this same query in different server that is not growing even 1 mb.
I checked the tempdb option the Trunc log on checkpoint is true.
Why this problem happening ?.
I have just dbo permission to access all the database.
Do you have any advice regarding this?.
Thanks,
Ravi
I have a database of 22 gb in sql 2000, my database option is set to full recovery mode, the problem i'm having is the tran log is growing too fast, this morning it was 24 gb, more than the database size. Can anyone help how I can keep it in a managable size?
Thanks in advance!!
Hi all
I have a DB with 1 data file, 1 log file and 1 index file.
data file is 3 GB but index file is 12 GB.
Index file is growing big day by day.
This cause performance of DB down.
What should I do to prevent index file become bigger and size of index file smaller?
Thanks in advanced
Thi Nguyen
My log file was 2x the size of my actual Database which is obviously too large on a DEV box. I know that my data can be easily recovered so I actually do not even want/need a log file.
After doing some investigation I found that I should turn my database into "Simple Recovery Mode" and after this I used a few scripts to truncate my log file. Things at this point looked great!
Unfortunately my log File is still growing even with this 'simple recovery mode'. So how do I stop this craziness from occurring?
I even unchecked the box 'allow autogrowth' on the database! However, I eventually get errors when creating records in the system because it complains about running out of room in the log file.
Code:
The transaction log for database 'ReportingDB' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases
The transaction log of a database grows until it runs out of disk space. If disk space is full, all databases on instance may get problems. Because of this i have set a limitation on how much it may grow, up to 40 GB. It grows in steps of 100 MB. It reaches its limit a couple of times a week, causing the application to hang.
The database file itself is about 2,3 GB large.
The SQL version is 10.50.4276, SQL2008R2 SP2.
The recovery model is FULL.
I have a backupjob that runs a FULL Backup at midnight.And a backupjob that runs a LOG backup every 30 minutes.Both finishes with success. However, the transaction log is never truncated, the unused space is never released.
I have checked for "long running jobs", it sometimes sys "backup_log", sometimes "active_transaction".
Could a workaround be to set the recovery mode to simple, and create a full-backup job that runs every 30 minutes for this database? It is a critical database....
Hi all.
I have a db application that has been running fine for months. Volumes have been gradually increasing and one day the system locked up.
A stored proc that typically ran in 3 or 4 minutes never returned. The tempdb kept expanding to fill available disk space (100GB). This was the offending statement inside the stored proc:
INSERT INTO cpp (CPPDate,MerchantLink,ReportNumber,FromDate,ToDate ,TransThreshold,DayThreshold,CPPType)
SELECT S.CPPDate,S.MerchantLink,s.ReportNumber, s.FromDate,s.ToDate, S.OccurThresh,s.DaysThresh,'D'
FROM #stuff S, Trans T with (nolock), Supplier P with (nolock)
where T.MerchantNumber in (SELECT MerchantNumber FROM Merchant WHERE MerchantLink = s.MerchantLink)
AND T.TranDate >= S.FromDate
AND T.TranDate <= S.ToDate
AND T.LoadDate <= @ReportDate2
AND (T.SupplierNumber = P.SupplierNumber
AND T.IncludeInCpp = 'Y'
AND P.CountryNumber IN (SELECT CountryNumber FROM REPORTCOMBO WHERE ReportNumber = s.ReportNumber))
GROUP BY CPPDate,Merchantlink,ReportNumber, FromDate,ToDate, OccurThresh, DaysThresh
HAVING COUNT(DISTINCT T.AccountNumber) >= OccurThresh
I realize that a "group by" uses the tempdb, but can't figure out why it would go away rather than returning an error.
I have a workaround in place now. I split this big query into several steps using a cursor. (slower and clumsier, but it works) Statistics are updated daily, i have tried defragging, and reindexing with no success.
Any thoughts would be appreciated. If you need any more details, please let me know.
Thanks in advance.
I have a sql2005 transactional replication replicating several databases between 4 servers, all of the replication are working fine except 1 that is going to a sql2000 sp4 enterprise database. The undistributed commands grows by several hundred thousand records a day, yet all of the records counts match exactly. At first this server was having a lot of lock issues, so i thought maybe that was the cause, but over the weekend i dropped replication and resnapshotted it. Without any activity ocurring on this database this weekend it continued to grow at the same rate
Hi there,
We have a Principle, Mirror and Witness set-up and all is working fine, however, the transaction logs for a few large databases just keep growing over a the course of the month until the disk is full. As I understand it, and having tried you can't dump the transaction logs while mirroring is configured, is there any way at all to commit and truncate the logs while mirroring is running or do I have to manually remove the mirroring each month, dump the transaction logs and then re-enable it again after doing the backup/restore?
The databases in question are about 6GB in data size and the transaction logs can grow to be about 60GB in a month.
Would a normal SQL Server 2005 backup truncate the logs if I configured this? At the moment we use Litespeed for SQL server for nightly backups.
Any advice would be very helpful.
Thanks
Ed
I am showing a Multi Value parameter on a Report, in a textbox that has CanGrow = True.
This is fine when there are only a few value in the parameter, but when many values have been selected, the fact that the textbox grows automatically to accommodate the contents means that this textbox can become quite large.
I am using the value of:
Join(Parameters!ParameterName.Label,", ")
Is there any way to either limit the number of values of a parameter when displaying in this way, or of setting a maximum size for an automatically grow textbox?
Or anything else that would solve this issue?
Thank you very much in advance!