Huge Performance Issue

Jan 23, 2007

I have a medical DB with the loads 150,000 transactions per month. Each month, I load the tranactions into a table for the current year. I also have to update records for prior months based on current month information.

For example, out of 186,000 dump records...150,000 will be loaded into the main table and 36,000 will be used to update records already loaded into the main table.

The tables have 90 columns, I have a clustered PK using [Soc_Sec_Number] & [Month] & [Row Index]. I need the row index counter (like auto number in MS Access) because I can have multiple transactions per month for the same Soc Sec Number.

===========================================================

My steps are

1) Load 150,000 records into main table (For december, this makes the table have 1,800,000 rows
2) Run queries for the remaining 36,000 rows to update records already loaded into the table containing the 1,800,000 rows.
3) The 36,000 queries have to be splits depending upon the update type code, So I am actually running 6 queries using 6,000 rows each against the 1,800,000 records.

The update queries are using inner joins with [Soc Sec #] and [Date], part of my composite Primary Key on both tables.

=============================================================
Problem

This process takes forever, about 4 hours per monthly update. As the months go out, the main table gets larger and the time increases. It took almost 24 hrs to get from January 2004 to June 2004.

I am running Sql Server on my PC, no seperate workstation. My PC has 2.8 GHZ with about 1 Gig in RAM. Could my PC specs be too low. I noticed that the task mamager shows sqlservr.exe using over 657,000 mb of RAM when running.

I also ran a simple Select MAX(Soc_Sec_Number) query that took over 5 minutes. This is way too long especially since Soc_sec_Number is part of the composite PK.

Could my queries actually take that long or are my pC specs too low. MY PC seemed to freeze after the JUNE update? Any help appreciated.

View 17 Replies


ADVERTISEMENT

Performance Issue With Huge Data

Mar 20, 2007

Hello All,

I am using SSIS to transfer data between two SQL Servers (2000). There is no transformation involved as the source and destination table structure is same. Even then the package execution takes lot of time.

The data in the tables is of the order of 66000000 the we were required to kill the package execution after it took more than 24 hours. The CPU usage was more than 13000s and disk I/O was well above 330000000. I am new to the tit-bits of SIS. Can anyone please tell me the reason as to why the package has gone so resource hungry.



Thanks in advance,

Atul

View 3 Replies View Related

Huge Performance Loss When Using Variables

Oct 26, 2007

was trying to performance tune a query and came up on this weird issue(Weird to me atleast.. possibly because I am a newbie:-))).

When I run the query mentioned below, I get the results in 1 second, while the same query, if I declare a variable for the value I am comparing, it takes 40 seconds. I am on SQL Server 2000. Both queries are being run in the same DB on the same tables and the only difference between the two is that the date has been declared as a variable. I am able to consistantly reproduce this. Tried casting the variable, declaring it as varchar , replacing the date by getdate() at both places, but the moment I use the variable, the performance goes for a toss. Has anyone came across any similar issues?
TIA
Callista

Query 1
select ae.Col1, ae.Col2, ae.Col3, ae.Col4, ge.Col4, c.Col5, 0
from table1 ae WITH (INDEX=ind_Col3 NOLOCK), table2 ge with (nolock), table3 c with (nolock)
where ae.Col1 = ge.Col1
AND ae.Col4 = c.Col1
AND ae.Col3 > '2007-10-25 14:18:12.380'
and ae.Col6 is not null

Query 2
declare @MyVariable datetimeselect @MyVariable = '2007-10-25 14:18:12.380'
select ae.Col1, ae.Col2, ae.Col3, ae.Col4, ge.Col4, c.Col5, 0
from table1 ae WITH (INDEX=ind_Col3 NOLOCK), table2 ge with (nolock), table3 c with (nolock)
where ae.Col1 = ge.Col1
AND ae.Col4 = c.Col1
AND ae.Col3 > @MyVariable
and ae.Col6 is not null

View 5 Replies View Related

Huge Performance Difference When Running UDF In Workstation Vs Server

Dec 13, 2007

Hi,

I created a CLR UDF that returns a large number of rows, when I run it from my VPC (XP, SQL Server Developer Edition and 1GB Memory) it takes approx 2 min and 30 secs to start displaying the rows (Using Management Studio), when I run the same query in our development server (Win 2003, SQL Server Enterprise Edition, 8 GB Memory and 8 Processors) it takes more than 15 min to start displaying the results, does anybody have an idea why is this happening?

Thanks in advance

View 2 Replies View Related

Transact SQL :: Huge Performance Difference For Same Select Between Environments

Jun 22, 2015

I have encountered a problem with a specific set of tables. The same select yields slightly differing execution plans in two different environments (instances). But the slight variation seems to contain a huge differences in stats. I don't know the significance of these stats. The two tables have the exact same indices.

This is the selcet statement:

SELECT 'xx' FROM DUKS.dbo.Profiler
WHERE DNA_Løbenummer IN
(SELECT DNA_Løbenummer FROM DUKS.dbo.Effektregister
WHERE Sagsnummer = '2015-00002')

View 17 Replies View Related

Huge Deletes In A Huge Table

Apr 3, 2000

SQL 7 SP1 NT4 SP5

I have a TRANSACTION table with 150 million rows.

I have a USER table.

Each user has about 600 records in the TRANSACTION table.

The TRANSACTION cluster index is on USERID + RECID . The second index is on USERID + Fieldx + Fieldy.

The TRANSACTION table gets about 1.4 million inserts in a normal day and about 40,000 updates.

I want to go through the USER table and delete all users who have not visited me in a while.

I want to do this without substantially hindering performance in a production environment. I can perform this over a week period or two if needed.

The best way I thought of doing this was to grab x amount of users in a cursor and loop through deleting their corresponding TRANSACTION records.

Does anyone have any ideas on a better way. What is going to happen to my indices during this time ?

Thanks !!!

View 3 Replies View Related

Huge Log

May 27, 2007

Dear all,



i have problems with log .

the mssql write 4 G log so plz how can i Eliminate log huge size

View 1 Replies View Related

Huge Log Files

May 9, 2008

 I have SQL Server 2005 Express Edition with Advanced Services running on a small web server. It all runs fine, but every now and then the log files grow and grow and eventually use up all the disk space of 30GB. As a quick fix, restarting SQL a couple of times clears out the logs and everything is up and running again. Any ideas on how to stop this happening?

View 5 Replies View Related

Huge Transaction Log - Do I Need This?

Nov 19, 2004

I just peeked at my DNN setup and I found that I have a transaction log about 98 gigs large, compared to a DNN database that is only about 250 megs. Crazy, huh?

Do you happen to know what I need that transaction log for? Can I just delete it or will it break my SQL db? Is there a way that I can keep only maybe a week of transactions in it so it doesn't grow so dang large?

Thanks in advance for your response!!

Tim x 4
(always learning!)

View 2 Replies View Related

Huge Transaction Log

Aug 10, 2000

One of my production databases is currently 51 mb. The transaction log is well over 5 gig. I have tried truncating and then shrinking the log through the use of SQL utilities. This does not work! How can I quickly resolve this problem without tampering with the production environment?

Thanks in advance.
Ray Reinders

View 1 Replies View Related

Huge Transaction Log

Sep 30, 2002

I am not a DBA and I run a personal web site that has gotten pretty large. I have never done anything to maintain my sql server, and now my transaction log is 10 Gigs and my data is only like 300 Megs. I am starting to get a memory leak with the sql service. What should I do? Is it bad to have a huge transaction log. I am not familiar with any of this stuff, so someone please point me in the right direction.

View 3 Replies View Related

What Should I Do With The Huge Log File

Oct 11, 2004

Hi all,
I found my database log file is 26GB and the database file is just about 280MB. We are doing full backup everyday. However, my sql server seems running very slow now and please advise:

1. How can I decrease/truncate my log file?
2. Would the huge size of the log file be the reasons slowing up my sql server?
3. Would anyone give me direction knowing more on the transaction log?
Thank you and appreciated!

View 4 Replies View Related

Log File HUGE!!

Oct 30, 2007

Hi guys, its my first post! Its also like my first time really diving into sql. We are using sharepoint on site here along with sql server 2005, one of our log files is 255 GBs and needs to be made smaller very fast!! We are almost out of disk space and the log is growing fast.

I am very new to sql and dont even know where to go to enter commands, so youll have to bear with me here. I've read about truncating and shrinking and some other things, I am just worried and dont want to mess anything up. I know this is probably a simple task, but like I said, with the truncate command I was reading about, I dont even know where to go to type it in!!! If someone could please help it would be much appreciated. Thanks so much.

View 14 Replies View Related

Huge Dataset, Only Need A Little

Dec 7, 2007

Hello, newbie here...

I have a huge dataset in MS SQL Server, over 11 million records of machine data taken every four seconds. I really don't need samples of this data at this interval. I'd like to run a query to retrieve my data for every 30 minute interval. I know enough SQL and DTS to SELECT was fields I want, and how to direct it to a CSV file, but I'm not sure how to manipulate the TimeStamp field in the query to only pull data every 450 records.

TIA

View 20 Replies View Related

Huge Issue - Well For Me Anyway

Dec 26, 2007

Hey everyone.

I have a problem and I am sure someone here can help. Lat night my DB was working fine, as it has been . This morning I get to the office and now everything has gone to hell in a handbag .

I can no longer connect to my sql2005 DB I get this error when trying to place an order on our order page.

"There was a problem with the website:An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. (provider: TCP Provider, error: 0 - No connection could be made because the target machine actively refused it.)
The error has been logged."

I also notice the sql agent will not start. Any the former owner of the Co. had a eval version of sql management studio the must have just expired(could this be causing it?)

Any help anyone can offer would be great.

View 12 Replies View Related

Huge .BAK FILE

Jan 28, 2008

I have a .bak file of 72gb. But my database size is only 32gb, I got this value from sp_spaceused?
Anyone know why the .bak file is so big?. Is it possible to reduce the size? How could i reduce it?



http://www.sqlserverstudy.com

View 9 Replies View Related

Huge T-SQL - Out Of Memory

Mar 25, 2008

Hello,
I have a very big T-SQL script (~24mb) and I need the SQL Server on my hosted site to execute it.

Right now, I have a web page that uses Sql.Connection.ExecuteNonQuery () to do it, but the .NET process runs out of memory when I load the file into a C# string.

How would I go about executing this T-SQL script on the server?
Is there such a command:
EXECUTE SCRIPT "myscript.sql" FROM DISC
?

thanks.

View 1 Replies View Related

Huge LDF File

Nov 13, 2007



Hi
this is regarding SQL Sever 2000. ( it was upgraded form sql7). its log file is increasing in very high manner. say 40 gb, 50 gb and now 57 gb. Mdf file is around 15 mb. we created back up and tried to restore to another system. its asking 57 gb free space. how to proceeed with file recovery. we have backups but it askes more space for log file. how to retrieve the data.
rgds
Pramod

View 3 Replies View Related

Huge SQL Queries In SQL 2005

Mar 20, 2008

Good evening:
 We're porting an old app written in ASP.NET  1.1433 and SQL 2000 to ASP.NET 2.0 and SQL 2005.  In the old app we have a few data grids that are populated from a dataset pulled from the database.  We use a SQL query that we build based on more than 10 different user inputs, the result of which is an enormously complicated SQL string.  We'd like to move this processing into a SPROC in the 2005 database.
Rather than writing stored procedures to create the SQL SELECT statement, is it possible to pass an entire select state ment to a SPROC and have it executed within?  We're trying to capitalize on paging in 2005 using ...ROW_NUMBER() OVER (ORDER BY PM ASC)... and building the string using IF ELSE statements is mind numbling complex and tedious.
And suggestions woudl be great.
Thanks,
Brad

View 4 Replies View Related

SQL Backup File Is HUGE!!!

Sep 25, 2004

One of my databases is approxiamtely 1.5 GB's but when I run a backup the backup file explodes to 38GB's.

What is the proper usage of Shrink Database, is this safe or is there another method to reduce the size?

View 2 Replies View Related

The Log File For My Database Is Huge

May 20, 2006

The log file for my database in SQL Server 2005 is huge. How do I get empty it or in effect shrink it or start it over?Thanks

View 1 Replies View Related

Problem On Huge Table.

Mar 14, 2001

We have a huge table which has 12 million records. And when I run the following script, it took 50 hours. Is there anyone who can help? Thanks.

update TableA
set In=e.In, EA=e.ea, We=e.we
from TableA c, TableB e
where c.code=e.code

TableA 12,000,000 records.
TableB 750,000 records.
And have clustered index on each code field.

View 6 Replies View Related

Alter A Huge Table

May 18, 2001

I need to alter a table (expand the column size for varchar(10) to varchar(255)) and the table has 200 million rows.
Please suggest me the best and the fastest method to achieve it. The database is on SQL 7.0


Thanks

View 1 Replies View Related

Huge Backup File

May 12, 2000

I am using SQL Server 7 and have about 5 databases. One of them has a data file of about 10 Meg, and most of the others are larger. I do a nightly backup to both a local and mapped drive. On both, the size of the backup file for this database is more than 500 Meg, but the rest appear to be an appropriate size. Does anyone know why this would be happening? The database works fine, it does not get a lot of insert/delete activity and I run DBCC every weekend. If anyone has any ideas I would sure like to hear from them.

View 1 Replies View Related

Huge Memory Usage With BCP?

Oct 17, 2000

Does anybody know why BCP on v6.5 grabs so much memory for SQL Server? I have a few table imports where the BCP process will consume over 460MB of RAM during the imports.

The BCP cmd file is executed via an xp_cmdshell call. The server has 2+GB of RAM, but the BCP process effectively flushes large amounts of data from the buffer. It takes quite along time for the cache to recover from this, and after this, the rest of the nightly processes run much slower, as they end up having to hit the drives to retrieve information that should already be in cache.

If anyone can shed some light on this it would be much appreciated.

Ian Dundas
DBA
Assante Asset Management

View 2 Replies View Related

Huge Select Into.. (Can I Choose Not To Log?)

Dec 12, 2000

I have a job that selects alot of data from one database into another.

Can I choose not to log this operation (doesn't need to be and the log fills up before it's done)

Thanks..

type of code:

Insert into database_1
Select * from database_2

View 1 Replies View Related

Huge Log File On SQL Server 7

Feb 21, 2001

I have a huge log file (285M) on SQL Server 7.
The database itself is about 10M.
How can I reduce the log file ?
Is it possible to build it again from scratch ?

I tried the Truncate Transaction Log but it didn't help.

View 2 Replies View Related

What Is The Best Way To Upload Huge File Into SQL 2K

Mar 13, 2002

I have a 2 GB text file(semicolon delimited), which I need to
pump into SQL 2K. What is the best way to achieve this in shortest time?

I tried with DTS BCP, it tooks 1 minute 14 secs to transfer 13457 records. This is only 1/2000 of the record i need to transfer. Please help.

Thanks.

regards,
Terry

View 2 Replies View Related

Transaction Logs Are Huge

Mar 19, 2008

Hi all our It Admin is having issues with the backups. He was doing full backups every 4 hours with backupExec, which means thats way too much. But now hes trying to do a simple recovery now, because obviously the transaction logs have not been truncated. Its a big mess, I'm not involved in this part, they handle the backing up and permissions. Transaction logs are huge??

Any suggestions??

View 12 Replies View Related

How To Compare Two Huge Tables

Aug 15, 2012

I need to compare if two developers did the job correctly and created identical tables.

The problem is more complex, but I will try to solve it somehow if I solve the problem of comparing two tables (let them be in different SQL Server 2008 databases) and their properties. No data needs to be compared.

View 6 Replies View Related

Import Of Huge XML File

May 22, 2013

I have an xml file of 44 Gb (Not Meg, its really GB) . Delivered by the Danish custom authorities.

My problem is simple - How to import such a beast? I have seen a limit of 2.1 Gb everywhere.

View 9 Replies View Related

Huge Table Backup

May 13, 2008

Hello
i want to ask about the huge table(table with many tera records) backup time cost , any one can help me please in determining the time cost nearly

View 2 Replies View Related

Delete Huge Log File

Feb 15, 2006

how to delete a very huge log file, to free up some harddisk space

View 3 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved