I'm Flooding My SQL Server With INSERT / UPDATE Requests. How Do I Optimize?
Sep 29, 2005
I have an application that calculates a bunch of numbers and then inserts them into a table (or updates a record in the table if it exists). In my test environment it is issuing 100 insert or update requests to the server per second and it could run for several hours.After a the first several hundred requests, the SQL server is bogging down (processor at 90-100%) and the application slows down while it waits on SQL to update the database.What would be the best way to optimize this app? Would it help to loop through all my insert/update requests and then send them as one big batch of statements to the server (per 1000 requests or something)? Is there a better way of doing this?Thanks!
View 1 Replies
ADVERTISEMENT
Feb 2, 2007
Hello, I have a bit of a problem. Some of my pages has way to many database requests, some pages upto 70 requests,and the loading time can be all the way up to 12seconds because of the many requests. So i am trying to optimize it abit.What i want to know if how to send several ID's to the database and collect the result from all of the ID's, i have a stored procedure like this:
public DataTable User_GetUsernameTypeTwo(string strUsername){OpenDBCon();mCom.Parameters.Clear();mCom.CommandText = "Profile_GetUsernameTypeTwo";mCom.Parameters.Add(new SqlParameter("@UserName", SqlDbType.NVarChar, 20));mCom.Parameters["@UserName"].Value = strUsername;myAdap = new SqlDataAdapter();myTable = new DataTable();myAdap.SelectCommand = mCom;myAdap.Fill(myTable);CloseDBCon();return myTable;}
To this i send a username and in return i get info about the user that then is presented on the page. If i have a gridview on the page with 20 posts, then it needs to make 20 requests to this file in order to collect all the info for all the users. Then i have one just like the one above that collects the photos of all the users. So my first thought was that i merge these two together, witch works perfect, reduces the database calls by around 40%. Then i thought, perhaps can i get all the info from all the users in 1 request, to reduce the load on the database even more, my problem is: How can i rewrite the one above to accept lets say 50 usernames (1 username can max have 20characters), and return the info from all of them. Since i use a stored procedure i cant with ease use WHILE (username = xxx) AND (username = yyy) AND (username = ccc), then i would have to add 50 Parameters and add each username to a seperate parameter and then look if the parameter is null or if it contains a username. So i need to find another way to send the usernames, so the sqlserver can add them to the where clause, perhpas using IN?
This is the sql server store procedure (it works perfect, when its one user i am requesting info about), i am using FULL OUTER JOIN because ALL fields i request can be NULL:
ALTER PROCEDURE [dbo].[Profile_GetUserpopupInfo] -- Add the parameters for the stored procedure here@UserName nvarchar(20)ASBEGIN-- SET NOCOUNT ON added to prevent extra result sets from-- interfering with SELECT statements.SET NOCOUNT ON;SELECT settings_username.color, settings_username.pic1, settings_username.pic2, profile_profilephoto.imageurl, profile_profilephoto.alttext FROM settings_username FULL OUTER JOIN profile_profilephoto ON profile_profilephoto.username = settings_username.username AND (profile_profilephoto.approved = 1) WHERE (settings_username.username = @UserName)END
Anyone got any idea of how to send a bunch of usernames to this stored proc? (it must be efficient since there is alot of users)
Patrick
View 1 Replies
View Related
Nov 27, 2006
Hi,
I'm using access tables to store my data. to retrive data I use jet odbc engine in c++.
I want to insert a large amuont of records (about 20 mega records) to my database in the fasts way i can. if i use sql syntax (insert into table ()....) it's takes for ages (about 500 records per second).
if i'm writing a csv file and then use import (via access) it's much faster but here I have two problems
1.I dont know how to use the access import tool from c++.
2.I dont think I can distributie the access import tool with my product.
so my questions are :
1. Does any know any tool that insert records in an optimize way?
2. How can I use the access import tool in c++?
3. Can i use the jet engine to import csv files?
thank ishay
View 3 Replies
View Related
Apr 16, 2015
If I have a table with 1 or more Nullable fields and I want to make sure that when an INSERT or UPDATE occurs and one or more of these fields are left to NULL either explicitly or implicitly is there I can set these to non-null values without interfering with the INSERT or UPDATE in as far as the other fields in the table?
EXAMPLE:
CREATE TABLE dbo.MYTABLE(
ID NUMERIC(18,0) IDENTITY(1,1) NOT NULL,
FirstName VARCHAR(50) NULL,
LastName VARCHAR(50) NULL,
[Code] ....
If an INSERT looks like any of the following what can I do to change the NULL being assigned to DateAdded to a real date, preferable the value of GetDate() at the time of the insert? I've heard of INSTEAD of Triggers but I'm not trying tto over rise the entire INSERT or update just the on (maybe 2) fields that are being left as null or explicitly set to null. The same would apply for any UPDATE where DateModified is not specified or explicitly set to NULL. I would want to change it so that DateModified is not null on any UPDATE.
INSERT INTO dbo.MYTABLE( FirstName, LastName, DateAdded)
VALUES('John','Smith',NULL)
INSERT INTO dbo.MYTABLE( FirstName, LastName)
VALUES('John','Smith')
INSERT INTO dbo.MYTABLE( FirstName, LastName, DateAdded)
SELECT FirstName, LastName, NULL
FROM MYOTHERTABLE
View 9 Replies
View Related
Feb 14, 2006
Hello,
I have been working as Sybase DBA for 5+ years, and I would very much like to add MS SQL Server to my resume. Given the common roots of the two RDBMS, it seems that the learning curve would not be as sharp as if I were going to learn Oracle or DB2. Can anyone out there know of any books that are geared toward learning MS SQL Server from a Sybase DBAs perspective?
Thank you,
- Chris Merz
View 3 Replies
View Related
Jun 5, 2007
Hi,
I want to know how to assign a normal user to take backup,Restore and attach and detach the database in sqlserver2000.
Please can anyone help me to find out which server role or database role will suites my requirements.
Regards
Md S.Hassan
View 1 Replies
View Related
Sep 13, 2007
Hello,
I see the following error on the sql server log of one of our sql servers running sql 2000 with sp4.
SQL Server has encountered 1964 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [h: empdb.mdf] in database [tempdb] (2). The OS file handle is 0x00000534. The offset of the latest long IO is: 0x0000002b09e000
Any idea as to what might be causing this error. Appreciate any comments.
Thanks!
View 3 Replies
View Related
Jan 15, 2008
Hi,I am looking for websites/forum threads/replies about SQL Server Optimization. Does anyone have any good links that can help me?I am Googling the net but thought I might as well ask someone on this forum since I am sure I am not the first one looking for this information.Thanks in advance
View 14 Replies
View Related
Apr 9, 2001
we are running SQL7.0. I execute a store procedure on Server A that insert a data into the table on Server B. I use a Link Server to connect from Server A to Server B. When I execute the SP, I got the messege 'The command(s) completed successfully'. When I select from the table was inserted, I don't see the data that was inserted. Can we update or insert tabel using Link Server.
Thanks.
View 1 Replies
View Related
Feb 6, 2007
HelloI used to work in a Sybase database environment. When I had to insert/update records in the database, I always used "insert on existingupdate", in this way, you didn't have to check whether a recordalready existed (avoid errors) and you were always sure that afterrunning the scripts, the last version was in the database.Now I'm looking for the same functionality in MS SQL Server, asked afew people, but nobody knows about such an option.Does anybody here knows the SQL Server counterpart of "insert onexisting skip/update"? If this doesn't exist, this is a minus forMS ;).Greetz,Bart
View 4 Replies
View Related
Dec 27, 1999
How can you optimize Access queries on linked SQL Server tables?
View 1 Replies
View Related
Sep 4, 2006
I designed a database for daily reporting system. It involves mainly INSERT operations and data retrieval. The problem here is the size of database is growing rapidly beyond the estimation. I changed the datatypes from int to tinyint, even after that i didn't find any difference in size of the database. Please suggest me solution by which i can optimize the database size also suggest me some tips by which i can estimate the database size. Please respond at the earliest possible.
My mail ID : akbar.t@tcs.com
Thanks in advance.
View 14 Replies
View Related
Jun 22, 2006
Hi,
How can i store the record insert/update timestamp in a SQL server 2000 db programacally. ? what are the date/time functions in ASP.NET 2.0 ? I know that this can be done by setting the default valut to getdate() function in SQL, but any other way on ASP page or code-behind page ???
Thanks,
Alex
View 3 Replies
View Related
Nov 27, 2006
I have current current sql server 2000 database containing some columns in big5. To display these cols correctly, my asp.net nust have directive with CodePage="1252" ContentType="text/html;charset=BIG5". I can not update, or insert big5 character into these columns via .aspx page. I'm using .net framework 2.0. Please help me, thanks a lot for any help.
View 7 Replies
View Related
Apr 9, 2015
I am building a table change log that will track each attribute update and include the original and new values.
[BatchYearMonthKey] [int] NULL,
[BatchYearMonthDayKey] [int] NULL,
[AccountID] [varchar](200) NULL,
[Attribute] [varchar] (200) NULL,
[Old_ValueAtrDefault] [varchar] (200) NULL,
[New_ValueAtrDefault] [varchar] (200) NULL,
[Old_ValueAtrLong] [varchar] (max) NULL,
[New_ValueAtrLong] [varchar] (max) NULL
The challenge that the spectrum of varchar lengths across the table. I have one attribute that requires varchar(max) and all other attributes (about 40) are varchar (200).
I am trying to accomplish the following:
Account ID Status
1 Enabled
Now changed to
AccountID Status
1 Disabled
My log table will look like the following:
[BatchYearMonthKey] BatchYearMonthDayKey] [AccountID] [Attribute] [Old_ValueAtrDefault] [New_ValueAtrDefault] [Old_ValueAtrLong] [New_ValueAtrLong]
201504 20150409 1 Status Enabled Disabled NULL NULL
My question:
I created two fields (Old_ValueAtrLong and New_ValueAtrLong) dedicated for the one attribute that is a varchar (max). I was trying to avoid storing [Status] for example that's a varchar(200) in a field that is varchar(max). Is this the right approach? Or are there other recommendations in how to handle storing the data in the most efficient manner?
View 9 Replies
View Related
Aug 26, 2015
I have one view which is based on couple of tables. Here is the definition of view. Which are the options i can use to optimize the view for better performance. This is one of the view which causing issue on database.
CREATE VIEW [dbo].[V_Reqs]
WITH SCHEMABINDING
AS
SELECT purchase.Req.RequisitionID, purchase.Req.StatusCode AS Expr2, purchase.Req.CollectionDateTime,
purchase.Req.ReportDateTime, purchase.Req.ReceivedDateTime, purchase.Req.PatientName, purchase.Req.AddressOne,
purchase.Req.AddressTwo, purchase.Req.City, purchase.Req.PostalCode, purchase.Req.PhoneNumber,
[code]....
View 3 Replies
View Related
Jan 8, 2006
I saw this tool for SQL-Server 2000 :
http://www.sql-server-performance.com/index_tuning_wizard_tips.asp
Is there anything similar for SQL-Server 2005 Express ?
Thank you very much for any help!
Regards,
Fabian
my favorit hoster is ASPnix : www.aspnix.com !
View 2 Replies
View Related
Mar 13, 2007
All:
I have created a stored procedure on SQL server that does an Insert else Update to a table. The SP starts be doing "IF NOT EXISTS" check at the top to determine if it should be an insert or an update.
When i run the stored procedure directly on SQL server (Query Analyzer) it works fine. It updates when I pass in an existing ID#, and does an insert when I pass in a NULL to the ID#.
When i run the exact same logic from my aspx.vb code it keeps inserting the data everytime! I have debugged the code several times and all the parameters are getting passed in as they should be? Can anyone help, or have any ideas what could be happening?
Here is the basic shell of my SP:
CREATE PROCEDURE [dbo].[spHeader_InsertUpdate]
@FID int = null OUTPUT,@FLD1 varchar(50),@FLD2 smalldatetime,@FLD3 smalldatetime,@FLD4 smalldatetime
AS
Declare @rtncode int
IF NOT EXISTS(select * from HeaderTable where FormID=@FID)
Begin begin transaction
--Insert record Insert into HeaderTable (FLD1, FLD2, FLD3, FLD4) Values (@FLD1, @FLD2, @FLD3,@FLD4) SET @FID = SCOPE_IDENTITY(); --Check for error if @@error <> 0 begin rollback transaction select @rtncode = 0 return @rtncode end else begin commit transaction select @rtncode = 1 return @rtncode end endELSE
Begin begin transaction
--Update record Update HeaderTable SET FLD2=@FLD2, FLD3=@FLD3, FLD4=@FLD4 where FormID=@FID;
--Check for error if @@error <> 0 begin rollback transaction select @rtncode = 0 return @rtncode end else begin commit transaction select @rtncode = 2 return @rtncode end
End---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Thanks,
Blue.
View 5 Replies
View Related
Mar 15, 2013
I am experimenting with using CDC to track user changes in our application database. So far I've done the following:
-- ENABLE CDC ON DV_WRP_TEST
USE dv_wrp_test
GO
EXEC sys.sp_cdc_enable_db
GO
-- ENABLE CDC TRACKING ON THE AVA TABLE IN DV_WRP_TEST
USE dv_wrp_test
[Code] ....
The results shown above are what I expect to see. My problem occurs when I use our application to update the same column in the same table. The vb.net application passes a Table Valued Parameter to a stored procedure which updates the table. Below is the creation script for the stored proc:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
if exists (select * from sysobjects where id = object_id('dbo.spdv_AVAUpdate') and sysstat & 0xf = 4)
drop procedure dbo.spdv_AVAUpdate
[Code] ....
When I look at the results of CDC, instead of operations 3 and 4, I see 1 (DELETE) and 2 (INSERT) for the change that was initiated from the stored procedure:
-- GET CDC RESULTS FOR CHANGES TO AVA TABLE
USE dv_wrp_test
GO
SELECT *
FROM cdc.dbo_AVA_CT
GO
--RESULTS SHOW OPERATION 1 (DELETE) AND 2 (INSERT) INSTEAD OF 3 AND 4
--__$start_lsn__$end_lsn__$seqval__$operation__$update_maskAvaKeyAvaDescAvaArrKeyAvaSAPAppellationID
--0x0031E84F000000740008NULL0x0031E84F00000074000230x02119Test26NULL
--0x0031E84F000000740008NULL0x0031E84F00000074000240x02119Test36NULL
--0x0031E84F00000098000ANULL0x0031E84F00000098000310x0F119Test36NULL
--0x0031E84F00000098000ANULL0x0031E84F00000098000420x0F119Test46NULL
Why this might be happening, and if so, what can be done to correct it? Also, is there any way to get the user id associated with the CDC?
View 7 Replies
View Related
Jul 17, 2015
I have Table Staffsubjects
with columns and Values
Guid AcademyId StaffId ClassId SegmentId SubjectId Status
1 500 101 007 101 555 1
2 500 101 007 101 201 0
3 500 22 008 105 555 1
I need to do 3 scenarios in this table.
1.First i need to update the row if the status column is 0 to 1
2.Need to insert the row IF SegmentId=@SegmentId and SubjectId<>@SubjectId and StaffId=@StaffId
3.Need to insert the row IF StaffId<>@StaffId And ClassId=@ClassId and SegmentId<>@SegmentId and SubjectId<>@SubjectId
I have wrote the stored procedure to do this. But the problem is If do the update. It is reflecting in the database by changing 0 to 1. But it shows error like cannot insert the duplicate
Here is the stored Procedure what i have wrote
ALTER PROCEDURE [dbo].[InsertAssignTeacherToSubjects]
@AcademyId uniqueidentifier,
@StaffId uniqueidentifier,
@ClassId uniqueidentifier,
@SegmentId uniqueidentifier,
@SubjectId uniqueidentifier
[Code] ....
View 8 Replies
View Related
Nov 2, 2015
If a column is set to allow nulls I know that a constraint can be used to supply a default (i.e. GetDate() ) when no value is provided but what about when an explicit NULL is provided in an INSERT or UPDATE statement?Is there any way other then an AFTER trigger to substitute a value for an explicitly provided NULL? In other words, assuming that dtAsof is a NULL enabled column, is there any way to over ride what the following will do to MYTABLE:
INSERT MYTABLE(sCol1, sCol2, sCol3, dtAsOf)
SELECT 'a', 'b', 'c', NULL
If there's no way to do this in SQL Server 2008R2 then what about later versions of SQL Server? Do any more recent versions have a way to deal with this? We have a third party app that uses a SQL Server back end and many of the tables have columns for storing audit like data such as date/time but many are left to NULL values and I'd really like to fix that in as passive a way as possible so as to not break the app that uses the database. I know a constraint with a default can be sued to over ride a null but not when a null is explicitly provided.
View 1 Replies
View Related
Nov 14, 2007
I have a web form with a text field that needs to take in as much as the user decides to type and insert it into an nvarchar(max) field in the database behind. I've tried using the new .write() method in my update statement, but it cuts off the text after a while. Is there a way to insert/update in SQL 2005 this without resorting to Bulk Insert? It bloats the transaction log and turning the logging off requires a call to sp_dboptions (or a straight-up ALTER DATABASE), which I'd like to avoid if I can.
View 6 Replies
View Related
Mar 28, 2015
Our system runs a SQL Server 2012 DB, it has a table (table_a) which has over 10M records. Our system have to receive data file from previous system daily which contains approximate 3M updated or new records for table_a. My job is to update table_a with the new data.
The initial solution is:
1 Create a table (table_b) which structur is as the same as table_a
2 Use BCP to import updated records into table_b
3 Remove outdated data in table_a:
delete from table_a inner join table_b on table_a.key_fileds = table_b.key_fields
4 Append updated or new data into table_a:
insert into table_a select * from table_b
As the test result, this solution is very inefficient. Step 3 costs several hours, e.g. How can I improve it?
View 9 Replies
View Related
May 8, 2015
i would like to know it's possible to find all transaction(insert, delete,update) on a database for a day. if yes what can i do.
View 2 Replies
View Related
Jan 24, 2014
Reformatting data in a PIVOT Table or find a better way to display.
--ORDERDETAIL TABLE
SKUO QTYO ORDERIDO
KUM 1 12345
KUS 2 12345
SUK 1 12345
KHN 4 12345
DRE 1 12345
[Code] ....
Number of SKU's in order could be over 1000.
Looking to change my current pivot table to allow an unlimited number of SKU's and add QTY.
Data I am looking to get. MAX of 15 SKUS Per line.
ORDERID SKU1 QTY1 SKU2 QTY2 SKU3 QTY3 SKU4 QTY4 SKU5 QTY5 SKU6 QTY6 SKU7 QTY7 SKU8
QTY8 SKU9 QTY9 SKU10 QTY10 SKU11 QTY11 SKU12 QTY12 SKU13 QTY13 SKU14 QTY14 SKU15 QTY15
12345 KUM 1 KUS 2 SUK 1 KHN 4 DRE 1 HGF 2 FDE 1 CDS 1 GYT 1 POI 3 LKH 2 TTT 4 JHG 8 YUI 2 WQE 1
12345 PMN 1 BVC 1 ABD 1
[Code] ....
CURRENT PIVOT ONLY GOES TO 150 - BELOW
SELECT PKGCUSTOM4, [1] AS [SKU1], [2] AS [SKU2], [3] AS [SKU3], [4] AS [SKU4], [5] AS [SKU5], [6] AS [SKU6], [7] AS [SKU7], [8] AS [SKU8], [9] AS [SKU9], [10] AS [SKU10],
[11] AS [SKU11], [12] AS [SKU12], [13] AS [SKU13], [14] AS [SKU14], [15] AS [SKU15], [16] AS [SKU16], [17] AS [SKU17], [18] AS [SKU18], [19] AS [SKU19],
[Code] ....
View 6 Replies
View Related
Apr 4, 2015
I am inserting updating few tables from snapshot and reading same bunch of tables from reporting using readcommitted . It is showing some deadlocks i think it is write in this situation as " x" is not compitable with "s" ,"is".
View 4 Replies
View Related
Oct 23, 2014
I'm working on inserting data into a table in a database. The table has two separate triggers, one for insert and one for update (I don't like it this way, but that's how it's been for years). When there is a normal insert, done via a program, it looks like the triggers work fine. When I run an insert manually via a script, the first insert trigger will run, but the update trigger will fail. I narrowed down the issue to a root cause.
This root issue is due to both triggers using the same temporary table name. When the second trigger runs, there's an error stating that a few columns don't exist. I went to my test server and test db and changed the update trigger so that the temporary table is different than the insert trigger temporary table, the triggers work fine. The weird thing is that if the temporary table already exists, when the second trigger tries to create the temporary table, I would expect it to fail and say that it already exists.I'm probably just going to update the trigger tonight and change the temporary table name.
View 1 Replies
View Related
Apr 5, 2006
Folks,
I have been hunting for a way to trace sql requests going to/from the sql express server (from VS.2005 using asp.net). I.E., I want to see the raw SQL commands that asp.net is generating.
Can anyone point me to a tool or article? Thanks,
--stuart
View 1 Replies
View Related
Mar 27, 2008
Hi. I'm new to SQL Server programming. I'm using it at my new job for the first time and catching on quickly. I've run into a problem where I run a request via interactive query and it times out. It's really annoying because it seems to happy a lot. Is this just a memory issue with my machine (it has 2.8 GB of RAM) or there a way around it?
Thanks!
View 12 Replies
View Related
Jul 23, 2005
I'm monitoring one of our servers, and on the whole it is performingwell. However, I'm puzzled by the number of LockRequests/sec thatPerfmon is recording. We frequently see values exceeding 50,000 andthe current peak is 533,616 (the average, as I type this, is 35,102).There are only 40 users on the system.sp_lock shows nothing like this number of locks; it shows somethingof the order of 30 locks maximum for each execution, which seems farmore reasonable.SQL Profiler seems to back up the Perfmon values; it records hundredsof "Lock Acquired" events per second during these peak periods.For example:NO. OFLOCKS OBJECT STARTTIME===== ====== ======================678User_T2005-04-15 09:03:22.863931User_T2005-04-15 09:03:22.877924User_T2005-04-15 09:03:22.89316EnquiryUser_T2005-04-15 09:03:22.893961User_T2005-04-15 09:03:22.910820User_T2005-04-15 09:03:22.9234NULL2005-04-15 09:03:22.923828User_T2005-04-15 09:03:22.9404NULL2005-04-15 09:03:22.940734User_T2005-04-15 09:03:22.957Can anyone think why such a comparatively small system should generatethese numbers of locks? Why does sp_lock NOT show the same level oflocking?
View 4 Replies
View Related
Jun 21, 2005
May be it's too late, but I think this requests could be scheduled at least
View 145 Replies
View Related
Apr 21, 2015
I have a situation where I have Table A, Table B.
View C is created by joining table A and table B.
I have written a instead of trigger D on view C.
I do not insert/update/delete on the view directly.
For every insert/update in table A /B the values should get insert/update in the view respectively. This insert/update on view should invoke the trigger.
And I am unable to see this trigger work on the view if any insert/update occurs on base table level.
Trigger is working only if any operation is done directly on the view.
View 2 Replies
View Related
Feb 15, 2008
Hello
I've to write an trigger for the following action
When a entry is done in the table Adoscat79 having in the index field Statut_tiers the valeur 1 and a date in data_cloture for a customer xyz
all the entries in the same table where the no_tiers is the same as the one entered (many entriers) should have those both field updated
statut_tiers to 1
and date_cloture to the same date as entered
the same action has to be done when an update is done and the valeur is set to 1 for the statut_tiers and a date entered in the field date_clture
thank you for your help
I've never done a trigger before
View 14 Replies
View Related
