Handle Paging With Large Datasets
Apr 10, 2008Im looking at this article http://www.dotnetjunkies.com/Article/EA868776-D71E-448A-BC23-B64B871F967F.dcik
and it seems like they are selecting the entire customers table into the temp table, correct ?
ADVERTISEMENT
Large Datasets
Mar 27, 2008Hi,
I have a table which has over 450,000 records in it, I have now split this into 4 so each table has around 100,000 records in it but I'm still having the problem of the data being returned really slowly.
What I need to do to this data is group it by a code and show the total for each code for every month of the year (this is currently based on one column and selecting the data accordingly). I have created views and put some indexes onto my table but the results are still being returned slowly. Does anyone have any suggestions of how I can speed this up?
Thanks
Gemma
Paging Large Recordsets
Oct 21, 2007Here's a question you'll never quit hearing: is there a convenient way to page through large recordsets in SQL Server 2000?
I'm writing some software which, for all intents and purposes, works like a messageboard: users can create threads, leave replies, and so on.
I have about a half million records in a few of my tables, and some of my queries return 1000s of results. I'd prefer not to return 1000s of records all at once, so I don't want to page my records in code; I'd rather page them in SQL Server. Naturally, I want to page replies. However, I don't know of a convenient way to page records in SQL Server 2000.
Beware Of Avg() For Large Datasets
Oct 31, 2006Not a question, just a post about something I ran into today that surprised me a little.
I have a huge dataset - about 700m rows, one column of which is a tinyint. I needed to take an average of that column, so I did this:
select avg(mytinyint) from mytable
which returned:
Msg 8115, Level 16, State 2, Line 6
Arithmetic overflow error converting expression to data type int.
It turns out, if the sum() over the column is bigger than MAX_INT = 2147483647, then avg() will throw this error. This surprised me a little - I guess I was assuming sql server would use bigint internally when computing integer avg, but it uses int.
so you have to do this:
select avg(cast(mytinyint as bigint)) from mytable
what's annoying to me is, it's impossible that an average of type T could be larger or smaller than the max/min allowed value of a type T. This means sql server devs could have internally always used bigint for sum and count, and then returned cast(sum/count as T) from avg() without possibility of overflow (unless sum or count was bigger than the max bigint! you are on your own if that happens i guess!).
Because of this, you have to cast your column to a bigint to make it work if your dataset is large enough, even if the column is only 1 measly byte! kind of a waste of space if you ask me. :)
here's a little example of what I'm talking about, that doesn't require you to import a 700m row dataset :)
declare @t table (id int)
insert into @t select 1 union all select 2147483647
select avg(cast(id as bigint)) from @t -- works
select avg(id) from @t -- fails
Paging Large Results In SQL 2005
May 29, 2006lets say we have more than 100 000 rows in Table1, and we want to view each 10 rows alone.... and by pressing on a NEXT button we will see the other 10 pages....
there is 2 buttons : NEXT and PREVIOUS
so can anyone tell me how to do that in SQL 2005, and what is correctly called.
I have found a code that does use ROW_NUMBER in order to view results between 2 numbers,
example: rows between 10 and 50....
but It is not what I want, so please I need some help, thank you
By Uncle Sam
Paging Large Results In SQL 2005
May 29, 2006lets say we have more than 100 000 rows in Table1, and we want to view each 10 rows alone.... and by pressing on a NEXT button we will see the other 10 pages....
there is 2 buttons : NEXT and PREVIOUS
so can anyone tell me how to do that in SQL 2005, and what is correctly called.
I have found a code that does use ROW_NUMBER in order to view results between 2 numbers,
example: rows between 10 and 50....
but It is not what I want, so please I need some help, thank you
By Uncle Sam
Methods For Handling Large Datasets
Apr 7, 2008Lets say you want to offer users access to a table that has over 1,000,000 records in it. You dont want to fetch the entire table into a dataset. Even with paging, it will still cause a performance lag, will it not ?Lets say there was a column that held last names. You could then divide up all the data in to alphabetical groups by letter. That will at least let you split up the data 26 different ways (letters in teh alphabet). Thats still roughly 38,000 possible records in the dataset at any given time. I suppose since the database and web server are very close to each other, physically, the latency shouldnt be too bad. However, Im worried about having several dozen or more users, each requesting datasets with tens of thousands of records. That would cause the memory use on the web server to spike most likely. Right now, its just one user do this sort of processing, but that might grow to include several dozens.ideas ?
View 3 Replies View RelatedSorting + Paging A Large Table In Stored Procedure
May 6, 2007As I said above, how do I put sorting + paging in a stored procedure.My database has approximately 50000 records, and obviously I can't SELECT all of them and let GridView / DataView do the work, right? Or else it would use to much resources per one request.So I intend to use sorting + paging at the database level. It's going to be either hardcode SQL or stored procedures.If it's hardcode SQL, I can just change SQL statement each time the parameters (startRecord, maxRecords, sortColumns) change.But I don't know what to do in stored procedure to get the same result. I know how to implement paging in stored procedure (ROW_NUMBER) but I don't know how to change ORDER BY clause at runtime in the stored procedure.Thanks in advance.PS. In case "ask_Scotty", who replied in my previous post, http://forums.asp.net/thread/1696818.aspx, is reading this, please look at my reply on your answer in the last post. Thank you.
View 3 Replies View RelatedHow To Handle Large Outputs From RDBMS
Nov 30, 2014Obviously Excel is the tool of choice for most people but with it's limited ability to leverage RAM (32 bit) and it's limitation with rows at just over 1 million what other choices do we have for viewing data?
My bosses boss created several OLAP universes and they seems to fly a lot fast than regular relational database. This still doesn't work with the fact the data can't be worked with unless you have a strong front end that can handle processing all those rows.
SQL Server 2008 :: How To Find Statements That Cause Large Memory Paging
Apr 22, 2015I am monitoring our production server, and noticed that periodically we have spikes of Memory Paging Rate (pages/sec).
How to find particular queries/stored procedures that causing this?
SQL Server 2012 :: Best Way To Handle Like Percentage On Column Too Large For Index
Sep 18, 2015We have a table to 100M rows and up until now we were fine with an non clustered index a varchar(4000) because we never went above 900 bytes (yes it is a bad design).We have the need to support international character sets now so the column was updated to nvarchar(4000) and now we have data past the 900 byte limit.
The data is long, seems useless but is needed by the business and they need to be able to search "where bigcolumn like 'test%'". With an index, even with a huge amount of data, it was 'fast'. Now of course without an index it is unusable. The wildcard is always at the end of the search. I made a full text index on the column and basic queries such as: select * from ourtable where contains(bigcolumn, 'AReallyLongStringofTextHere') works fine unless there is a space in the data. We loose thousands of returned rows because of spaces in the data.
I have tried select * from ourtable where contains(bigcolumn, '"AReallyLongStringofTextHere that includes spaces"') but not all of the data is returned. I get 112 rows with the contains statement. The table scanning statement of "select * from ourtable where bigcolumn like 'AReallyLongStringofTextHere that includes spaces%' returns 1939 rows.I understand that a full text index is breaking the long string up since it contains spaces. Is there a way to retain the entire string as 1 index entry or is there a way to fix my query to return all of the rows?
SQL Server 2008 :: How To Find Which Queries / Processes Causing Large Memory Paging Rate
Mar 30, 2015Our monitoring tool shows that our production system periodically experiencing large rate - up to 800 memory pages/sec. How to find out which particular queries, S.P., processes that initiate this?
View 3 Replies View RelatedError Log Peppered With --&&> 'The Conversation Handle Is Missing. Specify A Conversation Handle.'
Dec 3, 2007Hi
I'm using service broker and keep getting errors in the log even though everythig is working as expected
SQL Server 2005
Two databases
Two end points - 1 in each database
Two stored procedures:
SP1 is activated when a message enters the sending queue. it insert a new row in a table
SP2 is activated when a response is sent from the receiving queue. it cleans up the sending queue.
I have a table with an update trigger
In that trigger, if the updted row meets a certain condition a dialogue is created and a message is sent to the sending queue.
I know that SP1 and SP2 are behaving properly because i get the expected result.
Sp1 is inserteding the expected data in the table
SP2 is cleaning up the sending queue.
In the Sql Server log however i'm getting errors on both of the stored procs.
error #1
The activated proc <SP 1 Name> running on queue Applications.dbo.ffreceiverQueue output the following: 'The conversation handle is missing. Specify a conversation handle.'
error #2
The activated proc <SP 2 Name> running on queue ADAPT_APP.dbo.ffsenderQueue output the following: 'The conversation handle is missing. Specify a conversation handle.'
I would appreceiate anybody's help into why i'm getting this. have i set up the stored procs in correctly?
i can provide code of the stored procs if that helps.
thanks.
Conversation Handle Reuse And Conversation Handle XXX Not Found
Jan 18, 2008
We have implemented our service broker architecture using conversation handle reuse per MS/Remus's recommendations. We have all of the sudden started receiving the conversation handle not found errors in the sql log every hour or so (which makes perfect sense considering the dialog timer is set for 1 hour). My question is...is this expected behavior when you have employed conversation recycling? Should you expect to see these messages pop up every hour, but the logic in the queuing proc says to retry after deleting from your conversation handle table so the messages is enqueued as expected?
Second question...i think i know why we were not receiving these errors before and wanted to confirm this theory as well. In the queuing proc I was not initializing the variable @Counter to 0 so when it came down to the retry logic it could not add 1 to null so was never entering that part of the code...I am guessing with this set up it would actually output the error to the application calling the queueing proc and NOT into the SQL error logs...is this a correct assumption?
I have attached an example of one of the queuing procs below:
Code Block
DECLARE @conversationHandle UNIQUEIDENTIFIER,
@err int,
@counter int,
@DialogTimeOut int,
@Message nvarchar(max),
@SendType int,
@ConversationID uniqueidentifier
select @Counter = 0 -- THIS PART VERY IMPORTANT LOL :)
select @DialogTimeOut = Value
from dbo.tConfiguration with (nolock)
where keyvalue = 'ConversationEndpoints' and subvalue = 'DeleteAfterSec'
WHILE (1=1)
BEGIN
-- Lookup the current SPIDs handle
SELECT @conversationHandle = [handle] FROM tConversationSPID with (nolock)
WHERE spid = @@SPID and messagetype = 'TestQueueMsg';
IF @conversationHandle IS NULL
BEGIN
BEGIN DIALOG CONVERSATION @conversationHandle
FROM SERVICE [InitiatorQueue_SER]
TO SERVICE 'ReceiveTestQueue_SER'
ON CONTRACT [TestQueueMsg_CON]
WITH ENCRYPTION = OFF;
BEGIN CONVERSATION TIMER ( @conversationHandle )
TIMEOUT = @DialogTimeOut
-- insert the conversation in the association table
INSERT INTO tConversationSPID
([spid], MessageType,[handle])
VALUES
(@@SPID, 'TestQueueMsg', @conversationHandle);
SEND ON CONVERSATION @conversationHandle
MESSAGE TYPE [TestQueueMsg] (@Message)
END
ELSE IF @conversationHandle IS NOT NULL
BEGIN
SEND ON CONVERSATION @conversationHandle
MESSAGE TYPE [TestQueueMsg] (@Message)
END
SELECT @err = @@ERROR;
-- if succeeded, exit the loop now
IF (@err = 0)
BREAK;
SELECT @counter = @counter + 1;
IF @counter > 10
BEGIN
-- Refer to http://msdn2.microsoft.com/en-us/library/ms164086.aspx for severity levels
EXEC spLogMessageQueue 20002, 8, 'Failed to SEND on a conversation for more than 10 times. Error %i.'
BREAK;
END
-- We tried on the said conversation, but failed
-- remove the record from the association table, then
-- let the loop try again
DELETE FROM tConversationSPID
WHERE [spid] = @@SPID;
SELECT @conversationHandle = NULL;
END;
Paging
Oct 12, 2004Is there any way to implement a paging scheme such that only the required records are transferred from SQL Server to the asp.net app?
The only support I can find such as the DataAdaptor.Fill will bring all the records back from SQL Server and then create the page...
This obviously still takes time and memory based on the entire query, not the page size.
Any ideas?
SQL Paging
Jul 26, 2005I have a table with a lot of records. I want to make paging without passing all the data back to a dataset. I know how to select the top n rows but how do I select 10-20 for example.
View 3 Replies View RelatedT-SQL Paging
Dec 9, 2005Hello,
How can I do paging in my SQL Server stored procedure.
I have thought of a way, but I need to :
"SELECT TOP @Count..."
which is not allowed :S
What can I do to get around this?
Paging In Sql 2k
Dec 11, 2003I have noticed that the server i'm running SQL2k on is starting to page out of the norm. I can see that the regsvc and sqlservr svc are showing high page faults/sec. I have 3 gigs of ram and set the max that sql can use to 2 gigs. It is currently using only 168 MB and still will show high paging at random times. I know I can add more ram but that doesn't seem to be the problem. I have also stopped unnecessary services at the os level.
Any other suggestions to fix this?
Thanks in advance.
Paging In RS
May 9, 2008Hi there
I've managed to make it query that return a dataset that have 2 views utilising the "Filter" in RS. I treat this as a single record with multiple views.
Now let say if I have a stored precedure that pass 2 parameters one is called state and year and accepting 'ALL' for every possibility of State and Year and construct that into single dataset with 2 views similar like above.
How do I breakdown this in the reporting services so it will have paging?
This is a simple dataset:
RECORDID, ReportViewType, State, Year, VALUE
1, "VIEW1", "NSW", 1, null
1, "VIEW2", null, null, 10000
2, "VIEW1", "NSW", 2, null
2, "VIEW2", null, null, 11000
3, "VIEW1", "VIC", 1, null
3, "VIEW2", null, null, 11003
4, "VIEW1", "VIC", 2, null
4, "VIEW2", null, null, 11001
I would like to break down (paging) this per recordid. Each page obviosuly has 2 views using the same data set with different FILTER.
Do I need to put into a LIST then inside that list put 2 TABLES? Is this possible?!?!
Thanks
Where Did The Datasets Go In .NET 2.0???
Jul 5, 2006I typically use DataAdapters and Datasets from the toolbox when connecting to a SQL database. Since switching to VS 2005, I am not able to find (or add) these items to the tool box for web site development.
Any ideas on how I can get them back? Or is there a better alternative to retrieving the data so I can process it? Any help or suggestions would be appreciated.
Thank you
About Datasets
Apr 19, 2007Can anyone advise me on how to view the contents of a dataset in VS dotnet 2003..
I had filled a dataset from a query...But i don know what happens after filling the qry results.It might be helpful if someone tells me to view the contents of the dataset
Ado Datasets
Jun 6, 2007hi
i am using older ado datasets in a borland 6 program.
i need to retrieve the data per record count....
example is that i need all 1000 records but i only want to retrieve them 100 at a time....
how will i accomplish this?
Paging Query
Aug 2, 2006I have created a stored proc for paging on a datalist which uses a objectDataSource.
I have a output param itemCount which should return the total rows. Them I am creating a temp table to fetch the records for each page request. My output param works fine if I comment out all the other select statements. But returns null with them. Any help would be appreciated.
CREATE PROCEDURE [dbo].[CMRC_PRODUCTS_GetListByCategory]( @categoryID int, @pageIndex INT, @numRows INT, @itemCount INT OUTPUT )AS
SELECT @itemCount= COUNT(*) FROM CMRC_Products where CMRC_Products.CategoryID=@categoryID Declare @startRowIndex INT; Declare @finishRowIndex INT; set @startRowIndex = ((@pageIndex -1) * @numRows) + 1; set @finishRowIndex = @pageIndex * @numRows
DECLARE @tCat TABLE (TID int identity(1,1),ProductID int, CategoryID int, SellerUserName varchar(100), ModelName varchar(100), Medium varchar(50),ProductImage varchar(100),UnitCost money,Description varchar(1500), CategoryName varchar(100), isActive bit,weight money)
INSERT INTO @tCat(ProductID, CategoryID,SellerUserName,ModelName,Medium,ProductImage,UnitCost,Description,CategoryName, isActive,weight)SELECT CMRC_Products.ProductID, CMRC_Products.CategoryID, CMRC_Products.SellerUserName, CMRC_Products.ModelName, CMRC_Products.Medium,CMRC_Products.ProductImage, CMRC_Products.UnitCost, CMRC_Products.Description, CMRC_Categories.CategoryName, CMRC_Products.isActive,CMRC_Products.weightFROM CMRC_Products INNER JOIN CMRC_Categories ON CMRC_Products.CategoryID = CMRC_Categories.CategoryIDWHERE (CMRC_Products.CategoryID = @categoryID) AND (CMRC_Products.isActive = 1)
SELECT ProductID, CategoryID,SellerUserName,ModelName,Medium,ProductImage,UnitCost,Description,CategoryName, isActive,weightFROM @tCat WHERE TID >= @startRowIndex AND TID <= @finishRowIndexGO
Paging Problems
Dec 13, 2006I am using a stored procedure to page some objectsThe procedure looks like this:
CREATE PROCEDURE sw20aut.sw20_Kon( @sid_nr INT, @sid_stl INT = 35, @kid int )AS BEGIN SET NOCOUNT ON DECLARE @rader INT, @sid_antal INT, @ubound int, @lbound int SELECT @rader = COUNT(*), @sid_antal = COUNT(*) / @sid_stl FROM sw20aut.sw_kontakter WITH (NOLOCK) WHERE kund_id = @kid AND del = '0'
IF @rader % @sid_stl != 0 SET @sid_antal = @sid_antal + 1 IF @sid_nr < 1 SET @sid_nr = 1 IF @sid_nr > @sid_antal SET @sid_nr = @sid_antal SET @ubound = @sid_stl * @sid_nr IF(@sid_antal > 0) SET @lbound = @ubound - (@sid_stl - 1) ELSE SET @lbound = 0 SELECT CurrentPage = @sid_nr, TotalPages = @sid_antal, TotalRows = @rader
DECLARE @ename VARCHAR(64), @fname VARCHAR(64), @konid VARCHAR(64) SET ROWCOUNT @lbound SELECT @ename = enamn, @fname = fnamn, @konid = kon_id FROM sw20aut.sw_kontakter WITH (NOLOCK) WHERE kund_id = @kid AND del = '0' ORDER BY enamn, fnamn, kon_id SET ROWCOUNT @sid_stl SELECT kon_id, enamn, fnamn FROM sw20aut.sw_kontakter WITH (NOLOCK) WHERE enamn + fnamn + '~' + CAST(kon_id as VARCHAR(64)) >= @ename + @fname + '~' + @konid AND (kund_id = @kid AND del = '0') ORDER BY enamn, fnamn, kon_id SELECT startid = @konid SET ROWCOUNT 0END
The big problem is that i need to display objet with the same name. In my book the best identifier is the PK and therefor i have sorted as above by ordering after LastName, FirstName, ContactId
After som thinking ive reached the conclusion that this dont work if the idnumbers isnt of the same length. as long as they are(for example two people named John Smith, one with id = '23' and one with id = '87' it works. If there ids would have been '23' and '1203' it will not work correctly) of the same length it works fine.
What im wondering is if anyone have a good solution to this? Only thing i can think of is filling all idnumbers with zeros to equal length. Dont know how and if this will affect performance though. Anyone has a practical solution to this?
Paging Technique
Aug 8, 2007Questoin
I am using Sql Server 2000.
I have a table named Cities which has more than 2600000 records.
I have to display the records for a specific city page wise.
I don't want to compromise with performance.
Can anyone has the idea?
Waiting for your fruitful response.
Happy Day And Night For All
Muhammad Zeeshanuddin Khan
Paging , Sql Select From To ?
Feb 10, 2008Hello To make pagination I would like to retrieve only the record from x to y ...I couldn't find how to do to in sql , I was thinking so if there is a way to do it with a sqldatasourceI make my request , put it in a sqldatasource and bind it to my datalistis there a way to "filter the sqldatasource ?" to make what I need ? Thx in advance ?
View 4 Replies View RelatedCustom Paging
May 23, 2005Im in the process of trying to teach myself SqlServer, comming from Oracle. How the heck do I get the equivlent of %ROWNUM pseudo-column in SqlServer? Top just isn't doing it for me.
Oracle Example wrote:
Select * from foo where foo%ROWNUM > 10 and foo%ROWNUM <20;
Paging Performance
Feb 21, 2003I have a paging dilema with an ADO/ASP web based application.
Currently I am using the temp table and inserted index method of paging which works well but the pages that use this paging have a variety of filters on them and a largish subset of data available. This means that every time the page is refreshed the code is creating that temporary table, inserting all the data, selecting the subset and then dropping it.
I was looking for a more efficent way of getting paged data to the client. One of the alternatives I came across was using a server side forward only cursor and the ado getrows() method. This sounds good in princible but I don't know if I am going to get a performance hit by using a server side cursor as opposed to sending the entire recorset to the client and letting it page the results.
Would it be any better to use a stored procedure and pass the full sql statement to it. I can't actually write the sql into the stored procedure becuase it is all totally dynamic.
So I guess I have three options, temp tables, server side cursor and small amounts of data sent to the client or client side cursor and large amounts of data sent to the client.
Any ideas or recomendations?
Paging Question
Jan 5, 2007Is this a correct statement? When commit Charge total (k) is greater than Physical Memory total (k) then the server is paging badly, correct?
thanks.
Help Using Row_Number For Paging
Apr 25, 2008Hi,
My application runs this query using a stored proc
SELECT empid1,name1,joindate from emp where empid2=3
union
select empid2,name2,joindate from emp where id1=3
Now I want to implement paging for the same using Row_Number so that I can display the results in pages.
Can someone please help me write a query for the same. I tried playing with Row_Number but no luck with it.Basically I am not good with SQL and I had programatically implemented paging in asp.net by looping through all records returned by the query.
Thanks,
Ganesh
Paging (Performance)
Nov 28, 2006Hello, I have incorporated a paging query in my software. I got the query from here:
http://rosca.net/writing/articles/serverside_paging.asp
My web software ususlly responded in .005 - .02 seconds with about a 100 rows of data. When I put simulated data on my database I added about 2 million rows. when I did this -- every page that did not execute the paging query responded lightning fast. But the webpages that executed the paging query took over 5 seconds. I dont understand why this paging query brought my web application to its knees.
Does anyone know of a more efficient way to do paging. I have SQL server 2000. If it may be easier I can upgrade to SQL 2005. PLZ Let me know. Thanks
Paging Problem
Mar 14, 2006Hello, I have a datagrid with paging allowed, but when i click on page number 2 or 3, it still displays the records from the first page. How do I solve this?
Thank you.
Strategies For Paging
Sep 28, 2007hello, what are the strategies when designing tables that needspaging?in the past i used to useselect top 200 * from tablewhere id not in (select top 100 id from table)with SQL 2005, would u guys recommend using CTE and/or ROW_NUMBER?or any other advice?thanks
View 4 Replies View Related