Keeping A Table Moderatley Sized (partitions?)
Jan 29, 2008
Guys,
First off, I'm not very familiar with SQL Server. I need some guidance on what the best path to take is for this as it may not even be table partitions.
I have a huge table (155 million rows) and it's gotten so large than I can't even delete a large set of rows from it (i.e. delete everything older than 6 mo, which would be ~100 million rows). When trying to run a delete like this, it just goes for a LONG time and then just eventually runs out of memory.
The current data in this table can actually be completely cleared out soon (after Feb 1st) and I plan to do this with TRUNCATE TABLE, or just DROP and recreate. Once I do this, I want to create a way to keep this table moderately sized so it never grows that large again and it seems table partitions may be the way to go for this?
I'd like to keep the last 6mo of data in it (I have a datetime column to keep track of this). Anything older I'd like automatically removed. Can I do this with table partitioning? Create 6 partitions that store the 6 most recent months of data and everything older automatically gets dropped off?
If not partitions, what do you suggest to keep this DB modest size?
Thank you.
View 9 Replies
ADVERTISEMENT
Aug 28, 2007
Hi experts,
We have a huge table with around 250 million records and have implemented SQL server 2005's new table partitioning feature. Now the data seems to be evenly spread across 20 different filegroups ( each 5 GB approx ) for the same table that was occupying 100 GB itself in the PRIMARY filegroup earlier.
Still the query response times have not come down drastically but we could see a good improvement in the execution plans now.
WE ARE USING RAID 5 IN OUR PRODUCTION ENVIRONMENT. ANY IDEA / THOUGHT ON HOW TO PLACE THE PARTITIONED FILEGROUPS AND THE LOG FILES IN THE RAID 5 (BTW , I'm very new to RAID concepts , any detailed instruction would be helpful ).
Any help would be greatly appreciated.
Thanks,
Hariarul
View 8 Replies
View Related
Oct 31, 2006
Hi,
For my work I am now learning Sql server 2005 and I have been given a database that has been set up by someone else to work with. It is my job to get the database ready for use in reports.
My problem is that the current database has one huge table with almost 8GB of data. The table contains data from 2004 to present (and growing) from 14 different countries. The reports we use are mostly per country, but we also want to compare the 14 countries to eachother for say, whole 2006. At the moment the table is stored in one single file instead of using partitions.
I believe partitions can give a good performance boost when running the queries. But how do I do this? Currently the country codes are just plain text, can they be used for partitions?
Any advice would be welcome,
Thanks!
View 5 Replies
View Related
Feb 26, 2008
Hi Experts,
We have a very huge database that stores 12 years of data(120 Million records). But our application mainly accesses past 3 years data i.e , the queries would scan the 120 million records even when it actually has to scan 30 million records alone (for 3 years).
Since few other important applications needs access to all the 12 years data, we are in a position to have 12 years data in the same database.
Right now we are looking for an approach that would help us to efficiently access the 3 years data alone and boost the performance.
1. Will SQL server table paritioning help in this scenario ?
Or
2. Indexed views would help us ? Is it possible to create indexed views based on year range and access the views in the stored procedures ?
Any help would be greatly appreciated.
Thanks in advance,
Hariarul
View 4 Replies
View Related
Jun 4, 2007
Hi guys , assuming right now I already create partition function (PF_Date) and partition scheme (PS_Date). Let say I would like to implement the partition on the existing tables ( eg: transaction table which is in PRIMARY filegroup), how am I switch it from PRIMARY to PS_Date ? Is it I have to re-create the particular table then only able to put in the partition scheme? Hope can get any assistance here. Thanks alot.
Best Regards,
Hans
View 8 Replies
View Related
Nov 7, 2001
Hi
We need to keep track of all changes that are made to our tables.
The changes will be saved in a table that records:
- the table in which the change was made
- the name of the field that was changed
- the old data for the field
- the new data for the field etc..
I've seen a few examples that record the name of the table that was
modified but none that record done to the field level.
Can anybody give some guidance?
Thanks..
Wayne
View 1 Replies
View Related
Feb 5, 2004
I need to update a row but keep a lock on the table (so no one else can update it) while I do run some more code. In Oracle, it always locks whatever you update until you hit commit, but sql server works opposite. How do I tell it not to commit a statement, or how would I explicitly get a lock and then release it later?
View 4 Replies
View Related
Dec 31, 2011
I have a table that contains records of transactions with ID column is primary key
I use partition follow ID column, each partition have 1 million records.
CREATE PARTITION FUNCTION [pfTBLTRANS_ID](int) AS RANGE LEFT FOR VALUES (1000000, 2000000, 3000000, 4000000, 5000000, 6000000, 7000000, 8000000, 9000000, 10000000)
CREATE PARTITION SCHEME [psTBLTRANS_ID] AS PARTITION [pfTBLTRANS_ID] TO ([GROUP01], [GROUP02], [GROUP03], [GROUP04], [GROUP05], [GROUP06], [GROUP07], [GROUP08], [GROUP09], [GROUP10], [GROUP11])
But now I have more records with IDs that are greater than 11.000.000. So how can I add more partitions to this table ?
View 9 Replies
View Related
Apr 13, 2015
Perhaps this task is not for MDS.... But another tool for rapid development & startUp - we don't have. And nevertheless....
We created table managers_plan in MDS :
year
month
id_manager (domain attr)
POSÂ (domain attr)
plan_sum_USD
plan_unit
----------------------------
Entities:
Managers ~ 800Â records
POSÂ ~ 100 000 records
managers_plan Total records for
1 year = 100K x 12 = 1 200 000
managers_plan - table partitions Â
- Will bemade ?
View 3 Replies
View Related
Jun 1, 2015
SQL 2014 I've inherited a db that has several partitioned tables. They are partitioned by month. We're approaching the last partition, 11-30-2015, so we need to extend the tables. My question is how do I do this? There are Partition Schemes and Partition Functions setup at the db level. I've figured out how to ALTER those. Next I go to a table that I know is partitioned, right-click Storage and select Manage Partitions.Â
My only option is to "create a staging table for partition switching". Not knowing what switching is, I'm not sure if this is what I want to do. All I want to do is add new partitions to the table - and remove some of the old ones since they are empty due to archiving of data.
So, what is the proper steps to adding new partitions to a table that is already partitioned?
View 4 Replies
View Related
Jul 2, 2001
Hi, I am trying to find a way to capture all the status (Start time, execution time, Status messages etc) from executing a DTS package in to a table I will create in a database, does anyone know, where those information being kept?
When I excute the DTS package manually, a window will come up and show the status of each step within the DTS package. I am hoping to capture these information and load it to my log table.
Thanks in advance.
View 1 Replies
View Related
Aug 17, 1999
How can I instruct SQL Server to keep entire table in memory? ie the memory pages should not be swapped to HD.
View 2 Replies
View Related
Nov 20, 2007
Hi all,
I have the following function:
Code Block
create function udf_CountRows(@pTableName sysname)
returns int
as
begin
declare @ret int
select @ret = SUM(p.rows)
from sys.partitions p
inner join sys.objects o
on p.object_id = o.object_id
and o.[name] = 'Well'
return @ret
end
Can I trust sys.partitions to always return the correct value or does it suffer the same issue as sysindexes prior to SQL2005?
Thanks
Jamie
View 5 Replies
View Related
Jul 20, 2005
We currently have an SQL db running on a web server.One of these fields is a large(ish) amount of text data – up to 400characters – and has been cast variously as varchar, nchar and texttype to overcome a problem. The problem appears to be in retrievingthe data via ASP. I understand that ASP can handle string data of thissize so I am okay there.When the records are retrieved from the db, the data string length =0.I know the data is there because I have written a Delphi data managerwhich interrogates the db and shows all records and their contents.So if ASP can handle strings this size and the data is there, why do Iget a data length of zero bytes returned when I interrogate the recordset?Whichever way I cast this field I get the same result.I know the code is sound as it works locally through a MS SQL serveron my PS.Anyone have this problem or know what's causing it? I have logged asupport call with my hosting company, but they haven't replied as yetand I am stuck on an urgent project.Any suggestions?CheersGrant
View 1 Replies
View Related
Jun 22, 2015
I have some table that need to be partitioned and archive one of the partitions.
I did this in Oracle several years ago but not in SQL Server.
I'm looking for a basic example on how to do this.
I know the basic steps but the examples that I found on the Web were not quite what I'm looking for.
[url][/
Partition an existing SQL Server Table
url]
View 9 Replies
View Related
Sep 15, 2015
I have 3 columns. I would like to update a table based on job_cd and permit_nbr column. if we have same job_cd and permit_nbr, reference number should be same else it should take max(reference number) from the table +1 for all rows where reference_nbr column is null
job_cd permit_nbr reference_nbr
ABC1 990 100002
ABC1 990 100002
ABC1 991 100003
ABC1 992 100004
ABC1 993 100005
ABC2 880 100006
ABC2 881 100007
ABC2 881 100007
ABC2 882 100008
ABC2 882 100008
View 3 Replies
View Related
Oct 8, 2015
I have table having around 100 million rows.Everyday we have an ETL process in which table will be trucnated and relaoded. Will creating a partition on the table increase the inserting speed?
View 4 Replies
View Related
Feb 6, 2008
Somebody suggested I use the slowly changing dimension object to keep a table current between servers and databases.
I have two databases servers. One 2000 and the other 2005. I was told because security reasons I should not link them
The two tables are not identical. The destination table only has a handle of the columns and in some cases columns are not the same size.
Once a day. When new rows are introduced in the source table, I'd like to introduce those rows into the destination table. When rows are modified, I'd like to apply changes to the distination table.
I started playing with the slowly changing dimensions object, but am frankly confused connected the dots. Will I have an output oledb object for each command object?
Am i using the right the object for objective?
Are there any good tutorials or videos out there that will cover this?
Thanks for any help or information!
View 4 Replies
View Related
Sep 18, 2015
In an asp.net app I'm creating email attachments from ssrs reports and am rendering them In Word and Excel using the WORDOPENXML and EXCELOPENXML format extensions. When I render the report in excel, the image is sized correctly, however, when I render the same report in word, its very small. I will attach a screenshot of each, and in the word screenshot, I will select the image so you can see the image control's outline (the control's actual size). Notice the image inside of the border is very small. How can I resolve this?
first, excel:
and now in word:
View 3 Replies
View Related
Jan 9, 2002
Does anyone have any statistics on the performance gains one can get using raw partitions. The database in question is very IO intensive and performs about 1,000,000 inserts/updates per select.
Thanks...
View 2 Replies
View Related
Oct 12, 1999
Please help!
I have a server that has SQL Server installed on both C and D drives. The SQL Server software is currently running from the C drive and the live databases and backups are stored on the D drive.
I need to have everything on the D drive. Is there an easy way to make the registry point to the D drive without reinstalling SQL Server? The software will needs to run from the D drive because the C drive is running out of disk space. I will also need to delete the whole C:mssql directory.
Thanks.
View 2 Replies
View Related
Jun 19, 2008
assuming that you have two databases, the OLTP db and the OLAP db (take not that both have the same structure -- archiving purposes)... using table partitioning, is there a way where we can move 1 partition from the OLTP db to the OLAP db???
i'm actually trying to use this example with both tables in the DB.. I tried to modify to use two databases but sql server is unable to move the partition...
ALTER TABLE [Production].[TransactionHistory]
SWITCH PARTITION 1
TO [Production].[TransactionHistoryArchive] PARTITION 2;
SlayerS_`BoxeR` + [ReD]NaDa
View 9 Replies
View Related
Aug 23, 2007
Hi experts,
In SQL Server 2005 database we have partitioned a very big table into 30 partitions each holding few million of records.
Im just curious to know whether there are some configuration related to processors or system hardware in order to benefit from partitioning ? (Ex : If we have multiple processors Whether they need be configured to do a parallel processing ? )
Any real time experience (other than referring links) would be really helpful for me.
Thanks in advance,
Hariarul
View 1 Replies
View Related
May 21, 2008
Hi There,
I have a quick question regarding merging cube partitions.
If I create partitions sliced by date (let's assume we have year level partitions like 2006,2007,2008...)
Later, if I want to merge selected partition to another partition , for example I have history partition and 2006 partition and I want to merge 2006 to history partition then I can simply merge them using ' merge partition' through Management Studio.
My question is that in script, History partition has condition which is where clause and restricted by year level (i.g. WHERE date < '01-01-2006' ) ; however after merging , script won't change like WHERE date <'01-01-2007').
If so, whenever I merge partitions then I have to alter the script as well based on selected merge partion?
I need to refresh history partition once a month;however even if I merge 2006 partition , once I reprocess history partition then it only process what it was wrote in script. So, after reprocessing 2006 data won't appear in this history partition.
So, wondering it's mentatory to alter the script once partition is merged.
Please give me some comments on this issue.
Thanks in advance.
View 4 Replies
View Related
Apr 16, 2002
I am testing horizontal partitions to see whether it is a feasible
option for a project. IF I have a composite Primary Key and the
constraint column (a part of the Primary Key) that helps the
partitioned view is defined with DateTime Data Type, select on a restricted set of data through a partitioned view still tries to access all the tables instead of just one table that contains the data. Is this the case or am I
missing something ?
Any help on this is appreciated.
Here is what I am trying to do.
************************************************** **************
CREATE TABLE [dbo].[tst01] (
[Dt] datetime NOT NULL ,
[TID] int NOT NULL ,
[Nm] [char] (10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL
) ON [PRIMARY]
GO
CREATE TABLE [dbo].[tst02] (
[Dt] datetime NOT NULL ,
[TID] int NOT NULL,
[Nm] [char] (10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[tst01] ADD
CONSTRAINT [PK_tst01] PRIMARY KEY CLUSTERED
(
[Dt], [TID]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[tst02] ADD
CONSTRAINT [PK_tst02] PRIMARY KEY CLUSTERED
(
[Dt], [TID]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[tst01] ADD
CONSTRAINT [CK_tst01] CHECK (Dt between '11/1/2002' and '11/30/2002')
GO
ALTER TABLE [dbo].[tst02] ADD
CONSTRAINT [CK_tst02] CHECK (Dt between '12/1/2002' and '12/31/2002')
GO
insert into tst01 values('11/1/2002', 1, 'SS')
insert into tst01 values('11/2/2002', 2, 'KK')
insert into tst01 values('11/3/2002', 3, 'DD')
Go
insert into tst02 values('12/1/2002', 1, 'LL')
insert into tst02 values('12/2/2002', 2, 'MM')
insert into tst02 values('12/3/2002', 3, 'GG')
Go
CREATE VIEW vtst AS
SELECT * FROM tst01
UNION ALL
SELECT * FROM tst02
Go
SELECT * FROM vtst WHERE dt = '11/1/2002'
************************************************** *********
When I look at the Execution Plan, it shows that clustered index seek
would be performed on both the tables.
StmtText
---------------------------------------------------------------------------------------------------------------------------------
|--Concatenation
|--Filter(WHERE:(STARTUP EXPR(Convert([@1])<='Nov 30 2002
12:00AM' AND Convert([@1])>='Nov 1 2002 12:00AM')))
| |--Clustered Index
Seek(OBJECT:([test].[dbo].[tst01].[PK_tst01]),
SEEK:([tst01].[Dt]=Convert([@1])) ORDERED FORWARD)
|--Filter(WHERE:(STARTUP EXPR(Convert([@1])<='Dec 31 2002
12:00AM' AND Convert([@1])>='Dec 1 2002 12:00AM')))
|--Clustered Index
Seek(OBJECT:([test].[dbo].[tst02].[PK_tst02]),
SEEK:([tst02].[Dt]=Convert([@1])) ORDERED FORWARD)
Thanks,
Sathish
View 1 Replies
View Related
Feb 24, 2004
I'm considering using horizontal partitions to separate my data by year.
For example, SomeTable_2004, SomeTable_2003, etc. This works well for backups, maintenance, etc. because I'm working with 150+ GB of data. I'll be a partitioned view for queries.
However, I'm new at this and have a few questions. I would also like to do partitioned updates or inserts. But I need to make sure that the tables don't use similar primary keys. Does that make sense? I need to make sure that the primary keys from the first table are not used again in the second table.
SomeTable_2003
primary keys: 1,5,8,9,15
SomeTable_2004
primary keys: 2,3,4,10
I don't really care what keys are used on what table, as long as they are different. I have apps that already use this data, and I don't want to change the application logic.
Thanks,
T
View 1 Replies
View Related
Oct 1, 2015
how to set up the partitions.I have a transaction table with 50 million records that's very hard to query. it holds data for the last 4 years but the application only ever looks at the last 6 months so i believe this is and ideal candidate for partitioning.
Would it be better to
1) create a partition based on each year for all data so would have a 2015, 2014, 2013, 2012?
2) create 1 partition based on month for this years data then 3 based on year so would have jan,feb,march,april,may..., 2014, 2013, 2012
For 1) would you have to perform some maintenance at the turn of each year for accommodating the next years data. For 2) although this would give better performance as query's are mostly in the last 6 months wouldn't this have more maintenance to move month data to year partitions come the turn of the year and then create the next years months partitions.
View 9 Replies
View Related
May 21, 2007
I just inherited a dev box, and need to do some performance analyzing on a 40 gig db for a client. Time is of the essence!
My question is that this dev box only has one disk partition (c: drive). Is it a huge deal that I don't have the db system files on one drive, with the data files on another, and tempdb on another,etc.....
View 1 Replies
View Related
Mar 9, 2008
I have a requirement that I need to reload the last seven days worth of data each night to ensure that we pick up late arriving and updated records. To avoid having to do updates we delete the last seven days data and reload.
I was wondering if it is possible to set up the table as a partition, paritioned on a value (OLD, NEW) or similar.
The job would set the last day in the NEW partition to be old, the theory being that this would cause the rows to move to the OLD parition, and then truncate the new partion rather than deleting. The last seven days data could then be inserted into the empty new partition.
My questions is 1. Is my theory about the data moving from one partition to another correct. 2. Can I actually truncate and individual parition, 3. Do you think it will perform any quicker. We would expect data in the range of 100K to 500K rows in the seven days and will store up to 4 years of historical data.
Thanks for your thoughts
Stapsey
View 1 Replies
View Related
Oct 17, 2007
Hi champs!
The data i have is like this:
nr date value
------- -------------- --------
1 2007-10-03 45
1 2007-10-05 5
1 2007-10-11 -1
1 2007-10-30 23
2 2007-03-03 3
2 2007-03-13 -5
2 2007-03-03 6
3 2007-10-03 42
3 2007-10-03 11
.....
I want to summerize the value in each group and set the date to the 1'st og that month
i.e.
nr date value
------- -------------- --------
1 2007-10-01 72
2 2007-03-01 4
3 2007-10-01 53
any help is much appreciated.
thanks
View 3 Replies
View Related
Jul 8, 1999
I have been asked to move the indexes on our membership database tables to seperate partitions on the server. This is a new concept to me and thought I could use some advice on how to go about doing it.
Thanks in advance.
Brad Keck
View 2 Replies
View Related
Jul 31, 2015
I have 3 partitions using a year grouping. Current year, previous 4 years, older than 5 years. I have two measure groups, one is a distinct count, so I actually have 6 partitions.I also use usage based optimization to build my aggregations. Should each partition have a separate aggregation or should there be one for each measure group?
View 5 Replies
View Related
Oct 27, 2015
We are currently developing an OLTP application, which will need to purge data when it becomes older than 1 hour.Rather than having a process which deletes rows periodically (and risks locking the tables), I am considering using partitioning on a rolling 15 minute window.The idea is to have 5 active partitions, with the 5<sup>th</sup> one being swapped out, merged and a new one split in. This will allow data to live to a max of 1 hour 15 mins,which is acceptable.
Actually, I will have 8 partitions; there will be 4 partitions set in the future, just to ensure when the last partition is split, there isn’t any data movement, as the newest partition will be empty.I am wondering if there will be any performance issues due to partition swapping, merging and splitting every 15 minutes? The application will have a high volume of users when live. I think this should be a better option that continually deleting from the tables.
View 6 Replies
View Related
