Data Moving Between Partitions
Mar 9, 2008
I have a requirement that I need to reload the last seven days worth of data each night to ensure that we pick up late arriving and updated records. To avoid having to do updates we delete the last seven days data and reload.
I was wondering if it is possible to set up the table as a partition, paritioned on a value (OLD, NEW) or similar.
The job would set the last day in the NEW partition to be old, the theory being that this would cause the rows to move to the OLD parition, and then truncate the new partion rather than deleting. The last seven days data could then be inserted into the empty new partition.
My questions is 1. Is my theory about the data moving from one partition to another correct. 2. Can I actually truncate and individual parition, 3. Do you think it will perform any quicker. We would expect data in the range of 100K to 500K rows in the seven days and will store up to 4 years of historical data.
Thanks for your thoughts
Stapsey
View 1 Replies
ADVERTISEMENT
Oct 17, 2007
I have a table with millions of rows and about 70 columns that move through a number of states (11 possible states in all) from "New" via various states to "Processed" and eventually to "Archive" (there's a complicated state diagram that I won't bore you with)
Movement between states is based on a heap of business logic including the move to Archive (not just dates).
Different sorts of processing (querying and update both by users and overnight processing) are carried out on the data according to its state.
Maintaining the indexes for optimum performance across the board is a headache.
We have two problems in that we want better query performance and want to be able to easily switch out objects that are in the Archive state.
I had in mind partitioning the table (and its indexes) on state so that :
(a) Queries would be directed only at the appropriate partition (that is always use "where state=" as part of the query)
(b) The Archive partition could be swapped out of the table periodically
In my test setup 10 of 11 partitions are in [PRIMARY] but Archive is in a different filegroup.
Query performance is OK - execution plans look good.
However my update performance is now appalling when moving between any two states (10 times as long as on the unpartitioned table).
I understand that when you update a column which is used as a partition key it will cause the row to "move from one partition to another" as it says in another post.
Fine - because that's exactly what I want - logically.
I can also understand that moving from one filegroup (and hence the underlying file) to another must mean that the data has to physically move.
However is the data physically moving whenever you move between partitions or what's going on to cause such a degradation in performance ?
View 6 Replies
View Related
Jun 19, 2008
assuming that you have two databases, the OLTP db and the OLAP db (take not that both have the same structure -- archiving purposes)... using table partitioning, is there a way where we can move 1 partition from the OLTP db to the OLAP db???
i'm actually trying to use this example with both tables in the DB.. I tried to modify to use two databases but sql server is unable to move the partition...
ALTER TABLE [Production].[TransactionHistory]
SWITCH PARTITION 1
TO [Production].[TransactionHistoryArchive] PARTITION 2;
SlayerS_`BoxeR` + [ReD]NaDa
View 9 Replies
View Related
Jul 8, 1999
I have been asked to move the indexes on our membership database tables to seperate partitions on the server. This is a new concept to me and thought I could use some advice on how to go about doing it.
Thanks in advance.
Brad Keck
View 2 Replies
View Related
Oct 17, 2007
Hi champs!
The data i have is like this:
nr date value
------- -------------- --------
1 2007-10-03 45
1 2007-10-05 5
1 2007-10-11 -1
1 2007-10-30 23
2 2007-03-03 3
2 2007-03-13 -5
2 2007-03-03 6
3 2007-10-03 42
3 2007-10-03 11
.....
I want to summerize the value in each group and set the date to the 1'st og that month
i.e.
nr date value
------- -------------- --------
1 2007-10-01 72
2 2007-03-01 4
3 2007-10-01 53
any help is much appreciated.
thanks
View 3 Replies
View Related
Aug 1, 2006
First of all, we are using SQL Server 2005 with a SQL Mobile subscriber and we are attempting to use Data Partitions on our current database
schema which contains associative tables for many-to-many relationships.
We have two tables, a User table
and an Audit table. A user can be
assigned more than one Audit. An Audit
can be assigned to more than one User.
So an AuditUser associative table exists. If data partitions are used based on User,
then any Audits that are assigned to one or more users should be copied to the
proper partition for each User (the msmerge_current_partition_mappings table
with the proper partition_id values).
In order to insert records with such a schema, the following
steps occur in order:
Insert
new row into Audit table with new rowguidInsert
entry into AuditUser table associating the auditguid with every userguid that
is assigned this audit.
Merge replication triggers are fired on insert of the Audit
row and another one for the insert of the AuditUser row.
When the Audit row is inserted, the replication trigger follows
the following logic:
Inserts
a copy of that row into the msmerge_contents table. Evaluates
the row to determine which partition(s) this row should be copied to as
well (msmerge_current_partition_mappings table). To do this, it checks to see if the
AuditGuid is referenced in one or more AuditUser rows. Since we haven€™t inserted the AuditUser
row at this point, the trigger€™s logic doesn€™t find a partition to copy
this row to.
When the AuditUser row is inserted, the replication trigger performs
the same logic as with the Audit row, it:
Inserts
a copy of that row into the msmerge_contents table.Evaluates
the row to determine which partition(s) this row should be copied to as
well (msmerge_current_partition_mappings table). Since the row meets the criteria for one
or more partitions, it is copied to the msmerge_current_partition_mappings
table for each partition that exists.
When replication occurs, we see only the AuditUser rows
copied down to our device, and not the corresponding Audit rows. Now that we understand the triggers, it is
plain to see why. If the AuditUser row
could be inserted first, then the trigger on the Audit row would copy that row
into the proper partitions and all would work well. However, the Audit row must be inserted
first, so that foreign key relationship constraints are preserved.
It seems that the Update trigger on the AuditUser row
actually walks the relationships and copies any related child rows to the
msmerge_current_partition_mappings table.
View 3 Replies
View Related
Apr 13, 2015
Perhaps this task is not for MDS.... But another tool for rapid development & startUp - we don't have. And nevertheless....
We created table managers_plan in MDS :
year
month
id_manager (domain attr)
POSÂ (domain attr)
plan_sum_USD
plan_unit
----------------------------
Entities:
Managers ~ 800Â records
POSÂ ~ 100 000 records
managers_plan Total records for
1 year = 100K x 12 = 1 200 000
managers_plan - table partitions Â
- Will bemade ?
View 3 Replies
View Related
Sep 23, 2015
I have table named #t1 and three columns. CODE, Column1, and Column2.
create table #t1 (
CODE NVARCHAR(20),
COLUMN1 NUMERIC(18,2),
COLUMN2 NUMERIC(18,2)
)
And i have some data:
INSERT INTO #t1 (CODE,COLUMN1,COLUMN2)
VALUES ('432', 0,100),
('TOTAL FOR 432',0,100),
('4320001',0,250),
('4320001',50,0),
('4320001',0,140),
('4320001',300,0),
('TOTAL FOR 4320001',350,390),
('432002',200,0),
('432002',0,100),
('TOTAL FOR 432002',200,100)
drop table #t1
I want to have 4 column (named BALANCE). Balance must be column that represent running totals between two columns (Column1 - Column2) for each group of data. For each group total must start from zero.after total 432 it starts to count again for total 4320001 and again for total 432002. I'm using MS SQL SERVER 2014.
View 9 Replies
View Related
Feb 20, 2015
I am restoring a database with 10yrs worth of data which have monthly partitions but i would like to keep only 5yrs of data after the restore is done, what is the best/faster approach to delete the 5yrs data without deleting the partitions as that may cause the db in accessible.
View 9 Replies
View Related
Oct 8, 2015
I have table having around 100 million rows.Everyday we have an ETL process in which table will be trucnated and relaoded. Will creating a partition on the table increase the inserting speed?
View 4 Replies
View Related
Oct 23, 2007
I have 2 databases on one server that I want to consolidate into one database. I'm just learning SQL Server 2005. What is the easiest way to move my 3 tables from one database to a new one on the same server? Do I have use SSIS to do it, or can a simple query be written? I'm new so please be a little detailed in your answer. Thanks in advance for any comments.
View 2 Replies
View Related
Aug 3, 2000
I have a scenario where I need to refresh a database that is in 7.0 (converted from 6.5 database) from the original database. Is there an easy way to do this. I have tried creating a DTS package but the data never seems to make it accross.
View 1 Replies
View Related
Feb 16, 1999
Hello,
I'm having problems using the update wizard to move data from 6.5 sql server
(on another machine) to a 7.0 server sitting out a PDC. The wizard dies (and
passes me over to Dr Watson) when login fails for the 6.5 machine.
I am sure I have the right pasword (I've tried variations as well) and have updated the
hosts file so that the machine is known by it's name (I think one of the FAQ answers
suggested that)....
What other possibilities are there for moving the data ? I looked at bcp, but that seems
a rather long winded route (the data contains time stamps so I suppose there will be
a problem reading them in on the 7.0 side) ...
Many Thanks,
Paul.
View 1 Replies
View Related
Sep 4, 1998
Hi all!
What is the best way to move selected data from one database to another using SQL 6.5?
View 1 Replies
View Related
Sep 6, 2001
what would be the best way to move 59 million rows from one table to another. The table has no constraint, but has has three indexes. The table has only four columns. It will be going from SQL 2000 to SQL 2000.
Thank You
View 2 Replies
View Related
Dec 5, 2006
Hi all,
I want to transfer all the data from one of my Ms Access tables to SQL Server table, using C#.
How can i do it?
Thanks in advance,
J.Jasmeeta.
View 6 Replies
View Related
Feb 2, 2007
Hi all,
I have an Ms Access table and a MsSql table. I am running a windows service in my localhost where the data from Ms access table will be copied to Ms sql table for every one minute. Before copying the data, the Ms sql table will be flushed inorder to avoid replicates.
Now i want to copy only the latest records updated within 1 min in Ms access table, to Ms sql table.
My Ms access table
Name Id
jas 100
meena 101
viji 102
My Ms sql table
Name Id
jas 100
meena 101
viji 102
After 1 min, say 2 records are added to my Ms access table like,
Name Id
jas 100
meena 101
viji 102
bhuvana 103
pinky 104
Now i want to insert only the latest records from Ms access to Ms sql like,
Name Id
jas 100
meena 101
viji 102
bhuvana 103
pinky 104
how to do this? thanx in advance.
Jasmeeta.
View 4 Replies
View Related
Oct 10, 2007
I am wondering the best way to go about a task I have been assigned. We have two similar websites but each is located on a different network. One network is secure so it cannot be accessed on the normal WWW. The secure network will contain the master database. I need to write a program or do something with SQL server to retrieve all records from the WWW site and get them onto the secure database. I also in the future will need to update records from the WWW site if they have been updated. What is the easiest way to move data from one network to the other when I cannot connect to both databases simultaneously?
Thanks,
Matt
View 5 Replies
View Related
Jan 5, 2006
I have a web app that has been regulated to a disconnected PC. It's runing IIS and 2.0 with sql server express, but no connectivity. I have changes that are made to some of the data in the db (data, bot schema). There is one particular table that I cannot overwrite, and must extract the data.
What methods are available to do this swap of data between databases? I was thinking of doing something like this:
Track last date that remote db was updated. Upload updated database into data directory, loop through records for all affected tables, any date that was past logged date then update the record if it exists or insert new record, and then loop through the remote db and delete this records that dont exist in the updated db.
This seems intensive and slow - especially as the tables get bigger but I can't think of another solution that can be done by a user using sometime of web interface.
View 1 Replies
View Related
Jul 5, 2002
Our current sql server 7 is overloaded and I have been asked to move some data to a new box which is to be acquired. My query is as follows:
1what should be the ideal spec of a sql server to allow for windows 2000 in future. And how do I go about moving data from one to the other. I am new to sql and this is my first assignment.
Thanks
Eve
View 1 Replies
View Related
Jul 5, 2002
Our current sql server 7 is overloaded and I have been asked to move some data to a new box which is to be acquired. My query is as follows:
1. What should be the ideal spec of a sql server to allow for windows 2000 in future.
2. Is there any kind of monitoring agent available to see how heavily loaded the server is and what load each application/database is creating.
3. And how do I go about moving data from one to the other.
4. What do I do at the clients end to ensure they are able to access their database.
I am new to sql and this is my first assignment.
Thanks
Eve
View 1 Replies
View Related
May 25, 2001
I am trying to move a data and log backup file from my C drive to another drive to free my C drive. Does anybody know the easiest way to do it? Thanks.
View 6 Replies
View Related
May 27, 1999
Hi all,
This is (probably) a simple question, but I'm new to SQL Server scripting.
What I want be able to do is move data from one table in a database to another table in the same database, once one of the fields (a date field) reaches a certain value.
Specifically, we are inserting rows into a table that are stamped with an insert date and an expiry date. When the expiry date is reached, we want to move the applicable row from the original table to an identically structured table.
Ideally, we want this to be a script that is run daily.
Does anyone have any ideas or examples that we could use.
Any help is much appreciated!
Tim
View 3 Replies
View Related
Oct 15, 2001
What is the best method to move 1.500.000 rows from one table to another ??
Thanks,
Charles Roberto Boeing Mari
View 1 Replies
View Related
Nov 10, 2004
I imported a little over 9 million records into my database without "cleaning" the data.
I'm looking for suggestions on what would be the most efficient way to get all the fields to the correct types and to insert decimals on the fields that need them.
My only idea thus far is to make another table with the correct field types and append the data to the new table.
View 1 Replies
View Related
Oct 28, 2013
I have a server that will be going off-line in a couple of weeks, and in the meantime, I need to backup everything and move it to a new location. This involves a lot of data and databases.
What I have is 33 databases in MS SQL Server 2005. The target machine is MS SQL Server 2008. Each of these databases has a myriad of stored procedures and compiled functions that will need to be moved as well. Essentially the task is to completely replicate the entire MS SQL environment.
Is there a shortcut to making this happen or am I relegated to backing up each database individually and then restoring them individually?
View 4 Replies
View Related
May 21, 2008
I've been given the task of moving data from an old system to a new one. That's easy enough apart from some of the data fields do not match in the least:
Example: In the old system (tableOld) there is a field that contains text data such as 'Laboratory','Field','Market','Demo','Development', but this one field could contain one or more of these. In the new system (tableNew) there are boolean fields for each of the above.
What I need is a way to check for these and then make sure the correct field(s) gets ticked (1 entered into the field) in the new system.
Any suggestions, just trying to get my head around SSIS.
Thanks for any help provided.
View 2 Replies
View Related
Jul 23, 2005
Hi all,I have 5 databases, each about 20GB in size. I need to copy the data toa new server for reporting purposes. Initially I wanted to setuptransactional replication; however, the database schemas often change(adding or modifying tables) and in order to accomodate those changesthrough replication I would need to send a new snapshot each time achange occurs. This is very time consuming.So I decided instead to use log shipping, until I realized thereporting databases would be unavailable during the period when thetransaction log was being applied. That also will not work.So what does everyone else use? My only requirements are that thereporting databases be available during business hours and there bevery little administration when it comes to changing the databaseschema.Thanks,Josh
View 4 Replies
View Related
Jul 20, 2005
Access 2k -> SQL Server 2kMy client has an app that is A2k FE with A2k BE. They have asked meto move the BE to SQL Server.I have a bit of experience with SQL Server, and I'm happy withscripting the database etc.However, when it comes time to move the data itself, I have a teensylittle concern.Let's take our typical Customers -> Orders relation, where CustomerIDis the Foreign Key in the Orders table.Let's say I have 4 customers, but I used to have 6. The CustomerID isan AutoNum column, and they are 1, 2, 4, 6. However, when I insertthese records into the matching SQL Server table with an Identitycolumn, they will (presumably) be 1, 2, 3, 4.So what can I do about the matching orders? Is there any alternativeto this, which seeems very long winded.1. Insert the records as above, but copy the OLD Access AutoNumcolumn (called OldCustomerID) into a temporary column in the new SQLServer table for both the Customers and Orders tables.2. Insert the Orders records into the new SQL Server table.3. Run an update query thus:UPDATE OSET CustomerID = C.CustomerIDFROM Orders OINNER JOIN Customers C ON O.OldCustomerID = C.OldCustomerIDCan I do this any easier? For example, is there any way in SQL Serverto maintain the values of the AutoNum field in the Insert but havethem still be Identity fields? I know that I can import the AutoNumdata into a simple int column, and once the import is complete, turnON the identity, but is this reliable? Might I lose any dataintegrity?TIAEdward--The reading group's reading group:http://www.bookgroup.org.uk
View 4 Replies
View Related
Oct 4, 2007
Hello everyone!
I have two servers, one is 2005 standard, the other is 2005 express.
I'm using express to record test results and it will be keep recording non-stop.
So, what I want to do is I want to transfer data from express to standard using SSIS package, and as soon as data gets transferred the data to no longer exists in express.
I have a job running the "copy" package every minute, but I don't know how to delete the data in express.
Does anyone know how to help me out pleaseeeee..?
Thanks in advance.
MIKE!
View 6 Replies
View Related
Jun 25, 2006
Hi All,
I am completely new to VS2005 and .net2.0 but I am learning fast! But one thing that I am struggling with is the deployment of my website, or to be more precise the deployment of the data that was created during development on my local machine.
I have used VS2005 to create a simple personal site. I have created a single user (myself) in an admin and friends role. So far so good. The site works great on my local machine, all the default photos provided with the site display correctly etc and I can log in OK.
My remote server (ie my web hosts) provide me with a separate SQL server on which to install my databases. I have created a database on this server and configured it so that it contains all the correct tables and schemas etc. I have modified my web config file to connect to this database and when I browse to my default.aspx page it displays in my browser fine...EXCEPT that none of the data ( ie the photos and user data) is in the database so therefore no pictures display and I can't log in!!
What I can't figure out is how do I get all the data from my local app_data folder into my new remote database!!?? I can't find anything that explains how to do this. Surely this is one of the most important operations of all since data is what .net2.0 is all about!
Thanks in anticipation
View 2 Replies
View Related
Jun 24, 2004
I need to write stored procedure in SQL Server 2000 that moves data from table in SQL Server 2000 to same table in Access.
.mdb file is located on the same computer and I know its location (path).
View 2 Replies
View Related
Dec 5, 2006
Hi!
I have data in the following manner:
period course1 course2
4 NULL Eng
4 NULL Math
4 Phys NULL
4 Chem NULL
Is there any way I can show this data this way (the order is irrelevant - row1 can be joined with row3 or row4)?
period course1 course2
4 Phys Eng
4 Chem Math
4 Phys Eng
4 Chem Math
Basically, I want to be able to remove nulls.
Please let me know if any of you have come across such an issue.
Thanks so much!
-Parul
View 3 Replies
View Related
