I'm moving set of data from one partition to another what is the best way.
what all the things need to be considered
Note: The set of data will be all from one partition to another one partition
My current query:
UPDATEtable1
SET table1.partitioncolumn = @newpartitioncolumn
FROMtable1
INNER JOIN table2
ON table1.id = table1.id
AND table1.partitioncolumn = @oldpartitioncolumn
Hi all
I am wanting to continuously monitor a source table throughout the day and as data becomes available, process it and insert it into one of a number of tables.
I have tried achieving this using a FOR LOOP and setting the halt condition such that it is not stisfiable. However, this has a couple of problems:
1) It runs in a tight loop and consequently degrades system performance enormously.
2) I can't get transactions to work. I would like each iteration of the loop to spawn a new transaction under which the tasks in the loop can run. Therefore, if one of the tasks fails during such an iteration, only the updates affected by that iteration are lost.
Ideally, I would like to be able to put a wait statement within the loop container so that it runs every couple of seconds. And would also like to implement transactions as described above.
All help is appreciated.
Jays :-)
The secondary server for my availability group was recycled. When SQL Server came back online the data movement for a database was suspended. The error log shows:
"AlwaysOn Availability Groups data movement for database 'XXXXXXXXX' has been suspended for the following reason: "system" (Source ID 5; Source string: 'SUSPEND_FROM_RESTART'). To resume data movement on the database, you will need to resume the database manually. For information about how to resume an availability database, see SQL Server Books Online."
I was able to resume data movement with no issue. I would like to understand the technical reason as to WHY the data movement was put in the suspended state and left there upon recycle. I searched for an article that would list possible reasons (BOL, Google, Bing, etc..). I just could not find much information out there on 'SUSPEND_FROM_RESTART'.
I am aware that TDE protects data at Rest and not during communication or data in motion (UNLESS you use Encrypted communication channels using SSL certs etc). Hence I am thinking of doing data export from a TDE encrypted database to a database on the instance where TDE is not enabled or supported. I believe it works and need to take care of relationships between tables.The target database is hosted on SQL 2012 standard edition on which TDE is not supported.
View 4 Replies View RelatedHi all,
I'm getting this error on expanding the Databases node in SQL Server Management Studio Express Object Explorer:
Failed to retrieve data for this request. (Microsoft.SqlServer.Express.SmoEnum)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.Express.ConnectionInfo)
Could not continue scan with NOLOCK due to data movement. (Microsoft SQL Server, Error: 601)
A similar error message also appeared when I tried opening an existing data connection within Database Explorer in Visual Web Developer Express. (However, the web application ran fine and it managed to access the database normally.)
These errors only appeared recently. Any ideas how to go about solving this issue?
Thanks in advance.
Peace be on you,
In my organization, we have a database which exceed unto 2 GB. The application on which we are working is developing since the time of asp / cb 6 and in short it is the baby of many developers. It is a kind of ERP which has no documentation.
Problem 1:
=========
Now, to reduce the size of the database we have examined some of the orphan procedures, tables, and columns. Is it possible for me to create a DTS package which create the backup of all the orphan stuff to some other database and delete it from the actual one. Moreover, We still have doubts that some of fields which we think are orphan, might not be. So we need another DTS package which rollback all the changes. (Any Idea for that)
Problem 2:
=========
We want another DTS package which would move the data from one database to another and same rollback DTS for this one. (Any Idea for that)
Problem 3:
=========
How can we use DTS programming to do these tasks ? what benefits do I got from DTS Programming over DTS Wizard.
Any help would be greatly appreciated.
We have a stored procedure that failed with: Could not continue scan with NOLOCK due to data movement
We are running with ISOLATION LEVEL READ UNCOMMITTED. There are other jobs running, some of which might be hitting these tables (all using the ROWLOCK hint - though I know that's not guaranteed), however, this stored proc would not be going near the same rows. But even if they were, we'd be happy with either the before-look or after-look. This needs to be a low-impact job and should have minimal impact on ther other jobs, so we can't take out locks. Is there any hint we can use to do this? e.g. can we tell the query to just wait until the data has stopped moving then try again?
Hi folks,
Recently i've been working on a new project that would partition a large table 2 smaller tables. I then create a view to union the 2 smaller tables(table A, B). I've been getting a strange error when i try to update, insert, delete a record through the view. "View needs partitioning column"....i find this strange. Both of my table have a cluster primary key consisting of 3 columns, and one of the 3 columns(date field) consist of a check constraint. The constraint is used to determine what record goes into which table. Am i missing anything else? The really strange part is sometime it works, and sometimes i get the error message.
Any thoughts?
Joe R.
Hi ,
I have question regard data partition view .
Please see below sample from BOL + sample of execution plane .
I would like to ask what is the way to avoid the optimizer scan tables out of the scope (I would expect that the only table for this query will be SUPPLY1)
Thanks,
Eyal
--This example uses tables named SUPPLY1, SUPPLY2, SUPPLY3, and SUPPLY4, which correspond to the supplier tables from four offices, located in different countries/regions.
USE tempdb
GO
--create the tables and insert the values
CREATE TABLE SUPPLY1 (
supplyID INT PRIMARY KEY CHECK (supplyID BETWEEN 1 and 150),
supplier CHAR(50)
)
CREATE TABLE SUPPLY2 (
supplyID INT PRIMARY KEY CHECK (supplyID BETWEEN 151 and 300),
supplier CHAR(50)
)
CREATE TABLE SUPPLY3 (
supplyID INT PRIMARY KEY CHECK (supplyID BETWEEN 301 and 450),
supplier CHAR(50)
)
CREATE TABLE SUPPLY4 (
supplyID INT PRIMARY KEY CHECK (supplyID BETWEEN 451 and 600),
supplier CHAR(50)
)
GO
--create the view that combines all supplier tables
CREATE VIEW all_supplier_view
AS
SELECT *
FROM SUPPLY1
UNION ALL
SELECT *
FROM SUPPLY2
UNION ALL
SELECT *
FROM SUPPLY3
UNION ALL
SELECT *
FROM SUPPLY4
GO
INSERT all_supplier_view VALUES ('1', 'CaliforniaCorp')
INSERT all_supplier_view VALUES ('5', 'BraziliaLtd')
INSERT all_supplier_view VALUES ('231', 'FarEast')
INSERT all_supplier_view VALUES ('280', 'NZ')
INSERT all_supplier_view VALUES ('321', 'EuroGroup')
INSERT all_supplier_view VALUES ('442', 'UKArchip')
INSERT all_supplier_view VALUES ('475', 'India')
INSERT all_supplier_view VALUES ('521', 'Afrique')
GO
/* */
SELECT * FROM all_supplier_view WHERE supplyID BETWEEN 1 and 150
I have partitions that I have filled with data. I am not trying to figure out exactly how much data the partitions contain, and therefore I will be able to see if any of them are close to hitting their autogrow conditions. If I were looking at a single unpartitioned table, then I could maybe look at the table properties to determine data and index sizes, and compare that to the size of the mdf file size, but for partitions, then I am not sure how I would query this information out. Any pointers on how this information could be queried out of the system?
View 3 Replies View RelatedHi,
I have a sql server 7 running on a machine with two disk partitions (D: and E:).
The data files xxx.mdf and xxx.ldf are stored in D:, which has very few space available. I want to copy these files to E: but I get an error saying that it is not possible to change the source file of a database. Is it possible to do it or do i have to create another data file in E: and keep the old one in D:?
Thanks in advance,
browser
Hi All,
I am trying to understand, when would I do a vertical partition in a Dimensional Data Warehouse ? What are the things I need to consider, before I take the decision?
Necessity is the mother of all inventions!
I have a table named sales, num of rows are 20. total number of rows for stor_id is 5 rows
I assume that when I create a cursor like this one which will have 5 rows from the cursor Now if I print a statement , I assume that I will have the statement 5 times( one for each row) Am I correct...
well, if this is the concept, I am still having one printed statement.. I do not know why , am I doing something wrong?
thanks for help
CREATE PROCEDURE p_cursor @ord_nbr char(4)
declare rst cursor for
select * from sales
where stor_id= 123
open rst
fetch rst
if (@@fetch_status=0)
print ' I may have 5 rows '
close rst
deallocate rst
We're very interested in having our application use SQL/e but we can't have the 4 gig limit. It makes sense to me that SQL/e simply should not be able to access a file over the network, and then you wouldn't have any reason to put a 4 gig limit on it.
At that point it becomes a very flexible alternative for remote users that need to have a large amount of data (i.e. documents, images etc.) with them.
We'd love to start building an abstraction layer so that we can support both SQL Server and SQL/e so that we can support network and remote users and not have the nightmare that is SQL Server Express installation. (care of the windows installer group's bugs...)
Thanks! Hoping for a favourable answer!
Hi
Column with TEXT datatype is not stored in the same data row any way. I am wondering if there is any performance gain to put it in a seperate table. Thanks
Does anyone have a helpful link for using the partition processing data
flow task in SSIS? I am trying to process a monthly partition
from within my package and am getting the following error:
Error: 0xC113000A Errors in the high-level relational engine. Pipeline
processing can only reference a single table in the data source view.
If anyone has used this before and could point me in the right direction, I would appreciate it.
Thanks,
Nick
I am using a WriteBack Partition to receive data from various inputs and appends any new data that I add to the WB partition.
I am able to read the data immediately in the WB partition through a Fact partition query. This is working at this point as desired.
Eventually I want to move the data from the WB partition into Fact Partition. How can I do this, manually and through automation.
What is the syntax to verify that the partition data is loaded into the correct partition.
View 0 Replies View RelatedWhen you load the data into a new partition table, can it to done online without any downtime? because I have few tables that are around 250 gigs and more.
View 5 Replies View RelatedWhat is the syntax to verify Partition data load.
View 1 Replies View RelatedI’m looking for clearity on partition switching. The idea is to use many BULK INSERT statements into table dbo.X_n in parallel and when BULK INSERT for table dbo.X_n is completed, switch dbo.X_n into dbo.bigdaddy. I think this is the fastest way to upload a couple hundred GB of data.
In learning about partition switching (in part) from The Data Loading Performance Guide under Partition SWITCH, I hear the instructions to say copy the main table exactly to become a target. But in that same step (#1), I read that we need to change the default file group of the target (dbo.X_n) from the default file group. Then it says I need to match indexes and lists the filegroup as something we need to match with the main table.
As an overview of the partition switching strategy, I think the whole point of BULK INSERT with partitioning is to have seperate files (in same group) to enable concurrent uploading where each table has its own file. Once the upload is completed to a table (dbo.X_n) then we do the partition switch into the main table (dbo.bigdaddy). The data we just uploaded doesn’t actually move, just the metadata for it.
When I read the instructions linked above, I hear “Don’t have the same filegroup on your target as the main table. You must have the same filegroup on your target as the main table.”
Where am I disconnected?
I have transactional replication setup from server A to Server B. I wanted to move the subscriber from B to C. What could be the best approach.
1. Backup the DB from Server B and restore on Server C. set the replication between A & C.
2. setup the transaction replication between A & C along between A & B. Test A& C working fine and then remove B.
If I am going with approach 2 , I have to replicate data approx. 70 GB so If I ran both the replication on Server A that will stress as 140 GB of data moving out. How do I control this large movement ? Can the replication be manual synch?
Hi All,
I have SQL server 2005 Database which is having following Replication stuff.
1. 6 merge Subscribers
2. 5 Snapshot Subscribers (Push Susbribers)
3. 3 Transactional Publisher
Due to the Performance Issue, there is need to move SQL server from the Current Server to an Higher End Server.
I want to keep all the Replication settings after movement of the Database. Can anyone tell me how to acheive this requirement?
Is there a possibility to keep the Replication settings ? Even we can have the Same System Name to the New Server.
Awaiting response...
Thanks,
Thams.
I have a heavy database , More than 100 GB only for six month .every Query on it takes me along time and I dont have enough space to add more indexes.by a way I decided to do partitioning. I create a partition function , on date filed and all Data records per month was appointed to a separate file.And is partitioning only for Future data entry?
View 9 Replies View RelatedI’m looking for clearity on partition switching. The idea is to use many BULK INSERT statements into table dbo.X_n in parallel and when BULK INSERT for table dbo.X_n is completed, switch dbo.X_n into dbo.bigdaddy. I think this is the fastest way to upload a couple hundred GB of data.
In learning about partition switching (in part) from The Data Loading Performance Guide under Partition SWITCH, I hear the instructions to say copy the main table exactly to become a target. But in that same step (#1), I read that we need to change the default file group of the target (dbo.X_n) from the default file group. Then it says I need to match indexes and lists the filegroup as something we need to match with the main table.
As an overview of the partition switching strategy, I think the whole point of BULK INSERT with partitioning is to have seperate files (in same group) to enable concurrent uploading where each table has its own file. Once the upload is completed to a table (dbo.X_n) then we do the partition switch into the main table (dbo.bigdaddy). The data we just uploaded doesn’t actually move, just the metadata for it.
“Don’t have the same filegroup on your target as the main table. You must have the same filegroup on your target as the main table.”
I'm currently stuck with a table that has 350 mil records. Querying this table is insanely slow so I had a better look at existing yearly partitioning. I already managed to partition on a month level which increased the performance/querrying a lot. I did this on the staging table where I used an alter statement to split the 2015 partition by 12 months.
However, in our project we used Data Vault. This means that we have 4 tables (hub, sathub, link, satlink), all carrying 350 mil records. The problem is that altering the partition function does not work. The server cannot handle this action. What the best way is to do this, without having to drop/reload all tables.
Hello everyone,There's an interesting SQL problem I've come across that I'm currentlybanging my head against. Given the following table that contains itemlocation information populated every minute :location_id date_created=========== ============5 2000-01-01 01:00 <-- Don't need5 2000-01-01 01:01 <-- Don't need5 2000-01-01 01:02 <-- Need7 2000-01-01 01:03 <-- Don't need7 2000-01-01 01:04 <-- Need5 2000-01-01 01:05 <-- Need2 2000-01-01 01:06 <-- Don't Need2 2000-01-01 01:07 <-- Need7 2000-01-01 01:08 <-- Needhow would you generate a result-set that returns the item's locationhistory *without* duplicating the same location if the item has beensitting in the same room for a while. For example, the result setshould look like the following :location_id date_created=========== ============5 2000-01-01 01:027 2000-01-01 01:045 2000-01-01 01:052 2000-01-01 01:077 2000-01-01 01:08This is turning out to be a finger twister and I'm not sure if itcould be done in SQL; I may have to resort to writing a stored-proc.Regards,Anthony
View 9 Replies View Relatedi have a table named table1 sitting on a a partittion scheme px.
table1 is partitioned on u_id by tens,
first partion u_id=0 to 9
second partion u_id=10 to 19
third partion u_id=20 to 29
ans so on and so forth
i have a table named table2 with
records having u_id = 13 to 16
obviously belonging to the second partion.
i want to load table2 to table1 as fast as i could with
the following requirements:
all contents of partition 2 of table1 must be deleted before loading
table2 ceases to exist after the operation
how will i do that.
thanks.
Hi All,
Can someone take a look at my code and tell me what i'm doing in wrong. The script runs fine but when i go to table property it says the table is not partitioned. Thanks for your help.
create database [mypartition]
go
--CREATE FILEGROUP
USE [mypartition]
GO
ALTER DATABASE mypartition ADD FILEGROUP Y2000_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2001_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2002_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2003_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2004_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2005_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2006_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2007_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2008_filegroup
ALTER DATABASE mypartition ADD FILEGROUP Y2009_filegroup
--CREATE FILES
USE mypartition
GO
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2000,
FILENAME = 'F:ss_datadatadetail_2000.ndf',
SIZE = 2MB)
TO FILEGROUP Y2000_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2001,
FILENAME = 'F:ss_datadatadetail_2001.ndf',
SIZE = 2MB)
TO FILEGROUP Y2001_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2002,
FILENAME = 'F:ss_datadatamdetail_2002.ndf',
SIZE = 2MB)
TO FILEGROUP Y2002_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2003,
FILENAME = 'F:ss_datadatadetail_2003.ndf',
SIZE = 2MB)
TO FILEGROUP Y2003_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2004,
FILENAME = 'F:ss_datadatadetail_2004.ndf',
SIZE = 2MB)
TO FILEGROUP Y2004_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2005,
FILENAME = 'F:ss_datadatadetail_2005.ndf',
SIZE = 2MB)
TO FILEGROUP Y2005_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2006,
FILENAME = 'F:ss_datadatadetail_2006.ndf',
SIZE = 2MB)
TO FILEGROUP Y2006_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2007,
FILENAME = 'F:ss_datadatadetail_2007.ndf',
SIZE = 2MB)
TO FILEGROUP Y2007_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2008,
FILENAME = 'F:ss_datadatadetail_2008.ndf',
SIZE = 2MB)
TO FILEGROUP Y2008_filegroup;
ALTER DATABASE mypartition
ADD FILE
(NAME = mypartition_detail_2009,
FILENAME = 'F:ss_datadatadetail_2009.ndf',
SIZE = 2MB)
TO FILEGROUP Y2009_filegroup;
--CREATE PARTITION FUNCTION
USE [mypartition]
GO
CREATE partition FUNCTION detail_part_function (varchar(10)) AS
RANGE LEFT FOR VALUES('2001','2002','2003','2004','2005','2006','2007','2008')
GO
--CREATE PARTITION SCHEME
USE [mypartition]
GO
CREATE PARTITION SCHEME detail_part_scheme AS
PARTITION detail_part_function TO
(Y2000_filegroup, Y2001_filegroup,Y2002_filegroup,Y2003_filegroup,Y2004_filegroup,Y2005_filegroup,Y2006_filegroup,Y2007_filegroup,Y2008_filegroup,Y2009_filegroup)
GO
-- Now just create a table that uses the particion scheme
USE [mypartition]
GO
/****** Object: Table [dbo].[partitioned_table] Script Date: 05/14/2008 09:44:21 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[partitioned_table](
[id] [int] NOT NULL,
[fiscal_year] [varchar](10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
CONSTRAINT [PK_partitioned_table] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON detail_part_scheme(fiscal_year)
GO
SET ANSI_PADDING OFF
I can see lot of documentation on Range Partitioning. Is there any other type of partition supported in SQL Server 2005?
For example, I have a Fact table having Billion rows. It has a column called BATCH_ID. A BATCH_ID corresponds to 10-20 Million rows and it is a running sequence number like 1,2,3 etc. (not an identity column). Is there anyway I can specify a partition function with BATCH_ID column as an int value? Will the SQL Server automatically does the partition on each int value in that case? If not, what is the best way to do it?
Thanks in advance for all help in this
Hi,
I recently installed MSSQL 2000 and sp3a onto a windows 2003 server in a test lab. I configured one big c: partion on this os and installed the db in the default location.
I need to detach the db's on this server and re-attach them onto another MSSQL 2000/sp3a server running 2003 os with a partition scheme like this:
c: = 20 gigs for the os
e: = 600 gigs for the data
I could not re-attach the db's onto the e:default path odatabase
Is there a work-around for this? This makes sense to me as to why it is not working and was an install oversight on my part but there has to be a way to overcome this delima?
Thanks in advance,
damnit
Hi
Please help me how to do the Horizontal table partition??
I have to split the table in to multiple sub tables with same columns and less rows and then I have to use each sub table.
Thanks you in advance....
Regards
LakshmiPK
Hi everybody
Does anyone know where I can find code to PROCESS a Partition using DSO.
Thank a lot.