1.6 Billion Records To Copy
Mar 20, 2008
HELP. I've inherited some less-than-stellar dessign.
I have a table:
tbMD
ObjectID uniqueidentifier not null
Tag nvarchar(50) not null
Data nvarchar(400) not null
This table has 1.6Billion records and has filled up the drive (nearlly 1 TB in the DB, with about 600GB in this one table).
I need to move this table to another drive with ZERO down time.
I have created a new filegroup on a second drive array and created a new table (tbMD_TEMP) and my thought was to copy all the data from tbMD into tbMD_Temp then drop tbMD, and rename tbMD_Temp.
So, anyone have any great ideas on how to get this done very quickly?
View 5 Replies
ADVERTISEMENT
Jan 31, 2008
hi,
I want to Display the image records of each employee of having 10 images of each.the number of records are more then one billion.......
please provid me the optimized query that can i used in SP and display in Gridview.
View 7 Replies
View Related
Jul 30, 2007
Hi;
I want to know, if a table has 1 billion recoreds then it will work eaisly with .net objects like datasets etc?
if not, what is the normal amount of data for sql server which is easy to manage for system.
Adnan
View 2 Replies
View Related
Mar 28, 2008
I will be receiving data from a company called Loan Performance, that has one file/table that will hold 1 billion rows. They send data by period, and I plan to load the data via BCP via NT/DOS scripts. The 1 billion rows represents data for 200+ periods.Are the following design plans feasible1. Partition table by period value, I'm not sure of the max number of partitions per table in 2005, but I think we have periods data back to 1992 and a new one gets created every month, so the possibility of having > 1000 partitions exists. I plan on just pre-creating partitions for future data, instead of dynamically creating when a new period is sent.2. Load data via BCP in DOS shell scripts that will drop index (by partition), BCP in data, and they re-create indexes by partition, is this possible ? and will I see a performance increase as opposed to one huge table (I'm pretty much sure I will). There is usually one periods data present per day, but sometimes the vendor resends all data (would get loaded on weekend).I'm a bit unsure of where to start being I never worked with this amount of data. I worked with partitioning in Oracle a long time ago.I plan on having an 2XQuadCore 2.66Ghz CPU with 32GB of RAM and SQL2005EE 64Bit connected to 1 Terabyte SAN Disk.Thanks all,PMA
View 7 Replies
View Related
Aug 28, 2004
I have 1 table. In that table there are thousands of records. I have an ID that is autonumbered for a unique id.
What i need to do is at any given time select as many records from that table that have a certain Forein Key in it. Take all those records and copy them back into the same table but at that time i need to change some of the values including the Forien Key and the name.
Is there a bulk way to do this without having to loop? If so can you show me.
If not can someone show me the best way to get all the records and copy them back into its own talbe with a new Forien Key and Name.
View 2 Replies
View Related
Aug 19, 1999
After moving our database between servers, one table had a hiccup in it's
identity column. The number jumped from 761 to 1886863475 on the next
insert. This is a production server and I didn't catch it until yesterday.
So I have several days worth of numbers in a central table. I can't clean
them out, but I wondered if I could do the following:
DBCC CHECKIDENT ('table_name', RESEED, 800)
I tested it on our development server and it doesn't seem to cause any
problems. I know that when we get to 1.8 billion on the identity column
again we'll get an error, but I'm willing to risk it.
A few questions. 1) Am I insane for trying this? 2) Has anyone else ran into
a similar problem and what did you do to fix it? 3) If I do this, is there
anything in particular to watch out for? 4) What caused the jump in the
first place?
My other option is to change the datatype from int to decimal(19) or
something along those lines. This will upset our programmers and I'll have
to change all my foreign keys (not a big deal, just a pain).
Any input at all here would be appreciated.
Thanks,
Grant
View 1 Replies
View Related
Aug 31, 2001
Hi i am looking for some advice about sql databases and wondering if anybody can help me.
First the Background.
I am in the process of writing a payroll program that uses SQL as a data store. However rather then creating individual databases for each payroll that can be ran on the system i have created a core payroll table that generates a unique id for each payroll and then reference that thoughtout the system.
Now The Problem
The problem I now have is when I work out the final pay for the employees I want to make a quick copy of the various records just in case I need to undo the net pay calculations for adjustments to the data, it is effectively rolling back the transactions to the database but I want to be able to keep them till I decide that everything is correct.
I cant take a snapshot of the entrie database because the payrolls stored within may be in all sorts of states.
I need to keep the records for just the payroll I'm working on.
Any Ideas or Solutions !!!!!
Thanks
EJ Gibbons
"Where poets speak there hearts, then bleed for it" - U2
View 1 Replies
View Related
May 24, 2008
I have 2 tables.
TABLE 1: AttachmentTypeDefaults (fields: DefaultId (pk), AttachmentType)
TABLE 2: AttachmentType (fields: AttachmentTypeId (pk), CompanyId, AttachmentType)
The @CompanyId variable has to be the IDENTITY of a new record that was just INSERTed into another table just before (in this new SP.)
What I want to do is loop thru the AttachmentTypeDefaults table and INSERT every record into the AttachmentType table and use the @CompanyId for every new record. (AttachmentTypeId will create itself.)
How would I do this?
Sounds easy but I am new.
I would like it to be compatible with SQL Server 2000 also.
I tried a query like this, but I don't think I'm close...
declare @MyNewIdentity int
SET @MyNewIdentity = SCOPE_IDENTITY()
-- Now INSERT default attachments
INSERT INTO AttachmentType(CompanyId, AttachmentType)
VALUES(
SELECT @MyNewIdentity AS CompanyId, AttachmentType
FROM AttachmentTypeDefaults)
View 6 Replies
View Related
Feb 29, 2012
So I just got an email from Production Support saying an hour and a half downtime is unacceptable to move a half billion rows between 2 partitions because I am moving a clustered index and space is a consideration.
I can not use partition switching because the clustered index is changing.
This is what I am doing...
1. I am creating a new table with the new cluster on a new partiton
2. I am moving the records in 5K set based batches by doing a range search on the existing clustered index on the existing table.
3. I then reapply all of the nonclustered index from the original table to the new one.
4. I do a sp_rename swap out.
The same way I have done this many times before. Is there some new secret special sauce (other than partition switching) I can use?
View 14 Replies
View Related
Mar 30, 2007
Hi everyone. I hope that my question isn't too broad, but here it goes...
I am trying to figure out the best way to scale a SQL Server database so that it can handle a billion simultaneous users querying the same tables, and can easily scale to handle many billions of simultaneous users. The database must also have 999.99% availability. The number of licences and amount of hardware needed is not an issue.
Thanks in advance.
View 15 Replies
View Related
Aug 5, 2014
It was an interview question, what's the best way to update a table with 10 billion rows.
View 16 Replies
View Related
Jun 4, 2004
Im trying to copy details from a specific header as details of a different header (eg. all sales items from invoice #10 copied as sales items of invoice #11).
So far I have two stored procedures:
1) sp_copyDetailsOne
/*Create a recordset of the desired items to be copied*/
CREATE PROCEDURE sp_copyDetailsOne @invoiceIdFrom INT, @outCrsr CURSOR VARYING OUTPUT AS
SELECT itemId, itemPrice, itemDescription, itemQuantity
FROM tblSalesItems
WHERE (invoiceId = @invoiceIdFrom)
OPEN @outCrsr
2) sp_copyDetailsTwo
CREATE PROCEDURE sp_copyDetailsTwo @invoiceIdFrom INT, @invoiceIdTo INT
/*Allocate a cursor variable*/
DECLARE @crsrVar CURSOR
/*Execute sp_copyDetailsOne to instantiate @crsrVar*/
EXEC sp_copyDetailsOne @invoiceIdFrom, @outCrsr = @crsrVar OUTPUT
/*Loop through the recordset and insert a new record using the new invoiceId*/
FETCH NEXT FROM @crsrVar
WHILE @@FETCH_STATUS = 0
BEGIN
/*Run the insert here*/
INSERT INTO tblSalesItems
(invoiceId, itemId, itemPrice, itemDescription, itemQuantity)
VALUES (@invoiceIdTo , 5, $25.00, N'Black T-Shirt', 30)
/*Fetch next record from cursor*/
FETCH NEXT FROM @crsrVar
END
CLOSE @crsrVar
DEALLOCATE @crsrVar
My question comes on the Insert of sp_copyDetailsTwo, as you can see the values are hard coded and I want them pulled from the cursor. However I don't know how to do this - do I need varables or can I access the cursor values directly in my VALUES clause? Or is this whole approach needing overhauled. Any advice is welcome.
Thanks
View 2 Replies
View Related
Sep 24, 2012
In a job of migrating from an old database to a new one (with other structure, other server, other version) i'm copying from the source old tables and inserting into the new destination tables. The problem is that some records have inconsistencies (of any kind) and thus are not inserted due to foreign key, not null, etc validation. When a problem occurs none record is copied! and there is my question: How can i perform the copy in wich it copies the good records (without inconsistencies) and leave aside the bad records. I also want to know wich were not copied and better if in the copy process those were put in a temp table or exported to excel for further analisys o its data.
I'm using this model of "migration":
BEGIN TRY
INSERT INTO DESTINTATION_TABLE (
col_d1,
col_d2,
col-d3,
...)
SELECT
col_s1,
dbo.some_function(col_s2),
col_s3 * 100,
...
FROM SOURCE_TABLE join <other_table> ... where <some filters>
END TRY
BEGIN CATCH
print ERROR_MESSAGE()
END CATCH
(for now, with try/catch I've only get to know the error occurred, if some)
View 5 Replies
View Related
Dec 11, 2013
I have a parts table which has partid (GUID) column and parentpartId (GUID) column. Need to copy the records to the same table with new GUIDs for partids. How to do that? cursor or temp tables?
View 5 Replies
View Related
Jul 20, 2006
How could I go about doing the following:
I have a column with lets say
AHA1
AHA2
AHA3
AHE1
AHE2
ARL1
ARL2
How could I make a column next to it that is basically the same but cuts off anything after the third letter? So this new column would look like:
AHA
AHA
AHA
AHE
AHE
ARL
ARL
Thanks!
View 4 Replies
View Related
Oct 24, 2007
I have a table on an offline database having let say 2000 rows and same table on an online database with let say 3000 rows. Now I want to copy just those records (1000) from online database that are not in the offline database table. Also, both tables have same name, same schema etc.
View 3 Replies
View Related
Apr 6, 2008
Hi,
I already submitted this type of question before and i receive reply. But unfortunately i found out errors when performing on my system.
My problem regarding to this one:
Suppose i have two databases with same tables with different records and I would like to copy the records from one database to another data and vice-versa. So that both the tables contains same number of records inside the tables.
Example:
Database1 (EmployeeTable) contains 6 records.
Database2 (EmployeeTable) contains 10 records.
It should copy only those records which is not present in each other database. No duplicate records.
Before i was recommend to use Primary key, if it is not present use index.
Hope this time i could solve my problem.
Thanks.
Kashif Chotu
View 1 Replies
View Related
Aug 26, 2014
I have been asked to give the users the ability to copy a set of records within the database. The current setup is
Master Table: JSA1
JSAID: (PK, int, not null)
JOBTITLE(nvarchar(200), null)
PlantNumber(int, not null)
Detail Table: tblSteps
STEPID (PK, int, not null)
JSAID (FK, int, not null)
StepNo (int, null)
BasicSteps (nvarchar(200), null)
DBPhoto(varbinary(max), null)
The plant number field is a location based field that the application uses to filter/select data on for the end users. What they want to be able to do is to select a record, select another location from a dropdown list and then click a button that duplicates the master record and the detail records to the new location.
I am thinking that a stored procedure passing the JSAID and new Location number to do it, I am just not sure how to get the new ID when I go to copy the detail records.
View 4 Replies
View Related
Apr 22, 2015
I am using a SQL Server Agent jobs that run each morning to update the records in a table to match what they should be for that day. I built them and tested it using a test table called "testtable1". It worked fine. But when I switched over to our production table, it fails saying the table has to be decaled. What would be the difference. The production table has a "@" in front of the name, is that causing issues?
USE [Live_build]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
BEGIN
DELETE
FROM @ZIPLIST
INSERT INTO @ZIPLIST
SELECT * FROM tblZip3DSWed;
END
View 4 Replies
View Related
May 24, 2015
I am having three table tblTest,tblTestQuestion,tblAnswers
Each test can have multiple question and each Question can have multiple answers.
Now I am already having records in database. I wants to create clone copy of existing test except testdetails in tblTest because the test will be unique, and then insert questions and answers into their respective tables.
I was trying to create SP but stuck.
Please find below tables structure
[code="
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[tblAnswer](
[Code].....
View 5 Replies
View Related
Jul 9, 2007
I have an application which stores records on a local SQL Express and I need to move the records to a SQL 2000 database. I have the SQL 2000 server linked in the Express Management Console (under Linked Servers). I'm trying to use a stored procedure to accomplish this, but get an error "Invalid object name 'ngtxa4-rsmsz-01.newpurchase.tblRequest'." Express uses a table named tblTRequest in the TempPurchase database, while 2000 uses a table named tblRequest in a NewPurchase database. Here is the stored procedure I'm using:
USE [tempPurchase]
GO
/****** Object: StoredProcedure [dbo].[InsertRequestToMain] Script Date: 07/09/2007 08:54:56 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[InsertRequestToMain]
AS
BEGIN
INSERT INTO [ngtxa4-rsmsz-01].newpurchase.tblRequest
(fldRequestDate, fldRequiredBy, fldUserID, fldWorkAreaID, fldVendorID, fldClassID, fldEstimate, fldMemo, fldStatusID, fldStatusDate, fldSystemID,
fldtmpRequestID, fldUpdateCode)
SELECT fldRequestDate, fldRequiredBy, fldUserID, fldWorkAreaID, fldVendorID, fldClassID, fldEstimate, fldMemo, fldStatusID, fldStatusDate, fldSystemID,
tmpRequestID, UpdateCode
FROM tblTRequest
WHERE (fldHold = 0)
END
Any assistance with this would greatly be helpful. Thank you.
View 2 Replies
View Related
Apr 7, 2006
How can I copy a table from my sql server 2000 db to my sql server 2005 express edition?
I have a project in VS.NET 2005 and I have a db in App_Data folder. However, when I look into that folder, there is nothing visible. I now need to copy a table from my existing sql server 2000 to my db located in my project's App_Data folder.
Any help would be appreciated..
Regards,
View 1 Replies
View Related
Oct 22, 2015
I am copying data from database to an excel file through SSIS. database is MS SQL 2005 and BIDS is also 2005.However, the job doing this task fails every time.As per investigation, the result of the query is more than 100,000 rows and we know that excel has a limit of 65000 rows of data.Is there a way of setting the limit up? or something? a better approach maybe so that everything will be copied to the excel file successfully.
The PrimeOutput method on component "Source - Query" (1) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure. End Error Error: 2015-10-22 04:34:58.05 Code: 0xC0047021
Source: Data Flow Task
Description: SSIS Error Code DTS_E_THREADFAILED.
Thread "SourceThread0" has exited with error code 0xC0047038. There may be error messages posted before this with more information on why the thread has exited. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 4:30:00 AM Finished: 4:35:05 AM Elapsed: 304.844 seconds. The package execution failed. The step failed. "
View 1 Replies
View Related
Jan 20, 2007
Hi allI have two tables in SqlServer with Exactly Same Structure,I want to Copy all Records fromone of them to another one.I came across to "Insert....select..." statement But i have two problem
1) I don't know any thing about Columns name!!! i just know they have same structure and as far as i know , "Insert...select..." need the Column list to operate correctly, am i right?
2) these two table have One Prinary Key column with IDENTITY feature.
Any Help Greatly appriciated.Regards.
View 6 Replies
View Related
Sep 11, 2015
I've got to copy several records (almost a 100) in a table from one instance of SQL Server 2008 R2, to another instance of SQL Server 2008 R2, but on a different server. The table structure is identical. I've searched online and found examples of doing a bulk insert into a table from a .CSV file. However I don't know of a way of exporting records to a .CSV file using a SELECT statement.
I've heard of linked servers, but at least as far as I know linking the server would require administrative privileges that I don't have on either machine.
View 9 Replies
View Related
Mar 20, 2014
writing the query for the following, I need to collapse the continuity. If the termdate for an ID is one day less than the effdate of the next id (for the same ID) i need to collapse the records. See below example .....how should i write the query which will give me the desired output. i.e., get min(effdate) and max(termdate) if termdate is one day less than the effdate of next record.
ID effdate termdate
556868 1999-01-01 1999-06-30
556868 1999-07-01 1999-10-31
556869 2002-10-01 2004-01-31
556872 1999-02-01 2000-08-31
556872 2000-11-01 2004-01-31
556872 2004-02-01 2004-02-29
output should be ......
ID effdate termdate
556868 1999-01-01 1999-10-31
556869 2002-10-01 2004-01-31
556872 1999-02-01 2000-08-31
556872 2000-11-01 2004-02-29
View 0 Replies
View Related
May 11, 2006
Hi,
I set up DB mirror between a primary (SQL1) and a mirror (SQL2); no witness. I have a problem when I issue command:
alter database DBmirrorTest
Set Partner = N'TCP://SQL2.mycom.com:5022';
go
The error message is:
The remote copy of database "DBmirrorTest" has not been rolled forward to a point in time that is encompassed in the local copy of the database log.
I have the steps below prior to the command. (Note that both servers' service accounts use the same domain account. The domain account I login to do db mirror setup is a member of the local admin group.)
1. backup database DBmirrorTest on SQL1
2. backup database log
3. copy db and log backup files to SQL2
4. restore db with norecovery
5. restore log with norecovery
6. create endpoints on both SQL1 and SQL2
CREATE ENDPOINT [Mirroring]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = PARTNER)
7. enable mirror on mirror server SQL2
:connect SQL2
alter database DBmirrorTest
Set Partner = N'TCP://SQL1.mycom.com:5022';
go
8. Enable mirror on primary server SQL1
:connect SQL1
alter database DBmirrorTest
Set Partner = N'TCP://SQL2.mycom.com:5022';
go
This is where I got the error.
The remote copy of database "DBmirrorTest" has not been rolled forward to a point in time that is encompassed in the local copy
Thanks for any help,
KT
View 8 Replies
View Related
Feb 28, 2008
Hi!
I did:
alter database mydb set single_user with rollback immediate;
exec sp_detach_db @dbname='mydb', @keepfulltextindexfile='true';
then I tried to copy files to new location on other drives, same server but got
>>Cannot copy <myfile>: Access is denied
Make sure the disk is not full or write-protected and that the file is not currently in use<<
I also tried rename of file without success.
I also tried with db service stoppet (not preferred) without success.
How to find out, which process locks the files?
Best regards
View 7 Replies
View Related
Aug 30, 2005
How do I transfer/copy the stored procedures in my Test DB to my LIVE DB? IT won't allow me to export keeps giving me an error.
View 4 Replies
View Related
Nov 4, 2006
Hello,
if i have a given database (a model) and i want to copy this database in the same database instance. Is it ok to copy the mdf and ldf file and attach the files with a new database name in the same instance.
Or is the datebase name part of the .mdf file?
Regards
Markus
View 6 Replies
View Related
Jan 23, 2012
I am attempting to use the copy wizard to copy databases from SQL Server 2005 to SQL Server 2008 R2 w/ FP1.
The copy fails with a login failure to SQL Server 2005. I have a user id & password under Windows for both servers. I have a user id and password under SQL security with the called for admin security rights.
The 2005 server has two instances, 20 databases, two dozen maintenance plans, and over a hundred users. I really would like to use the utility so I don't have to recreate everything manually.
View 1 Replies
View Related
Feb 1, 2007
Hi~,
Before implementing memory based bulk copy insert with IRowsetFastLoad interface of SQL Server 2005 OLE DB provider, I want to know some considerations.
- performance : compared with T-SQL's "BULK INSERT ..." and bcp utility
- SQL Server's resource usage : when running memory based bulk copy, server resource's influence
- server side action(behavior) : when server is busy, delayed-update means IRowsetFastLoad::Commit(true) method can insert right after?
- row-count : The rowcount limitation can be inserted by IRowsetFastLoad::InsertRow() method before IRowsetFastLoad::Commit
- any other guide lines
View 1 Replies
View Related
Mar 18, 2014
I have a situation where deleting old records is blocking updating latest records on highly transactional table and getting timeout errors from application.
In details, I have one table called Tran_table1 in OLTP database. This Tran_table1 is highly transactional table, it will receive data for insert/update continuously
While archiving 2 years old records from Tran_table1 into Tran_table1_archive in batches(using DELETE OUTPUT INTO clause), if there is any UPDATEs on Tran_table1,these updates are getting blocked and result is timeout errors in application.
Is there any SQL Server hints to avoid blocking ..
View 3 Replies
View Related
