Detecting && Deleting Duplicates In Batch Vs Proc
Aug 22, 2006
I know how to detect & delete dups/or >dups in test with a select clause, this works fine in a small table, but if the table has a million rows say, it sounds like a proc would be faster: my question is: How do I display those rows in a proc for detecting what the problem is. The print stmt. doesn't seem to work and I wondered if I had to go through the process of building an output stream. The proc creates okay but I'm stuck after that part.
thx
Kat -- very rough code below
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_NULLS ON
GO
create proc dupcount
@count int
as
set nocount on
select categoryID,
CategoryName,
Count(*) As Dups
from Categories
group by Categoryid, CategoryName
having count(*) >1
set @count = @@Rowcount
print convert(varchar(30),CategoryID) + ' ' + convert(varchar(30),@count) + CategoryName
/*set @count = @@RowCount
IF @count > 1
print convert CategoryID, CategoryName, convert(varchar(30),@count)*/
View 18 Replies
ADVERTISEMENT
Apr 17, 2001
Hi,
does anyone out there know how to delete dulicate records in a table? I've looked in BOL and I can't seem to find it. Please advise.
thanks, mark
View 2 Replies
View Related
Sep 21, 2004
Hey all.
I have a table with 100,000 plus records in it, and some are duplicates. Is there any way to delete one of them and not the other. For instance, if I duplicate the table I could run this query.
<cfquery name="query1" datasource="datasource">
DELETE DISTINCT
FROM tablename
WHERE FirstName in ( SELECT FirstName from tablename1 where tablename1.FirstName = tablename.FIRST_NAME AND tablename1.LastName = tablename.LAST_NAME AND tablename1.State = tablename.STATE)
</cfquery>
However, it doesn't work. I know the distinct is not correct. But does anyone know how to achieve this, I have looked all over, and everything I try deletes both records. I was thinking of using some kindof count statement, but it still deletes both of them. Please help. Thanks
View 1 Replies
View Related
Jul 25, 2006
Hello,I have a stored procedure that deletes duplicatesin one table:.....ASBEGINDELETE FROM mytableWHERE Id IN(SELECT Max(id)from mytablegroup by date, idsenshaving count(*) >1)ENDsometimes it happens that I have >2 rows with duplicated values.How can I write a new stored procedure that delete all rows withduplicated infomrations (leaving only one row with those values)?ThanksM.A.
View 3 Replies
View Related
Jan 6, 2014
I need to remove duplicate data from around 25 tables. I want to use a while loop to go through all tables. If I list out all of the column names the query runs fine, but since there are 25 tables some with 50 plus columns I was hoping to use something like the following, which errors out because my sub queries return more than one result.
SELECT q.* from
(Select ROW_NUMBER() OVER ( Partition BY (SELECT [name] AS [Name] FROM syscolumns
WHERE id = (SELECT id FROM sysobjects
WHERE type = 'U'
AND [Name] = 'Orders')
)
Order by (select top 1 [name] AS [Name] FROM syscolumns
[Code] ....
View 9 Replies
View Related
Jul 20, 2005
Suppose that I have a table that contains a lot of records that areidentical except for an id field and a date-time-stamp field. ForexampleId Unit Price DTS1 A 1.00 Date 12 A 1.00 Date 23 A 1.00 Date 34 B 1.25 Date 45 B 1.50 Date 56 B 1.50 Date 67 C 2.75 Date 78 C 2.75 Date 89 C 2.75 Date 910 C 3.00 Date 10I want to cull out records that are duplicates in the units and pricefields. I want to use the max DTS as the criteria for which record ina set of "duplicates" will remain. So, If I get the right query, Ishould return withId Unit Price DTS1 A 1.00 Date 14 B 1.25 Date 45 B 1.50 Date 57 C 2.75 Date 710 C 3.00 Date 10Is this possible using a single query? If so, how? I am sure that Ican do this using code, but it will involve a bunch of loops andprocess time. I would prefer a cleaner, more elegant way. Thanks forany help.Jerry
View 4 Replies
View Related
Aug 9, 2006
Hello all. I have a table with two coulmns CODE and DESCRITPION. Can anyone suggest how i can go about deleteing the entire record where two or more codes are the same?
Thanks.
View 4 Replies
View Related
Sep 20, 2006
I have asked this question before and got some great answers, I just wanted to ask this again. I can detect dups easily, is there a way to get a total number of all dups so that I can delete a certain number at a time, then check the total again for verification that that specific number of dups are gone? I'm still using 2000. If I delete dups, won't it delete the 1 row I want to keep? And I do have a unique identity column.
thx,
Kat
View 3 Replies
View Related
Nov 15, 2007
I've got a table with a unique column, "id". I've got the id values of about 300,000 records. These records need to be DELETEd from this table. Is there a way to do this in batch? I can't imagine the only way to do it is:
DELETE FROM Table WHERE id = 1 OR id = 2 OR id = 3... OR id = 300000
View 10 Replies
View Related
Jul 27, 2015
How would you differentiate duplicates in a table without deleting.
I have a table name ANGEL that contains duplicate value and i want to append a letter V to the duplicates instead of deleting them.
View 5 Replies
View Related
Oct 16, 2015
In another forum post, a poster was deleting large numbers of rows from a table in batches of 50,000.
In the bad old days ('80s - '90s), I used to have to delete rows in batches of 500, then 1000, then 5000, due to the size of the transaction rollback segments (yes - Oracle).
I always found that increasing the number of deleted rows in a single statement/transaction improved overall process speed - up to some magic point, at which some overhead in the system began slowing the deletes down, so that deleting a single batch of 10,000 rows took more than twice as much time as deleting two batches of 5,000 rows each.
good rule-of-thumb numbers (or even better, some actual statistics and/or explanations) as to how many records should be deleted in a single transaction/statement for optimum speed? 50,000 - 100,000 - 1,000,000 or unlimited? Are there significant differences between 2008, 2012, 2014?
View 9 Replies
View Related
Jan 27, 2006
Is there a way to delete records from table passing parameter as tablename? I won't to delete all records from a table dependent on table selected. i'm trying to do this with stored procedure...Table to delete depends on the checkbox selected.
Current code(works)
Public Function DelAll()
MZKDB = MZKHRFin
If sloption = "L" Then
sqlConn.ConnectionString = "Server=" & MZKSrv & ";Initial Catalog=" & MZKDB & ";Integrated Security=SSPI;"
ElseIf sloption = "S" Then
sqlConn.ConnectionString = "Server=" & MZKSrv & ";User id=sa;Password=" & MZKPswd & "; Initial Catalog=" & MZKDB & ";"
End If
sqlConn.Open()
sqlTrans = sqlConn.BeginTransaction()
sqlCmd.Connection = sqlConn
sqlCmd.Transaction = sqlTrans
Try
sqlCmd.CommandText = sqlStr
sqlCmd.ExecuteNonQuery()
sqlTrans.Commit()
frm2.txtResult.Text = frm2.txtResult.Text & " " & TableName & " Prior records have been deleted from the database." & vbCrLf
SetCursor()
Catch e As Exception
Try
sqlTrans.Rollback()
Catch ex As SqlException
If Not sqlTrans.Connection Is Nothing Then
frm2.txtResult.Text = frm2.txtResult.Text & " " & TableName & " An exception of type " & ex.GetType().ToString() & " was encountered while attempting to roll back the transaction." & vbCrLf
SetCursor()
End If
End Try
frm2.txtResult.Text = frm2.txtResult.Text & " " & TableName & " Records were NOT deleted from the database." & vbCrLf
SetCursor()
Finally
sqlConn.Close()
End Try
ResetID()
End Function
If cbGenFY.Checked Then
sqlStr = "DELETE FROM FIN_FiscalYear"
TableName = "dbo.FIN_FiscalYear"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenFY()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenFund.Checked Then
sqlStr = "DELETE FROM FIN_Fund"
TableName = "dbo.FIN_Fund"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenFund()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenFunc.Checked Then
sqlStr = "DELETE FROM FIN_Function"
TableName = "dbo.FIN_Function"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenFunc()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenObject.Checked Then
sqlStr = "DELETE FROM FIN_Object"
TableName = "dbo.FIN_Object"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenObject()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenCenter.Checked Then
sqlStr = "DELETE FROM FIN_Center"
TableName = "dbo.FIN_Center"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenCenter()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenProject.Checked Then
sqlStr = "DELETE FROM FIN_CodeBook"
TableName = "dbo.FIN_CodeBook"
DelAll()
sqlStr = "DELETE FROM FIN_BudgetAccnt"
TableName = "dbo.FIN_BudgetAccnt"
DelAll()
sqlStr = "DELETE FROM FIN_Budget"
TableName = "dbo.FIN_Budget"
DelAll()
sqlStr = "DELETE FROM FIN_Project"
TableName = "dbo.FIN_Project"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenProject()
timeStepStop = Date.Now
TableName = "dbo.FIN_Project"
DispOneCounts()
End If
If cbGenProgram.Checked Then
sqlStr = "DELETE FROM FIN_Program"
TableName = "dbo.FIN_Program"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenProgram()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenGL.Checked Then
sqlStr = "DELETE FROM FIN_gl"
TableName = "FIN_gl"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenGL()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenRevenue.Checked Then
sqlStr = "DELETE FROM FIN_Revenue"
TableName = "FIN_Revenue"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenRevenue()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbGenBank.Checked Then
sqlStr = "DELETE FROM FIN_VendorBankAccnt"
TableName = "dbo.FIN_VendorBankAccnt"
DelAll()
sqlStr = "DELETE FROM FIN_VendorBank"
TableName = "dbo.FIN_VendorBank"
DelAll()
sqlStr = "DELETE FROM FIN_bankAdd"
TableName = "dbo.FIN_bankAdd"
DelAll()
sqlStr = "DELETE FROM FIN_bankTERMS"
TableName = "dbo.FIN_bankTerms"
DelAll()
sqlStr = "DELETE FROM FIN_bank"
TableName = "dbo.FIN_bank"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenBank()
timeStepStop = Date.Now
TableName2 = "dbo.FIN_bankTERMS"
TableName3 = "dbo.FIN_BankAdd"
TableName4 = "dbo.FIN_VendorBank"
TableName5 = "dbo.FIN_VendorBankAccnt"
DispTwoCounts()
End If
If cbFinAP.Checked Then
sqlStr = "DELETE FROM FIN_Period"
TableName = "FIN_Period"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenPeriod()
timeStepStop = Date.Now
DispOneCounts()
End If
If cbFinVM.Checked Then
sqlStr = "DELETE FROM FIN_vendorClass"
TableName = "FIN_vendorClass"
DelAll()
sqlStr = "DELETE FROM FIN_vendorAdd"
TableName = "FIN_vendorAdd"
DelAll()
sqlStr = "DELETE FROM FIN_vendor"
TableName = "FIN_vendor"
DelAll()
sqlStr = "DELETE FROM FIN_AddressType"
TableName = "FIN_AddressType"
DelAll()
sqlStr = "DELETE FROM FIN_VendorStatus"
TableName = "FIN_VendorStatus"
DelAll()
sqlStr = "DELETE FROM States"
TableName = "States"
DelAll()
sqlStr = "DELETE FROM Country"
TableName = "Country"
sqlStr = "DELETE FROM FIN_IndustrialCodes"
TableName = "FIN_IndustrialCodes"
DelAll()
ClearCounts()
timeStepStart = Date.Now
GenIndCodes()
timeStepStop = Date.Now
DispOneCounts()
DelAll()
ClearCounts()
timeStepStart = Date.Now
FinVendStat()
timeStepStop = Date.Now
TableName = "FIN_VendorStatus"
DispOneCounts()
ClearCounts()
timeStepStart = Date.Now
FinAddrType()
timeStepStop = Date.Now
TableName = "FIN_AddressType"
DispOneCounts()
ClearCounts()
timeStepStart = Date.Now
GenCountry()
timeStepStop = Date.Now
TableName = "Country"
DispOneCounts()
ClearCounts()
timeStepStart = Date.Now
GenState()
timeStepStop = Date.Now
TableName = "States"
DispOneCounts()
ClearCounts()
timeStepStart = Date.Now
FinVM()
timeStepStop = Date.Now
TableName = "FIN_Vendor"
TableName2 = "FIN_VendorAdd"
DispTwoCounts()
End If
If cbFinbudget.Checked Then
sqlStr = "DELETE FROM FIN_BudgetAccnt"
TableName = "FIN_BudgetAccnt"
DelAll()
sqlStr = "DELETE FROM FIN_Budget"
TableName = "FIN_Budget"
DelAll()
sqlStr = "DELETE FROM FIN_CodeBook"
TableName = "FIN_CodeBook"
DelAll()
ClearCounts()
TableName = "FIN_Budget"
timeStepStart = Date.Now
FinBudget()
timeStepStop = Date.Now
DispOneCounts()
ClearCounts()
TableName = "FIN_Codebook"
TableName2 = "FIN_budgetAccnt"
timeStepStart = Date.Now
FinCodeBook()
timeStepStop = Date.Now
DispTwoCounts()
ClearCounts()
End If
View 4 Replies
View Related
Oct 22, 2014
I have a table with 22 million Business records. I can see that there are duplicates when I group by BusinessName and Address and Phone. I'd like to place only the duplicates into a table, with a ranking, oldest business key gets a ranking of 1.
As a bonus I'd like each group to have a distinct group name (although not necessary, just want to know how to do this)
Later after I run more verifications to make sure these are not referenced elsewhere I'll delete everything with a matchRank > 1 out of the main Business table.
DROP TABLE [dbo].[TestBusiness];
GO
CREATE TABLE [dbo].[TestBusiness](
[Business_pk] INT IDENTITY(1,1) NOT NULL,
[BusinessName] VARCHAR (200) NOT NULL,
[Address] VARCHAR(MAX) NOT NULL,
[code]....
View 9 Replies
View Related
Aug 9, 2007
Hi
i have to delete the master table data without deleting the child table records,is there any solution for this, parent table has relation with the child table.
regards
vinod.t.v
View 9 Replies
View Related
Jan 26, 2015
Is there a query or a way to convert duplicates value in a column to non duplicates.
View 14 Replies
View Related
Jul 18, 2014
/****** Object: StoredProcedure [dbo].[dbo.ServiceLog] Script Date: 07/18/2014 14:30:59 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER proc [dbo].[ServiceLogPurge]
-- Purge records dbo.ServiceLog older than 3 months:
-- Purge records in small portions to avoid locking production tables
-- for a long time. The process takes longer, but can co-exist with
-- normal usage of the tables.
[Code] ...
*** Getting this error below when executing the code ***
Msg 102, Level 15, State 1, Procedure ServiceLogPurge, Line 45
Incorrect syntax near 'Failed:'.
View 9 Replies
View Related
Nov 17, 2005
Good Morning,
View 3 Replies
View Related
Aug 7, 2007
HELP,
I need to take a variable from a tabel in SQL Server pass to a Batch file and execute the batch file. Right now I can exec the batch file with XP_CMDSHELL but how can I pass the variable to the batch file and loop through all the variables.
Please help
Phil
View 4 Replies
View Related
Aug 10, 1998
Is there a TSQL function or value that is assigned to some sort of internal variable when part of a distributed transcation commits? If so, how do I access it?
Marc
View 1 Replies
View Related
Jan 10, 2005
Is there a way in detecting changes in the database?
thanks..
View 14 Replies
View Related
Oct 15, 2007
Does anyone have a good way to detect LOB datatypes for a table or index? I am doing index rebuilds with the ONLINE option and need to work around indexes with LOB datatypes in them...
Any suggestion?
View 3 Replies
View Related
Sep 27, 2007
Hello all,
Is there a preferred method for detecting SQL Server 2005 SP2? I do installation/deployment and during my install script I would prefer to skip the SP2 install if the instance is already upgraded.
I had tried to check HKLMSoftwareMicrosoftMicrosoft SQL ServerInstance NamesSQL to first find out what MSSQL.[n] our instance took.
After that I was attempting to check HKLMSoftwareMicrosoftMicrosoft SQL ServerMSSQL.[n]SetupSP but there have been cases where the key SP did not exist.
I decided I could just install it under the condition that no key existed at all, assuming since the 'SP' key did not exist, the instance was not upgraded. In tests however, after a fresh install of SP2, the key was not there.
I guess the question is further defined as: does installing SP2 place a flag that it exists anywhere else? Is it supposed to write to HKLMSoftwareMicrosoftMicrosoft SQL ServerMSSQL.[n]SetupSP?
Thanks,
-Mike
View 4 Replies
View Related
Jul 23, 2005
Can someone please point me to some code (preferably C#, but C++ or C or VBwill work) that will detect if the (local) instance of SQL Server isrunning on a machine or not? Many thanks!-- Rob
View 2 Replies
View Related
Jan 2, 2008
I am trying to use DDL triggers to detect all changes to a table.
Sometimes a user directly or (more likely) through the SQL Server Management Studio GUI renames a column via sp_rename.
I can't seem to figure out how to detect such renames using a DDL trigger. Is there any way or is this a limitation of the DDL trigger system?
View 9 Replies
View Related
Nov 23, 2007
Edit: Newer mind. I tested this query more after writing this, and now it seems to work!I hope it continues to work. In following query, else is never executed.CREATE PROCEDURE Put_into_basket( @Product_code varchar(20))ASBEGIN SET NOCOUNT ON;IF NOT EXISTS(SELECT * FROM dbo.t_shopping_basket WHERE Product_code=@Product_code) BEGIN INSERT dbo.t_shopping_basket (Product_code, Name,Price) SELECT Product_code, Name,Price FROM dbo.t_product WHERE Product_code= @Product_code ENDELSE --this part is never executed BEGIN UPDATE dbo.t_shopping_basket SET Quantity=Quantity+1 WHERE Product_code=@Product_code ENDENDGO The query should test if there is a record or row with Product_code=@Product_code. If there is not, that is the first part, one such row is inserted. Quantity has a default value of 1. Insertion works, one row is inserted. At least sort of. If there is already record, That's later part, Quantity is increased by 1. That too works, if ran separately.But when I test query, it never runs the quantity+1 part.
View 3 Replies
View Related
Nov 16, 2005
Is it possible, using VB code, to retain a user on a particular page
until the status of a certain field in a database is changed?
The scenario I am working on is that a person makes a reservation and
has to wait until that reservation has been accepted or rejected by the
administration.
Dim dsResv As DataSet
dsResv = objResv.DALgetResvStatus(resvId)
While (dsResv.Tables(0).Rows(0).Item("resvStatus") = "pending")
dsResv = objResv.DALgetResvStatus(resvId)
If
dsResv.Tables(0).Rows(0).Item("resvStatus") = "denied" Then
failure.Visible = True
LinkButton1.Visible = True
ElseIf dsResv.Tables(0).Rows(0).Item("resvStatus") = "reserved" Then
success.Visible = True
LinkButton1.Visible = True
End If
End While
This is how I tried to do it, but it doesnt seem to work.
Any suggestions?
View 2 Replies
View Related
Jun 12, 2006
Can anyone provide me with some SQL that will identify rows from a table where a varchar column named "Notes" contain Carriage Returns?
I know that with report writer SQR I can translate CR's to white space but I do not know of any Sybase function that will allow me to do the same, any ideas on this well would be appreciated.
View 3 Replies
View Related
Oct 1, 2013
I have 2 tables:
Table 1 Customers with column CustomerId
Table 2 Purchases with columns PurchaseId and CustomerId
I want to select all customers who have made just 1 purchase, what would be the sql for that? I know I can join the tables to get the customers with purchases, but I don't know how to limit it to those with just 1 purchase (and exclude all the other customers).
View 9 Replies
View Related
Apr 17, 2008
Folks,
I have a SSIS package that works fine 90 % of the time. Every once in a while it gets hung and when I notice that the package has been running for several hours I check the job activity monitor and notice that the package is in executing state for several hours..my question is ...Is there a way I can check for this incidence through a c# app ? Does the package name show up in the process task if so I can poll to see if the process corresponding to the package has been executing for > 30 mins then I can kill it.
Any suggestions would be great.
Thanks
View 6 Replies
View Related
Aug 13, 2007
I have added a new column to a table, but it is not showing up in the column mapping tab of the lookup editor.
It does show up in the preview.
Without deleting and recreating the component (with all the ensuing broken metadata fixes 'downstream' that this always entails), how can I get it to recognise the change?
In future I will do all LUs as a sql statement so at least I can control columns. This again defeats the purpose of the drag and drop environment..... If it were a harry potter character, it would be a dementor
View 5 Replies
View Related
Jul 20, 2007
I am using the Konesans Checksum transformation ( http://www.sqlis.com/21.aspx ) to detect changes in my big (many columns, type 2 SCD) dimensional table.
But I am running into collossions
The checksum transformation, sometimes misses a small change in the record, for instance when a certain flag is set or unset. Is there a more robust checksum generator? Of any other suggestions on to solve this?
thx
View 12 Replies
View Related
Feb 7, 2006
I have a .NET app recently ported from 1.1 to 2.0 with a pending feature request. I'd first like to state that I'm not asking anyone for the programmatic answer, unless of course, you'd like to provide it. I'm simply asking the best way to accomplish this task with the tools I have available to me (VS 2005, SQL 2000 and 2005). I do not consider myself a professional .NET or T-SQL programmer but I do have a good understanding of the technologies so that I can find what I need to know once I know I'm going about something the right way.
The application I wrote, from scratch, manages News, Featured Connections, and FAQs for a web site. It was designed in a multi-layer approach where there are:
items -> assigned to categories -> assigned to users
So if user Fred is granted rights to category A, when he authenticates to the app he can Add/Change/Delete any item in category A. Simple enough. But also what Fred can do is see all the other items other users have created and *schedule* any relevant item related to his web page to appear within his own items even tho he has no rights to change that item in the admin interface.
The feature request is a notification service, so that Fred gets an e-mail when someone has added a new item or edited an existing one. I don't want Fred to get an e-mail *every* time an item is created or edited, that would be big bother and would generate lots of e-mails. What I'd like to do is send Fred an e-mail sometime in the middle of the night that informs him of the items added or changed since his last login, so that at his choosing he can go into the admin interface and schedule one or more of those items for his page.
Should I just write a new console app that does this, compile it and schedule the .EXE to run at midnight? Or is there a better way to leverage the new features in SQL 2005 (extended strored procs, triggers) and .NET to do this a better way? Currently, the DB is in SQL 2000 but I have no problems moving it to SQL 2005 if that helps me in some way. Thanks in advance for your comments.
...Bill
View 1 Replies
View Related
Aug 5, 1999
How do you find out which service pack is installed on SQL Server?
We are installing sp5a but we get an error saying 'cfgchar.exe could not execute' and at the end it says ' can not install successfully'.
How can I understand If I really installed sp ?
View 2 Replies
View Related