Assigning Group Numbers For Millions Of Data

Mar 27, 2006

I have a table with first name, last name, SSN(social security number)
and other columns.
I want to assign group number according to this business logic.
1. Records with equal SSN and (similar first name or last name) belong
to the same group.
John Smith 1234
Smith John 1234
S John 1234
J Smith 1234
John Smith and Smith John falls in the same group Number as long as
they have similar SSN.
This is because I have a record of equal SSN but the first name and
last name is switched because of people who make error inserting last
name as first name and vice versa. John Smith and Smith John will have
equal group Name if they have equal SSN.
2. There are records with equal SSN but different first name and last
name. These belong to different group numbers.
Equal SSN doesn't guarantee equal group number, at least one of the
first name or last name should be the same. John Smith and Dan Brown
with equal SSN=1234 shouldn't fall in the same group number.

Sample data:
Id Fname lname SSN grpNum
1 John Smith 1234 1
2 Smith John 1234 1
3 S John 1234 1
4 J Smith 1234 1
5 J S 1234 1
6 Dan Brown 1234 2
7 John Smith 1111 3


I have tried this code for 65,000 rows. It took 20 minute. I have to
run it for 21 million row data. I now that this is not an efficient
code.


INSERT into temp_FnLnSSN_grp
SELECT c1.fname, c1.lname, c1.ssn AS ssn, c3.tu_id,
(SELECT 1 + count(*)
FROM distFLS AS c2
WHERE c2.ssn < c1.ssn
or (c2.ssn = c1.ssn and (substring(c2.fname,1,1) =
substring(c1.fname,1,1) or substring(c2.lname,1,1) =
substring(c1.lname,1,1)
or substring(c2.fname,1,1) =
substring(c1.lname,1,1) or substring(c2.lname,1,1) =
substring(c1.fname,1,1))
)) AS group_number
FROM distFLS AS c1
JOIN tu_people_data AS c3
ON (c1.ssn = c3.ssn and
c1.fname = c3.fname and
c1.lname= c3.lname)


dist FLS is distinct First Name, last Name and SSN table from the
people table.

I have posted part of this question, schema one week ago. Please refer
this thread.

http://groups.google.com/group/comp...6eb380b5f2e6de6

View 5 Replies


ADVERTISEMENT

Assigning Unique Sequence Numbers Across Different Tables

Oct 11, 2007

I have a procedure which updates a sequence number in a table such as the one below.


Seq Sequence_Id

------ ------------------
NextNum 1


This is the procedure ...

create procedure DBO.MIG_SYS_NEXTVAL(@sequence varchar(10), @sequence_id int)
as
begin

update mig_sys_sequences
set
@sequence_id = sequence_id = sequence_id + 1
where
seq = 'CSN'

return(@sequence_id)
end


The purpose of this is to generate a sequential number each time the procedure is called. This number would then be used in a number of different tables to allocate a unique id so that the id is unique across the different tables.


1). What is the most efficient way of allocating these unique ids? The tables that I plan to update will already be populated with data.

2). How would I call the above procedure from an UPDATE statement?

Many thanks,

Fred

View 1 Replies View Related

SQL Server 2012 :: Assigning Numbers For A Range During INSERT

Jun 25, 2014

Given a Table1 with two columns 'Name' with some N rows of data and another Table2 with one column 'SeqNo' with N rows, each of which contains a unique integer which can be ordered monotonically, I want to do an INSERT into some Table3 with two columns 'Name' and 'SeqNo' such that each INSERT'd row gets one of the unique integers.

E.g. -

Table1 contains 'Fred','Tom','Mary','Larry'
Table2 contains 6000978,6000979,6000980,6000981

INSERT INTO Table3
SELECT Table1.Name
Table2.SeqNo
FROM Table1

And I want to get Table3

'Fred',6000978
'Tom'.6000979
'Mary',6000980
'Larry',6000981

How can I reference Table2 so that Table2.SeqNo will 'line up' properly? Note that the ordering of the SeqNo values isn't mandatory as long as each SeqNo is assigned to one and only one row.

On edit: Table2 isn't required, it's just the way I started thinking about it. It would be nicer to just have two integer vars, @StartSeqNo = 6000978 and @EndSeqNo = 6000981 for he example above. Either way is fine.

View 3 Replies View Related

SQL Server 2012 :: Assigning A (Group ID) Based On Consecutive Values?

Jul 31, 2014

I have a data set that looks something like like this:

Row# Data
1 A
2 B
3 B
4 A
5 B
6 B
7 A
8 A
9 A

I need wanting to assign a group ID to the data based on consecutive values. Here's what I need my data to look like:

Row# Data GroupID
1 A 1
2 B 2
3 B 2
4 A 3
5 B 4
6 B 4
7 A 5
8 A 5
9 A 5

You'll notice that there are only two values in DATA but whenever there is a flip between them, the GroupID increments.

View 2 Replies View Related

Count Top N Group Numbers

Jun 27, 2007



I have a group in my report. This group is showing only the Top 50 results. Inside of group row is a total column. I would like to display the total of the totals in the table footer. Something like this:



This is my data:



Creature Kind Number

Frog Amphibian 3

Cat Feline 10

Lizard Amphibian 20

Cow Mammal 8



Group is on Kind. Limited to the Top 50 Sum(Number). So I want the report to look like this:



Amphibian 23

Feline 10

Mammal 8



Total Creatures 41



I can't get the correct Total Creatures because the data is limited to the Top 50 Sum(Number). So right now Total Creatures is adding up to be every creature in the database. I just want the Total Top 50 Sum(Number) Creatures.



I hope that makes sense.



Sarah

View 1 Replies View Related

How To Put Serial Numbers For Results In Group

Apr 4, 2012

I need to put the serial numbers for results in group in SQL Server 2000. Please see below:

My input:
procedureid procname
1 A
1 B
2 A
2 B
2 C
2 D
3 A
3 B
3 C

Output I need:
procedureid procname serial_num
1 A 1
1 B 2
2 A 1
2 B 2
2 C 3
2 D 4
3 A 1
3 B 2
3 C 3

Here is my table:

create table po(
procedureid int,
procname varchar(10),
)
insert into po values (1,'A')
insert into po values (1,'B')

[Code] ....

View 11 Replies View Related

Bulk Insert Data In Millions - Lock Issue

Oct 11, 2005

Hi all,Hope there is a quick fix for this:I am inserting data from one table to another on the same DB. Theinsert is pretty simple as in:insert into datatable(field1, field2, field3)select a1, a2, a3 from temptable...This inserts about 4 millions rows in one go. And since I had the'cannot obtain lock resources' problem, several methods were suggestedby some web sites:1) one to split the insert into smaller chunks (I have no idea how Ican spit a insert to insert only n records at a time..)2)to use waitfor - which I did but did not fix the error.3)use bulk insert (in t-sql) - I dont know how to do this?As I see I am simply trying to move data from one table to another(ofcourse lots of data) in SQL Server 2000 and I dont see one simplesolution to the locking problem.any ideas on how best I can do this will save my day!thanks all.

View 9 Replies View Related

SQL Server 2008 :: Getting Row Numbers For Each People Group?

Feb 18, 2015

I have a set of rows in a table like for example

Client ID Client Name Date Score
1 Smith 12/31/2014 25
1 Smith 10/15/2014 45
2 John 08/11/2014 55
2 John 06/18/2014 15
3 Rose 04/15/2014 12
4 Mike 07/23/2014 28
5 Mary 01/5/2014 56
6 Lisa 08/1/2014 54
6 Lisa 05/10/2014 34

Now I want to use Row Number function or any way where I can get the result as below

Client ID Client Name Date Score RowNo
1 Smith 12/31/2014 25 1
1 Smith 10/15/2014 45 2
2 John 08/11/2014 55 1
2 John 06/18/2014 15 2
3 Rose 04/15/2014 12 1
4 Mike 07/23/2014 28 1
5 Mary 01/5/2014 56 1
6 Lisa 08/1/2014 54 1
6 Lisa 05/10/2014 34 2

View 3 Replies View Related

Passing A Coma Delimited Group Of Numbers To A Collection For Sql

Sep 13, 2005

I have a sql statement and one of the arguments I want to pass is a comma delimited set of numbers.  It keeps getting turned into a string.  How do I keep that from happening.  Here is kind of what it looks likeSelect FirstNamefrom Userwhere NameID in (5,6,7)or Select FirstNamefrom Userwhere NameID in (@NameIDList)There is no error code just nothing returns.  If I take out the @ANameIDList and put the values I want, it returns the correct results.Thanks,Bryan PS the link to the original thread it here http://forums.asp.net/1046154/ShowPost.aspx

View 1 Replies View Related

SQL 2012 :: Data Size Of Table Is Not Reduced Even After Deleting Millions Of Rows

Sep 21, 2015

I have deleted nearly 30 million rows from a table. But however when I used the sp_spaceused command to calculate the data occupied by the table I don't see any difference in the data size of the table. In fact the data has increased to few MBs after the deletion, but not much.

View 8 Replies View Related

Assigning Data From An SQL Query To A Variable

Apr 27, 2006

Hello all,
for a project I am trying to implement PayPal processing for orders
public void CheckOut(Object s, EventArgs e)    {        String cartBusiness = "0413086@chester.ac.uk";        String cartProduct;        int cartQuantity = 1;        Decimal cartCost;        int itemNumber = 1;        SqlConnection objConn3;        SqlCommand objCmd3;        SqlDataReader objRdr3;        objConn3 = new SqlConnection(System.Web.Configuration.WebConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString);        objCmd3 = new SqlCommand("SELECT * FROM catalogue WHERE ID=" + Request.QueryString["id"], objConn3);        objConn3.Open();        objRdr3 = objCmd3.ExecuteReader();        cartProduct = "Cheese";        cartCost = 1;        objRdr3.Close();        objConn3.Close();                cartBusiness = Server.UrlEncode(cartBusiness);        String strPayPal = "https://www.paypal.com/cgi-bin/webscr?cmd=_cart&upload=1&business=" + cartBusiness;        strPayPal += "&item_name_" + itemNumber + "=" + cartProduct;        strPayPal += "&item_number_" + itemNumber + "=" + cartQuantity;        strPayPal += "&amount_" + itemNumber + "=" + Decimal.Round(cartCost);        Response.Redirect(strPayPal); 
 
Here is my current code. I have manually selected cartProduct = "Cheese" and cartCost = 1, however I would like these variables to be site by data from the query.
So I want cartProduct = Title and cartCost = Price from the SQL query.
How do I do this?
 
Thanks
 

View 1 Replies View Related

Computing And Assigning A Value To A Textbox From Other Data Regions

May 18, 2007

I am new to SSRS, so perhaps its a trivial question. I was wondering that since all controls have names in the report, is it possible to programatically access values of different textboxes, do some computation and then assign to another text box? I know how to do it using the Aggregate functions and operators, but am not sure if I can access values from textboxes within two different tables and assign the computed value to a third text box on the page (not belonging to any table or other control).



somethig like.... txtTotal.Value = FormatCurrency(txtSalesTotal.Value) - txtDiscount.Value));



Any ideas??



DNG.

View 6 Replies View Related

Formatting Numbers In A Mixed Column (numbers In Some Cells Strings In Other Cells) In Excel As Numbers

Feb 1, 2007

I have a report with a column which contains either a string such as "N/A" or a number such as 12. A user exports the report to Excel. In Excel the numbers are formatted as text.

I already tried to set the value as CDbl which returns error for the cells containing a string.

The requirement is to export the column to Excel with the numbers formatted as numbers and the strings such as "N/A' in the same column as string.

Any suggestions?



View 1 Replies View Related

SQL 2012 :: Accurate Sorting Data Each Time With Millions Of Records Without Time Field?

Apr 25, 2014

Sample Table

USE [Testing]
GO
/****** Object: Table [dbo].[Testing] Script Date: 4/25/2014 11:08:18 AM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON

[Code] ....

It seems to work fine with one million records.

Each primary key is unique, but the begindate is non-unique, and i guess even if i use datetime2 and add nanoseconds, from what i have read, there is a chance that i could have a duplicate datetime since the date is imported via XML from multiple sources.

View 7 Replies View Related

Problem Assigning Value To Package Variable From Data Flow Script Component

Sep 28, 2005

In my Script Component properties I have included "ClientReportGroupId" as a ReadWrite variable. This variable is declared as a Package Variable.

View 23 Replies View Related

The Return Of Problem Assigning Value To Package Variable From Data Flow Script Component

Jul 10, 2006

I have a Data Flow Script Component(Destination Type) and in the properties I have a read/write variable called User::giRowCount

User::giRowCount is populated by a Row Count Component previously in the Data Flow.


After reading http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=97494&SiteID=1 it is very clear that you can actually only use  variables in the PostExecute of a Data Flow Script Component or you will get an error
"Microsoft.SqlServer.Dts.Pipeline.ReadWriteVariablesNotAvailableException: The collection of variables locked for read and write access is not available outside of PostExecute."




What I need to do is actually create a file in the PreExecute and write the number of records = User::giRowCount as second line as part of the header, I also need to parse a read/write variable such as gsFilename to save me hardcoding the path

(Me.Variables.gsFilename.ToString),(Me.Variables.giRowCount.ToString)

 -they must go in the PreExecute sub --workarounds please-here is the complete script component  that creates a file with header, data and trailer --Is there any workaround

Thanks in advance Dave
 
Imports System
Imports System.Data
Imports System.Math
Imports System.IO
Imports System.Text
Imports System.Configuration
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
 
Public Class ScriptMain
    Inherits UserComponent
    'Dim fs As FileStream
    Dim fileName As String = "F:FilePickUpMyfilename.csv"
    'Dim fileName = (Me.Variables.gsFilename.ToString)
 
    Dim myFile As FileInfo = New FileInfo(fileName)
    Dim sw As StreamWriter = myFile.CreateText
    Dim sbRecord As StringBuilder = New StringBuilder
 
 
    Public Overrides Sub PreExecute()
 
        sbRecord.Append("RECORD_START").Append(vbNewLine)
 
    End Sub
 
 
 
    Public Overrides Sub ParsedInput_ProcessInputRow(ByVal Row As ParsedInputBuffer)
 
        sbRecord.Append(Row.ProjectID.ToString)
        sbRecord.Append(Row.TransactionRefNum.ToString)
        sbRecord.Append(Row.BillToCustomerNum.ToString)
        sbRecord.Append(Row.BillToAccountNum.ToString)
        sbRecord.Append(Row.BillToLineNum.ToString)
        sbRecord.Append(Row.BillToReassignmentNum.ToString)
        sbRecord.Append(Row.ChargeCode.ToString)
        sbRecord.Append(Row.NotificationMethod.ToString)
        sbRecord.Append(Row.AdjustmentAmount.ToString)
        sbRecord.Append(Row.AdjustmentDate.ToString)
        sbRecord.Append(Row.ReparationGivenFlag)
        sbRecord.Append(Row.BillingSystemProcessingErrorCode.ToString).Append(vbNewLine)
       
    End Sub
 
    Public Overrides Sub PostExecute()
        sbRecord.Append("RECORD_COUNT").Append((vbTab))
        sbRecord.Append(Me.Variables.giRowCount.ToString).Append(vbNewLine)
      sbRecord.Append("RECORD_END").Append(vbNewLine)
       'Now write to file before next record extract
        sw.Write(sbRecord.ToString)
        'Clear contents of String Builder
        sbRecord.Remove(0, sbRecord.Length)
 
 
       'Close file
        sw.Close()
 
    End Sub
 
 
End Class

Has anyone got a workaround

thanks in advance

Dave

View 6 Replies View Related

Assigning User Name And Password Form The Databse To Ssis Package For Data Flow Operations And Execute Sql Statement

May 8, 2008

hi in my package, some sql operations need the special user name and admin privilage. so how do i create my ssis package so that when it executes it takes the given username and password from the table in some database.

View 8 Replies View Related

Query Analyzer Shows Negative Numbers As Positive Numbers

Jul 20, 2005

Why does M$ Query Analyzer display all numbers as positive, no matterwhether they are truly positive or negative ?I am having to cast each column to varchar to find out if there areany negative numbers being hidden from me :(I tried checking Tools/Options/Connections/Use Regional Settings bothon and off, stopping and restarting M$ Query Analyer in betwixt, butno improvement.Am I missing some other option somewhere ?

View 7 Replies View Related

I Need To Update A Table With Random Numbers Or Sequential Numbers

Mar 11, 2008



I have a table with a column ID of ContentID. The ID in that column is all NULLs. I need a way to change those nulls to a number. It does not matter what type of number it is as long as they are different. Can someone point me somewhere with a piece of T-SQL that I could use to do that. There are over 24000 rows so cursor change will not be very efficient.

Thanks for any help

View 6 Replies View Related

SQL Server 2012 :: Update Statement Will Not Update Data Beyond 7 Million Plus Rows Out Of 38 Millions Rows

Dec 12, 2014

I run the following statement and it will not update beyond 7 million plus rows and I have about 38 million to complete. I keep checking updated row counts and after 1/2 day it's still the same so I know something is wrong because it was rolling through no problem when I initiated it. I need to complete ASAP so it's adding to my frustration. The 'Acct_Num_CH' field is an encrypted field (fyi).

SET rowcount 10000
UPDATE [dbo].[CC_Info_T]
SET [Acct_Num_CH] = 'ayIWt6C8sgimC6t61EJ9d8BB3+bfIZ8v'
WHERE [Acct_Num_CH] IS NOT NULL
WHILE @@ROWCOUNT > 0
BEGIN
SET rowcount 10000
UPDATE [dbo].[CC_Info_T]
SET [Acct_Num_CH] = 'ayIWt6C8sgimC6t61EJ9d8BB3+bfIZ8v'
WHERE [Acct_Num_CH] IS NOT NULL
END
SET rowcount 0

View 5 Replies View Related

SQL 2012 :: Assign Consecutive Numbers To A Block Of Data?

Mar 17, 2015

I want to assign consecutive numbers to a block of data, where block of data is based on days consecutive to each other i.e., one day apart.

Date format is: YYYY-MM-DD

Data:
TestId TestDate
----------- -----------------------
1 2011-07-21 00:00:00.000
1 2011-07-22 00:00:00.000
1 2011-07-27 00:00:00.000

[Code]....

The OrderId is the column I am trying to obtain using my following cte code, but I can't work around it.

View 4 Replies View Related

Generate List Of All Numbers (numbers Not In Use)

Feb 21, 2007

I have an 'ID' column. I'm up to about ID number 40000, but not all are in use, so ID 4354 might not be in any row. I want a list of all numbers which aren't in use. I want to write something like this:

select [numbers from 0 to 40000] where <number> not in (select distinct id from mytable)


but don't know how. Any clues?

View 1 Replies View Related

T-SQL (SS2K8) :: Distribute Data Into Groups Based On Existing Numbers?

Aug 11, 2014

i've been looking at moving one of our processed from excel (+vba) into t-sql to make life easier but am stuck.

We have lots of tasks that are assigned to work groups which we want to distribute evenly across the work groups. This is a simple task for ntile.. However when these tasks are no longer required they are removed which leaves the groups uneven. When new tasks are added we want to still try to keep these groups balanced.

EG Existing groups :

GroupName - Task Count
Group1 - 1000
Group2 - 999
Group3 - 998

If we were to add 6 new tasks they would have more assigned to Group 2 & 3 as they have less than group 1.

Task 1 - Group3
Task 2 - Group3
Task 3 -Group2
Task 4 - Group1
Task 5 - Group2
Task 6 - Group3
Sample tables
Create table GroupTable
(GroupID int, Name varchar(200) )
Insert into GroupTable values (1,'Group1')
Insert into GroupTable values (2,'Group2')
Insert into GroupTable values (3,'Group3')

Create table Jobs(jobid int identity(1,1), name varchar(100),Groupid int)

--Existing tasks

Insert into Jobs(name,Groupid) values ('Task1',1)
Insert into Jobs(name,Groupid) values ('Task2',1)
Insert into Jobs(name,Groupid) values ('Task3',1)
Insert into Jobs(name,Groupid) values ('Task4',1)
Insert into Jobs(name,Groupid) values ('Task5',2)
Insert into Jobs(name,Groupid) values ('Task6',2)
Insert into Jobs(name,Groupid) values ('Task6',2)
Insert into Jobs(name,Groupid) values ('Task7',3)

-- New tasks

Insert into Jobs(name) values ('TaskA')
Insert into Jobs(name) values ('TaskB')
Insert into Jobs(name) values ('TaskC')
Insert into Jobs(name) values ('TaskD')
Insert into Jobs(name) values ('TaskE')
Insert into Jobs(name) values ('TaskF')

This gives us 6 unassigned tasks and a uneven group assignment

GROUPNAME TASK_COUNT
<none> 6
Group1 4
Group2 3
Group3 2

This means the new tasks will be assigned like this

TaskA - Group3
TaskB - Group3
TaskC - Group2
TaskD - Group1
TaskE - Group2
TaskF - Group3

View 5 Replies View Related

SQL Server 2008 :: Assign Consecutive Numbers To A Block Of Data

Mar 17, 2015

I want to assign consecutive numbers to a block of data where block of data is based on days consecutive to each other i.e., one day apart.

Date format is: YYYY-MM-DD

Data:

TestId TestDate
----------- -----------------------
1 2011-07-21 00:00:00.000
1 2011-07-22 00:00:00.000
1 2011-07-27 00:00:00.000
1 2011-07-29 00:00:00.000
1 2011-07-30 00:00:00.000
1 2011-07-31 00:00:00.000

[Code] ....

My Attempt:

WITH cte AS
(
SELECTTestId,
TestDate,
ROW_NUMBER() OVER
(
PARTITION BYTestId

[Code] .....

Expected Output:

TestId TestDate OrderId
----------- ----------------------- --------------------
1 2011-07-21 00:00:00.000 1
1 2011-07-22 00:00:00.000 1
1 2011-07-27 00:00:00.000 2
1 2011-07-29 00:00:00.000 3
1 2011-07-30 00:00:00.000 3

[Code] ....

The OrderId is the column I am trying to obtain using my following cte code, but I can't work around it.

View 7 Replies View Related

SQL Server 2014 :: Index Dates To Numbers With A Large Data Set?

Jun 16, 2015

I am trying to index dates to numbers with a large data set.

The first colums is index, the next is FactorsS, the next is value and the next is Date and the last is Lag.

Would it be difficult to write code that would determine the lag values. The lag value is based on the date value.

Index FactorS Value Date Lag
1 XYZ 2.3 12/31/2014 1
2 XYZ 1.4 12/30/2014 2
3 XYZ 3.3 12/29/2014 3
4 ABC 1.8 12/31/2014 1
5 ABC 2.2 12/30/2014 2
6 CBA 1.7 12/31/2014 1
7 CBA 1.8 12/30/2014 2
8 CBA 1.9 12/29/2014 3
9 CBA 2.1 12/28/2014 4

View 9 Replies View Related

Data Mining :: Updating Two Columns With Sequence Numbers With Where Clause

Jun 29, 2015

I have a question  in SQL server. For example I have a table which has two column like following table and I don't know how can I update theses two column with identity numbers but just the fields which are equal 111.

Example table:
numez numhx
111 111
111 111
0 111
111 0
111 0

and my results should be:
numez numhx
1 2
3 4
0 5
6 0
7 0

View 3 Replies View Related

How To Append A New Values Like Uniqueidentifiers, Sequence Numbers To Data Flow?

May 9, 2006

Hi,

I would like to know different possible ways in appending extra values like new uniqueidentifiers, sequence numbers, random number. Can you please tell what type of data flow components helps us ?



View 5 Replies View Related

Dataflow To Excel - Convert Numbers Stored As Text To Numbers Excel Cell Error

Mar 27, 2007

I'm trying to write data to excel from an ssis component to a excel destination.

Even thought I'm writing numerics, every cell gets this error with a green tag:

Convert numbers stored as text to numbers

Excel Cells were all pre-formated to accounting 2 decimal, and if i manually type the exact data Im sending it formats just fine.

I'm hearing this a common problem -

On another project I was able to find a workaround for the web based version of excel, by writing this to the top of the file:

<style>.text { mso-number-format:@; } </style>

is there anything I can pre-set in excel (cells are already formated) or write to my file so that numerics are seen as numerics and not text.

Maybe some setting in my write drivers - using sql servers excel destination.


So close.. Thanks for any help or information.

View 1 Replies View Related

Inserting Millions Of Rows

May 26, 2004

hi everybody,

i have an application that generate a lot rows from 1 mellion to 2 mellions rows
i wana insert this record in MS SQL server in a fast way

i am currentlly loop through this records while it is loaded in dataset
building a command text that generate insert query for each row
and run it against SQL server

but it takes a lot of time to be finished
is there r a way to bulk insert this data?

thanks 4 ur help.
Bolos

View 7 Replies View Related

DTS That Does HouseHolding With Millions Of Records

Sep 3, 2006

Hi all,I was given a task to create a houseHolding logic under a table thathave millions records.first let me explain what is a house holding:let's say I have 2 records that have the same phone number, that meanthat both records are under the same household, but this can get morecomplicatedthis article explain ithttp://www.teradata.com/t/page/115924/index.htmlif anyone worked with household he knows that you need to scan thetable many time to get all the house holds, I used a dts to do it.I tested the dts on 11 records like the article did and that workgreat, but once I went to million records each loop is taking me 2 houror so....a and I have no idea how how many loops I will have to do.if anyone out there worked with household queries and used sql, yourimput would help me allotthanks.

View 3 Replies View Related

Migration Millions Of Records

Aug 19, 2007

Hi

Please guide me for the below given problem.


While doing migration by using cursors for the below given sample data its taking more hours to complete the process. Therefore want to know is there any way I can do it in simple query.


ACNo Amount Balance CalType
A001 10 10 +
A001 10 20 -
A001 40 40 +
A001 10 30 -
A002 90 90 +
A002 20 110 +
A002 40 150 +
A003 10 30 +
A003 10 40 +
A003 10 30 -
A004 40 40 +
A004 10 30 -


Iam having Amount value alone and Balance has to be calculated value based on CalType. At the same time the Balance has to be reset as 0 when AcNo has changed.


Please guide me for the faster approach.


Regards,
Mohanraj

View 3 Replies View Related

Storing Millions Of Tiles

Jul 27, 2007

Hi everyone,

I have been trying to store millions of files (layer tiles) in NTFS now for a while with little success as the read/write speed drops off the chart or NTFS itself gets corrupted. My questions are:

1) Is there any way to store millions of files successfully in NTFS?
2) What does VE use to store its tiles (I am guessing SQL Server)?
3) If VE does not use SQL Server to store tile images, has anyone tried it and what are the pros/cons?

Any help would be greatly appreciated...

Thanks in advance,

Matt

View 2 Replies View Related

10 Millions Rows - 45 Minutes??

Feb 21, 2008



Hi everyone - I have an ETL package which loads about 10 million rows from SQL 2005 staging tables to new, empty tables (no indexes or constraints) in another SQL 2005 DB to be SWITCHED into the main partioned data tables.

Both databases reside on the same SQL Server instance - it is a dev server so the disk aren't super fast/SAN speeds but it has plenty of RAM/CPU & SCSI disks.

The insert takes about 45 minutes - can I get this working any faster?? or is this typical for 10 million rows?? I've messed about withe the data flow a few times but I can't seem to get any significant improvements.

Any tips anyone??

I perform several lookups on dimensions - these are not cached.

I do query the source table concurrently with different WHERE clauses & run two pipelines processing the data into 2 destination tables.

Would it be better to query the base table once & use a conditional split instead of the two separate queries??

I also mulicast from each pipeline & use a UNION ALL to log some of the rows from each pipeline to anther destination table.

Hope this makes sense?? Any ideas or tips on how I can speed up this kinda transform would be appeciated..

I'm using oleDB connections.

Hope ths makes some kinda sense!! Thanks for any advice!!

Sinister Pengiun

View 5 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved