Importing Unique Data && MAX Data To Table Using DTS

Nov 28, 2005

I am creating a DTS package that is combining several tables, converting one column of data to a new column removing all special characters, then exporting the unique data based on this column and another column, and the max of other duplicates to a new table.

Now that I have the data in this table, I want to import any data that is not in my main table.

This "CLEANED" table does not have a designated "key" column, but the table I want to import the unique items does have an ID column that is also a primary key column.

DTS seems to want me to have a Key column to reference when importing from the CLEANED table to the MAIN table.

How would I go about checking the MAIN table against the CLEANED table, having DTS import only the unique items from the CLEANED table that are not present in the MAIN table based on three columns? The rest of the columns I want to just extract the MAX data from the duplicates.

Now here is the query I use to extract the unique values from the "CLEANING" table to get the data to the "CLEANED" table, but do not know how to use this to import into the MAIN table using something similar.


Code:



select
partno2,
MAX (partno) as partno,
alt,
MAX (C_alt) as C_alt,
Max (cmpycd) as cmpycd,
MAX (type) as type,
compFN,
MAX (pndesc) as pndesc,
MAX (equipment) as equipment

into tbl_CLEANED
from tbl_CLEANING
group by partno2, alt, compFN
ORDER BY partno, compFN



The three main columns I need to check against are:
partno2
alt
compFN
I have named the columns the same in both tables.

partno2 is the column that has been copied from partno with all special characters & spaces removed. This is the main column I am using as a reference for unique values, then if no match, I have it check against the alt column, then the comFN column. If there are no matches in any of these columns, then I want to extract the data to the MAIN table.

How can I compare these tables and import only unique info to the MAIN table?

In addition, how can I also check items that are the same in both tables and update the MAX info for the other columns (not the three I use for reference - these I need to leave alone) and update those if there is more data in the CLEANED table then in the MAIN table?

View 3 Replies


ADVERTISEMENT

Importing Data From Another Table

Mar 23, 2006

Hi All,I'm coming from using MySQL, and in their dialect you could pull data from one table to another using the following: INSERT INTO Table1 (fname, lname)VALUES(    SELECT fname, lname    FROM Table2    )Let's assume Table1 is a simple table with the fields ID (PK/Identity), fname, and lname.  This query would grab all the first and last names out of Table2 (fname and lname fields) and insert them into Table1, generating the ID for each new row.How would I do this in T-SQL?

View 1 Replies View Related

Need Help With Importing XML Data To A Table

Jul 23, 2005

Using some VB sample code on the Internet I have the followingthat works well and exports a set of records to an XML file:Dim oCmd, sSQL, oDomSet oDom = CreateObject("Msxml2.DOMDocument.4.0")Set oCmd = CreateObject("ADODB.Command")oCmd.ActiveConnection = "Provider=SQLOLEDB;Data Source=(local);InitialCatalog=TestXML;UID=sa;Password=123456"sSQL = "<ROOTxmlns:sql=""urn:schemas-microsoft-com:xml-sql""><sql:query>" & "select *from tblTest for xml auto</sql:query></ROOT>"oCmd.CommandText = sSQLoCmd.Dialect = "{5D531CB2-E6Ed-11D2-B252-00C04F681B71}"oCmd.Properties("Output Stream") = oDomoCmd.Execute , , 1024oDom.Save "C: emp estdts.xml"This is my first day of using XML in SQL Server and I need help on howto imitate INSERT statements by basically importing data from an xml file.With some changes I managed to write VB code that seems to be readingthe XML file I exported earlier but I can't seem to know what to do to beable to take the data only of the XML and INSERT it into a table.The code I used is the one found in the SQL Online Help, search in INDEXfor "OPENXML" and then choose "USING OPENXML"If you have sample code or can point me to a link that has sample code toachieve what I want, I would appreciate your help.Thank you

View 2 Replies View Related

Integration Services :: Load Incremental Data Into Fact Table When Source Table Not Have Timestamp And Unique Key

Sep 24, 2015

I have a transaction table having about 40 crore rows in source. It don't have timestamp and unique key columns. It have only Bill_month and Bill_Year columns. Actually for loading this table into staging I have added a new datetime column by adding default bill_date as 01. Then

* First we delete last 3 month data from staging tables.
* Get last 3 months data from source table.
* Load that 3 months data from source to staging table. 

We do this because we only get update for last three months data. Now I have to include this transaction table as Fact table in DW. What will be the best practice for loading the fact table by picking data form staging table. Also we have to look up with dimensions for Foreign Keys. 

* Should I implement the same method of deleting last 3 months records and loading them again. 

View 3 Replies View Related

Importing Split Data Into Table

Apr 4, 2006

Hi,

I have Data split into 3 text files with 3 fields repeated in each to link then (key). I want to import this data into one table.
I used DTS to create 3 tables with the data. Now i want to combine the 3 tables into only one (that i already created). How can i do this? Note: the field names in the source tables are different from the destination table.

Thanks

Guy

View 6 Replies View Related

Help For Importing CSV Data File To Table

May 30, 2007

Hi all-

I am in need of some help importing a .CSV file into a SQL Server 2005 Enterprise Edition.
The problem is I already implemented Bulk Insert task in SSIS but it is not importing any data. My detailed layout is as follows :
In SSIS package1 -
In Control Flow Bulk Insert Task has been inserted
Properties of Bulk Insert Task:
Connection adtc009d.ganny
Destination Table ganny.dbo.t4
Format
Format Specify
Row Delimiter {CR}
Column Delimiter Comma{,}
Source Connection
File r.csv
Options
Options Check Constraints
Maxerrors 20

This bulk insert task is connected to Data flow task, if we click edit to data flow task, data flow section will come, here Flat file source & OLE DB Destination is there. Flat file source is connected to OLE DB Destination.

Properties of Flat File
Connection Manager
Flat file connection Manager
here by clicking new link flat file properties to this.
Preview
by clicking preview all data are visible
Properties of OLE DB Destination editor
Oledb connection manager adtc009d.ganny
Data access mode: Table or View - fast load
Name of Table or view dbo.t4

After designing all this then if I start debugging I could able to get records are imported to a table.
Please suggest me where I am going wrong.

Thanks in advance
Karna

View 3 Replies View Related

Importing Data From Web Page Into SQL Server Table

Sep 2, 2005

Hi,I have to find a way to do the following : a company provides for my company a web page displaying data separated with ;I need to automatically update a table in SQLServer with this data, ie delete the old data, read the web page containing new data as text, and insert these data into the table.I would actually prefer not to trigger this import process "by hand" ; if there's a solution to schedule it ...Thanks for your helpJohann

View 3 Replies View Related

Importing Data In Table With Identity Column

Nov 13, 2000

Whenever I import (or even append from another table) data to a table that has an Identity column, I get an error: Cannot insert NULL values in Identity column.

Isn't the Idenity column supposed to incement automatically without me having to provide a value? Using INSERT provides an Identity value automatically, using import however doesn't.

How can I overcome this problem? (I now delete and recreate the Identity column - really BAD practice!)
Thank you.

View 4 Replies View Related

Importing Excel Data Into New Table In MS SQL Server

Jul 10, 2007

I have installed SQL Server Managemert Server Express .... I am wondering how to import data from an excel spreadsheet?



.. Silent Running

View 3 Replies View Related

Importing Textfile Data To Existing Table (append?)

Jul 25, 2005

I have an existing table I need to add data to. The data is in a text file, and the existing table already has data in it (I don't want to delete this I want to add to it).

I used Microsoft's import utility but this created a seperate table with generic fieldnames (column01, column02, ect). Is there a step in this wizard I missed?

View 2 Replies View Related

Issue In Importing Data From Excel Into Database Table

Mar 7, 2008



Hi

I am facing issue while loading data from excel into Sql database table.
The format of spreadsheet is like

Date -
Name - A B C
Section Marks
X1 10 10 10
X2 20 30 10

X3

I need to load this data into table with columns

Date
Name
Section
Marks

Please share your thoughts on how to load this kind of data into table.

Thanks
Anshu

View 1 Replies View Related

Advice On Importing Access Data Into MSSQL Table Using Code

Aug 2, 2004

Hi,

I'm about to embark on writing some code in perl or VBscript that automatically synchronises a constantly updated Access database with an MSSQL database.

I know MSSQL has an import tool built into Enterprise manager but I'm wondering if theres a stored procedure that does this?

The way I'm thinking of doing it is to read the all the access tables into separate hash arrays and then INSERTing them into the MSSQL database after checking for any duplicates. This all sounds a bit time consuming (there are a large number of tables) and processor intensive.

If anyones done anything like this before, I'd love to hear their views......!

Thanks!

View 9 Replies View Related

Master Data Services :: Unique Identifier In MDM Profisee Stage Table

Apr 13, 2015

I am new to MDM profisee tool and currently working for Addres verification project for my organization.I wanted to clear my doubts here about Unique Identifer in Stage table and how it works.. Here what i understand till now:

Step 1) I created an Entity using MDM profisee UI and it generated a stage table in MDS database called stg.Address_leaf
Step 2) I have loaded data from external source to MDS stage table using ETL and passed Import type as 2 and Import status id as 0
Step 3) I have run store procudure system generated something stg.udp_Address_leaf to load the model and passed the version name as Version_1, Log flag as '1' and Batch tag as 'Address'

Now my below are my questions:
1) What is the field i can use in MDS stage table to populate my unique intifier value coming from source? (lets say Address_Id is my Unique value for all the records coming from source)
2)  Where/how Unique Identifier is useful in this process? Will this be useful in next time load from stage to Model?
3) If i truncate and load my MDS stage table in next run and few earlier records has been updated how it will update those records in Model? will this process (code present in SP) recognize by Unique Identier column present in MDS stage table?

View 3 Replies View Related

Integration Services :: Importing Data From Word Document (Table Format)?

Aug 4, 2015

how to import data from word document into sql server table in SSIS.

View 18 Replies View Related

Importing Data In Datatable Using SSIS Package Trigger On Insert Is Not Firing On That Table

Oct 6, 2007

Hi
I am Importing data in datatable using SSIS package . I made trigger on that table on insert. The trigger on insert is not firing on that table
Please help
Thanks
CP

View 1 Replies View Related

Integration Services :: Importing Text File Into Table - Random Data Order

Aug 3, 2015

I'm importing comma-delimited text files into a SQL table. The data imports in a seemingly random order. One time I import and the lines appear one way and the next time I import they import another way.

Is there a way to force the text files to import in the same order the data is found in the file?

View 10 Replies View Related

Delete Rows With Duplicate Column Data But Unique Row Data

May 25, 2000

Hello,

This probably has been addressed before but I was unable to get the search to work properly on this site.
I am needing a script/way of deleting all rows from a DB with the exception of one record left for each row that has duplicate column data. Example :
Row 1
Field1 = 12345 Field2 =xxxxx Field 3=yyyyy Field4=zzzzz etc.
Row 2
Field1 = 12345 Field2 =zzzzzz Field 3=xxxxxx Field4=yyyyyy etc.
Row3
Field1 = 12345 Field2 =20202 Field 3=11111 Field4=zzzzz etc.
Row 4
Field1 = 54321 Field2 =xxxxx Field 3=yyyyy Field4=zzzzz etc.
Etc. Etc.

I want to be able to find the duplicates for Field1 and then delete all but 1 of those rows.( I don't care which one I keep just so only one is left.) The data in the other fields may or may not be unique.

I know how to find the duplicates it's just the deleting part I am having problems with. Any help would be much appreciated. Thanks,

Kerry

View 3 Replies View Related

Integration Services :: SSIS - Managing Data Integrity When Importing Sharepoint Data

Sep 28, 2015

I setup this package to import data from a Sharepoint list to a SQL Server data table. The primary key of my SQL table is mapped to the Title column of my Sharepoint list. There is a possibility that duplicate values will be entered in the Title field of the Sharepoint list. So when importing data into my table via SSIS, my package always error-out when there it comes across duplicate values. how you others have managed data integrity when importing from a Sharepoint list with the Title column being mapped to the primary key of a table.

View 4 Replies View Related

Memo Data Type Import Error While Importing Data From Access File Into SQl Server 2005

Sep 10, 2007

I have one column in SQL Server 2005 of data type VARCHAR(4000).

I have imported sql Server 2005 database data into one mdb file.After importing a data into the mdb file, above column
data type converted into the memo type in the Access database.

now when I am trying to import a data from this MS Access File(db1.mdb) into the another SQL Server 2005 database, got the error of Unicode Converting a memo data type conversion in Export/Import data wizard.

Could you please let me know what is the reason?

I know that memo data type does not supported into the SQl Server 2005.

I am with SQL Server 2005 Standard Edition with SP2.

Please help me to understans this issue correctly?

View 4 Replies View Related

Data Access :: Importing Huge Data From One Database To Another Daily

Jul 7, 2015

We have a daily process, which copies millions of rows of data from one DB to another over Linked Server. Just checking on the best practise, are there more efficient ways than the Linked server to copy millions of rows of data from one DB to another? I checked bulk insert but that transfers the data from the file to DB not DB to DB. 

View 6 Replies View Related

Importing Data From Oracle To Sql Loosing Data After The Decimal Point

Jun 18, 2007

I have created a simple package that uses a sql command to pull data from an oracle database and inserts the data into a sql 2005 table. Some of the data fields that i am pulling from contain two digits after the decimal point, however this data is lost when it gets into sql. I have even tried putting the data into a flat file, and still the data is lost.

In the package I have a ole db source connection which is the oracle database and when i do the preview i see all the data I need. I am very confused and tried a number of things to get the data into sql, but none work. Any ideas would be very helpful.

thanks

View 6 Replies View Related

Import Data Where Current Data Has Unique Identity

Aug 10, 2000

I have data for online catalogue in SQL 7.0. The web grogrammer asked me to add a unique key for reference. I used int datatype with identity seed of 1 and increment of 1. This works fine BUT when I try to import new data I get an error because the csv file has no column and therefore no value for the unique field which will not allow null by definition.

How can I maintain a unique field to act as primary key in my data when
I want to add (and delete) data that doesn't have this field.

I tried adding the uniqueidentifier field but this gives error message.

The only work round is to delete the unigue field altogether and then add the new data and afterwards create a new unique field. At 600000 + lines of data, this is time and memory consuming

Any help appreciated,Thanks, Keith

View 1 Replies View Related

Data Format Issue While Importing Data From Excel To SQL

Jul 17, 2007

hi



when i m importing data from excel to Sql using DTS the column which has text content was not imported as same in excel sheet. whereas a special character is appearing in between the lines. the text field contains multiple lines but the conetent is imported in single line .

ex:









ARIZONA
ALABAMA
STATE


but i m getting imported

as :
ARIZONA ALABAMA STATE

How to Format a single column while importing?



Regards

Raj

View 1 Replies View Related

Importing Data From Various Data Sources With Non Standard Formats

Mar 19, 2007

Hi all :)

I'm wondering if SSIS will be the solution to the problem I'm working on.

Some of our customers give us an Excel sheet with data they want to insert or update in the database.

I've created a package that will take an Excel sheet, do some data conversion so the data types match up and after that I use a Slowly Changing Data component to create the insert/update commands.

This works great. If a customer adds a new row to the Excel sheet or updates an existing row changes are nicely reflected in the database.

But now I€™ve got the following problem. The column names and the order of the columns in the Excel sheet are not standard and in the future it could happen a customer doesn't even use an Excel sheet but something totally different.

Can I use SSIS for this? Is it possible to let the user set the mappings trough some sort of user interface? I€™ve looked at programmatically creating the package but I€™ve got to say that€™s quit hard to do€¦ It would be easier to write the whole thing myself than to create the package trough code ;)

If not I thought about transforming the data in code before I pass it on to the SSIS package in something like XML. That way I can use standard column names and data types.

So how should I solve this problem? Use SSIS or not?

Thnx :)

Wouter de Kort

View 6 Replies View Related

How Can I Specify The Data Type When Importing Excel Data Via DTS?

Jun 11, 2006

I'm new to SQL and DTS packages. I am trying to import data from an excel spreadsheet to an SQL server table via DTS package. It seems that the excel task looks at the first few records in a column to determine the datatype for that column. If the first few records are text, the entire column is imported as text. If numeric, the entire column is imported as numeric.
There are about 25,000 records. In one field, the most important one, about half of the records begin with letters and the rest are all numbers. It is the subscriber ID field, and some subscriber IDs are all numbers, some are letters and numbers. The entire column should be imported as text. However, when I run the transform data task from the excel connection, none of the records that are all numbers are imported. I end up correctly importing only 13,000 of the 25,000 records. The rest are imported with the subscriberID field as <NULL>.
I tried using the CAST or CONVERT function in the SQL query, but get the error message "Undefined Function."

Can anyone give me some help? Thanks,
Jim

View 4 Replies View Related

How Can I Specify The Data Type When Importing Txt File Data Via DTS?

Jun 27, 2006

hello,
I create a txt file with a bash script, and i need to use it in a DTS package. But, i don't know how i can specify the type of my column. So in the transformations task, i have an error due to an incompatible type. what can i do to fix this error ?
thanks,

View 8 Replies View Related

Creating A Stored Procedure That Will Summarize Data In A Table Into A Table Reflecting Period Data Using An Array Type Field

Sep 20, 2007

I am attempting to create a stored procedure that will launch at report runtime to summarize data in a table into a table that will reflect period data using an array type field. I know how to execute one line but I am not sure how to run the script so that it not only summarizes the data below but also creates and drops the table.

Any help would be greatly appreciated.

Current Table

Project | Task | Category | Fiscal Year | Fiscal Month | Total Hours
---------------------------------------------------------------------------------------------------------
Proj 1 | Task 1 | Cat 1 | 2007 | 01 | 40
Proj 1 | Task 1 | Cat 2 | 2007 | 02 | 20
Proj 1 | Task 1 | Cat 3 | 2007 | 03 | 35
Proj 1 | Task 1 | Cat 1 | 2008 | 01 | 40
Proj 1 | Task 1 | Cat 2 | 2008 | 02 | 40
Proj 1 | Task 1 | Cat 3 | 2008 | 03 | 40

Proposed Table

Project | Task | Category | Fiscal Month 01 | Fiscal Month 02 | Fiscal Month 03 | Fiscal Year
---------------------------------------------------------------------------------------------------------------------------------------------------
Proj 1 | Task 1 | Cat 1 | 40 | 0 | 0 | 2007
Proj 1 | Task 1 | Cat 2 | 0 | 20 | 0 | 2007Proj 1 | Task 1 | Cat 3 | 0 | 0 | 35 | 2007
Proj 1 | Task 1 | Cat 1 | 40 | 0 | 0 | 2008

Proj 1 | Task 1 | Cat 2 | 0 | 40 | 0 | 2008
Proj 1 | Task 1 | Cat 3 | 0 | 0 | 40 | 2008

Thanks,
Mike Misera

View 6 Replies View Related

Integration Services :: Purge Data In Transaction Table Or Delete Some Data And Store In Separate Table

Aug 18, 2015

How to purge data  in transaction table or we can delete some data and store in separate table in data warehouse?

View 7 Replies View Related

Importing Data

Feb 12, 2005

I have created a DTS package for SQL Server, saved this as a VB file and upgraded this to the .NET framework.

It is now saved in a file called Shortages.vb.

What I want to do now is add this to an existing ASP.NET project and be able to call this DTS function by the click of a button.

Is this possible and how can I go about it? Can I just add it to the Click event of a button?

View 3 Replies View Related

Importing Data

Mar 17, 2002

Hallo every one,

I', working for a school project.
The setuation is an old database that has to be repaird.

All the data from each table of the old data base are stored in different tabdelimited text files.

I designed a new database, but the tables are a little different.
When I try to import the data I get some errors, because of the changed tables.

An example of my new setuation is a new table thas as colums from 2 different tabdelimited text files.

My question is, how do I solve thos problem!
Do I have to use DTS?
What do I have to do.

tnx
icheron.

View 1 Replies View Related

Importing Data

Jan 12, 2001

How can you can import results into table from system procedures.

For Ex. if you exec sp_spacesued, how do I import the results into the table.

Thank You

View 1 Replies View Related

Importing Data???

Sep 29, 2004

I have a process that calls a proc that BCP's a delimited file into a table. Well the SOX police say a header and footer must be added to the file. Needless to say this screws my BCP process. Does anyone know how to strip a header and footer record from a text file using transact sql or have any other suggestions to strip the records?

View 6 Replies View Related

Importing Data

Dec 4, 2006

I am trying to import data from excel into my server, but get this error message:

Error during transformation 'directcopyXform' for row number 1. Errors encountered so far in this task 1. TransformCopy 'DirectCopyXform' conversion error: Conversion invalid datatypes on coulumn pair 19 (source column '*9' (DBTYPE_WSTR( destination column 'F19' (DBTYPE_R8)

Anyone get me in the right direction?

Scott

View 6 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved