In the FLAT FILE source, I have to find the duplicate rows based on the two fields say, "bill number" & "invoice date".
The rows within flat file has like "bill number" which is duplicated on the same "invoice date".
If duplicate rows found then move the duplicate rows into another Flat File.
If not found then move the rows into Sql Server Table.
Pls provide the solution. Thank you
I would like to locate duplicate rows within a specific table. This table has 12 diffrent rows.
BASENO - POSITION - SEQ - PROD -STYL - DESCR - FIELD01 THRU FIELD05 - VALUE01 THRU VALUE05 - FORMTYPE - ANSWER
I have been playing with the following query but can't seem to get it perfect to locate my dups within sql. Can someone help me with the querry?
I'm playing with the following querry.
SELECT
<list of all columns>
FROM
tablename
GROUP BY
<list of all columns>
HAVING
Count(*) > 1
Can somone possible input my column names into this querry that would possibly get it to locate my dups? I'm missing somehting and not sure what.
Thanks for any help
SQL Newbie
I'm using this to find duplicates where a person has the same email but varying firstname and lastnames:
select distinct t1.booking_id, t1.first_name, t1.last_name, t1.email_add, t1.booking_status_id
from [aren1002].[BOOKING]
as t1 inner join [aren1002].[BOOKING]
as t2
on t1.last_name=t2.last_name and t1.booking_id<>t2.booking_id
where t1.booking_status_id = 330
order by last_name asc
Sample data:
3927 Greg Smith greg@emailno1.com 303
5012 John Smith greg@emailno1.com 303
6233 John Smith greg@emailno1.com 303
4880 Dulcie Abuud dulcie@theiremail.com 303
However it is listing the non duplicate rows, For example: The record with Abuud as the last name, doesn't have any duplicates in the table, so I don't want it listed.
The data should be like this:
3927 Greg Smith greg@emailno1.com 303
5012 John Smith greg@emailno1.com 303
6233 John Smith greg@emailno1.com 303
I have some duplicate records in my flat file. But i don't want to load those duplicate rows into my destination.
View 2 Replies View RelatedI'm writing to a flat file destination (CSV file) which contains 2 header rows, lets call it Col1 and Col2.
For some reason, the header rows seem to get duplicated in the output - i.e.
Col1,Col2
A,B
Col1,Col2
C,D
Is there any way to resolve this?
I don't want the file to be overwritten everytime since its used for record-keeping purposes.
Thanks
i have a table like below
create table staff_attendance
(
attendance_id int,
attendace_date datetime,
staff_id int,
working_year int,
hours int
)
values like
1 2014-06-30 00:00:00.0ST10121
2 2014-06-30 00:00:00.0ST10122
3 2014-06-30 00:00:00.0ST10122 ----same entry like previous one
4 2014-07-01 00:00:00.0ST10121
5 2014-07-01 00:00:00.0ST10122
6 2014-07-02 00:00:00.0ST10121
7 2014-07-02 00:00:00.0ST10122
8 2014-06-30 00:00:00.0ST10221
9 2014-06-30 00:00:00.0ST10222
10 2014-07-01 00:00:00.0ST1022 1
11 2014-07-01 00:00:00.0ST102 22
12 2014-07-02 00:00:00.0ST102 21
13 2014-07-02 00:00:00.0ST102 22
I Need to find the duplicate rows like same entries which is having more than 1 rows.... how do i find?
I need find out the number of columns in flat file before i process that particular file.I have file name in @filename variable and file path is @filepath variable.But do not not that how i will check the column name in before i will process that file.
@filePath = C:DatabaseSourceFilesCAHCVSSourceFiles
And i am using for each loop container to read the file one by one and put the file name in @filename variable.and my file name like
Product_20120607060930.txt
Product_20130708060930.txt
[code]....
Now what i have to do is i need to make sure that ID,Name,City,County,Phone is there in flat file.if it is not there then i have to send mail to client saying that file is not valid.I need to also calculate the size of flat file.
Hi, I am trying to import data from a csv files to a OLE DB Destination. The csv files contains all transactional changes . For example for a particular record the firstname, lastname, email address records change within the same csv file. I need to save only the last updated record from the csv files. I have tried "slowly changing dimensions" but these dont work when there is duplictes within the same csv file. Also have tried 'Sort' but this only stores the first occurance.
Any ideas how i can store the latest changed data within 1 csv file.
Hi!
I have a problem that I can´t understand.
I have rows in a table in a SQL Server 2005 that I want to send to a Flat file. When I am executing the task for the first time it is OK and works perfectly. But the problem is when I am trying to do it again. I seems like the path to the file and the filename dissappears after the first time and the execution failes with an error message that tells me that the file is invalid and don´t exists.
I don´t now why this is happening and it´s such an easy thing to do and I can´t make it work.
So please help me!
Regards,
Johanna
I have a text file that come from our client that is Column deliminated by ~ and row deliminated by {CR}{LF}.
There is a comment field that appearently is not cleaned up and has {CR}{LF} within the comment field.
I am new to SSIS and I'm wondering if there is a way to detect and correct the bad rows?
example file formet:
ORDERID~DATE~Comment~Address
1~2/3/2007~Some Comment~1234 oak st
2~2/3/2007~Some messed
up comment~345 oak st.
3~2/3/2007~Another comment~3214 asdf blvd.
Thank you.
Hi,
I noticed something strange today. I created a pkg that reads a flat file and writes the rows to a table.
In checking the data in the file against what's in the table, I noticed that the rows were inserted in a different order than they are in the file.
All the rows appear to be in the table correctly, but they're just not in the same order as in the file. I've never seen this before. But I checked very carefully, and this is indeed the case.
Is this normal??
Thanks
Hello,
Is there a way (perhaps a property) to capture the number of rows selected from a Flat File Data Flow Source without having to develop a script to loop through the rows and count them?
Thanks a lot,
Grace
Hi,
In a dataflow there is a Flat File Source that loads a file with 500000 rows but when I execute the package it only loads 249999, I have verified the file and every looks fine, it is a csv and ms-excel recognized it perfectly.
I'm wondering if there is any technical limitation regarding to the amount of rows per file?
and what else can I verify for discover the problem?
thanks
hi
I have a flat file as below
Date 20070606
Empid Salary X1 x2 x3 x4 x5
100 10000 .............................
where 10000 is the salary got on date 20070606 by empid 100
.x1,x2,x3...are the remaining columns in the file.I need to extract the date and also continue reading the remaining data.How can i do this???
Any help?
thanks
ami
I finally put together a SSIS package that takes a Text File and successfully imports its data into the right table. My question is, where in the package's properties can I find the option to Delete all rows from Destination Columns prior to Importing. I have looked everywhere in the Package Explorer for this setting. Thanx in advance.
View 3 Replies View RelatedI tried to load a fixed width flat file with around 300,000 rows. However, only the first 8xxxx rows were loaded to the destineation table and the rest row were loading blank records. There was no error message showing during package execution. I've tried to split the file in half and the result was the same. So it wasn't the data file problem.
Would there be any buffering issue I need to cater for inside the package? Thanks!
Hello,
Using ssis, how can I limit the number of rows I wish to import from a flat file?
thanks in advance
I got a flat file like:Â
Location CurrencyRates
ALBERTA #15U.S.$ 0.2930 1 Can$ 0.0900
BRITISH COLUMBIA #14U.S.$ 0.6891 2 Can$ 0.2117
MANITOBA #18U.S.$ 0.4557 3 Can$ 0.1400
If there a way I can use SSIS to transfer this file to something like:
locationUSDUSDrateflagCADCADrateALBERTA #15U.S.$ 0.29301Can$ 0.0900BRITISH COLUMBIA #14U.S.$ 0.68912Can$ 0.0900MANITOBA #18U.S.$ 0.45573 Can$ 0.1400
Sorry if this question had already been answered previously. I was unable search the forum on this topic. How will I merge these and then configure the first row as Column names (As this helps to map to the destination column names automatically)
Hi,
I have a data task with the following requirements:
1) Run query against database to retrieve rows
2) Add header and footer row to the result set. The footer row must contain a count of the records.
3) Write the rows to a fixed width file if there were any data rows
I have got to the point that I can create the file (using a set of tasks that includes derived columns, sorts, aggregation and merges). However the file is created regardless of whether there were data rows returned.
I can't check the row count before proceeding as this isn't set until the data task ends. And if I try to split them into separate data tasks (so that I can access this variable and perform conditional execution) it becomes harder to access the original rows.
Do you have any recommendations on the best way to achieve this?
It all seems to be very complex and I'm starting to feel that it would be easier to do this outside of SSIS...
Please help me to keep the faith!
For those interested this is a slightly simplified version of what I have so far (all within a single data task):
1.Run dummy sql to create header row 2.Run main SQL to retrieve rows
| |
| 3.Multicast
| | |
| | 4.Create footer row by doing sum() in aggregate task
| | |
| 5.Merge body and footer
| |
6. Merge header with body and footer
|
7. Create fixed width file
Just attempting to import a simple tab delimited text file into my SQL Server 2005 database using the SQL Server Import and Export wizard. Column names are specified within the first line of the file. The Header Rows to Skip field value is listed as 0, but the wizard indicates that "The field, Header rows to skip, does not contain a valid numeric value".
Why isn't zero (0) a valid numeric value? I don't want to skip any rows. PLUS, I get the same error when trying to export to a text file although the header rows to skip field does not exist. I can increase the number to 1 or more, but the wizard will skip part of my data .. unacceptable.
What am I missing here? I installed SP1 of SQL server 2005, but that did not help.
Thanks in advance.
All,
I'm having an issue with the Flat File Data Flow Source returning only a limited set of the rows that are in the flat file. Basically, I connect to the flat file fine, it goes to retrieve the data (tab delimited file) and only returns 190 of 392 rows. Is there a limitation on the # of rows this data flow source can retrieve or something? I've look all through the settings and properties of the task as well as the connection manager and nothing is obvious as to what is causing this. Hopefully someone ou tthere has run into this before and can help me retrieve all rows. Thanks in advance!
bakerz
Hi all
I have a flat file.I am trying to set the value for the property "HeaderRowsToSkip" during runtime.I have set an expression for this in my "flat file connection manager". But this is not working.The connection manager is not able to take the value during runtime.
My expression is as follows:
DataRowsToSkip : @[user:: Var]
where "Var" is my variable which gets the value from the rowcount component and trying to set it back to the "HeaderRowsToskip" property.
I ve even tried setting the value to the "HeaderRowsToSkip" property in the expression builder.
Its not working....
Can anyone help me out in solving this????
Thanks in advance
Regards
Suganya
Hi,
I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.
Did someone else faced this issue?
Thanks,
AQ
Hi,
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
I am transferring data from an OLEDB source to a Flat File Destination and I want the column width for all of the output columns to 30 (max width amongst the columns selected), but that is not refected in the Fixed Width Flat File that got created. The outputcolumnwidth seems to be the same as the inputcolumnwidth. Is there any other setting that I am possibly missing or is this a possible defect?
Any inputs will be appreciated.
M.Shah
Hello All,I am trying to create a DTS package.I have two tables tbl_A and tbl_B with similar data/rows but noprimary keys.tbl_A is master.I would like this package to query tbl_A and tbl_B and find1)all rows in tbl_A that are different in tbl_B, 2)all rows in tbl_Athat are not present in tbl_B and3)all rows in tbl_B that are not present in tbl_A, and then just showthose rows.Can this be done with a simple UNION?Perhaps this could produce a temp Table that can be dropped once theDTS package exists successfully.The 2nd part after all the above rows are retrieved is that I wouldlike to add an addional Column to the retrieved data called STATUSwhich has 3 possible values(letters) at the end of each row...M (modified) means that row exists in tbl_B but has 1 or moredifferent columnsA (add) means this row exists in tbl_A but not in tbl_BD (delete) means this row exists in tbl_B but not in tbl_AI'm hopping this DTS package would output a nice comma seperated TXTfile with only...1) rows from tbl_A that are different in tbl_B (STATUS M)2) rows from tbl_A that are not present in tbl_B (STATUS A)3) rows from tbl_B that are not present in tbl_A (STATUS D)Can a DTS package in MS SQL be used to perfom all of the above tasks?I would very much appreciate any help or any advise.Thanks in advance :-)
View 7 Replies View RelatedHi all,
I m using SSIS and i am transfering the data from Flat File Source to the OLE DB destination File. The source file contain some corrupt data which i am transfering to the other Flat file destination file.
Debugging is succesful but i am not getting any error output in the Flat file destination file.
i had done exactly which is written in the msdn tutorial of SSIS.
Plz tell me why i am not getting the error output in the destination flat file?
thanx
First, a couple of important bits of information. Until last week, I had never touched SISS, and therefore, I know very little about it. I just never had the need to use it...until now. I was able to convert my first 3 flat files to SQL2005 tables by right clicking on "SISS Package" and choosing "SISS Import and Export Wizard". That is the extent of my knowledge! So please, please, please be patient with me and be as descriptive as possible.
I thought I could attach some sample files to this post, but it doesn't look like I can. I'll just paste the information below in two separate code boxes. The first code box is the flat file specifications and the second one is a sample single line flat file similar to what I'm dealing with (the real flat file is over 2 gigs).
My questions are below the sample files.
Code Snippet
Record Length 400
Positions Length FieldName
Record Type 01
1,2 L=2 Record Type (Always "01")
3,12 L=10 Site Name
13,19 L=7 Account Number
20,29 L=10 Sub Account
30,35 L=6 Balance
36,37 L=1 Active
37,41 L=5 Filler
Record Type 02
1,2 L=2 Record Type (Always "02")
3,4 L=2 State
5,30 L=26 Address
31,41 L=11 Filler
Record Type 03
1,2 L=2 Record Type (Always "03")
3,6 L=4 Coder
7,20 L=14 Locator ID
21,22 L=2 Age
23,41 L=19 Filler
Record Type 04
1,2 L=2 Record Type (Always "04")
3,9 L=7 Process
10,19 L=10 Client
20,26 L=6 DOB
26,41 L=16 Filler
Record Type 05
1,2 L=2 Record Type (Always "05")
3,16 L=14 Guarantor
17,22 L=6 Guar Account
23,23 L=1 Active Guar
**There can be multiple 05 records, one for each Guarantor on the account**
and the single line flat file...
Code Snippet
01Site1 12345 0000098765 Y 02NY1155 12th Street 03ELL 0522071678 29 04TestingSmith,Paul071678 05Smith, Jane 445978N 05Smith, Julie 445989N 05Smith, Jenny 445915N 01Site2 12346 0000098766 N 02MN615 Woodland Ct 04InfoJones,Chris 012001 01Site3 12347 0000098767 Y 02IN89 Jade Street 03OWB 6429051282 25 04Screen New,Katie 879500
As you can see, each entry could have any number of records and multiples of some of the record types, with one exception, every entry must have a "01" record and can only have one "01" record. Oh, and each record has a length of 400.
I need to get this information into a SQL 2005 database so I can create a front end for accessing the data. Originally, I wanted one line for each account and have null values listed for entries that don't have a specific record. Now that I've looked at the data again, that doesn't look like a good idea. I think a better way to do it would be to create 5 different tables, one for each record type. However, records 2 through 5 don't have anything I can make a primary key. So here are my questions...
Is it possible to make 5 tables from this one file, one table for each of the record types?
If so, can I copy the Account number in record 01, position 13-19 in each of the subsequent record types (that way I could link the tables as needed)?
Can this be done using the SISS Import and Export Wizard to create the package? If not, I'm going to need some very basic step by step instructions on how to create the package.
Is SISS the best way to do this conversion or is there another program that would be better to use?
I know this is a huge question and I appreciate the help of anyone who boldly decides to help me! Thank you in advance and I welcome anyone's suggestions!
How can I take this example Flat file and parse out each section to a new flat file? Each section starts with HD (header row)
http://www.webfound.net/flat_file_example.txt
e.g. an example output file based on above (cutting out the first section) would be:
http://www.webfound.net/flatfile_output.txt
Also, I'll need to grab a certain value in each header row (certain position in the 100 byte header row) to use that as part of the filename that's outputed. I assume it would be better to insert these rows into a temp table then somehow do a search on a specific position in the row...but that's impossible? The other route is to insert each row into a temp table separated out by fields but that is going to be too combursome because we have several formats to determine separation of fields based on the row type so I'd have to create many temp tables and many components in SSIS when all we want to do is again:
1) output each group (broken by each header row) into it's own txt file
2) use a field in the header row as part of the name of the output txt file (e.g. look at the first row, whcih is a header row in flat_file_example. txt. I want to grab the text 'AR10' and use that as part of the filename that I create
Any suggestions on how to approach this whole process in SSIS...the simplest approach that will work ?
Dear Gurus,I have table with following entriesTable name = CustomerName Weight------------ -----------Sanjeev 85Sanjeev 75Rajeev 80Rajeev 45Sandy 35Sandy 30Harry 15Harry 45I need a output as followName Weight------------ -----------Sanjeev 85Rajeev 80Sandy 30Harry 45ORName Weight------------ -----------Sanjeev 75Rajeev 45Sandy 35Harry 15i.e. only distinct Name should display with only one value of Weight.I tried with 'group by' on Name column but it shows me all rows.Could anyone help me for above.Thanking in Advance.RegardsSanjeevJoin Bytes!
View 4 Replies View RelatedHi,
Does anybody know the SQL query to find the duplicate records?
Many Thanks in advance!
Hello board,
I was wondering if anyone can tell me an easy way to find duplicate records on sql. The thing is this, at work we have a database (table) which includes tracking numbers, I need a easy way to be able to search this table for duplicate tracking numbers and print them out. I currently access this table to edit some data by using the following path “Start > Programs > Microsoft SQL Server > Enterprise Manager” then work my down the tree to “Databases > Master > Tables” on tables I do a right click and “open table/query”. Any help would be most appreciated. Believe me I’m very “SQL illiterate”
Bill
:confused: