Is there any faster and efficient way for exporting or unloading the data from a table (Contains about 400 to 500 million rows) into a flat or raw file. I am using SQL server 2005 and the Export Utility in SQL Server is very slow.
Any help is greatly appreciated...
First, a couple of important bits of information. Until last week, I had never touched SISS, and therefore, I know very little about it. I just never had the need to use it...until now. I was able to convert my first 3 flat files to SQL2005 tables by right clicking on "SISS Package" and choosing "SISS Import and Export Wizard". That is the extent of my knowledge! So please, please, please be patient with me and be as descriptive as possible.
I thought I could attach some sample files to this post, but it doesn't look like I can. I'll just paste the information below in two separate code boxes. The first code box is the flat file specifications and the second one is a sample single line flat file similar to what I'm dealing with (the real flat file is over 2 gigs).
My questions are below the sample files.
Code Snippet
Record Length 400
Positions Length FieldName
Record Type 01
1,2 L=2 Record Type (Always "01")
3,12 L=10 Site Name
13,19 L=7 Account Number
20,29 L=10 Sub Account
30,35 L=6 Balance
36,37 L=1 Active
37,41 L=5 Filler
Record Type 02
1,2 L=2 Record Type (Always "02")
3,4 L=2 State
5,30 L=26 Address
31,41 L=11 Filler
Record Type 03
1,2 L=2 Record Type (Always "03")
3,6 L=4 Coder
7,20 L=14 Locator ID
21,22 L=2 Age
23,41 L=19 Filler
Record Type 04
1,2 L=2 Record Type (Always "04")
3,9 L=7 Process
10,19 L=10 Client
20,26 L=6 DOB
26,41 L=16 Filler
Record Type 05
1,2 L=2 Record Type (Always "05")
3,16 L=14 Guarantor
17,22 L=6 Guar Account
23,23 L=1 Active Guar
**There can be multiple 05 records, one for each Guarantor on the account**
and the single line flat file...
Code Snippet
01Site1 12345 0000098765 Y 02NY1155 12th Street 03ELL 0522071678 29 04TestingSmith,Paul071678 05Smith, Jane 445978N 05Smith, Julie 445989N 05Smith, Jenny 445915N 01Site2 12346 0000098766 N 02MN615 Woodland Ct 04InfoJones,Chris 012001 01Site3 12347 0000098767 Y 02IN89 Jade Street 03OWB 6429051282 25 04Screen New,Katie 879500
As you can see, each entry could have any number of records and multiples of some of the record types, with one exception, every entry must have a "01" record and can only have one "01" record. Oh, and each record has a length of 400.
I need to get this information into a SQL 2005 database so I can create a front end for accessing the data. Originally, I wanted one line for each account and have null values listed for entries that don't have a specific record. Now that I've looked at the data again, that doesn't look like a good idea. I think a better way to do it would be to create 5 different tables, one for each record type. However, records 2 through 5 don't have anything I can make a primary key. So here are my questions...
Is it possible to make 5 tables from this one file, one table for each of the record types?
If so, can I copy the Account number in record 01, position 13-19 in each of the subsequent record types (that way I could link the tables as needed)?
Can this be done using the SISS Import and Export Wizard to create the package? If not, I'm going to need some very basic step by step instructions on how to create the package.
Is SISS the best way to do this conversion or is there another program that would be better to use?
I know this is a huge question and I appreciate the help of anyone who boldly decides to help me! Thank you in advance and I welcome anyone's suggestions!
Hi,
I was wondering what SQL code would allow me to extract my results into a .txt.
For example:
Select * from employees
(What should I write here to enable me to import it into a .txt file and save it on a network drive?)
Thanks
Hi,
I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.
Did someone else faced this issue?
Thanks,
AQ
Hi,
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
I am transferring data from an OLEDB source to a Flat File Destination and I want the column width for all of the output columns to 30 (max width amongst the columns selected), but that is not refected in the Fixed Width Flat File that got created. The outputcolumnwidth seems to be the same as the inputcolumnwidth. Is there any other setting that I am possibly missing or is this a possible defect?
Any inputs will be appreciated.
M.Shah
Hi all,
I m using SSIS and i am transfering the data from Flat File Source to the OLE DB destination File. The source file contain some corrupt data which i am transfering to the other Flat file destination file.
Debugging is succesful but i am not getting any error output in the Flat file destination file.
i had done exactly which is written in the msdn tutorial of SSIS.
Plz tell me why i am not getting the error output in the destination flat file?
thanx
Hi,
I have a set of flat files and transforming it to SQL server. If I do that in 2000 it was done with in 45 seconds for 1.5 M records. If I do the same in SSIS it takes 3 minutes. Why there is difference in time that too lower when compared to the previous version. I used the data access mode as "Fast load". Am I missing anything while doing through SSIS?
I am looking to evaluate a text field in the row and change it to an integer in the sql table based on the text.
What is the best data flow tranformation for me to use inbetween the flat file source and the ole db destination?
After deleting all the test data from all tables in a SQL 2000 database, is there a way to reset all the auto-incrementing fields back to zero in one shot? In Access, you can run the Compact and Repair option. Also, in Sybase SQL, there was an "unload/reload" option to reduce the database size. Is there a similar function in SQL2000? Thanks for all the help
View 1 Replies View RelatedOk,I revamped my query, I thought I fa different way.....
Select
RecordID,
Type, (Traditional, Teclar, AllClaims)
Year,
Month,
TotalCharges,
AdjustedTotalSavings,
AverageClaim,
PercentSavings,
DHSIncome
From sumClaimsSettlement
What I need the end result to look like...
Type Q1 Q2 Q3 Q4
Traditional
Teclar
All Claims
Under the Quarter cloumn will be Total Charges. What is the better way to do this? Should I go with a Pivot table or make it into a flat file?
Any one can help for how to create sql table dynamically as per flat file format...
its required very urgently
I'm using sqlserver 2000 enterprise edition. I am an oracle dba and we have some tables in sqlserver 2000 that we need to write out to the flat file. I have a procedure in oracle to do this for oracle tables. But, how would I do this in sqlserver 2000. I have 10 columns on this table and I only want 3 columns data to be dumped on the flat file. We are on NT sever 4.0.
Thanks,
Ranjan
Several years ago, I am sure that I had a query that either read from or wrote to a text file. There was some way to:
select * into 'c:\out.txt' from SomeTable
Does anyone know if this functionality exists? If so, what is the syntax?
TIA
I need to do some advanced formatting in the text file, because the data in the text file is bit complex. I have some knowledge of using BCP, BULK INSERT and bit about FORMAT FILES but I want to do some advanced formatting(using IF condition and all) to pull the data from text file.
Can anyone please tell me how can I perform bit advanced formatting using BULK INSERT?
Cheers
What I would LIKE to do is noted in the subject line. What I'm findingis that "edit SQL" appears to only be an option if I am creating atable. If I select "append to" the option to edit SQL shades itself asunavailable.The reason I'd like this is that there is a datum in the flat file thatindicates whether that record should be appended to that table notedabove. There are other ways of dealing with this "problem" but it wouldbe nice to be able to control it using SQL, in the DTS import/exportwizard.If the source of my data is an SQL table, I can generate an SQL queryto specify what fields to import in an append, to check for existingvalues, etc...Is there a way around this? I can reserve a table for data transfers,regularly overwrite it with new data from text file inputs, and use SQLto insert select fields from that transfer table to other databasetables. (From this "transfer" table, data needs to be inserted intofour separate tables in our database).I hope this is clear. If it CAN'T be done this way, it's okay...just alittle ugly with the need to re-create the transfer table.
View 3 Replies View RelatedHi All,
I have read what people have posted on this but don't seem to find exactly what i want. I tried the methods from this website like many suggest http://www.sqlis.com/default.aspx?311. But all this seems to do insert new records but does not update the existing records.
This is what i want to do. I have an online store that uses an SQL DB with a products table. I have an excel/csv spreadsheet with the same columns as in the sql database. It is much quicker for me to update the spreadsheet than logging in to sql to do it. So i would like to update all the product prices on flat file to the sql table. There will be more updating than adding new rows.
Please can you help.
Thanks
Hi,
I noticed something strange today. I created a pkg that reads a flat file and writes the rows to a table.
In checking the data in the file against what's in the table, I noticed that the rows were inserted in a different order than they are in the file.
All the rows appear to be in the table correctly, but they're just not in the same order as in the file. I've never seen this before. But I checked very carefully, and this is indeed the case.
Is this normal??
Thanks
Hi Guys,
I hope this is easy stuff for you..
First of all i searched the forum but didnt exactly find what iam searching for:
I have a File folder which contains 1..n Files of the same type.
The files contain a DateValue at the beginning of each row. I now want to read the first Record of the file - extract the datevalue and search in my Importtable if there are any records with that Date. If there are allready Records with this date i know i allready imported that file and skip it in my For each File Container. If no records where found I want to copy them from the file to the table.
So I have a flatfile source and thoght I just make an oledb command task afterwards which looks like "Select count(*) from Import where Processdate = ?. and then a conditional split if the count == 0 or not... but i have problems getting the Count value out auf the OLD DB Command Task because everytime I try to add an outputcolumn i get the message: "An Output cannot be added to the output collection" and since there is no possibility to map an expression to the result...
I tried to workaround the problem using a lookup task.. but that seems to be the wrong way.
thanks for your help
bye
AS
We have project in SSIS:
Control Flow
Load Data Flow Task to Copy Data Flow Task to Scrip Task
Data Flow under
Load Data Flow Task has
Flat File Source to Row Count1 to OLE DB Destination (ODS database)
Variable name for Row Count1 is RowCount
Data Flow under
Copy Data Flow Task has
OLE DB Source to RowCount2 to OLE DB Destination (WIMS database)
Variable name for Row Count2 is RowNumber
Data Flow under Script Task is code:
' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.VisualBasic
Imports System.Diagnostics
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.IO
Public Class ScriptMain
' The execution engine calls this method when the task executes.
' To access the object model, use the Dts object. Connections, variables, events,
' and logging features are available as static members of the Dts class.
' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
'
' To open Code and Text Editor Help, press F1.
' To open Object Browser, press Ctrl+Alt+J.
Public Sub Main()
Dim varMyRowCount As Variable = Dts.Variables("RowCount")
Dim varMyRowNumber As Variable = Dts.Variables("RowNumber")
Dim varPackageName As Variable = Dts.Variables("PackageName")
Dim varStartTime As Variable = Dts.Variables("StartTime")
Dim varInstanceID As Variable = Dts.Variables("ExecutionInstanceGUID")
Dim varMailMsgtext As Variable = Dts.Variables("MailMsgText")
Dim PackageDuration As Long
Dim Filenum As Integer
Dim FilNam As String
'<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
' Event log needs
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Dim sSource As String
Dim sLog As String
Dim sEventMessage As String
Dim sMachine As String
'<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
PackageDuration = DateDiff("s", varStartTime.Value, Now())
sSource = "RowCountReporting from SSIS"
' We need the message posted to the Application event log.
sLog = "Application"
sEventMessage = "Flat File Row Process: " & Chr(10) _
& " case Rows:" + varMyRowCount.Value().ToString + Chr(10) _
& "Wims Rows Processed: " & Chr(10) _
& " case Rows:" + varMyRowNumber.Value().ToString + Chr(10) _
& "Started: " & varStartTime.Value().ToString _
& Chr(10) _
& "Current Time:" & System.DateTime.Now _
& Chr(10) _
& "=============================================" _
& Chr(10) _
& "Package Run Duration in seconds: " & PackageDuration _
& Chr(10) _
& "Execution GUID: " & varInstanceID.Value().ToString
sMachine = "."
varMailMsgtext.Value = sEventMessage
If Not EventLog.SourceExists(sSource, sMachine) Then
EventLog.CreateEventSource(sSource, sLog, sMachine)
End If
Dim ELog As New EventLog(sLog, sMachine, sSource)
ELog.WriteEntry(sEventMessage, EventLogEntryType.Information, 777, 2)
Dts.TaskResult = Dts.Results.Success
End Sub
End Class
We want to verify flat file count should be same as the data load to WIMS database.
Now if I run this job I get the error message:
Error: Failed to lock variable "RowCount" for read access with error 0xC0010001 "The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created.".
My co-work said even I get balance from RowCount1 and Rowcount2 still need to have query from table in Wims database because for SQL server sometimes process counts not same as the data results load in table so need select count(*) from table. How and where I can put at the job.
My design is row count from flat file put one variable then row count from final table put the other variable then send e-mail to me show both variable for testing.
For real job I need if both variables same ok else send me an e-mail.
Any one has any idea? I will very appreciate.
Thanks.
Grace
I'm rather new to ssis and I've been reading and testing but didn't find a solution for this problem.Supose I've got a table Customer with some fields. One of the fields is CustID.I want to create as many flat files as there are Customers in the table with flat file name set to the CustID.If you could point me in a good direction, It would be nice.Greetings from Belgium
View 4 Replies View Related
Hi all,
I'm trying to work out how to update a table from a flat file source.
I could use the simple way of using OLE DB Command with
UPDATE <tblName> SET Column1 = ?, Column2 = ?, <etc> WHERE ID = ?
then using the mapping to join the parameter fields.
However, this (maybe!) a slow process when you have potential 500k+ records in the source file (and approx 150 files to process!)
What I would like to do is the following statement but I'm not sure if this is possible?
UPDATE <tblName> SET Column1 = sourceColumn1, Column2 = sourceColumn1, <etc> FROM <sourcefile> WHERE ID = sourceID
I've also thought about importing the sourcefile into a Recordset Destination and trying to use the variable in the SQLCommand but I'm not sure how to do this...
Does anybody have any ideas on how this could work or an alternative solution?
Thanks in advance....
If the system not able to convert the data from one data type(string,int) to another data type then i need to move the reocord into my error_table.
How to redirect the bad data into another table, while extracting the flat file data ?
Bad Data means if the system not able to convert the data from one data type to another data type.
Say €œX2R322€? data can€™t convert as DATE data type but "070899" can be converted as DATE data type.
Thank you very much.
I am running a set of SQL statements on a SQL server, to insert flat file data into a SQL table. The flat file is already FTP'ed to the SQL server. I seem to be getting an error, which is possibly pointing to a permissions issue
The statements:
BULK INSERT [Jedox_prod].[dbo].[B_BP_Customer]
FROM 'c:jedox_dailyjdcom4401.txt'
WITH
(
FIRSTROW = 2,
MAXERRORS = 0,
FIELDTERMINATOR = '|',
ROWTERMINATOR = '
'
)
GO
The error is :
Msg 4861, Level 16, State 1, Line 1
Cannot bulk load because the file "c:jedox_dailyjdcom4401.txt" could not be opened. Operating system error code 3(failed to retrieve text for this error. Reason: 1815)
If it is permissions issue, how do I overcome this?
This question is around how we can get the data types and lengths populated into the flat file source columns.
In Connection Manager, you have your flat file defined. You can choose "Suggest Types...", and the minimum lengths and correct data types will be returned from within the data in the flat file.
Is there some way to automate this data type definition, but coming from the other direction (coming from the destination table that we are loading)?
For example, you have mapped the columns that will be loaded. Can you then reverse engineer the data types and lengths for the columns in the flat file from the destination table?
TIA
I have a flat file with header and detail information, it is actually employee punch card data. I need to parse the header line which contains the Employee ID and don't save it to a table just save the value. Then with the detail line, parse the different data elements and save them along with the employee ID to one table. Then continue until the next header line is read.
The file looks something like this:
FINNEY,RONNIE 0001005420
Mon 09/03 700a HOL 8.00
Tue 09/04 630a*E 326p 8.50 8.50
Wed 09/05 645a 330p 8.00 16.50
Thu 09/06 639a 2.40 18.90
HALL,MARK 0001005601
Mon 09/03 700a HOL 8.00
Tue 09/04 608a*E 257p 8.40 8.40
Wed 09/05 601a*E 259p 8.50 16.90
Thu 09/06 606a*E 3.30 20.20
JONES,WILLA JEAN 0001005702
Mon 09/03 700a HOL 8.00
Tue 09/04 556a*E 326p 9.10 9.10
Wed 09/05 600a*E 328p 9.00 18.10
Thu 09/06 554a*E 3.50 21.60
So I think I need a data flow transformation object that let's me save the Employee ID into a variable available when the next record is read. What type of transformation would be best?
Hi All,
I have a table A. I need output subset of a table A to a flat file using query, like:
select A.* from A inner join B on... ..... inner join C......where left(A.id, 3) = B.sid.... AND B.num between 100 and 200).
How can I do this in SSIS? Which data flow item I may need?
Thanks
I have a table dbo.Sales that contains all sales records. There is a column in that table called ItemNumber that I'd like to match with ItemNumber in a flat file and update the ItemCost based on the ItemCost column in the flat file.
So while there will be many sales records for each ItemNumber, I need to loop through and update the ItemCost in that sales record based on the corresponding ItemCost in the flat file. Does this make sense? I really need this for court and I can't figure out how to do it. I took a SQL course about 7 years ago but have forgotten everything.
Database Name: BTData
Database Table: dbo.Sales
Database Columns: ItemNumber (match on this), ItemCost (update this)
FlatFile Name: InventoryCosts.txt
FlatFile Columns: ItemNumber, ItemCost
There will be many sales records for each ItemNumber in the database table. I need to update each one with correct cost based on the item number and cost mapping from flat file.
Hi all,
I am facing a problem on validating the data from a flat file while inserting the data into the destination table of sql server 2005 database. In my package, i have to validate the input data whether the values are coming as null or not, before inserting into the destination database. The flat file may not contain data for all NOT NULL columns. I have to find out that row(s) and reject the record. If the rows are coming as Null for the Not Null columns, the OLEDB Destination throws OLEDB exception for the constraint.
To resolve this, i have an script component in data flow, to check whether the input data is coming as null. I have added the output column of boolean type to the script component, it will be assigned to TRUE when there is null for the Not Null column in the script.
And in the follwing conditional split, i am checking the flag for TRUE to reject the record.
Is there any other way to handle this validation?
Regards
Madhav
Hi -
I have a SSIS package that dumps data from an internal table to a flat file output using standard data flow tasks. The entire table is output - no special SQL. Most of the time the records are placed in the output file in the same order as the internal DB table, but occasionally the order appears to be more random. When that happens, the record order in the internal table is correct - it is just the output.
I can find no properties that seem to affect this. I would appreciate any hints and advice that anyone can give me. Has anyone else encountered this same problem?
Thanks in advance,
Jim Sweeney
I am new to SSIS and am having trouble with automatically setting up the destination output columns.
I am sure there must be an easy way to do this.
My table (source) has 86 columns in it of varying lenghts.
In my connection magagers, I have created one for the SQL Server (source data) and one for the flat file (destination output).
I have also created an OLE DB source data object and a destination Flat File object and set them up to the respective connection managers.
Finally I have linked the source to the destination.
Now when I look at the source, it shows me all 86 columns.
When I open up destination, there are no columns set up.
Problem: do I have to type in all the columns manually in the connection manager for the Flat File?
I would think there would be some automatic way that it would self-populate the columns over to the flat file destination.
i have this flat text file that has a number of packed decimal
field type. How do I load that text file into a sql table.
thanks
I am sorry, I am posting this message again, since I did not get anyreply.I want to export a table into a "fixed width" file using SQL 2005import export wizard.This is the version I have:SQL Server 2005 - 9.00.2047.00For some reason it joins all the rows together. For EX: if the tableis like this:Create table Mytable (col1 varchar(50) null, col2 varchar(60) null,col3 varchar (100) Null)Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")It is not exporting every row in a single line. Actually if I open itin "Ultra Edit", it is all in one line.I used to do this regularly with SQL 2000 import export wizard and itexported every row in one line.I looked at the setting:The header row delimiter has {CR}{LF}Code page has 1252 Ansi-Latin.In the Advanced tab:String:dt_str.I tried changing the header row delimiter to just {CR} or just {LF}.Also I tried changing the string to dt_text and nothing seems to help.Please help.Thank you
View 1 Replies View Related