Hi, i'm still rather new to sql server and sql server express. My problem is that I have 50 fixed width files(actually was LIS file type, converted to txt using notepad). Each text file has about 50 columns of data. What would be the best way to get this into sql server db?
thank you in advance.
I can't use DTS nor DTSwizard as I need to put it in a .sql and run it through a command line via .bat file (it's more for the users).
Each row ends with an EOL character, the fields are all fixed width, but I have a little problem here, some rows are empty but just with a EOL character.
How shall I go about it?
many thanks! :D
I'M HAVING AN ISSUE, UNDERSTANDING, THE CONNECTION STRING.
I WANT TO CONNECT TO AN INSTANCE OF SQLEXPRESS ON A REMOTE SERVER WITH A FIXED IP ADDRESS
THE TCP PORT IS OPEN TO 1433
I OPENED THE PORT ON THE ROUTER AND THE WINDOWS 2003 FIREWALL
MY CODE IS AS FOLLOWS:
S = "Provider=SQLNCLI;"
S = S & "DATA SOURCE=44.66.777.888,1433SQLEXPRESS;"
S = S & "INITIAL CATALOG=TESTDB;"
S = S & "Persist Security Info=false;"
S = S & "UID=TEST999;"
S = S & "Pwd="TEST999888;"
CnMgt.ConnectionString = S
CnMgt.Open S
I'VE FOLLOWED THE STEPS OUTLINED IN ARTICLE 914277
CAN SOMEONE HELP?
THANK YOU
PS. IF THE CLIENT IS LOCAL, I HAVE NO ISSUE OPENING THE DATABASE.
I DO NEED TO OPEN THE DB FROM FROM CLIENTS.
I wasn't sure where to put this topic so I put it here since I figured it is a question that would apply to virtually any version even though I am using SQL Server 2005.
We have a vendor that sends us a fixed width text file every day that needs to be imported to our database in 3 different tables. I am trying to import all of the data to a staging table and then plan on merging/inserting select data from the staging table to the 3 tables. The file has 77 columns of data and 20,000+ records. I created an XML format file which I sampled below:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="RetNo" xsi:type="CharFixed" LENGTH="6"/>
[Code] ....
The data file is a fixed width file with no column delimiters or row delimiters that I can tell. When I run the following insert statement I get the error below it.
BULK INSERT myStagingTable
FROM '.........myDataSource.txt'
WITH (
FORMATFILE = '.........myFormatFile.xml',
ERRORFILE = '.........errorlog.log'
);
Here is the error:
Msg 4832, Level 16, State 1, Line 1
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
what is the best way to import fixed length text file to sql server using SSIS?
I was trying to using text file source and ole db destination..but since the text file has no columns and have different length per column and per line( it show only one column becasue it all concatnated), I can not map it to destination column..
How can I import it?
Here is the example of text file ( fixed with row delimeter)that i need to import to different columns...
010000000000000000001164.00023 YV
02004101 1 2008-04-OLL 43456 0000000001 2008-04-08
030000100000000000000000000007.00
047890 7556 YYU 779
Hey All,
Similar to a previous post (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=244646&SiteID=1), I am trying to import data into a SQL Table.
I am trying to program a small application that will import product data obtained through suppliers via CD-ROM. One supplier in particular uses Fixed width colums, and data looks like this:
Example of Data
0124015Apple Crate 32.12
0124016Bananna Box 12.56
0124017Mango Carton 15.98
0124018Seedless Watermelon 42.98
My Table would then have:
ProductID as int
Name as text
Cost as money
How would I go about extracting the data with an XML Format file? I am stumbling over how to tell it where to start picking up data for a specific column.
Is there any way that I could trim the Name column (i.e.: "Mango Carton " --> "Mango Carton")?
I don't know if it makes any difference, but I've been calling SQL from my code by doing this:
Code in C# Form
SqlConnection SqlConnection = new SqlConnection(global::SQLClients.Properties.Settings.Default.ClientPhonebookConnectionString);
SqlCommand cmd = new SqlCommand();
cmd.CommandType = CommandType.Text;
cmd.CommandText = "INSERT INTO PhonebookTable(Name, PhoneNumber) VALUES('" + txtName.Text.ToString() + "', '" + txtPhoneNumber.Text.ToString() + "')";
cmd.Connection = SqlConnection;
SqlConnection.Open();
cmd.ExecuteNonQuery();
SqlConnection.Close();
RefreshData();
I am running Visual Studio C# Express 2005 and SQL Server Express 2005.
Thanks for your time,
Hayden.
Hi,
View 5 Replies View RelatedHi,
I try to use Data Transaction Service (SQL Server 7) to copy information from an Sql Server Table to an text file (fixed field). When I run the process, no problem, the text file is created and a Ok message appears. But when I look in my text file, every time a field was NULL or empty in the source table, the following fields are not aligned !
Somebody knows what the problem ?
Thanks
Vincent
Ps: I'm french so excuse me for my english
I have a request from a vendor to export data out of my SQL Server 2K database view to a 'flat fixed file'.
What kind of file is this exactly, not a .csv ? Does EM have the capabilities through the DTS wizard, by choosing the output to a text file and fixed width ?
Thank you
I need to write data into a fixed column length file and was wondering the best (most efficient) way to tackle this. For example, the first few pieces of the report I'm working on now would be:
PacketID - Starting position 1, Field length 9
TransactionID - Starting position 10, Field length 9
Group number - Starting position 19, field length 10
PID/SSN - Starting position 29, field length 10
For the PID/SSN, if I have a PID it'll be 10 digits and fill the field length, if I don't I use SSN which is only 9 digits and enter a space as the 10th digit. Obviously if I don't have certain pieces of information I'll just need spaces of the specified length to satisfy the file format. I'm using SQL 2005. Thanks in advance for any help provided.
I have approximately 13 columns. Each Column has a start position and end position.. I created this in a table and defined the position, it's still not working for me.
FiceCode char(6), -- starting position 1, field length 6
StateStudID char(10), -- starting position 7, field length 10
CampusStudID char(10), -- starting position 17, field length 10
LastName char(25), -- starting position 27, field length 25
[Code] .....
I need a text output file that will define each start position.I also used: right(replicate('0',25) + cast(last_name as char(25)), 25) in my sql statement.when I add the first_name, I can't get it to start in position 52.
I am attempting to import a fixed width file into a SS2005 table and am having problems when importing a date that has no value in it. The table will allow nulls.
The date is in dd/mm/yyyy foramt and when there is no date then there are 10 spaces. When transforming the data I TRIM the data down using a derive transform script so all there is, is an empty string. When the file attempts to load I get the following message:
[OLE DB Destination [2238]] Error: There was an error with input column "paid_date" (2306) on input "OLE DB Destination Input" (2251). The column status returned was: "The value could not be converted because of a potential loss of data.".
How can it potentially lose data when there is nothing to lose?
I need some way of converting the empty string into a null. Has anyone got any ideas for me?
I have a fixed width file that I only need a middle field imported.
The field I need starts at character 116 and is 15 character wide. What should my format file look like?
I have a format file that can import a field before it but this is data I don't need. How do you skip the first field?
I have a fixed-length flat file that contains about 30 columns. I have got it pretty well figured out using the flat file connection tool, but I am having trouble with the end of the line.
I know when I look at the file it is a CrLf that separates the rows, and SSIS only seems to understand this to a certain extent. It knows to go to the next line, but it also adds two rectangles to the lines, like this:
Col1 Col2 Col3 Col4 Col5
aaaaaa bbbbb ccccc ddddd eeeee
[][]aaaaa bbbbb ccccc ddddd eee
ee[][]aaaaa bbbbb ccccc ddddd e
eeee[][]aaaaa bbbbb ccccc ddddd
While this does create a cool pattern, it is a pain in the butt. The only solution I have found is adding two more spaces to the last column in the table, but the ?s just get appended there.
If anybody has any clue how to get rid of them, that would be great.
Thanks in advance
What is the best way to export fixed length data to text file in SSIS?
View 12 Replies View RelatedI have a table with 3 columns (ID Int , Name Varchar(25), Course Varchar(20))
My source data looks like below
ID Name Course
1 A Java
1 A C++
2 B Java
2 B SQL Server
2 B .Net
2 B SAP
3 C Oracle
My Output should look like below...
ID Name Course(1) Course(2) Course(3) Course(4)
1 A Java C++
2 B Java SQL Server .Net SAP
3 C Oracle
Basically need t-sql to Convert non fixed rows to non fixed columns...
Rule: IF each ID and Name have more than 1 course then show it in new columns as course(1) course(2)..Course(n)
Create SQL:
Create table Sample (ID Int null , Name Varchar(25) null, Course Varchar(20) null)
Insert SQL:
INSERT Sample (ID, Name, Course)
VALUES (1,'A','Java'),
(1,'A','C++'),
(2,'B','Java'),
(2,'B','SQL Server'),
(2,'B','.Net'),
(2,'B','SAP'),
(3,'C','Oracle')
Export to Fixed width text file
I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
Hi, Does anybody know how to import a fixed field length ASCII text file which is 370 bytes into a SQL Table by DTS?
Thnaks,
Mano
Will be getting new file for download from vendor to process in future. When I use DTS or Import wizard in sql server I get "could not find the selected row delimiter within the first 8kb of data, is the selected delimited valid? This is for a fixed length file. If I answer yes and continue everything is fine, until I get to the end of the record which it can't find. It basically lumps the records together. What is interesting, that if I import the same file in Access 2000, I don't get this problem. ANyone seen this before? Could not find anything on MSDN
View 2 Replies View RelatedHi,
I am trying to upload a fixed field text file to a sqlserver table using the DTS wizard. The txt file has 111 columns and the total length of a single row is 5897. The problem is when I use the wizard to specify the starting and ending of each column, its not allowing me to specify the columns beyond the position 4095.
Is there a limitation on this? if so is there a work around ? to solve this.
Any help on this is truly appreciated.
Thanks much. :)
I'm trying to do an insert using Bulk Insert with a fixed length file.I'm using a format file.I'm getting the following error message:Cannot perform bulk insert. Invalid collation name for source column 16in format file '\wbhq.comdfsdviDataIntGOPFilesGOPFormatFile. txt'.Any suggestions are appreciated.Thanks!JenniferFormat File Contents:8.0161SQLCHAR02""0Space""2SQLCHAR04""1YearID""3SQLCHAR02""0Space""4SQLCHAR02""2PeriodID""5SQLCHAR02""3CompanyID""6SQLCHAR01""0Dash""7SQLCHAR04""0Space""8SQLCHAR01""0Dash""9SQLCHAR04""4UnitID""10SQLCHAR01""0Dash""11SQLCHAR04""5AccountCode""12SQLCHAR05""0Space""13SQLCHAR01""6AccountType""14SQLCHAR029""0Space""15SQLCHAR016""7GLAmount""16SQLCHAR0105"
"0Space""Bulk Insert Statement:BULK INSERT FlatFile_GOPFROM '\wbhq.comdfsdviDataIntGOPFilesGLPAM.GOP'WITH(FORMATFILE ='\wbhq.comdfsdviDataIntGOPFilesGOPFormatFile. txt')Table Definition:CREATE TABLE [dbo].[FlatFile_GOP] ([YearID] [smallint] NOT NULL ,[PeriodID] [smallint] NOT NULL ,[CompanyID] [smallint] NOT NULL ,[UnitID] [smallint] NOT NULL ,[AccountCode] [int] NOT NULL ,[AccountType] [char] (1) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,[GLBalance] [money] NOT NULL) ON [PRIMARY]GOFile Contents:2007 210- -0002-3000 G196395.102007 210- -0002-3700 B1484.002007 210- -0002-3700 G1571.132007 210- -0002-3800 B157457.002007 210- -0002-3800 G161577.73
I have a requirement to import a file of rows containing fixed length data. The problem is that each row can be one of 5 different formats (i.e. different columns) -- where the "type" of row is indicated by the first two characters of the row. Each row gets inserted into its own table.
Could I use a simple Conditional Split to route the rows? Or is the split for routing similiar rows? Anyways, problems are never this simple...
In addition, each "grouping" of rows is related. The "first" row is considered the "primary" row (and gets a row id via IDENTITY, whereas the remaining rows in the group are "secondary" rows and have foreign key references back the the primary rows id.
Given (using spaces to separate columns and CrLf to show "grouping"):
01 MSFT blah blah
02 blahblah blahblahblah
03 boring boringblah
01 AAPL blah blah
02 blahblah blahblahblah
03 boring boringblah
01 CSCO blah blah
02 blahblah blahblahblah
02 blahblah blahblahblah
03 boring boringblah
So, the first 3 lines are all related to a MSFT record which needs to be spread across multiple tables. The next three lines are all related to AAPL, And the next FOUR lines (yes, each record can have zero, one, or more secondary rows) are related to CSCO.
(If this is still not clear, all the "01" rows will be written to [Table1] with each row having an IDENTITY value. All the "02" rows will be written to [Table2] the a FK pointing to the correct [Table1] row. All the "03" rows will be written to... and so on.
Any ideas would be appreciated.
Hi,
I was trying to import a fixed-width file to a sql 2005 table.
The total record lenght is 1500. I was trying to import it to a single column.
The strange thing that's happening is: SSIS is inserting only the first 32 chars of the record and the remaining are gone. I tried using nvarchar(max) and varchar(max) but of no use.
I think something somewhere is going wrong but I was unable to figure it out. Earlier I was able to load a similar file into a single column table.
My Header row delimiter is {CR}{LF}
The preview pane shows the complete record but when it transfers to the table, I'm getting 32 chars only.
Can anybody suggest any ideas to figure this out?
Thanks,
Siva.
Export to Fixed width text file
I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
Hello,
I have a series of fixed width files, all with the same schema. I need to import the data into a SQL Server table. Each record in the flat file begins with 'D1'. The length of each record (string) is 380. There are cases where the record ends after position 193, and a new record appears in the current string beginning at position 194. So at position 194 'D' appears, and '1' appears at position 195.
In the flat file, I need to insert a line break after position 193 if position 194 = 'D' and if position 195 = '1'. I'm guessing I would do this with a Script Component Transformation. Once the file is edited, then I can bring the data into the table.
What might the script look like? If you have any suggestions, samples, or know of examples on the web you can point me to, please share.
Thank you for your help!
cdun2
Currently we're working on an SSIS package to extract data from a SQL Server database to several fixed width flat files.
Some of the data needs to be formatted/converted in a certain way
DateTimes need to be formatted in ISO8601Booleans need to be 0/1 instead of False/True...Has anybody any idea what the preferred approach (best practice) would be to do these conversions?Convert everything in the select query? What about readability of your query?
Do it somewhere in the package? If so, how?....
Hi,
I am trying to import a text file that has fixed length fields.
It also has column headers repeated in the file.
The text file is delimited by <CR><LF>.
Question 1
There are certain rows that end abruptly after a column for ex:
Row 1 <col1 data>.....<col2 data>.....<col3 data><CR><LF>
Row 2 <col1 data>.....<col2 data><CR><LF>
This seems to be throwing off the text file import as row 1 seems to be importing alright but row 2 gets ignored.
This not the behaviour I want. I want row 2 to also be imported and have a default of NULL for the columns that
are not specified.
Does anyone know how to achieve this?
Question 2
This is not that serious, but currently, I do not know of a way to ignore repeating column headers.
It would be nice if there was a way, but I can always resort to T-SQL based data cleaning after the import.
Thanks.
Shailen Sukul
|BSc|Mcsd.Net|Mcsd|Mcad|
Software Architect / Developer
Ashlen Consulting Services
http://www.ashlen.com.au
Hi All,
I have a file that has fixed row size of 148 and fixed column size, but the file has no end of line character. I know it is wierd but a client has made the file and refuses to change the format. So I am stuck with reading it the way it is.
In Enterprise Manager, I used the Import/Export wizard and I specified fixed length and it let me specify 148 as the lenght of each line. Then it recognized the file and I was able to read it in.
I saved the DTS package and I can run it over and over again using dtsrun. However I want to do the same thing using Bulk Insert. How do you specify fixed row length for Bulk insert and how do you give it individual column lengths?
Thanks,
Shab
I'm using SSIS to do bulk inserts from fixed width files to about 20 tables in my SQL database.
The problem I'm running into is in creating Format Files for the bulk insert task to use. I've gotten the bcp command to create format files that will read csv files, but I can't seem to figure out how to get it to create one for fixed-width.
I know it can be done: http://msdn2.microsoft.com/en-us/library/ms191234.aspx
At the bottom (Section F) it shows an XML format file for reading a fixed-width file. When I manually create one of these to match one of my tables, the bulk insert worked fine.
Closest I've come is with this ( [] bracketed items are correct values, just censored here):
C:Program FilesMicrosoft SQL Server90ToolsBinn>bcp [database].[owner].[table] format
nul -c -f C:TableFMT.xml -x -S[Server] -U[Username] -P[Password]
My question is, what is the bcp command to create this sort of XML format file?
Hi,
There's a lot of information on importing data from text files, but not a lot on exporting data to text files... I've checked but found no info on this.
I'm trying to export data from SQL Server to a fixed-width flat file and wondering if I'm doing it the right way.
I use a view as source (using a OLEDB connection manager) and I can see the data without problem.
I defined a Flat File Destination (using a flat file connection manager). When setting up the flat file connection manager, I am asked for a file... Does this mean one should create manually a template file with the desired output format? So I used a production file as template since we're replacing an existing process.
After having set up everything, I run the SSIS only to see all the data on the same row. There are no CRLF...
When I create the file connection manager, there's no way to mention the row delimiter. In the properties I see a "Row Delimiter" field and when I try with "{CR}{LF}" it makes no difference. Interesting to note that, contrary to the HeaderRowDelimiter field, the RowDelimiter field has no drop-down control to give choices.
So I had to return the CRLF as the last field of the source view (SELECT .... ,'CRLF' = CHAR(13) + CHAR(10) FROM ...) to make it work.
Seems odd... Is it the way to go?
Thanks
Hi,
I'm trying to extract data from a Flat File which is as fixed length as they come. The file has a header, which simply contains the number of records in the file, followed by the records, with no header delimeter (No CR/LF, nothing).
For example a file would look like the following:
00000003Name1Address1Name2Address2Name3Address3
So this has 3 records (indicated by the first 8 characters), each consisting of a Name and Address.
I can't see a way to extract the data using a flat file connection, unless we add a delimeter for the header (not possible at this stage). Am I wrong?
Any suggestions on possible solution would be much appreciated - I'm thinking Ill have to write a script to parse the file manually.
Thanks in advance,
Scott
When I use SQL 2000 DTS Export to create a fixed length flat file, the data rows are delimited by carriage return-line. Which means that when I open the flat file in a text editor like UltraEdit or WordPad, the data rows are broken out nicely (row ends at the max row length position and new row starts at position 0).
But when I use SSIS to create the file, the whole file is displayed as one line in WordPad. The data rows don't end at the max row lenght position in ultraEdit neither. From Flat File Connection Manager's Preview page, I can see the data rows are displayed properly.
Now I wonder if the flat file destination is a true fixed length file.
Hi,
I am trying to write to a fixed format flat file using Flat File Destination Data Flow Component. I have all required information gathered from more than one sources. But when I tried to format the columns to a big string that will make up one line in the flat file, I could not figure out how to do that. Couple of issues that I am facing are:
How to padd different columns? For example, One interger column has could be 1 to 10 character long in my case. When I convert to string, dont know how to padd the remaining characters i.e. if the value of integer is '1234', it should be written to file as '1234 ' . Which transformation is best in this case, if available?
How to convert T-SQL datetime to a specific date and time format to write in the flate file? I have to write these date formats depending upon one of the parameters passed.
Also, I dont want to put a delimiter at the end of each column, just the new line characters at the end of each record.
Some of the columns has some unwanted characters (like new line characters) how to find them and remove them from the string.
Can we directly write columns to a specific position in the flat file? e.g. col 1 a position 1 and col2 starts at postion 20 etc.
Your co-operation will be appreciated.
Thanks,
Paraclete