BCP And Treatment Of NULLs When Export To Fixed Format Text File
Jul 23, 2005
We are using the bcp utility (via APIs) to export data from a SQL table
in a fixed format text file. BCP is inserting spaces for a field if
the field contains a NULL. This is fine with us except at the end of
the line, there are no spaces for that field just the end-of-row
terminator prematurely, so it looks like that field is not present and
messes up another piece of software we pump the text file into down
stream.
Example -- The last row illustrates the problem.
123-49-890 Mary Smith Raleigh NC
999-88-123 Henry Ax Boston MA
456-99-123 Sue Kite WA
789-88-126 Andy Yates Philadelphia
We have thought about using a SQL query to convert the NULL data
explicitly to spaces, but were wondering is there a switch or something
in our format file to get around this.
Export to Fixed width text file I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
Export to Fixed width text file I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
Similar to a previous post (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=244646&SiteID=1), I am trying to import data into a SQL Table.
I am trying to program a small application that will import product data obtained through suppliers via CD-ROM. One supplier in particular uses Fixed width colums, and data looks like this:
Example of Data
0124015Apple Crate 32.12
0124016Bananna Box 12.56
0124017Mango Carton 15.98
0124018Seedless Watermelon 42.98 My Table would then have: ProductID as int Name as text Cost as money
How would I go about extracting the data with an XML Format file? I am stumbling over how to tell it where to start picking up data for a specific column. Is there any way that I could trim the Name column (i.e.: "Mango Carton " --> "Mango Carton")?
I don't know if it makes any difference, but I've been calling SQL from my code by doing this:
Code in C# Form
SqlConnection SqlConnection = new SqlConnection(global::SQLClients.Properties.Settings.Default.ClientPhonebookConnectionString); SqlCommand cmd = new SqlCommand();
SqlConnection.Open(); cmd.ExecuteNonQuery(); SqlConnection.Close(); RefreshData(); I am running Visual Studio C# Express 2005 and SQL Server Express 2005.
We're having issues exporting a set of data from SQL to a fixed width flat text file by just doing a right click on the DB, then choosing Tasks > Export Data. You can not specify a row delimiter when you choose a Fixed Width format. The only way around this that we've found is to specificy char(13) and char(10) at the end of the SQL select statement. Without row delimiters you end up with 1 giant record rather than 20,000 regular sized records. Is there any other way around this that we're missing?
Using Ragged Right is not an option either since the record lengths will be inconsistent if the last field doesn't contain a consistent length to the data.
I have a flat file. It's fixed-with with CRLF record delimiters (a.k.a. Ragged Right format).
Some fields are null, and represented by the text NULL.
I'm trying to import the file into SQL via an OLE DB connection. The target table is a SQL 2000 data table. Two of the fields in the target database are of type smallint.
When I run PREVIEW on the data source (Flat File), everything looks good & correct. I added the convert columns task to convert my strings to smallint. This is where things go haywire.
After linking everything up, the conversion gives me a "Cannot convert because of a possible loss of data." All of my numbers are < 50, so I know this isn't the case. Another SSIS bogus error
My first instinct is the SSIS doesn't understand that NULL means null. I edited the file and replaced all instances of NULL with 4 emtpy string chars. Still no good. It seems to be having a hard time parsing the file now.
I dropped the convert task and tried editing the data source, and set the two smallint fields to smallint instead of string (SSIS formats). I get the same conversion error.
Changing the NULL values to 0 fixed the problem, but they're not 0. They're null.
Short of creating another script that converts all zeros to NULL using the aforementioned hack, I'm out of ideas.
I'm I missing something or is SSIS just incapable of handling nulls in fixed-width flat file formats?
I'm using SSIS to do bulk inserts from fixed width files to about 20 tables in my SQL database.
The problem I'm running into is in creating Format Files for the bulk insert task to use. I've gotten the bcp command to create format files that will read csv files, but I can't seem to figure out how to get it to create one for fixed-width.
I know it can be done: http://msdn2.microsoft.com/en-us/library/ms191234.aspx At the bottom (Section F) it shows an XML format file for reading a fixed-width file. When I manually create one of these to match one of my tables, the bulk insert worked fine.
Closest I've come is with this ( [] bracketed items are correct values, just censored here): C:Program FilesMicrosoft SQL Server90ToolsBinn>bcp [database].[owner].[table] format nul -c -f C:TableFMT.xml -x -S[Server] -U[Username] -P[Password]
My question is, what is the bcp command to create this sort of XML format file?
I am trying to write to a fixed format flat file using Flat File Destination Data Flow Component. I have all required information gathered from more than one sources. But when I tried to format the columns to a big string that will make up one line in the flat file, I could not figure out how to do that. Couple of issues that I am facing are:
How to padd different columns? For example, One interger column has could be 1 to 10 character long in my case. When I convert to string, dont know how to padd the remaining characters i.e. if the value of integer is '1234', it should be written to file as '1234 ' . Which transformation is best in this case, if available? How to convert T-SQL datetime to a specific date and time format to write in the flate file? I have to write these date formats depending upon one of the parameters passed. Also, I dont want to put a delimiter at the end of each column, just the new line characters at the end of each record. Some of the columns has some unwanted characters (like new line characters) how to find them and remove them from the string. Can we directly write columns to a specific position in the flat file? e.g. col 1 a position 1 and col2 starts at postion 20 etc.
I am sorry, I am posting this message again, since I did not get anyreply.I want to export a table into a "fixed width" file using SQL 2005import export wizard.This is the version I have:SQL Server 2005 - 9.00.2047.00For some reason it joins all the rows together. For EX: if the tableis like this:Create table Mytable (col1 varchar(50) null, col2 varchar(60) null,col3 varchar (100) Null)Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")It is not exporting every row in a single line. Actually if I open itin "Ultra Edit", it is all in one line.I used to do this regularly with SQL 2000 import export wizard and itexported every row in one line.I looked at the setting:The header row delimiter has {CR}{LF}Code page has 1252 Ansi-Latin.In the Advanced tab:String:dt_str.I tried changing the header row delimiter to just {CR} or just {LF}.Also I tried changing the string to dt_text and nothing seems to help.Please help.Thank you
I have approximately 13 columns. Each Column has a start position and end position.. I created this in a table and defined the position, it's still not working for me.
FiceCode char(6), -- starting position 1, field length 6 StateStudID char(10), -- starting position 7, field length 10 CampusStudID char(10), -- starting position 17, field length 10 LastName char(25), -- starting position 27, field length 25
[Code] .....
I need a text output file that will define each start position.I also used: right(replicate('0',25) + cast(last_name as char(25)), 25) in my sql statement.when I add the first_name, I can't get it to start in position 52.
Hi-I have a sql database (2005) that I need to extract a report from that looks somehintg like SELECT * From Empl_Hours WHERE some_flag <> 'true' .The thing works fine, but the problem is this: I need to insert a record in the 1st row that looks like "Static_text"+row_count() +"more_static_text"where row_count is the actual # of rows that were retrieved. Thanks in advance for any help.DAn
Will be getting new file for download from vendor to process in future. When I use DTS or Import wizard in sql server I get "could not find the selected row delimiter within the first 8kb of data, is the selected delimited valid? This is for a fixed length file. If I answer yes and continue everything is fine, until I get to the end of the record which it can't find. It basically lumps the records together. What is interesting, that if I import the same file in Access 2000, I don't get this problem. ANyone seen this before? Could not find anything on MSDN
I am trying to import a text file that has fixed length fields. It also has column headers repeated in the file.
The text file is delimited by <CR><LF>.
Question 1
There are certain rows that end abruptly after a column for ex: Row 1 <col1 data>.....<col2 data>.....<col3 data><CR><LF> Row 2 <col1 data>.....<col2 data><CR><LF>
This seems to be throwing off the text file import as row 1 seems to be importing alright but row 2 gets ignored. This not the behaviour I want. I want row 2 to also be imported and have a default of NULL for the columns that are not specified.
Does anyone know how to achieve this?
Question 2
This is not that serious, but currently, I do not know of a way to ignore repeating column headers. It would be nice if there was a way, but I can always resort to T-SQL based data cleaning after the import.
I would like to read from a Text File using SSIS Integration Package.
The file has a fixed number of columns, let's say 3 columns. There is no row header and each columns length is fixed. There is no delimiter as well.
Here is the sample of the file contents: John Doe USA Mary Monroe UK Andy Archibald Singapore
Here is the hints to read the file contents 123456789 0123456789 0123456789 ============================== John Doe USA Mary Monroe UK Andy Archibald Singapore
If you notice, from the 1st column until the 9th column, it's reserved for the first name. The 10-th column until the 19th column, it's reserved for the last name. Finally the 20-th column until the 29th column is reserved for the Origin Country.
Since there's no delimiter inside the flat file contents, i have difficulty in parsing this text using SSIS Package.
Please let me know if you need any necessary information.
what is the best way to import fixed length text file to sql server using SSIS?
I was trying to using text file source and ole db destination..but since the text file has no columns and have different length per column and per line( it show only one column becasue it all concatnated), I can not map it to destination column..
How can I import it?
Here is the example of text file ( fixed with row delimeter)that i need to import to different columns...
I am trying to export data from a query in SQL Server 2005 SSIS to a flat file destination. Everything works fine except the rows returned from my query are written to the flat file in one long string (i.e., without line breaks). I have tried appending a new line character to the rows returned from the query but that only throws an error when the package is executed. My rows returned from the query are 133 characters wide (essentially only one column per row) so I have set the properties accordingly for a fixed width file format with 133 character wide rows.
Any suggestions or ideas on how to correct this would be greatly appreciated.
I have a text file I am trying to import into SQL Server using OLEDB connection.
It's a fixed field text file, ragged right format. One of my columns maps to a numeric column in the DB. In some spots in the file, it is blank, in others there is actual numeric data.
I can't get it to import. If I set the text file column to numeric, I get an error "That value could not be converted because of a potential loss of data." If I set the text file column to string, I get a similar error from the OLE DB provider, "Invalid character value for cast specification"
I have tried telling it to retain nulls in the data flow and the other way as well. Can someone tell me what I am doing wrong?
Hi all, I need to export/generate a data file in dbf format from SQL Server 2000 table. I wonder how can this be done inside SQL Server 2000? Would DTS helps? Please advise.
Has anyone out there worked on a project to export data from a SQL Server Database into the SAS JMP file format?
I want to create an SSIS package to take snapshots of our database at regular intervals and export the data directly into a SAS JMP File. I have no idea how to go about doing this.
What is the easiest way to accomplish this task with SSIS?
Basically I have a stored procedure that unions multiple queries between databases. I need to be able to export this to a text file on a daily basis and add a total records: row to the end of the text file.
Using SQL 2005. Need to create fixed lenght text file from table. Was able to create the file, but all the data was in one big line. Selected fixed length with field names. How can I get my text file to have field names, fixed lenght with each record on it's only line. Thank you. David
Anbody please help I am trying to export a text file to a table using enterprise manager and all tasks But the process keeps adding strange charater like squares at the end of each line and also replaces each empty line in the text file with a record in the table with that square type character. I used the following code to delete all rows with that character (as a work around) but no joy. I am losing hope.
I am new to ssis. I try to create a package completely by vb.net to export a table in sql server to text file. i got the following error while i run the package,
An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for ODBC Drivers" Hresult: 0x80004005 Description: "[Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified". The AcquireConnection method call to the connection manager "OLEDBSrc" failed with error code 0xC0202009. component "OLE DB Source" (1) failed validation and returned error code 0xC020801C. One or more component failed validation. There were errors during task validation.
i have posted my code below,
Dim pkg As New Package
Dim OLEDBConMgr As ConnectionManager
Dim FileConMgr As ConnectionManager
Dim SrcComponent As IDTSComponentMetaData90
Dim SrcInstance As CManagedComponentWrapper
Dim DesComponent As IDTSComponentMetaData90
Dim DesInstance As CManagedComponentWrapper
pkg.PackageType = DTSPackageType.DTSDesigner90
Dim e As Executable = pkg.Executables.Add("DTS.Pipeline.1")
Dim thMainPipe As TaskHost = e 'as Task Host
Dim DataFlowTask As MainPipe = thMainPipe.InnerObject 'as MainPipe
I got this code from another one of the MSDN forms. When I run the report and try to export using this format, it still gives me a csv file instead of tab delimited file.
Can someone please help me fix this code so I can get tab delimited text files. Thanks a lot, -Rohit
Hi, How can I convert a text file (.txt) into SQL in ASP.net 2.0 ? The sample of the file format is like that ... 09/03/2007 08:41 "Fung, Kitty" Granted Access D1 Main 2354 111 09/03/2007 08:42 "Ng, Jaclyn" Granted Access D1 Main 21906 18 09/03/2007 08:42 "Leung, Agnes" Granted Access D1 Main 21920 18 Cheers
Hello. I am in the process of migrating an old app to SQL Server. The old app reads hundreds of different flat file formats. One of the more complex ones is a multi-format delimited file. For example:
01^Bob Johnson^123 Main St^Anytown^St 02^Book1^$20 02^Book2^$30 03^Gift Cert^Happy Birthday^$100
This file is delimited with the ^ character. Note that the first 2 characters identify the row type. All 01 rows have data in the format: Name, Street Address, City, State. All 02 have data in the format: Book name, price. Etc.
Any clever ideas on how to parse this? I tried setting it up as a flat file source with the ^ delimiter. It doesn't work - in this example it wraps the third row to the end of the second row and keeps adding columns to fill out the row.
The only option that I can think of is to pull the entire row into one long column, and then use a script component to manually substring each column out.
If I tries to extract data as Delimited the columns are not containing the same data for each row as there are some wired characters in the "Description" filed.
How can I extract this data as proper csv file? If not is there a way to extract only ther first line in detail row as shown below
When using DTS (in SQL 7) to export via OLE DB a large varchar to a text file, it clips it at 255 chars. No other data access drivers seem to work, either. This is lame! I cannot use bcp as a work around, because i want to use quoted comma-delimited, which it doesn't support, and I am using query-based export, where the query calls a stored proc, which bcp also doesn't support.
Are there any new versions of MDAC that fix this? Anyone know a workaround? My current hack fix is to split my field into 2, but this is a grubby fix that hassles my reciptients.
This is a pretty fundamental limitation to a major product!
Hi, I need to export data from SQL server 2000 database into text file uisng ç Delimited. Because my destination database will be teradata. Could you let me know if you have any method for this. Thanks
