SQL Server 2014 :: Bulk Insert A Fixed Width File
Jul 29, 2015
I wasn't sure where to put this topic so I put it here since I figured it is a question that would apply to virtually any version even though I am using SQL Server 2005.
We have a vendor that sends us a fixed width text file every day that needs to be imported to our database in 3 different tables. I am trying to import all of the data to a staging table and then plan on merging/inserting select data from the staging table to the 3 tables. The file has 77 columns of data and 20,000+ records. I created an XML format file which I sampled below:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="RetNo" xsi:type="CharFixed" LENGTH="6"/>
[Code] ....
The data file is a fixed width file with no column delimiters or row delimiters that I can tell. When I run the following insert statement I get the error below it.
BULK INSERT myStagingTable
FROM '.........myDataSource.txt'
WITH (
FORMATFILE = '.........myFormatFile.xml',
ERRORFILE = '.........errorlog.log'
);
Here is the error:
Msg 4832, Level 16, State 1, Line 1
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
View 1 Replies
ADVERTISEMENT
May 14, 2007
I can't use DTS nor DTSwizard as I need to put it in a .sql and run it through a command line via .bat file (it's more for the users).
Each row ends with an EOL character, the fields are all fixed width, but I have a little problem here, some rows are empty but just with a EOL character.
How shall I go about it?
many thanks! :D
View 2 Replies
View Related
Dec 6, 2013
I have like 100 columns and most of them work fine, but some will not work. I need to select the columns with correct width for fixed width flat file.
Here is typical SQL statment that works for most of them.
left(RTRIM(A.City) + Replicate(' ', 25) ,25) as [City]
The above one is not working, but many other works fine like the address. Why would it work for some, but not others?
It works fine if I use any char, but not space. i.e.
left(RTRIM(A.City) + Replicate('*', 25) ,25) as [City]
View 5 Replies
View Related
Jan 10, 2007
I'm trying to do an insert using Bulk Insert with a fixed length file.I'm using a format file.I'm getting the following error message:Cannot perform bulk insert. Invalid collation name for source column 16in format file '\wbhq.comdfsdviDataIntGOPFilesGOPFormatFile. txt'.Any suggestions are appreciated.Thanks!JenniferFormat File Contents:8.0161SQLCHAR02""0Space""2SQLCHAR04""1YearID""3SQLCHAR02""0Space""4SQLCHAR02""2PeriodID""5SQLCHAR02""3CompanyID""6SQLCHAR01""0Dash""7SQLCHAR04""0Space""8SQLCHAR01""0Dash""9SQLCHAR04""4UnitID""10SQLCHAR01""0Dash""11SQLCHAR04""5AccountCode""12SQLCHAR05""0Space""13SQLCHAR01""6AccountType""14SQLCHAR029""0Space""15SQLCHAR016""7GLAmount""16SQLCHAR0105"
"0Space""Bulk Insert Statement:BULK INSERT FlatFile_GOPFROM '\wbhq.comdfsdviDataIntGOPFilesGLPAM.GOP'WITH(FORMATFILE ='\wbhq.comdfsdviDataIntGOPFilesGOPFormatFile. txt')Table Definition:CREATE TABLE [dbo].[FlatFile_GOP] ([YearID] [smallint] NOT NULL ,[PeriodID] [smallint] NOT NULL ,[CompanyID] [smallint] NOT NULL ,[UnitID] [smallint] NOT NULL ,[AccountCode] [int] NOT NULL ,[AccountType] [char] (1) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,[GLBalance] [money] NOT NULL) ON [PRIMARY]GOFile Contents:2007 210- -0002-3000 G196395.102007 210- -0002-3700 B1484.002007 210- -0002-3700 G1571.132007 210- -0002-3800 B157457.002007 210- -0002-3800 G161577.73
View 1 Replies
View Related
Jan 2, 2008
Hey All,
Similar to a previous post (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=244646&SiteID=1), I am trying to import data into a SQL Table.
I am trying to program a small application that will import product data obtained through suppliers via CD-ROM. One supplier in particular uses Fixed width colums, and data looks like this:
Example of Data
0124015Apple Crate 32.12
0124016Bananna Box 12.56
0124017Mango Carton 15.98
0124018Seedless Watermelon 42.98
My Table would then have:
ProductID as int
Name as text
Cost as money
How would I go about extracting the data with an XML Format file? I am stumbling over how to tell it where to start picking up data for a specific column.
Is there any way that I could trim the Name column (i.e.: "Mango Carton " --> "Mango Carton")?
I don't know if it makes any difference, but I've been calling SQL from my code by doing this:
Code in C# Form
SqlConnection SqlConnection = new SqlConnection(global::SQLClients.Properties.Settings.Default.ClientPhonebookConnectionString);
SqlCommand cmd = new SqlCommand();
cmd.CommandType = CommandType.Text;
cmd.CommandText = "INSERT INTO PhonebookTable(Name, PhoneNumber) VALUES('" + txtName.Text.ToString() + "', '" + txtPhoneNumber.Text.ToString() + "')";
cmd.Connection = SqlConnection;
SqlConnection.Open();
cmd.ExecuteNonQuery();
SqlConnection.Close();
RefreshData();
I am running Visual Studio C# Express 2005 and SQL Server Express 2005.
Thanks for your time,
Hayden.
View 1 Replies
View Related
Jul 16, 2004
Hi All,
I have a file that has fixed row size of 148 and fixed column size, but the file has no end of line character. I know it is wierd but a client has made the file and refuses to change the format. So I am stuck with reading it the way it is.
In Enterprise Manager, I used the Import/Export wizard and I specified fixed length and it let me specify 148 as the lenght of each line. Then it recognized the file and I was able to read it in.
I saved the DTS package and I can run it over and over again using dtsrun. However I want to do the same thing using Bulk Insert. How do you specify fixed row length for Bulk insert and how do you give it individual column lengths?
Thanks,
Shab
View 3 Replies
View Related
Feb 13, 2007
Hi,
I am trying to create a program that transfers tables to flat files.
At this point in time, I have suceeded in created one that creates delimited files.
However, I am now trying to create fixed-width files as you can do with the SSIS designer, but programatically.
Is there a way to programatically determine the width of a column from the source table? I can not seem to find any kind of function or member that stores this information or allows me to retrieve it.
I know what I need to change in order to set a width for a column, but I just don't know how to find the width without just asking the user to provide one.
View 5 Replies
View Related
May 2, 2006
I am attempting to import a fixed width file into a SS2005 table and am having problems when importing a date that has no value in it. The table will allow nulls.
The date is in dd/mm/yyyy foramt and when there is no date then there are 10 spaces. When transforming the data I TRIM the data down using a derive transform script so all there is, is an empty string. When the file attempts to load I get the following message:
[OLE DB Destination [2238]] Error: There was an error with input column "paid_date" (2306) on input "OLE DB Destination Input" (2251). The column status returned was: "The value could not be converted because of a potential loss of data.".
How can it potentially lose data when there is nothing to lose?
I need some way of converting the empty string into a null. Has anyone got any ideas for me?
View 1 Replies
View Related
Oct 19, 2007
I have a fixed width file that I only need a middle field imported.
The field I need starts at character 116 and is 15 character wide. What should my format file look like?
I have a format file that can import a field before it but this is data I don't need. How do you skip the first field?
View 3 Replies
View Related
Mar 28, 2007
Export to Fixed width text file
I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
View 4 Replies
View Related
Jun 14, 2007
Hi,
I was trying to import a fixed-width file to a sql 2005 table.
The total record lenght is 1500. I was trying to import it to a single column.
The strange thing that's happening is: SSIS is inserting only the first 32 chars of the record and the remaining are gone. I tried using nvarchar(max) and varchar(max) but of no use.
I think something somewhere is going wrong but I was unable to figure it out. Earlier I was able to load a similar file into a single column table.
My Header row delimiter is {CR}{LF}
The preview pane shows the complete record but when it transfers to the table, I'm getting 32 chars only.
Can anybody suggest any ideas to figure this out?
Thanks,
Siva.
View 3 Replies
View Related
Mar 29, 2007
Export to Fixed width text file
I am trying to export a table to a fixed lenght text file, there is only flat file option and that does not put LF/CR at the end of row, is there any solution?
View 5 Replies
View Related
Dec 7, 2006
Hello,
I have a series of fixed width files, all with the same schema. I need to import the data into a SQL Server table. Each record in the flat file begins with 'D1'. The length of each record (string) is 380. There are cases where the record ends after position 193, and a new record appears in the current string beginning at position 194. So at position 194 'D' appears, and '1' appears at position 195.
In the flat file, I need to insert a line break after position 193 if position 194 = 'D' and if position 195 = '1'. I'm guessing I would do this with a Script Component Transformation. Once the file is edited, then I can bring the data into the table.
What might the script look like? If you have any suggestions, samples, or know of examples on the web you can point me to, please share.
Thank you for your help!
cdun2
View 2 Replies
View Related
May 29, 2006
Currently we're working on an SSIS package to extract data from a SQL Server database to several fixed width flat files.
Some of the data needs to be formatted/converted in a certain way
DateTimes need to be formatted in ISO8601Booleans need to be 0/1 instead of False/True...Has anybody any idea what the preferred approach (best practice) would be to do these conversions?Convert everything in the select query? What about readability of your query?
Do it somewhere in the package? If so, how?....
View 1 Replies
View Related
Jan 22, 2008
I'm using SSIS to do bulk inserts from fixed width files to about 20 tables in my SQL database.
The problem I'm running into is in creating Format Files for the bulk insert task to use. I've gotten the bcp command to create format files that will read csv files, but I can't seem to figure out how to get it to create one for fixed-width.
I know it can be done: http://msdn2.microsoft.com/en-us/library/ms191234.aspx
At the bottom (Section F) it shows an XML format file for reading a fixed-width file. When I manually create one of these to match one of my tables, the bulk insert worked fine.
Closest I've come is with this ( [] bracketed items are correct values, just censored here):
C:Program FilesMicrosoft SQL Server90ToolsBinn>bcp [database].[owner].[table] format
nul -c -f C:TableFMT.xml -x -S[Server] -U[Username] -P[Password]
My question is, what is the bcp command to create this sort of XML format file?
View 1 Replies
View Related
Feb 1, 2007
Hi,
There's a lot of information on importing data from text files, but not a lot on exporting data to text files... I've checked but found no info on this.
I'm trying to export data from SQL Server to a fixed-width flat file and wondering if I'm doing it the right way.
I use a view as source (using a OLEDB connection manager) and I can see the data without problem.
I defined a Flat File Destination (using a flat file connection manager). When setting up the flat file connection manager, I am asked for a file... Does this mean one should create manually a template file with the desired output format? So I used a production file as template since we're replacing an existing process.
After having set up everything, I run the SSIS only to see all the data on the same row. There are no CRLF...
When I create the file connection manager, there's no way to mention the row delimiter. In the properties I see a "Row Delimiter" field and when I try with "{CR}{LF}" it makes no difference. Interesting to note that, contrary to the HeaderRowDelimiter field, the RowDelimiter field has no drop-down control to give choices.
So I had to return the CRLF as the last field of the source view (SELECT .... ,'CRLF' = CHAR(13) + CHAR(10) FROM ...) to make it work.
Seems odd... Is it the way to go?
Thanks
View 7 Replies
View Related
Aug 29, 2007
We would like to use the bulk insert function to import large CSV files into a SSE database however we have serious concerns regarding giving all our users these high privleges. Is there some way around this can we give them the privleges temporarily do the insert and take it away again or some other solution.
View 5 Replies
View Related
Dec 12, 2006
More SSIS woes. DTS was so much easier.
I have a flat file. It's fixed-with with CRLF record delimiters (a.k.a. Ragged Right format).
Some fields are null, and represented by the text NULL.
I'm trying to import the file into SQL via an OLE DB connection. The target table is a SQL 2000 data table. Two of the fields in the target database are of type smallint.
When I run PREVIEW on the data source (Flat File), everything looks good & correct. I added the convert columns task to convert my strings to smallint. This is where things go haywire.
After linking everything up, the conversion gives me a "Cannot convert because of a possible loss of data." All of my numbers are < 50, so I know this isn't the case. Another SSIS bogus error
My first instinct is the SSIS doesn't understand that NULL means null. I edited the file and replaced all instances of NULL with 4 emtpy string chars. Still no good. It seems to be having a hard time parsing the file now.
I dropped the convert task and tried editing the data source, and set the two smallint fields to smallint instead of string (SSIS formats). I get the same conversion error.
Changing the NULL values to 0 fixed the problem, but they're not 0. They're null.
Short of creating another script that converts all zeros to NULL using the aforementioned hack, I'm out of ideas.
I'm I missing something or is SSIS just incapable of handling nulls in fixed-width flat file formats?
TIA
View 7 Replies
View Related
Aug 28, 2007
We're having issues exporting a set of data from SQL to a fixed width flat text file by just doing a right click on the DB, then choosing Tasks > Export Data. You can not specify a row delimiter when you choose a Fixed Width format. The only way around this that we've found is to specificy char(13) and char(10) at the end of the SQL select statement. Without row delimiters you end up with 1 giant record rather than 20,000 regular sized records. Is there any other way around this that we're missing?
Using Ragged Right is not an option either since the record lengths will be inconsistent if the last field doesn't contain a consistent length to the data.
Thanks,
Mike
View 16 Replies
View Related
Sep 14, 2007
I have a text file that is comma delimited and im pulling it in with a flatfile connection manager. I want to read some of the data, then output another flat file but in a fixed column width. What settings do I made to the connection manager of the output flatfile ?
View 9 Replies
View Related
Oct 19, 2007
I am new to SSIS and am having trouble with automatically setting up the destination output columns.
I am sure there must be an easy way to do this.
My table (source) has 86 columns in it of varying lenghts.
In my connection magagers, I have created one for the SQL Server (source data) and one for the flat file (destination output).
I have also created an OLE DB source data object and a destination Flat File object and set them up to the respective connection managers.
Finally I have linked the source to the destination.
Now when I look at the source, it shows me all 86 columns.
When I open up destination, there are no columns set up.
Problem: do I have to type in all the columns manually in the connection manager for the Flat File?
I would think there would be some automatic way that it would self-populate the columns over to the flat file destination.
View 3 Replies
View Related
Oct 30, 2006
I am trying to export data from a query in SQL Server 2005 SSIS to a flat file destination. Everything works fine except the rows returned from my query are written to the flat file in one long string (i.e., without line breaks). I have tried appending a new line character to the rows returned from the query but that only throws an error when the package is executed. My rows returned from the query are 133 characters wide (essentially only one column per row) so I have set the properties accordingly for a fixed width file format with 133 character wide rows.
Any suggestions or ideas on how to correct this would be greatly appreciated.
Thank you,
Michael
View 3 Replies
View Related
Dec 7, 2006
I am sorry, I am posting this message again, since I did not get anyreply.I want to export a table into a "fixed width" file using SQL 2005import export wizard.This is the version I have:SQL Server 2005 - 9.00.2047.00For some reason it joins all the rows together. For EX: if the tableis like this:Create table Mytable (col1 varchar(50) null, col2 varchar(60) null,col3 varchar (100) Null)Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")Insert into MyTable values ("abcdef", "12345", "8900")Insert into MyTable values ("xxxxxxx", "11111111", "22222222")Insert into MyTable values ("yyyyyyyyy", "5555555555555555","6666666666")It is not exporting every row in a single line. Actually if I open itin "Ultra Edit", it is all in one line.I used to do this regularly with SQL 2000 import export wizard and itexported every row in one line.I looked at the setting:The header row delimiter has {CR}{LF}Code page has 1252 Ansi-Latin.In the Advanced tab:String:dt_str.I tried changing the header row delimiter to just {CR} or just {LF}.Also I tried changing the string to dt_text and nothing seems to help.Please help.Thank you
View 1 Replies
View Related
Jun 12, 2006
How do I delete columns in a fixed width column file connection object, after I've saved it I can't remove columns anymore?
View 1 Replies
View Related
May 19, 2008
Hi All,
I have a simple SSIS package that runs a query on the db and outputs a fixed width flat file. I have all my column widths defined and in the connection manager i can preview the output. Everything looks great. All the fields fall where they should and each record is on it's own line.
When i run the SSIS program and then go open my text file with a text editor the ouput is all on the same line. I have tried changing my file format from fixed width to ragging right and adding a row delimiter but that doesn't work either. I feel like i'm missing something small here. It could even be an issue w/ my text editor (although i've tried to open the text file in multiple editors). In the flat file connection manager I have my file defined to be 187 characters long, So figure every 187 characters it should output a new line (it should add the carraige return right?).
Has anyone encountered an issue like this?
Any help would be much appreciated.
View 4 Replies
View Related
Jun 21, 2007
Hi-I have a sql database (2005) that I need to extract a report from that looks somehintg like SELECT * From Empl_Hours WHERE some_flag <> 'true' .The thing works fine, but the problem is this: I need to insert a record in the 1st row that looks like "Static_text"+row_count() +"more_static_text"where row_count is the actual # of rows that were retrieved. Thanks in advance for any help.DAn
View 3 Replies
View Related
Jun 1, 2007
Is there a maximum width for fixed width column?
I'm trying to read in a flat file (which, admittedly, has one very wide column), and it keeps breaking because of truncation when it tries to read in the file.
Any ideas?
Jim Work
View 5 Replies
View Related
Jul 29, 2014
I need to load the following data into a SQL table. This is how the vendor is able to provide it to us.
CRCorp Daily Report,,,,,,
,,,,,,
Facility,Location,Purchase Order #,Vendor,Inventory #,Date Ordered,Extended Cost
09-Mowtown 495 CRST,09-402A Women's Imaging,327937,"BARD PERIPHERAL VASCULAR, INC.",113989,7/25/2014,650
09-Mowtown 495 CRST,09-402A Women's Imaging,327936,"WB MASON CO., INC.",112664,7/25/2014,8.64
01-Mowtown 499 CRST,01-302B Oncology,327947,McKesson General Medical,n/a,7/25/2014,129.02
[Code] ....
I have attempted to bulk insert it into this table with no luck.
CREATE TABLE POMaster
(Facility VARCHAR(75),
Location VARCHAR(75),
PONum INT,
VendorNm INT,
INVENTORYNUM VARCHAR(25),
orderDte DATE,
ExtendedPrice NUMERIC(10,2)
)
GO
It does not like the double quotes. How to make this format work? Do I need a format file?
View 2 Replies
View Related
Jul 22, 2015
I'm testing, with SQL 2014 on the same DB, a procedure that extracts data from a table into a file and Loads data from that file into a different table which has the same columns as the initial table (I use a function to create the create table statement from the source table and change the name of the destination table)
when doing my bcp -c the record with the special character "é" doesn't make it to the file.
when doing a bcp -w the record with the special character makes it to the file but the bulk insert omits the whole record.
The result file in the case that makes it to the file is unicode. I'm not using a format file (don't see the need for it)
The bulk insert into the destination table that contains identical columns as the source (a mixture on int, varchar, char) --> didn't work
I also tried by building the columns of the destination table with nvarchars -- Still doesn't work.
I tried the bulk insert with:
codepage = 'ACP, 'RAW' --> didn't change anything. still didn't work.
It's a complicated process that takes 1 XML record that contains information + the Create Table Statement (to eventually be able to this on a different server/DB) + the title Row for each column + the Data... Each of these are created with a BCP command (all with the same options). they are then appended to each other with a copy /B c:file1.txt + c:File2.txt + c:File3.txt + c:File4.csv c:ResultFile
Once the result file is created I bulk insert the 2 first rows in one table "TableA"
create the tmp table "TABLE B" with the create table statement that is in "TableA"
and do another bulk insert of the remainder of the file into the newly created table.
What else can I try? Should I be creating a format file? what are the benefits of a format file?
It's a very long procedure that does both Extract and Load (with 12 parameters) not sure what I should put here.
View 6 Replies
View Related
Oct 31, 2005
I have a request to produce a SQL report that will be produced with fixed column width. For example see below Position Element LengthField Format 1-2 Column1 2 Alphanumeric 3-4 Column2 2 Alphanumeric 5-13 Column3 9 Chars Any idea how I can produce the report specified above? Thanks
View 1 Replies
View Related
Jun 6, 2006
I'm sending the results of an SSIS data flow to an fixed-width flat file output, but instead of getting separate rows of data, like so:
row1data...
row2data...
row3data...
etc...
I get:
row1data...row2data...row3data...etc...
Is there some setting I'm missing in either the flat file output or the file connection to turn this on?
View 3 Replies
View Related
Mar 5, 2015
I have to perform a bulk Import on a regular Basis and have created a script to do this. The Problem is that the .csv file has 12 Columns and the table to Import into has 14. To Workaround this discrepancy I have decided to use a Format file. The Problem is that how to create one.
View 3 Replies
View Related
Mar 8, 2005
Hi,
I have a customer who wants to send a fixed width file with 1400 characters per row.
Before I tell the 'OK', I was wondering if there is any limit to the length of rows when using a DTS .txt source.
Anyone know?
Thanks!
View 7 Replies
View Related