Importing Text Files Into SQL Server - Some Problems

Mar 22, 2007

Hi all,

Sorry if this is in the wrong forum but i didn't know if i should put it in the Data Mining forum.

My problem is two fold.

Scenario:

I have 11 text files with standard data format. They range in size from 20megs to 300+ megs (containing 100's of thousands of lines of data)

I build a simple DTS Package to automate the importing of the text files into a database table with the same format as the text files.

Running the files through my DTS produced errors on all files. The error was :
Too many columns found in the current row; non-whitespace characters were found after the last defined column's data.

The error also gave an approximate position of the problem row. So i opened up a text editor that can handle large files and took out about 5000 records before and after the problem area. So i now had a new test file that had 5000 records and contained what "should" be the problem row. I ran this file and it imported correctly. It did not produce the error that i got previously. However it did import things incorrectly. Going through only one of the files i found several rows (over two dozen) that had half the data on one row, then the other half on the other row. Obviouslly there is a problem here with the way the data was exported. I wanted to know if anyone knows of any programs i can get that will analyze these text files for inconsistencies. I have considered writing one in C# but want to see if there is something already out there that can help me.

Basically i need to look at these text files and find Rows that are "incomplete"

the files come from a unix based system, use LF for row delims and Chr(29) for Column delimiters.



any ideas?



thanks,

mcm

View 4 Replies


ADVERTISEMENT

Importing Text Files To Sql Server Using Asp.net/Vb.net

Sep 20, 2004

Hi,
Can anyone help? Need to upload a text file to a sql database but keep getting errors.
I'm creating a page that will allow users to to bulk import and update to a MsSql database. The users provide a text file every so often with new/update information. So i want to use a DTS package to transform the infomation, and create a table in the database, then check against existing/non existing records, if the record exist, update it, if not insert it. I'm using Visual Studio.Net, ASP.Net and coding in VB.Net.

Anyone know where i can find documentation/code regarding the above?
I will be greatful for any help.

View 5 Replies View Related

Problems Importing Text Files With Double-quotes As Text Qualifier

Jul 14, 2006

I have text data files from a third party and they use comma as field delimiters and enclose the text for each column in double-quotes. Not a problem for most of the data files until they start sending files where there is " within the column values. SSIS package fails with the error:

The column delimiter for column "Column 1" was not found.

Any ideas on how to resolve this issue will be greatly appreciated.Thankspcp

View 15 Replies View Related

Importing Text Files

Apr 10, 2001

Hi,

I want to import multiple text files into a single table. I know I have to use BCP or DTS. But, I want import all files at once, instead of one at a time. And the file names are in sequence, viz. file1, file2, file3 etc.
Can anybody tell me, How I can achieve this.

Thanks

View 4 Replies View Related

Importing Text Files

Jul 7, 2000

Has anyone written a SQL script to import ASCII comma or space delimited?

I am trying to load a PROGRESS exported file (Unix) into SQL SERVER 6.5 (NT).

Much appreciated,

Nickd

View 1 Replies View Related

Importing Text Files

Sep 19, 2001

I have to import 18000 text files into a sql database.
Each file contains 10 fields and around 5000 records.
I am currently doing this with DTS.

What I am wondering is this: Is DTS the most efficient i.e. quickest way to import all this data. Bearing in mind there is about 90 million records to import in all.

I would appreciate the benefit of somebody elses experience when dealing with this type of thing.

Cheers,

Brookesy

View 2 Replies View Related

Help Needed With Importing Text Files

Jun 6, 2005

I have a load (180,000+) of text files whose contents need to go into a SQL server database.Whats the best way of doing this? Using a c# console program and if so, using FileStream or StreamReader? Or using a feature of SQL server itself. The text files themselves are less that 1k and are literally less than 200 characters.The problem is, I've tried a WinForm and although I can detect what files are there, as soon as I attempt to open one for reading, everything stops working and won't insert anything to the database.

View 1 Replies View Related

Importing Text Files - DTS Help Needed

Oct 31, 2006

Hi allCould someone help me with the following problem? Hours of googlingyesterday couldn't get me the answer. I'm using SQL 2000 and DTS andtrying to import a huge fixed width text file.File is >1m rows and >200 columns and is defined by a proprietory (i.e. notbcp produced) format specification of the formName Start LengthFld1 0 20Fld2 19 5Fld3 24 53and so on.Tbe only way I've found to define the columns so that DTS can import thefile properly is to go through the wizard and click on the starts of eachcolumn. I don't want to use bcp if possible (I did enough of that onSQL7) - but surely there's a way to get DTS to read from a format file so Idon't have to click 200 times (with all the ensuing errors I could make).Any help greatly appreciated.CheersRob

View 2 Replies View Related

Importing Text Files And Converting Datatypes

Oct 8, 2004

Hi ya... About to tear my hair out.

I thought i fixed this problem, as it WORKED about two days ago, but now I'm getting errors.

I have a series of text files, some are delimited, some are fixed width. I previously was able to import these files thru a dts package by creating the table in a stored procedure. Exple:
CREATE TABLE [Pol_Dump073104]
(
[Product_Type] varchar (12) NULL,
[Benefit] float NULL,
[Base_Premium] varchar (12) NULL,
[Rider_Premium] float NULL,
[Contract_Value] float NULL,
)

I then import the text file into the table and then cast the float datatypes as money in a select into statement to do aggregate functions on the money fields. this worked previously, but now I'm receiving this: 'Error converting datatype char to money.' I tried to convert to float and received the same error, only with float where the word money was.

Please help! :mad:

Thanks

View 5 Replies View Related

Importing Text Files With Different Number Of Columns

Oct 6, 2007

Hi!

I have a bit of a problem with importing (DTS Package) from a text file into an existing DB.

The file has no column names and every file only contain 1 row. This row varies in number of columns.

My first thought was to create a table with the maximum numbers oc columns that can appear in the text file.
An example:

Table with 4 columns; Col001, Col002, Col003 and Col004

The first text file:
123,Peter,Ocean Street,NY

The second text file:
356,John,Park Avenue


My aim is that the second file should append to the table, leaving Col004 as (NULL)

Any ideas how to get this to work?

Expekt

View 3 Replies View Related

How Do I Automate Importing All Text Flat Files Into SQL 7 Table????

Jan 19, 2000

How do I automate importing "All Text Flat Files" into a SQL 7 table. The key is that there is no validation neccessary for the data and I do not want to manually import the data. I just to delimited the data and import it using either a script or a schedular of some type that can do it for me. Some Please Help

View 1 Replies View Related

Problem Importing Text Files With Binary Zeros (0x00) Via SSIS(SQL2005). It Is All Fine When Using DTS(SQL2000) Though.

Apr 24, 2008

Hi.

There is a "text" file generated by mainframe and it has to be uploaded to SQL Server. I've reproduced the situation with smaller sample. Let the file look like following:
A17 123.17 first row
BB29 493.19 second
ZZ3 18947.1 third row is longer
And in hex format:
00: 41 31 37 20 20 20 20 20 ”‚ 31 32 33 2E 31 37 20 20 A17 123.17
10: 66 69 72 73 74 20 72 6F ”‚ 77 0D 0A 42 42 32 39 20 first row™ͺ—™BB29
20: 20 00 20 34 39 33 2E 31 ”‚ 39 20 20 73 65 63 6F 6E 493.19 secon
30: 64 0D 0A 5A 5A 33 20 20 ”‚ 20 20 20 31 38 39 34 37 d™ͺ—™ZZ3 18947
40: 2E 31 20 74 68 69 72 64 ”‚ 20 72 6F 77 00 69 73 20 .1 third row is
50: 6C 6F 6E 67 65 72 ”‚ longer

I wrote "text" in quotes because sctrictly it is not pure text file - non-text binary zeros (0x00) happen sometimes instead of spaces (0x20).

The table is:

CREATE TABLE eng (


src varchar (512)

)

When i upload this file into SQL2000 using DTS or Import wizard, the table contains:

select src, substring(src,9,8), len(src) from eng
< src ><substr> <len>
A17 123.17 first row 123.17 25
BB29 493.19 22
ZZ3 18947.1 third row 18947.1 35

As one can see, everything was imported, including binary zeros. And though SELECT * in SSMS truncates strings upon reaching 0x00's, still all information is stored in tables - SUBSTRINGs show that.

When i upload this file into SQL2005 using SSIS or Import wizard the result is following:
< src ><substr> <len>
A17 123.17 first row 123.17 25
BB29 4
ZZ3 18947.1 third row 18947.1 25

This time table is half-empty - all characters behind binary zeros in respective rows are lost.

I stumbled upon this problem while migrating my DTSes to SSIS packages. Do you think there is some workaround, or i need to turn on some checkbox or smth else could help? Please...

View 8 Replies View Related

Linked Server To Text Files: Is Possible To Detect Changes Made To Those Files? (SQL Server 2005)

Sep 3, 2007

Hi gurus,

I've created a linked server (and set up the corresponding schema.ini file) in order to perform bulk-inserts from some CSV text files into SQL tables (from my standpoint the text files are just for reading purposes). The linked server works fine (I can select the data in the files without a problem).

Now the question: is possible to automatically detect when one or more of those files change in order to start the import process automatically? Something like having a trigger created on the CSV files Or there's no easy way to do that so I have, to say something, to create a Job that periodically checks if the files have changed programatically (say, recording each file's timestamp everytime is imported and comparing the recorded value with the current one, or whatever)?


Thanks a lot in advance!

View 1 Replies View Related

Importing Txt Files With Sql Server 2000

Jul 23, 2005

HI!I am importing .txt files. How can i check the errors? I have created alog file, but the problem is that i lose some characters.I import for example:CodeABCFZHJHNfrom a text file, but sometimes Code can be 4 caracters longI import this 3 characters long now. When i add the same structuredtext file with some rows lenght 4, it skips the last character, but iget nothing in the log file.please helpxgirl

View 4 Replies View Related

Importing Csv Files To SQL Server Express

Nov 22, 2007

Hello,

I receive 4 .csv file downloads periodically (3 times per day) via email from our corporate database. I open each file in Excel, save as excel files, import to Access, replace the previous tables...run action queries, generate reports combined with production data from CSRs and supervisors. ALL DONE MANUALLY!!!

Here's where I am now:
-I've recently switched over to SQL Server Express.
-Used SSMA to bring tables over from previous Access database.
-Exercised the option to LINK these tables to the original Access database.(I'll explain why in a moment)
-Created ADPs for front end data entry use.
-Imported old Access database forms into the new ADPs previously used in Access.
-Connected to the new Server Express.
-I've eliminated my concurrent user problems by doing this.
HOWEVER, I am still bound to using old Access/Jet database to generate reports based on periodic downloads from corporate .csv files.

Here's the question:
What is the best way to import the csv files being sent to me via email into SQL Server Express? I've tried DTS. Seems to me you can't save AND actually use the packages later since it's the Express edition..... Importing manually 4 files, 3 times per day is a very tedious option I'd like to avoid. Any ideas?
Oh, by the way. Corporate has told me they would be willing to post the files to an FTP site instead of emailing the files. That's about as much help as I'm going to get from them. Can SQL Server Express be set up to run stored procedures (triggers) on a hot folder?
Thanks for your help.

David

View 3 Replies View Related

Importing Binary Files Into Sql Server Database

Sep 26, 2005

I work for a company that makes heat transfers for the imprinted apparel market. We're developing a database of merchandise images for all of our non-design inventory. Using Access we're going to be inserting thumbnails of psd (photoshop) files. We're wondering if there is any way to import multiple psd's into the sql server database into matching records like matching a column named "filename" and the actual filename of the file without having to upload each file individually. We want to be able to dump the files from the database of the matching records, also. This way, once our catalog designer has found which designs they need to put into the new catalog, it will dump the psd's for us. The same for our staffer who does color separations.

Any suggestions out there? If you need me to post further of what we're trying here, I will. This is for the bossman.

View 4 Replies View Related

Importing Text In SQL Server

Apr 23, 2006

I need to import a large textfile into sql server 2005.Some of the text fields contain special characters...like:





Île-de-Francewhen I import a text like this into SQL Server I get:ΓƒΕ½le-de-FranceWhat can I do to prevent this from happening? (other fieldtype?)

View 3 Replies View Related

Importing Excel 2007 And/ Or DBF Files Into SQl Server 2005

May 12, 2007

Greetings,

I'm having a tough time importing some of my legacy database into sql.
I have a number of dbase (IV) files I need to get into SQL. I have tried building a SSIS package with either an foxpro oledb connection or a jet 4.0 one, none of them work bec. of inconsistencies in the data format in my tables (e.g. date fields, etc).

I have tried to save the .dbfs as excel 2007 files, taking advantage of the larger space that comes with '07. Problem is you can't use the import/export wizard with 2007 for some reason and I haven't been able to create a package with the access 12 oledb as I have read.

I have to get some crucial data out of that old system and into the new one and I can't seem to be able to import them properly.

Any hints on what I should do ? (maybe I'm doing something awfully wrong)

Thank you for taking the time to answer my question,
Val

View 12 Replies View Related

Importing Text File Into SQL Server Problem

Aug 10, 2005

Hello,I am trying to load a text file into SQL Server but the text file seems to be in an unsual format that SQL Server is having a problem reading. I have tried the various options for delimited and fixed file formats.Any ideas would be appreciated.Sample of the file: Dn DCHB;… b`  DCHCVDR  SMGSWP04JOB08748SMA 704DSEARS VDR SWEEP 4   RDSSWSM REPTPROCSTEP1   V-1 &Π &Π BRTA_UA46  200508082345079999  BANNER PAGE                 ;… b`  DCHCVDR  SMGSWP04JOB08748SMA 704DSEARS VDR SWEEP 4   RDSSWSM REPTPROCSTEP1   V-1 &Π &Π BRTA_UA46  20050808234507 420                              ;… b`  DCHCVDR  SMGSWP04JOB08748SMA 704DSEARS VDR SWEEP 4   RDSSWSM REPTPROCSTEP1   V-1 &Π &Π BRTA_UA46  20050808234507 425                              ;… b`  DCHCVDR  SMGSWP04JOB08748SMA 704DSEARS VDR SWEEP 4   RDSSWSM REPTPROCSTEP1   V-1 &Π &Π BRTA_UA46  20050808234507 440                              

View 1 Replies View Related

Importing Text File To SQL SERVER TABLE

Dec 15, 2000

Hello

I am writing program in VC++ through SQl-DMO calls.My problem is when i when i tranfer(import) a text file(comma seperated) into SQl server through a SQl-DMO method called ImportData which is a method of Bulk copy object.Its is not able to convert the data field in the text file to corresponding value datetime in SQl server whereas other data types are working perfectly.


This is the record i need to convert:

90,MichaelB,Wintriss,Inspection,Paper,11,Job101,1, {ts '2000-12-10 15:54:56.000'},D:public233 and 247233.mcs,

and this is the date field
{ts '2000-12-10 15:54:56.000'}

Whereas if i export a table in SQl server in Binary mode and then import the file back it works but when do it as text it gives the above error

Pls help me in this i would be very thankful to you.

Note: I am using SQL Server 7.0 version

Regards
Jitender Singh

View 1 Replies View Related

Importing Text File Into SQL SERVER 2005

May 8, 2008



I want to be able to import data from a text file, into SQL Server 2005, using OPENROWSET. Can you pl give the the syntax for this. What I HAVE IS
select * --into #tmp1
From OpenRowSet
( 'Microsoft.Jet.OLEDB.4.0',
'Text;\ABC.TXT
)

View 2 Replies View Related

Error While Importing Text File Into Sql Server 2000

Mar 29, 2007

hi all,

While importing into sql server 2000 from a text file i am getting an error message like not enough disk space available to perform this operation but there is enough space (around 18gb). Please advice why this is happenning as my work is stuck



thanks and regards

jk

View 4 Replies View Related

SQL Server 2012 :: SELECT INTO - Importing A Text File Into A New Table?

Jun 6, 2015

how to import a text file with a list of NI numbers into a new table with a column to list all the NI numbers? I think I use the Select INTO clause, but not sure how to do this?

View 1 Replies View Related

Problem Importing Csv Delimited Text File Into A Sql Server 2005 Table

Apr 25, 2006

I am using the Bulk Insert command and trying to import a CSV delimited text file into a table and I am having problems with the quote field delimiters ", " The command below works but it takes in all the "" quotes as well and the field delimiter comma , works only if the commas are the separators only. If I have a comma within a address field for example then the data gets imported into the wrong fields. What can I use to identify that the text qualifier is ". I don't see where I can use the bulk insert command to determine this. Is there another command that I can use or am I using this command incorrectly. I thank you in advance for any response or suggestion you may have.

BULK INSERT AdventureWorks.dbo.MbAddress

FROM 'a:mbAddress.txt'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR=',',

ROWTERMINATOR='',

CODEPAGE = '1252',

KEEPIDENTITY,

KEEPNULLS,

FIRSTROW=2)

Here is a sample ascii file I am importing as well you can see that 6330 has a extra comma in the address line.

"AddressAutoID","Memkey","Type","BadAddress","Address1","Address2","Address3","City","State","Zip","Foreign","CarrierRoute","Dpbc","County","CountyNo","ErrorCode","ChangeDate","UserID"
6317,26517,1,0,"1403 W. Kline Ave","","","MILWAUKEE","WI","53221","","",0.00,"MILWAUKEE",79,"",1/25/2006 0:00:00,"admin"
6318,26225,1,0,"501 Dunford Dr","","","BURLINGTON","WI","53105","","",0.00,"RACINE",101,"",1/25/2006 0:00:00,"admin"
6319,20101,1,0,"2115 Cappaert Rd #35","","","MANITOWOC","WI","54220","","",0.00,"MANITOWOC",71,"",1/25/2006 0:00:00,"admin"
6320,23597,1,0,"728 Woodland Park Dr","","","DELAFIELD","WI","53018","","",0.00,"WAUKESHA",133,"",1/25/2006 0:00:00,"admin"
6321,23392,1,0,"7700 S. 51st St","","","FRANKLIN","WI","53132","","",0.00,"MILWAUKEE",79,"",1/25/2006 0:00:00,"admin"
6322,26537,1,0,"W188 S6473 GOLD DRIVE","","","MUSKEGO","WI","53150","","",0.00,"WAUKESHA",133,"",1/26/2006 0:00:00,"admin"
6323,25953,1,0,"3509 N. Downer Ave","","","MILWAUKEE","WI","53211","","",0.00,"MILWAUKEE",79,"",1/26/2006 0:00:00,"admin"
6324,19866,1,0,"10080 E. Mountain View Lake Rd. #145","","","SCOTTSDALE","AZ","85258","","",0.00,"MARICOPA",13,"",1/27/2006 0:00:00,"admin"
6325,25893,1,0,"W129 N6889 Northfield Dr. Apt 114","","","MENOMONEE FALLS","WI","53051-0517","","",0.00,"WAUKESHA",133,"",1/27/2006 0:00:00,"admin"
6326,26569,1,0,"8402 64th Street","","","KENOSHA","WI","53142-7577","","",0.00,"KENOSHA",59,"",1/27/2006 0:00:00,"admin"
6327,24446,4,0,"83 Sweetbriar Br","","","LONGWOOD","FL","32750","","",0.00,"SEMINOLE",117,"",1/30/2006 0:00:00,"admin"
6328,19547,1,0,"4359 MERCHANT AVENUE","","","SPRING HILL","FL","34608","","",0.00,"HERNANDO",53,"",2/8/2006 0:00:00,"admin"
6329,26524,1,0,"264 Lakeridge Drive","","","OCONOMOWOC","WI","53066","","",0.00,"WAUKESHA",133,"",2/10/2006 0:00:00,"admin"
6330,23967,1,0,"3423 HICKORY ST","100 Tangerine Blvd., Brownsville, TX 78521-4368","Texas Phone Number: 956-546-4279","SHEBOYGAN","WI","53081","","",0.00,"SHEBOYGAN",117,"",2/15/2006 0:00:00,"admin"
6331,25318,1,0,"3960 S. Prairie Hill Lane Unit 107","","","Greenfield","WI","53228","","",0.00,"MILWAUKEE",79,"",2/20/2006 0:00:00,"admin"
6332,24446,1,0,"83 Sweetbriar BR","","","LONGWOOD","FL","32750","","",0.00,"SEMINOLE",117,"",2/21/2006 0:00:00,"admin"
6333,26135,1,0,"P.O. Box 8 127 Main Street","","","CASCO","WI","54205","","",0.00,"KEWAUNEE",61,"",2/21/2006 0:00:00,"admin"




View 7 Replies View Related

Import Data From Text Files Into SQL Server...?

Jun 6, 2005

Hi,i wanna develop an web-database application with ASP.NET,C#, SQL server 2000.i already have some data whichs been in text format(text file) and now, i want to import the same into my database.the problem is, the text file has got many line breaks and also its not well formated to import it using DTS.Can any one help me out in importing the same.thanks in advance

View 3 Replies View Related

Import Text Files Into SQL Server Tables

Jan 31, 2007

I have created a DTS package which imports text file into single sql server table with 8 columns (SourceData). The DTS package uses 'Test1.txt' file. Now i have around 200 text files (Test1,Test2,.....Test200). I need to import them one by one into 'SourceData' table. Could you pls. help me out in getting solved this mistery.

Thanks,

Hemal

View 1 Replies View Related

Import Text Files Data Into SQL Server

Feb 17, 2007

I need to extract data from text files (around 200) and import into sql server tables. I tried using SSIS foreach loop container but could not manage it. Can anyone guide me how this can be done?

All help appreciated.

Thanks,

View 4 Replies View Related

DTS: Copying Data From Text Files To A SQL Server Table

Sep 10, 2001

Hi all,

I got a situation here.....

From a source table (in SERVER1) I get ids of candidates and from another source (in SERVER2) I get their CVs (text files stored in various Folders). My destination table (in SERVER3) has two fields, CandidateId & CandidateCV.

I have to transfer the data in above fashion for nearly 1 million records.
How can I write a DTS package which picks up the text file from SERVER2 based on the CandidateId which comes from SERVER1? Probably I need some kind of looping mechanism which changes the candidate id & his CV file.

Can anyone help???

Thanks...

View 2 Replies View Related

Importing DBF Files

Feb 16, 2007

With DTS it was easy to import a DBF file, but its a huge pain with SSIS. Is there any way on the horizon to import DBF files as easily as DTS did?

View 1 Replies View Related

Importing CSV Files

Jul 23, 2005

Hello,I need to import a bunch of .csv files. The problem I am having is the"non data" information in the files creating bogus rows and columndefinitions. Here is an example of the csv file.CBOT - End-of-Day Futures Bulk Download 2001.2 Year U.S. Treasury Notes FuturesDateSymbolMonth CodeYear CodeOpen20010103ZTH2001102.0937520010104ZTH2001102.0312520010105ZTH2001102.28125In this case, there are bogues rows created with the text at thebeginning of the file, and also the column names get placed into a rowas well. My question is; how do you import the file and strip out the"non-data" data? so that only the actual data gets inserted into the db?Thanks,TGru*** Sent via Developersdex http://www.developersdex.com ***Don't just participate in USENET...get rewarded for it!

View 4 Replies View Related

Importing .prt Files To SQL

Jul 20, 2005

What is the best way to import a structured .prt export file from anaccounting package into SQl Server??Tabs, Fixed col length???

View 1 Replies View Related

Importing .MDF And .LBF Files

Sep 14, 2006

I have the original copies of the .MDF and .LBF of another database that has a bunch of data in that I'm trying to important into MSSQL Express for a project I'm working on. I no longer have a running version of this database just the .MDF and .LBF files and I have no clue what version of MSSQL they were originally created with. If anyone knows a way to import this database please help.

View 1 Replies View Related

Exporting Databse Table Data In SQL Server 6.5 Into Text Files

Mar 11, 1999

Hi,

I have to export the table data from my databse into text files as I nedd to put it in Informix database using a sheel script. Is there a way by which I can do this.

Is there any other way by which I can put the data from SQL Server to Informix.

Any takers,

Thanking you in advance.

Bye for now,

Himauhu

View 1 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved