T-SQL (SS2K8) :: Import To Table From Varying Tab Delimited Text Files

Feb 10, 2014

I need to import data to a MSSql table from massive (read: a million and a half rows, every single day) logs that come in .txt format separated in tabs with a ";" symbol and then have some stored procedures analyze that data to generate some reports in an excel file with that info. The text files include the column headers in the first row and the data starts on the second one.

The challenge is that the text files differ in column order and count every single day.

The analysis that I need to do only needs about 15 columns from the nearly 90-120 that those files include, and those columns sadly happen to be in a different order in those files.

View 8 Replies

Import A CSV Delimited Text File Into A Table

Jan 18, 2007

Hi,

Could you help me to write a script to import a CSV delimited text file into a sql server table.?

Thanks,

carlos

View 2 Replies View Related

Import Text File Report With Varying Content And Records

Oct 23, 2014

I get error reports in simple text files like the one below in relatively the same format. The only thing that varies is the number error reasons as there can be any number of error reasons for a file. Usually there is only one but there can be a handful. What is the best way to capture the error description and count of errors no matter how many there are? I want to take these items and update a table I have in sql server 2008r.

Report Message example:
************************************************** *****

Original File Name: some.file.YYYYMMDD.d.incr.02of02.1.dat
Source File ID: file02YYYYMMDD
File Receipt Date: 10/17/2014
Total records received: 1331136
Total records loaded: 1329987

ERROR REASONS
Error code: EBBW002 Error desc: Duplicate Record Total records: 1146
Error code: EABC001 Error desc: Invalid Length Record Total records: 1
Error code: ERRCM10 Error desc: Missing First Name Total records: 2

Total number of Errors encountered during the ODS update processing: 1149
************************************************** *****

View 2 Replies View Related

Import Data From Two Text Files Into One Database Table

Feb 12, 2008

I am learning SQLServer Integration Services.

I created a file People.txt containing firstName, LastName seperated by a pipe.

------------------content-----------
John | Doe
Mike | James
Adam | Smith
-----------------------------------------

and another one called gender.txt

------------------content-----------
M
---------------------------------------

I will would like to create integration services package that compines each record of the first file with the record of the second file and inserts the result into table.

--------------Result table content------------------

John
Doe
M

Mike
James
M

Adam
Smith
M

-----------------------------------------------------------

Thanks

View 5 Replies View Related

Import Multiple Text Files Into Single Table

Jan 29, 2007

How to import multiple text files (residing in single folder) into SQL Server table? I know how to import single file but not sure how multiple files could be loaded? Pls. guide.

Thanks,

HShah

View 1 Replies View Related

Best Practice For Ragged Delimited Text Import

Aug 9, 2007

I have tab delimited text files which may have optional fields (meaning they can be not present at all) to the right of the required fields that I care about. It would appear that using a Flat File Connection with Delimited Format (tab) set will choke if it is initially configured with a file that has something like:

data data data

and it then encounters

data data data optionaldata

It chokes. I know this could be parsed line by line, but that seems silly. It seems like there should be a way to ignore columns beyond a certain point (e.g. Format "Delimited Ragged Right").

Is there some way to do this with a directly with a flat file connection?

Thanks,

--Andrew

View 1 Replies View Related

Help In Using ADO.NET To Read From Text Files(CSV Delimited)

Mar 2, 2007

Hello there

I have written a small code that is used to read records from a txt or a csv file, but when i try to read from a txt or csv file which is in unicode, i get the following error

No value given for one or more required parameters........this error is recieved on the following line

dtadpt.Fill(DataSet.Tables("table2"))

The code for reading from a file is as follows

Dim dtrow As DataRow

Dim cmdtxt, file, path

Dim dtadpt As New System.Data.OleDb.OleDbDataAdapter

Dim Command As System.Data.OleDb.OleDbCommand

Dim Conn As System.Data.OleDb.OleDbConnection

file = 'name of the file'

path = 'path of file'

Conn = New System.Data.OleDb.OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;" & _

"Data Source=" & path & ";" & "Extended Properties=""text;HDR=YES;FMT=Delimited""")

For Each dtrow In DataSet.Tables("table1").Rows

Dim id

id = dtrow.Item("id")

cmdtxt = "Select * from " & file & " where id = " & id

Command = New System.Data.OleDb.OleDbCommand(cmdtxt, Conn)

dtadpt.SelectCommand = Command

Next

DataSet.Tables.Add("table2")

dtadpt.Fill(DataSet.Tables("table2"))

this code works if i choose a csv or txtfile which is in ANSI format. But i want to read from a unicode file because i have some information in that unicode format that i want to store

any idea how this can be possible

thanks and best regards

Saad

View 1 Replies View Related

How Do I Import A Varying Column Width Flat File Into A Table Using SSIS?

Apr 24, 2008

How do i import a Varying Column Width Flat file into a Table using SSIS?

I have a flat file that has 4 columns with varying width
Like I should read the file as
Col 1 - (1 to 10 Characters)
Col 2 - (12 to 21 Characters)
Col 3 - (22 to 35 Characters)
Col 4 - (36 to 38 Characters)
At the end of the record is a "LF"

I think "Fixed Width" Columns allow me to define a standard column length for all the columns.. Right?

Any thoughts on how to?

View 9 Replies View Related

Import Into Comma Delimited Text File (very Urgent)

Aug 28, 2001

I want to join differnet tables and import the data into comma delimited text file. There will be lot of checks like if then else to manipulate data. I want to use stored procedure but don't know how to output to text file. Is there any utility which can be used in stored procedure. In future this will be run as an automated job.
Thanks in advance.

View 1 Replies View Related

T-SQL (SS2K8) :: Export Query Results To Pipe Delimited Text File

Sep 5, 2014

I've got a query that returns the data I need. I want to put the query in a stored procedure such that, when the SP runs I get a pipe delimited text file on disk. I don't really want to mess with SSIS, etc. Is there a Q&D way to do this?

View 1 Replies View Related

How To Import In Special Character Delimited Text File By Using SSIS ?

Sep 26, 2007

Hi,

I would like to know how to import in the custom delimited text file by using SSIS.
For example, instead by using tab or comma delimited, I use this character : '�'
The reason is the delimited format that SSIS provided is too common such as colon, semi colon, tab, comma and pipeline.
I have the data that the user also key in the pipeline there. So I am thinking to separate the field by using this special character, but cannot see if there is anyway to import in by using SSIS.

Please help to share the solution on this :

A�B�C
1�2�3

thanks
best regards,
Tanipar

View 8 Replies View Related

Import Text Files

Aug 30, 1999

Hi ppl,
I'm trying to import a fixed lenght text file into a sql 7.0 table using the SQL command "BULK INSERT".

The text file is fixed length and end with a carriage return and linefeed. Fields aren�t delimited.

Wich definition of the "FIELDTERMINATOR" should i use ?

I need some Help here!!!
Thanks.

View 1 Replies View Related

Import Text Files

Apr 11, 2007

hi friends ,

Thank you for the help and support you have given me. Now i am confronted with a new problem. I have to import some textfiles to SQL Server Tables . I have to create a tool to automate the porting using C# .The columns in textfile is seperated with pipe"|" . I f any body knows this please help me .

Thanks $ Regards

Nevin

View 1 Replies View Related

Import Text Files

Apr 11, 2007

hi friends ,

Thank you for the help and support you have given me. Now i am confronted with a new problem. I have to import some textfiles to SQL Server Tables .The columns in textfile is seperated with pipe"|" . I f any body knows this please help me .

Thanks $ Regards

Nevin

View 5 Replies View Related

Issues Importing A Text File (tab Delimited) To A SQL Table

Dec 7, 2005

I have a text file I am trying to import to a table. This text file is in a tab delimited format. I am using DTS to import the data to a new table I made. The fields are varchar and are set to allow nulls & allow 8,000 characters per field.

The error I am getting is that the data exceeds the allowed amount (or something like that) in col4.

Now I have checked everything in column 4 and nothing exceeds 5,000 spaces/characters combined. I have checked the entire sheet (in excel) for that fact, and there is not one single column/row/cell that exceeds 5,000 spaces/characters combined.

What the heck could be causing SQL to tell me I am trying to import too much data in one column when there is nothing that even comes close to 8,000 characters & spaces combined?

View 3 Replies View Related

Bulk Load From Text Delimited File To SQL Table

Dec 13, 2007

Hi,

I am new to SSIS but i have avg working knowledge in sql.
My problem is as follows ,I have a text pipe dilimited file in some folder and the number of columns and the name of the column is not consistant. It can have N number of column and it can have any column names. I need to load this text file data into a sql table. All that i want is to load this file to SQL Database with some temp name. Once i get the table in SQL Database, i can match the column names of both taget table and this temp table and only push those column which matches with the target table. For this i can frame Dynamic SQL. This part is clear to me.

Now the problem is , I developed a SSIS pacakge to push the text file to SQL Table. I am able to do this. But if i change the column names or added new column SSIS is not able to push the new columns. Is this functionality available in SSIS, is it can be dynamic like this?

I hope i am clear with my prob... if need any clarification please let me know

thanks in advance

Mike

View 3 Replies View Related

Import Multiple Text Files?

Jul 20, 2006

Hello,

I am kind of new to Sql Server 2005.

I figured out how to use the import data wizard to import a delimited text file.

But I need to find a way to import many delimited text files at once.

does anybody know if this can be done in Sql Server 2005? and how?

thanks in advance,

View 1 Replies View Related

Can DTS Import Varying File Names?

Oct 5, 2000

I receive several TXT files daily that need to update information in SQL Server databases. The process requires that all TXT files be appended to a master file and also update individual files' information based on the TXT file name. For example:

File TABLE1_x_ddmmyy.TXT (ddmmyy = date, x = "O" or "B") is to be appended to the master file and also update SQL table "TABLE1" by setting a flag for those records in the table that match a unique key that is provided in the TABLE1.TXT file.

In VFP, I had the following process in place:
a) open the TXT file.
b) read its file name and open the corresponding VFP file
c) update the VFP file based on the key provided in TXT
d) append the key to the master file.
e) repeat c-d for next record in TXT
f) repeat c-e for next TXT file

Using the same process with ADO takes a considerable time since I am processing one line at a time.

Is there any way to do this using a DTS package of some sort? How can I read the TXT file names in SQL Server?

Thank you.

View 3 Replies View Related

Import Excel With Varying Worksheet Name Using DTS

Jun 6, 2007

Hi Everyone,

I'm trying to create a DTS package that will let me import an Excel file. The user will be able to name the file the same name every time. But can the DTS package read a different worksheet name each time? Right now, if I use the Excel connection object in DTS designer, it wants to hard code the worksheet name.

Thanks,

Eric

View 1 Replies View Related

T-SQL (SS2K8) :: Varying Calculation For Fixed Number Of Fields

Aug 19, 2014

I'm writing a query that will be used in Jasper Ireports, but prefer to have the values done ahead of time using SQL rather than relying on the report to do the lifting.The fields are pretty straight forward, only the display is where I have a question.

Fields Used: PERIOD ('MON-yyyy') and VALUE

The results must start with the CURRENT PERIOD (AUG-2014) in one column and the VALUE for the current period multiplied by 1/12 (VALUE*(1/12)).The next column should return the VALUE for CURRENT PERIOD - 1 (JUL-2014) and multiply by 2/12 (VALUE*(2/12))

This should continue for the last 11 months and would end with OCT-2013 with the value being multiplied (VALUE*(11/12)).Is the easiest solution to this a CASE statement looking at PERIOD then PERIOD minus one month, minus two months...etc?

View 1 Replies View Related

Import Data From Text Files Into SQL Server...?

Jun 6, 2005

Hi,i wanna develop an web-database application with ASP.NET,C#, SQL server 2000.i already have some data whichs been in text format(text file) and now, i want to import the same into my database.the problem is, the text file has got many line breaks and also its not well formated to import it using DTS.Can any one help me out in importing the same.thanks in advance

View 3 Replies View Related

Import Text Files Into SQL Server Tables

Jan 31, 2007

I have created a DTS package which imports text file into single sql server table with 8 columns (SourceData). The DTS package uses 'Test1.txt' file. Now i have around 200 text files (Test1,Test2,.....Test200). I need to import them one by one into 'SourceData' table. Could you pls. help me out in getting solved this mistery.

Thanks,

Hemal

View 1 Replies View Related

Import Text Files Data Into SQL Server

Feb 17, 2007

I need to extract data from text files (around 200) and import into sql server tables. I tried using SSIS foreach loop container but could not manage it. Can anyone guide me how this can be done?

All help appreciated.

Thanks,

View 4 Replies View Related

Problem Importing Csv Delimited Text File Into A Sql Server 2005 Table

Apr 25, 2006

I am using the Bulk Insert command and trying to import a CSV delimited text file into a table and I am having problems with the quote field delimiters ", " The command below works but it takes in all the "" quotes as well and the field delimiter comma , works only if the commas are the separators only. If I have a comma within a address field for example then the data gets imported into the wrong fields. What can I use to identify that the text qualifier is ". I don't see where I can use the bulk insert command to determine this. Is there another command that I can use or am I using this command incorrectly. I thank you in advance for any response or suggestion you may have.

BULK INSERT AdventureWorks.dbo.MbAddress

FROM 'a:mbAddress.txt'

WITH (

DATAFILETYPE = 'char',

FIELDTERMINATOR=',',

ROWTERMINATOR='',

CODEPAGE = '1252',

KEEPIDENTITY,

KEEPNULLS,

FIRSTROW=2)

Here is a sample ascii file I am importing as well you can see that 6330 has a extra comma in the address line.

"AddressAutoID","Memkey","Type","BadAddress","Address1","Address2","Address3","City","State","Zip","Foreign","CarrierRoute","Dpbc","County","CountyNo","ErrorCode","ChangeDate","UserID"
6317,26517,1,0,"1403 W. Kline Ave","","","MILWAUKEE","WI","53221","","",0.00,"MILWAUKEE",79,"",1/25/2006 0:00:00,"admin"
6318,26225,1,0,"501 Dunford Dr","","","BURLINGTON","WI","53105","","",0.00,"RACINE",101,"",1/25/2006 0:00:00,"admin"
6319,20101,1,0,"2115 Cappaert Rd #35","","","MANITOWOC","WI","54220","","",0.00,"MANITOWOC",71,"",1/25/2006 0:00:00,"admin"
6320,23597,1,0,"728 Woodland Park Dr","","","DELAFIELD","WI","53018","","",0.00,"WAUKESHA",133,"",1/25/2006 0:00:00,"admin"
6321,23392,1,0,"7700 S. 51st St","","","FRANKLIN","WI","53132","","",0.00,"MILWAUKEE",79,"",1/25/2006 0:00:00,"admin"
6322,26537,1,0,"W188 S6473 GOLD DRIVE","","","MUSKEGO","WI","53150","","",0.00,"WAUKESHA",133,"",1/26/2006 0:00:00,"admin"
6323,25953,1,0,"3509 N. Downer Ave","","","MILWAUKEE","WI","53211","","",0.00,"MILWAUKEE",79,"",1/26/2006 0:00:00,"admin"
6324,19866,1,0,"10080 E. Mountain View Lake Rd. #145","","","SCOTTSDALE","AZ","85258","","",0.00,"MARICOPA",13,"",1/27/2006 0:00:00,"admin"
6325,25893,1,0,"W129 N6889 Northfield Dr. Apt 114","","","MENOMONEE FALLS","WI","53051-0517","","",0.00,"WAUKESHA",133,"",1/27/2006 0:00:00,"admin"
6326,26569,1,0,"8402 64th Street","","","KENOSHA","WI","53142-7577","","",0.00,"KENOSHA",59,"",1/27/2006 0:00:00,"admin"
6327,24446,4,0,"83 Sweetbriar Br","","","LONGWOOD","FL","32750","","",0.00,"SEMINOLE",117,"",1/30/2006 0:00:00,"admin"
6328,19547,1,0,"4359 MERCHANT AVENUE","","","SPRING HILL","FL","34608","","",0.00,"HERNANDO",53,"",2/8/2006 0:00:00,"admin"
6329,26524,1,0,"264 Lakeridge Drive","","","OCONOMOWOC","WI","53066","","",0.00,"WAUKESHA",133,"",2/10/2006 0:00:00,"admin"
6330,23967,1,0,"3423 HICKORY ST","100 Tangerine Blvd., Brownsville, TX 78521-4368","Texas Phone Number: 956-546-4279","SHEBOYGAN","WI","53081","","",0.00,"SHEBOYGAN",117,"",2/15/2006 0:00:00,"admin"
6331,25318,1,0,"3960 S. Prairie Hill Lane Unit 107","","","Greenfield","WI","53228","","",0.00,"MILWAUKEE",79,"",2/20/2006 0:00:00,"admin"
6332,24446,1,0,"83 Sweetbriar BR","","","LONGWOOD","FL","32750","","",0.00,"SEMINOLE",117,"",2/21/2006 0:00:00,"admin"
6333,26135,1,0,"P.O. Box 8 127 Main Street","","","CASCO","WI","54205","","",0.00,"KEWAUNEE",61,"",2/21/2006 0:00:00,"admin"

View 7 Replies View Related

T-SQL (SS2K8) :: Import Values From Excel Into Table?

Nov 18, 2014

I've already created a table and i wanna to insert values in that more than five hundred row ,that values are stored in Excel files, Here I've the doubt is it possible to insert values from excel sheet? I've current data base of ms sql 2000, if it is possible means, how to insert values using query?

ex:

create table test
(
item_code int,
item_name varchar(50),
bill_qty float,
bil_date datetime
)

In that excel file also had the same column name name.

View 4 Replies View Related

Import Of Multiple CSV Files In One MS SQL Table

Jul 20, 2005

Hi all,I have de following application to do :I receive several .csv files from another application in a determined folderof my PC.Those files are named with the format log1.csv logs2.csv logs...The number of file is variable but the internal format is always : time_sec;levelSo the files content a field that may be used as unique key in the target database.I'm trying to build a DTS package that should import periodicallyall the CSV's present in the folder and then destroy them if donesuccessfully.Apparently its not so simple than I supposed. I have always to give the nameof the table I want to import.any idea?

View 1 Replies View Related

IMPORT Multiple CSV Files To SQLSERVER Table

Jan 21, 2005

Dear All,

I am importing all the files from a particular folder to a table on my database KB. It is working perfectly if i use it on the same system where the DB exists and not working from the network.

USE TESTDB

--Table Creation Starts here

Create table Account([ID] int IDENTITY PRIMARY KEY, Name Varchar(100),
AccountNo varchar(100), Balance money)

Create table logtable (id int identity(1,1),
Query varchar(1000),
Importeddate datetime default getdate())

--Table Creation ends here

---Stored Procedure Starts here

Create procedure usp_ImportMultipleFiles @filepath varchar(500),
@pattern varchar(100), @TableName varchar(128)
as
set quoted_identifier off
declare @query varchar(1000)
declare @max1 int
declare @count1 int
Declare @filename varchar(100)
set @count1 =0
create table #x (name varchar(200))
set @query ='master.dbo.xp_cmdshell "dir '+@filepath+@pattern +' /b"'
insert #x exec (@query)
delete from #x where name is NULL
select identity(int,1,1) as ID, name into #y from #x
drop table #x
set @max1 = (select max(ID) from #y)
--print @max1
--print @count1
While @count1 <= @max1
begin
set @count1=@count1+1
set @filename = (select name from #y where [id] = @count1)
set @query ='BULK INSERT '+ @Tablename + ' FROM "'+ @Filepath+@Filename+'"
WITH ( FIELDTERMINATOR = ",",ROWTERMINATOR = "")'
--print @query
exec (@query)
insert into logtable (query) select @query
end

drop table #y

--sp ends here

Exec usp_ImportMultipleFiles 'c:myimport', '*.csv', 'Account'

If i use the above Exec like

Exec usp_ImportMultipleFiles '\kb-02C$MyImport', '*.csv', 'Account'
I am getting the following error:

Could not bulk insert because file '\kb-02C$MyImportAccess is denied.' could not be opened.
Operating system error code 5(Access is denied.).

C Drive and MyImport folder is shared on system kb-02

Would appreciate your valuable HELP.

thanking your valuable help in advance.
K006B

View 2 Replies View Related

Integration Services :: Import Varying Number Of Tables Each Time From One Database To Different Database

Sep 9, 2015

I am new to SSIS. I have been struggling with this for the past one week. I have a weird task. I need to import several tables from one database to a different server with a new database name. We need to do this at the end of every year. The main problem here is that the number of tables varies every year. You may not have all the tables as last year or may have more tables. So I need to create a dynamic task that takes care of this every year without changing the package.

I have performed the following tasks **

1. Create a new dynamic database. ( I have used Execute SQL Task to do this) 2. Copy all the table structures ( I have used Execute SQL Task to do this)

3. Import Data. This is the main problem. I was trying to create a dynamic connection string with variables as suggested in several forums but I finally came to know that this cannot be done if the table structures are different as the metadata cannot be refreshed at runtime.

4. The final step to create a process to validate the data (the count from each table for both source and destination. I think this can be done with Sql task.

What is the best method to do this? My DBA does not like “Transfer SQL Objects Task” or “transfer Database Task”. I would like to create this as a dynamic process.

View 5 Replies View Related

T-SQL (SS2K8) :: Create Separate MS Excel Files By Looping Through Large Table

Jun 24, 2014

I have a master table containing details of over 800000 surveys made up of approximately 400 distinct document names and versions. Each document can have as few as 10 questions but as many as 150. Each question represents one row.

My challenge is to create a separate spreadsheet for each of the 400 distinct document names and versions containing all the rows and columns present in the master table. The largest number of rows would be around 150 and therefore each spreadsheet will not be very big.

e.g. in my sample data below, i will need to create individual Excel files named as follows . . .
"Document1Version1.xlsx" containing all the column names and 6 rows for the 6 questions relating to Document 1 version 1
"Document1Version2.xlsx" containing all the column names and 8 rows for the 8 questions relating to Document 1 version 2
"Document2Version1.xlsx" containing all the column names and 4 rows for the 4 questions relating to Document 2 version 1

I assume that one of the first things is to create a lookup of the distinct document names and versions assign some variables and then use this lookup to loop through and sequentially filter the master table data ready for creating the individual Excel files.

--CREATE TEMP TABLE FOR EXAMPLE

IF OBJECT_ID('tempdb..#excelTest') IS NOT NULL DROP TABLE #excelTest
CREATE TABLE #excelTest (
[rowID] [nvarchar](10) NULL,
[docName] [nvarchar](50) NULL,

[Code] .....

--Output

rowIDdocNamedocVersionquestionblankField
1document11q1NULL
2document11q2NULL
3document11q3NULL
4document11q4NULL
5document11q5NULL
6document11q6NULL

[Code] .....

View 9 Replies View Related

Excel 2010 - Import Multiple Files To Table?

Sep 23, 2014

I have over 600+ Excel .xlsx file that I have been trying to import to Sql database table. I've been trying to complete this task with SSIS but no luck yet. I have seen several videos and read articles but when I run the package the source is validated but I always get an error in the destination. I am using Excel 2010 and SQL Server 2012.

View 3 Replies View Related

Transact SQL :: Import Bulk CSV Files In Database Table

Nov 12, 2015

I have more than 500 CSV files with a similar structure [Same column name and same data format]. I would like to load these files in a database table on the SQL Server 2014 database.

View 21 Replies View Related

T-SQL (SS2K8) :: Convert Non-delimited Column Value Into Rows

Nov 14, 2014

I have a scenario wherein one of the column values in a row contains a string value which is non-delimited (as shown below). I need to split them by 2 characters and generate as many rows as count of set of 2 digits in that string.

Current record

RecordID Name CountyList
1 ABC 00
2 CDE 01020304
3 XYZ NULL

Expected output:

RecordID Name CountyList
1 ABC 00
2 CDE 01
2 CDE 02
2 CDE 03
2 CDE 04
3 XYZ NULL

I already have a solution in place to run it thru cursor and then do a while loop on the CountyList column by taking 2 digit value using Substring function (keeping start position dyanamic and jumping 2 positions).

View 2 Replies View Related

Text Files To SQL Table ...

Mar 30, 2006

Hello!, I have two applications that import data from text files to Access 97 tables. One of this applications is in Access 97 and the other in VB6, know they are requesting me to import this text information to a SQL table. I do not have any experience with SQL Server, can some one give a idea of what I could do to accomplish this?, I would appreciate a lot anyone's help!, thanks! ...

View 4 Replies View Related