How To Import Hundreds Of CSV Files Into SQL Server?
Feb 5, 2004
Hi,
I have about 500 CSV files, mostly they are unser same structure. I need to import them all into SQL server. I create a new table for each one based on the firstline as the column name. I can do them one by one but it is very time consuming and boring. Is there a fast way of doing this?
If someone give me a information, I promise to share all of MY CSV files. They are the zone chart from UPS and I have edit all of them.
I have a task to import *.log files in to the Microsoft SQL Server and need help in this regards.
This import should be the best table structure/format for the web log (considerations for import time, size, indexing, querying, reporting, etc.), and best utilize all available space on the server
Hi, I try to import csv files to Sql Server using .net. The code is as following: string strCsvConn = @"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=;Extended Properties='text;HDR=Yes;FMT=Delimited(,)';"; using (OleDbConnection cn = new OleDbConnection(strCsvConn)) { string strSQL = "SELECT * FROM " + strFileName; OleDbCommand cmd = new OleDbCommand(strSQL, cn); cn.Open();using (OleDbDataReader dr = cmd.ExecuteReader()) { // Bulk Copy to SQL Server using (SqlBulkCopy bulkCopy = new SqlBulkCopy(strSqlConn)) { bulkCopy.DestinationTableName = strSqlTable; bulkCopy.WriteToServer(dr); } } } And the data is as following (simplified): Model,Serial AFICIO 3045,K9464900965 AFICIO 3045,K9464900932 Fax 5510L,A3761290041 Fax 2210L,A4978800008 AFICIO 3025,K8565201014 AFICIO 3025,K8565102398 The result of the 2nd column is: 9464900965, 9464900932, null, null, 8565201014, 8565102398 - either the first character is missing or the whole entry is missing. One more weird thing is that some other files work fine, though I am not able to tell any difference between them. Any idea is hoghly appreciated. shz
Various excel files with the same structure will reside in a folder. These excel files should be exported daily in a Sql server table. If the data within the excel file according to an id is new then all data should be appended to the table, otherwise if the id exists in the sql server table then it must update the remaining corresponding fields.
I know how to export an excel file to a sql server table, i need some help in comparing the excel file data with the sql table so to make the appropriate updates.
Hi,i wanna develop an web-database application with ASP.NET,C#, SQL server 2000.i already have some data whichs been in text format(text file) and now, i want to import the same into my database.the problem is, the text file has got many line breaks and also its not well formated to import it using DTS.Can any one help me out in importing the same.thanks in advance
I am a new user of SQL Server 2000. Please point me where I am able to get good online sources to be familiar with the SQL Server 2000. I wanted to import the primary file (MDF)and a log file (LDF), which are stored on two different floppy disks into the local SQL server 2000. Please direct me where I should start. Your help is greatly appreciated.
I have created a DTS package which imports text file into single sql server table with 8 columns (SourceData). The DTS package uses 'Test1.txt' file. Now i have around 200 text files (Test1,Test2,.....Test200). I need to import them one by one into 'SourceData' table. Could you pls. help me out in getting solved this mistery.
I need to extract data from text files (around 200) and import into sql server tables. I tried using SSIS foreach loop container but could not manage it. Can anyone guide me how this can be done?
Hello,I am trying to import a CSV file into my SQL Server database, this file was originally generated by another database table (on another server) with the same structure, the table contains two columns of real datatype with Allow Null Value setto true for those columns, the CSV file contains the value NULL for theses columns, I am facing a problem when importing this file. This may be because DTS tries to represent values as strings then to convert them to real datatype which results in transforming the value "NULL" to real, I receive an error message saying.Error during Transformation 'DirectCopyXform' for Row number 1. Errors encountered so far in this task: 1. TransformCopy 'DirectCopyXform' conversion error: Conversion invalid for datatypes on column pair 8 (source column 'Col008' (DBTYPE_STR), destination column 'zip_longitude' (DBTYPE_R4)). TransformCopy 'DirectCopyXform' conversion error: Conversion invalid for datatypes on column pair 7 (source column 'Col007' (DBTYPE_STR), destination column 'zip_latitude' (DBTYPE_R4)).How can I work around this problem? Any help would be appreciable
Brief overview...Running SQL Server 2003 Server Enterprise 64 bit - All Service Packs and patches current SQL Server 2005 Enterprise Edition 64 bit Build Microsoft SQL Server 2005 - 9.00.3054.00 (X64) Mar 23 2007 18:41:50 Copyright (c) 1988-2005 Microsoft Corporation Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)
I cannot import any SSIS packages nor crete any new folders under stored packages. I hve googled the news groups and looked at BOL to no avail. HELP!!!!
I'm a new user of SQL Server 2005. I have the full version installed. I also have SQL Server Business Integration Dev Studio installed. My OS is Windows XP.
I'm importing a series of 5 flat files into a database on one of the SQL Servers we have. My goal is to get 5 different tables (though perhaps I should do one and add an extra field to distinguish each import) into the database for further analysis.
I tried doing an import via DTS Wizard. There are no column names in the flat file so I defined them during the import process (all 58 of them). When I got to the end, I had an option to save the import process as a SSIS (SQL Server Integration Service) Package on:
SQL SERVER (I don't have permission for this)
or
FILE SYSTEM (did this one)
I saved the Package locally in hopes of being able to go back in, change the source file and destination table of the package and quickly get the other 4 flat files imported.
My problems are:
1) I couldn't find how to run the *.DTSX Package file to run in SQL Server Studio (basically reuse the Package with minor changes and saving me having to redefine the same 58 columns on each flat file import)
2) Tried but didn't understand how to run it in SQL Server Bus Intel Dev Studio (i.e. understanding the mapping and getting the data types right so it wouldn't error out)
3) Don't know how to make the necessary changes so that the Package handles the next source file and puts in a new destination table (do I need to do 5 CREATE TABLES so this Package has a place to run to?)
4) Does the Package need to be part of a Project to run (I haven't found how to take an existing Package and make it part of a Project/Solution)?
5) Is there a good book or online resource for just getting the basics of using SQL Server 2005 and SQL Server Business Intelligence Development Studio?
I'm really at a loss after spending a day fruitlessly on it scouring the help files, forums and experimenting around.
Hope somebody can point me in the right direction.
I just spent some time working out how to do a seemingly simple task. I€™m sharing the steps I took to do this in hopes it saves other SQL Server 2005 users (especially newbies like myself) time.
My original question posed on several SQL newsgroups was based on this goal:
I'm importing a series of 5 flat files (all with same file layout) into a database on one of the SQL Servers we have using SQL Server 2005 (SQL Server Management Studio) . My goal is to get 5 different tables. I want to do this without having to redo all the layout criteria 4 additional times.
Below are the steps I followed to get a solution (all done in Microsoft SQL Server Management Studio):
Create the Package (data import)
1) Use the SQL Server Import Export Wizard (equivalent to SQL Server 2000 Data Transfer Wizard) to import your first flat file. At the CHOOSE DATA SOURCE window browse for your file. 2) Under the Advanced tab, you can set your Column attributes (€œoutput column width€? or €œdata type€? to name a few). I highlighted all the columns and selected €œstring [DT_STR]€? for data type. To avoid truncation errors, I selected 255 for output column width. You can name the columns whose data you are most concerned with (I did import all the available fields). 3) After choosing a server destination you will have a €œSELECT SOURCE TABLES AND VIEWS€? window pop up. Under the €œMapping€? column you can choose to tweak your mapping further editing in SQL (see Edit SQL button). I didn€™t. 4) The €œSAVE AND EXECUTE PACKAGE€? will pop up. The €œExecute Immediately€? box should be checked and you should check the €œSave SSIS Package€? (SQL Server Integration Services). When you do, select €œFile System€? for where to save this import-file-package to. 5) Click OKAY for the Package Protection Level and the €œSAVE SSIS PACKAGE€? window will appear. Browse for a path on your local computer to save to.
Modify Package (data import) for Next Use
6) In SQL Server Management Studio, browse for the Package and open it.
Preparation for SQL Task €“ box
7) You should see a screen that shows two boxes (€œPreparation for SQL Task€?) and (€œData Flow Task€?). 8) Right click on the former and select €œEdit€?. 9) On the €œSQL Statement€? row, click into the right column and select the €œ€¦€? box 10) Change the destination table (the table you will create with this package) to a meaningful name and click OK. 11) Click OK for the €œSQL Task Editor€?
Data Flow Task - box
12) Right click on the €œData Flow Task€? box and select €œEdit€?. 13) Three boxes will appear €œSourceConnectionFlatFile€?, €œData Conversion 1€?, and €œDestination - <whatever table name your original data import went to>€?. Below them is a section that displays €œConnection Managers€?
SourceConnectionFlatFile - editing
14) The first thing you will want to do is change the import source to a new flat file. You do this by going below the boxes under the €œConnection Managers€? window and right clicking on €œSourceConnectionFlatFile€? and then selecting €œEdit€? 15) Browse for the new €œFile Name€? and select it. 16) A €œMicrosoft SQL Server Management Studio€? window will pop up asking you if you want to €œkeep or reset the existing metadata€?. The metadata is just your column definitions and choosing €œYES€? to keep this makes sense if you are doing data imports on files with the same file layout. 17) Still in the €œFlat File Connection Manager Editor€? window, change the €œConnection Manager Name€? to something meaningful (I add <_> at the end and then the name of the table the flat file is going to) and click OK.
SourceConnectionFlatFile €“ box (editing)
18) Right click on the €œSourceConnectionFlatFile€? box and select €œEdit€?. 19) Your newly named €œFlat File Connection Manager€? should appear in select box. 20) Click OK, right click again on the €œSourceConnectionFlatFile€? box and select €œShow Advanced Editor€?. 21) Under the €œConnections Manager€? tab, your newly named €œFlat File Connection€? should appear (the prior step is necessary for the advanced editor to recognize your change). 22) Under the €œComponent Properties€? tab, on the €œName€? row, click into the right column and rename to something meaningful (notice the €œIdentification String€? row description changes too once you click out of the €œName€? row) 23) Under the €œColumn Mappings€? tab, just confirm you are mapping your flat file fields (€œAvailable External Columns€?) to a destination table€™s fields (€œAvailable Output Columns€?). 24) Under the €œInput and Output Properties€? tab you can check in €œFlat File Source Output€? to make modifications to either your €œExternal Columns€? or your €œOutput Columns€? €“ you shouldn€™t need to for a simple import. ((NOTE: any changes you make here would likely need to be consistent with the column properties found under the €œConnection Manager Window€? for the €œSourceConnectionFlatFile€? as well as the €œData Conversion 1€? box under the €œData Flow Tasks€? window, so exercise caution 25) NOTE: This process has worked for me by making my source columns all €œstring [DT_STR]€? data type and the output columns all €œUnicode String [DT_WSTR]€? data type.
Data Conversion 1 €“ box (editing)
26) There is nothing you need to do here. By right clicking on the €œData Conversion 1€? box and selecting €œEdit€?, you can see and change the data type of the output columns (the ones in the table your importing the flat file to). There are probably more edits one can do but they€™re beyond what I€™ve learned.
Destination - <whatever table name your original data import went to> €“ box (editing)
27) Right click on the €œDestination - <whatever table name your original data import went to>€? box and select €œShow Advanced Editor€?. 28) Select the €œComponent Properties€? tab. 29) Select the right column at the €œName€? row and change the name to something meaningful (ie. related to the source file name or the table name you€™re importing to). 30) Select the right column at the €œIdentification String€? row and it will update to this change. 31) Select the right column at the €œOpenRowSet€? and change it to the name of the table you are importing your flat file to (this should be consistent with table name under step 10). 32) Click OK 33) Select FILE and select €œSave As€¦€? and then give your package a new name that€™s meaningful (this will be helpful if you have to rerun the import of the flat file later).
Run (execute) the Revised Package (data import)
34) Go back to SQL Server Management Studio and open the Object Explorer 35) Connect to an €œIntegration Services€? component. This should essentially be a local instance (not sure where it is on the local computer or in SQL Server Management Studio on the local computer). 36) In €œObject Explorer€? go down to your €œIntegration Services€? object and expand it. 37) Expand €œStored Packages€? 38) Right click on €œFile System€? and select €œImport Package€? and an €œIMPORT PACKAGE€? window will appear 39) For €œPackage Location€? choose €œFile System€? and then browse for the €œPackage Path€? 40) Click into the €œPackage Name€? and it defaults to your Package€™s file name. 41) Click OK and the Package is imported. 42) Right click on the newly imported Package and select €œRun Package€? 43) An €œExecute Package Utility€? window appears 44) Select €œExecute€? and the package runs.
What are the downsides to have many multiple views on a SQL Server Database?
It was somebody's design decision to create 10 views for each vendor in our database- that could range from 20 to 500 vendors ... which means we could potentially have 5000+ views in our database.
We are running a hosted application where we currently have over 700 databases and expect to go to over 2000 within a year. This is all running currently on a 2 node fail-over cluster connected to a SAN.
I'm basically looking for advice on growth startegy from anyone that may have a similar setup. Should I buy the biggest boxes I can afford and keep piling the databases on the same server? Should I add clusters and split the databases in bunches of, say, 500?
Performance is not bad right now but I'm not sure how long that will continue and I would rather plan than fight fires.
i have a table with a few hundred columns. Each SELECT statement, I list each of the columns, this is taking lots and lots of space and it is difficult to review the code due to its length...Below, I have to list out every column when I only want to use case logic on 1 column. In the next step I will have to list out every single column again
SELECT ACCT1_NO ,ACCT1_DT ,ACCT1_RISK ,ACCT1_RISK_WEIGHT ,ACCT1_CUSTOMER_NAME ,ACCT1_CUSTOMER_ADDRESS ,ACCT1_CUSTOMER_ST = CASE WHEN ACCT1_CUSTOMER_ST = ' ' THEN 'DC' END ,ACCT1_CUSTOMER_CITY ,ACCT1_CUSTOMER_ZIP --,THIS CONTINUES DOWN FOR ANOTHER 150 OR SO ACCTS. INTO #A1 FROM STAGE.CUSTOMER_ACCTS
Is there a way I can tell SQL to take all of the columns and then list the column where I want to do my case statement. Something like the code below
(which will fail as ACCT1_CUSTOMER_ST will be listed twice. SELECT * ,ACCT1_CUSTOMER_ST = CASE WHEN ACCT1_CUSTOMER_ST = ' ' THEN 'DC' END INTO #A1 FROM STAGE.CUSTOMER_ACCTS
I'm looking for a way to store a large chunck of text (200 lines) in a variable which can be called in different objects within the database.
For example: I have several stored procedures that create the same temp table which exists of 200 column names over and over again.
It would have to look like this
Import myScript (I keep thinking of Import like used in .NET)
and myScript could exist of:
CREATE #MyTable( ...)
or even some simple text or a part of a sql statement.
I'm familiar with synonyms but you can't use that in this scenario. Then I thought of functions. The scalar function returns a result, not what I want here. Table valued function return tables, not what I want.
Hi i have a question. How can i import dump from a text file , to an exsiting database, i´m will not useing DTS, i what to use BULK cmd in sql queryanalyser etc.
I have SQL 2000 running under windows 2000 server.
Question: I'm receving CSV files from the field. I would like to built SQL database from daily CSV files. I've been doing this manually :( however, is there anyway I can do this task automactically so that once new daily CSV file ~200KB appear in the folder, progarm automatically insert recent file to SQL database...:) Do I need to write VB progarm for this? Please help...
I have created a stored procedure to handle xml data. One of the inputs to this procedure is xmldata which is basically the contents of the xml in a file. Now I would like to create a SSIS package that takes the xml files from the network and uses this SP. Should I have a file system task inside a for loop container? How do I get started on this please? Thanks
I have created a stored procedure to handle xml data. One of the inputs to this procedure is xmldata which is basically the contents of the xml in a file. Now I would like to create a SSIS package that takes the xml files from the network and uses this SP. Should I have a file system task inside a for loop container? How do I get started on this please? Thanks
Hi, I have about 300-400 XML files I want to load in my SQL database (2005). The following code will load one (1) file. How do i do a mulitple collections? INSERT INTO MEL (DATA) SELECT * FROM OPENROWSET (BULK'C:TempCHAPTER1.xml', SINGLE_BLOB) AS TEMP Thanks,
I'm new to SQL server. I would like to ask how to do the following. Now I have many CSV files in a directory, which are uploaded through ftp from my client computers daily. How can I automatically import those CSV files to the SQL server? Can I do it by a script or a stored procedure or something else?
I have data in excel files I want to import it to existing SQL Server database. I can use SQL Server built in import tool but before Importing, the data needs to be validated. What is the best way to do this?
I have 8GB of text files which are basically log files from the past few years. There is 24 text files per directory which are labeled for every day (so they are not all in 1 folder). It would make reading them much easier if I could import them to SQL but I only seem to be able to import 1 at a time? (with the wizards :eek: )
Surely there is a way to mass import without all the costly applications that google searches give me? cheers :P
I have a job that calls about 5 DTS's. Each DTS imports a text file. The problem is that some of the files may not be there every day. How do I exit the DTS without an error if the file does not exist, but I want my job to keep running...
I have a script which imports the contents of a csv file from our CRM system and updates a table in my database. This works OK but the problems I have are that a) sometimes there is more than one file in the folder, and b) that I wish to move any csv files that have been imported into an archive folder. The csv files arrive with a time/datestamp and I currently rename them manually to FREXPORT before importing (the name is in the format FREXPORT_20141101_1217.csv).How do I:
1) get it to process the file without me having to manually rename the file(s) each time, 2) if there is more than 1 file in the folder process all the files and 3)move the correctly processed files to an archive folder which is: importarchive?
Ultimately, I would like the script to be run as a scheduled job, so it also has to deal with the fact that sometimes there will be no files to import too.
I have several hundred .csv files that have a specific cell that I need to get into a SQL table. These files are named after the date on which they were created...ie 8252005 would be todays date. Im looking for a way to import this cell to SQL... the same cell in each file.... Thanks for any help
I'd like to set up a dts package to import .csv files and on testing one .csv with only one row I am getting this error:
Error Source: Microsoft Data Transformation Services Flat File Rowset Provider
Error Description: Too many columns found in the current row; non-whitespace characters were found after the last defined column's data.
data set from .csv: -31,0,"CENTER","","","64 ROCK DR. #4100","BETH","MD.","20417","0470-5",30109,10598,"52492000 083","1018","406800","6896CE","7007","G00887",1,13983.24,2139.66,42741.9,150101.36,830317.41,1562843.99,30128.45,2,506,0,0,0,0,0,0,37.87,1,0,0,476.39,472.15,0,1191.12,0,90073.46,43038.97,0,309926.59,-14880.49,20060102,20090101,"A","","",""," ","C62454",0,0,0,0,0,0,0,0,8.39,0,0,0,0,"",0,"00000000",""
