Import From Excel To SQL Server - Can It Be More Efficient ?
Oct 21, 2004
Hello All,
Iam new to the world of ASP .Net. Right now iam building an application that will IMPORT about 5,000 records from an Excel spreadsheet to a table in MS SQL Server. Right now the code works correctly, but i feel it is not efficient and takes a little bit of more time in doing the import. Could you guys throw some light on how i can make the code run more faster ? Someone suggested me that i can use DataAdapter and update the table in the database thru an update method available with it. I dont know how to do it? Could anyone share with me a snippet of code that does this ?
Here is my code:
Private Sub ProcessRecords()
Dim ds2 As New DataSet
' readExcelSheet is a user-defined function that reads a spreadsheet and returns a DataSet object
ds2 = readExcelSheet("C:InetpubwwwrootProject1Book2.xls", "SELECT * FROM [Sheet1$]")
Dim myConnection As SqlConnection = Connection() ' user-defined function that returns a SQLConnection object
myConnection.Open()
Dim strSQL As String = "insert_member" ' stored procedure that inserts records
Dim myCommand As New SqlCommand(strSQL, myConnection)
I'm a new user of SQL Server 2005. I have the full version installed. I also have SQL Server Business Integration Dev Studio installed. My OS is Windows XP.
I'm importing a series of 5 flat files into a database on one of the SQL Servers we have. My goal is to get 5 different tables (though perhaps I should do one and add an extra field to distinguish each import) into the database for further analysis.
I tried doing an import via DTS Wizard. There are no column names in the flat file so I defined them during the import process (all 58 of them). When I got to the end, I had an option to save the import process as a SSIS (SQL Server Integration Service) Package on:
SQL SERVER (I don't have permission for this)
or
FILE SYSTEM (did this one)
I saved the Package locally in hopes of being able to go back in, change the source file and destination table of the package and quickly get the other 4 flat files imported.
My problems are:
1) I couldn't find how to run the *.DTSX Package file to run in SQL Server Studio (basically reuse the Package with minor changes and saving me having to redefine the same 58 columns on each flat file import)
2) Tried but didn't understand how to run it in SQL Server Bus Intel Dev Studio (i.e. understanding the mapping and getting the data types right so it wouldn't error out)
3) Don't know how to make the necessary changes so that the Package handles the next source file and puts in a new destination table (do I need to do 5 CREATE TABLES so this Package has a place to run to?)
4) Does the Package need to be part of a Project to run (I haven't found how to take an existing Package and make it part of a Project/Solution)?
5) Is there a good book or online resource for just getting the basics of using SQL Server 2005 and SQL Server Business Intelligence Development Studio?
I'm really at a loss after spending a day fruitlessly on it scouring the help files, forums and experimenting around.
Hope somebody can point me in the right direction.
I just spent some time working out how to do a seemingly simple task. I€™m sharing the steps I took to do this in hopes it saves other SQL Server 2005 users (especially newbies like myself) time.
My original question posed on several SQL newsgroups was based on this goal:
I'm importing a series of 5 flat files (all with same file layout) into a database on one of the SQL Servers we have using SQL Server 2005 (SQL Server Management Studio) . My goal is to get 5 different tables. I want to do this without having to redo all the layout criteria 4 additional times.
Below are the steps I followed to get a solution (all done in Microsoft SQL Server Management Studio):
Create the Package (data import)
1) Use the SQL Server Import Export Wizard (equivalent to SQL Server 2000 Data Transfer Wizard) to import your first flat file. At the CHOOSE DATA SOURCE window browse for your file. 2) Under the Advanced tab, you can set your Column attributes (€śoutput column width€? or €śdata type€? to name a few). I highlighted all the columns and selected €śstring [DT_STR]€? for data type. To avoid truncation errors, I selected 255 for output column width. You can name the columns whose data you are most concerned with (I did import all the available fields). 3) After choosing a server destination you will have a €śSELECT SOURCE TABLES AND VIEWS€? window pop up. Under the €śMapping€? column you can choose to tweak your mapping further editing in SQL (see Edit SQL button). I didn€™t. 4) The €śSAVE AND EXECUTE PACKAGE€? will pop up. The €śExecute Immediately€? box should be checked and you should check the €śSave SSIS Package€? (SQL Server Integration Services). When you do, select €śFile System€? for where to save this import-file-package to. 5) Click OKAY for the Package Protection Level and the €śSAVE SSIS PACKAGE€? window will appear. Browse for a path on your local computer to save to.
Modify Package (data import) for Next Use
6) In SQL Server Management Studio, browse for the Package and open it.
Preparation for SQL Task €“ box
7) You should see a screen that shows two boxes (€śPreparation for SQL Task€?) and (€śData Flow Task€?). 8) Right click on the former and select €śEdit€?. 9) On the €śSQL Statement€? row, click into the right column and select the €ś€¦€? box 10) Change the destination table (the table you will create with this package) to a meaningful name and click OK. 11) Click OK for the €śSQL Task Editor€?
Data Flow Task - box
12) Right click on the €śData Flow Task€? box and select €śEdit€?. 13) Three boxes will appear €śSourceConnectionFlatFile€?, €śData Conversion 1€?, and €śDestination - <whatever table name your original data import went to>€?. Below them is a section that displays €śConnection Managers€?
SourceConnectionFlatFile - editing
14) The first thing you will want to do is change the import source to a new flat file. You do this by going below the boxes under the €śConnection Managers€? window and right clicking on €śSourceConnectionFlatFile€? and then selecting €śEdit€? 15) Browse for the new €śFile Name€? and select it. 16) A €śMicrosoft SQL Server Management Studio€? window will pop up asking you if you want to €śkeep or reset the existing metadata€?. The metadata is just your column definitions and choosing €śYES€? to keep this makes sense if you are doing data imports on files with the same file layout. 17) Still in the €śFlat File Connection Manager Editor€? window, change the €śConnection Manager Name€? to something meaningful (I add <_> at the end and then the name of the table the flat file is going to) and click OK.
SourceConnectionFlatFile €“ box (editing)
18) Right click on the €śSourceConnectionFlatFile€? box and select €śEdit€?. 19) Your newly named €śFlat File Connection Manager€? should appear in select box. 20) Click OK, right click again on the €śSourceConnectionFlatFile€? box and select €śShow Advanced Editor€?. 21) Under the €śConnections Manager€? tab, your newly named €śFlat File Connection€? should appear (the prior step is necessary for the advanced editor to recognize your change). 22) Under the €śComponent Properties€? tab, on the €śName€? row, click into the right column and rename to something meaningful (notice the €śIdentification String€? row description changes too once you click out of the €śName€? row) 23) Under the €śColumn Mappings€? tab, just confirm you are mapping your flat file fields (€śAvailable External Columns€?) to a destination table€™s fields (€śAvailable Output Columns€?). 24) Under the €śInput and Output Properties€? tab you can check in €śFlat File Source Output€? to make modifications to either your €śExternal Columns€? or your €śOutput Columns€? €“ you shouldn€™t need to for a simple import. ((NOTE: any changes you make here would likely need to be consistent with the column properties found under the €śConnection Manager Window€? for the €śSourceConnectionFlatFile€? as well as the €śData Conversion 1€? box under the €śData Flow Tasks€? window, so exercise caution 25) NOTE: This process has worked for me by making my source columns all €śstring [DT_STR]€? data type and the output columns all €śUnicode String [DT_WSTR]€? data type.
Data Conversion 1 €“ box (editing)
26) There is nothing you need to do here. By right clicking on the €śData Conversion 1€? box and selecting €śEdit€?, you can see and change the data type of the output columns (the ones in the table your importing the flat file to). There are probably more edits one can do but they€™re beyond what I€™ve learned.
Destination - <whatever table name your original data import went to> €“ box (editing)
27) Right click on the €śDestination - <whatever table name your original data import went to>€? box and select €śShow Advanced Editor€?. 28) Select the €śComponent Properties€? tab. 29) Select the right column at the €śName€? row and change the name to something meaningful (ie. related to the source file name or the table name you€™re importing to). 30) Select the right column at the €śIdentification String€? row and it will update to this change. 31) Select the right column at the €śOpenRowSet€? and change it to the name of the table you are importing your flat file to (this should be consistent with table name under step 10). 32) Click OK 33) Select FILE and select €śSave As€¦€? and then give your package a new name that€™s meaningful (this will be helpful if you have to rerun the import of the flat file later).
Run (execute) the Revised Package (data import)
34) Go back to SQL Server Management Studio and open the Object Explorer 35) Connect to an €śIntegration Services€? component. This should essentially be a local instance (not sure where it is on the local computer or in SQL Server Management Studio on the local computer). 36) In €śObject Explorer€? go down to your €śIntegration Services€? object and expand it. 37) Expand €śStored Packages€? 38) Right click on €śFile System€? and select €śImport Package€? and an €śIMPORT PACKAGE€? window will appear 39) For €śPackage Location€? choose €śFile System€? and then browse for the €śPackage Path€? 40) Click into the €śPackage Name€? and it defaults to your Package€™s file name. 41) Click OK and the Package is imported. 42) Right click on the newly imported Package and select €śRun Package€? 43) An €śExecute Package Utility€? window appears 44) Select €śExecute€? and the package runs.
I am writing a custom application which will allow users to upload data in a Excel to SQLServer.
The problem I have is that Excel can come with a number of worksheets. I would ideally like to read from the default work sheet - is there a way I can specify that my code should only pick up the default worksheet?
Greetings All,I have a excel file which is originally a sqlserver table that wasexported as a excel file. I have added more data to this excel fileand now want to import it again to its original table,i.e, it willoverwrite current data in the table but with no change in the schema.How should I handle the issue of PKs in the current table that will beover-written. I know sqlserver dose not adjust PKs when data isover-written, like my case.MTIA,Grawsha
I am trying to import an xlsx spreadsheet into a sql 2008 r2 database using the SSMS Import Wizard. When pointed to the spreadsheet ("choose a data source")  the Import Wizard returns this error:
"The operation could not be completed" The Microsoft ACE.OLEDB.12.0 provider is not registered on the local machine (System.Data)
How can I address that issue? (e.g. Where is this provider and how do I install it?)
Hi all, I am trying to import excel file to SQL server using web application. I have been browsing all day long trying to get some helps but I found none that really solves my problem. :( I am aware that I can use DTS in SQL server, however I want to build a web application for it. I can upload the file on the server, my problem is I want to load the data in excel file dynamically; I will need to create a table in the server dynamically everytime I import an excel file. Please help!!!Thanks Irma
ARRGGH, I'm going to kill Excel or SQL Server!! I've spent an hour on this and I can't get it working. I'm importing some address data, and several of the zip codes are importing as NULL, even though in the Excel sheet, they look fine! Excel is complaining, something about the zip codes are numbers entered as text..whatever.. so I edited my SQL import and made it so the ZIP field gets created as varchar 50 instead of float. It still won't import right.
I need to import some excel files in the Database via DTS. The problem is, that I don't know the names of the Excel Sheets (there are multiple sheets in the excel file). Is it possible to get the names of those sheets? (via activeX or SQL Query or anything, but from a DTS package)
Various excel files with the same structure will reside in a folder. These excel files should be exported daily in a Sql server table. If the data within the excel file according to an id is new then all data should be appended to the table, otherwise if the id exists in the sql server table then it must update the remaining corresponding fields.
I know how to export an excel file to a sql server table, i need some help in comparing the excel file data with the sql table so to make the appropriate updates.
Hi, please any one is expert in Data Base help me, i would like to import two Data columns from Excel file 2003 instead of two columns in a table in Sql Server Express 2005 , How Can i do that? and what is the way if I have Sql Developer 2005, thank you .
I am trying to import an Excel file - when I pick the file I get the message "Could not open file for reading. Close any other application that may be locking the file." I have verified that the file is not open - I have even rebooted the machine - still the same message - what am I doing different? Please advise. Thanks
My database has mirgrated from Access to a SQL Server. As of now, we use SharePoint for our data entry form and it works great exporting an Excel workbook to my desktop and then import to the SQL Server and drop it in the tables that apply. The management team wants to get rid of SP and have all 110 project owners use a template Excel workbook I created and then anytime they have new projects or changes to existing they use this template and email it to me. The problem is the template has partial text, combox boxes, checkboxes and drop-down boxes. In Access I can spend a couple weeks creating VBA code to import this but not sure how this is done in SQL Server. The 2nd issue is why in the heck would they even want to go this route. This seems to be a coding nightmare, then I have to store all the Excel workbooks somewhere for history sake. It just seems to me that keeping the SP site is much smarter for them and for me. My question is not related to SP because I am going to push to keep that up anyway. I just want to know is it possible for SQL Server to import a workbook that has form functions on it easily or is it a stored procedure coding nightmare?
Hi everyone got a problem here! I have an existing data in excel and it is more than 10,000 cells that I need to import to my new SQL Database. How can I transfer those records easily without using INSERT commands in SQL? Because I'm afraid it's too hard to do.
I need to import data from more than 10 excels having the same format in to a single sql server table.
I tried to use
INSERT INTO MyTempTable SELECT * FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0', 'Excel 11.0;Database=C:Book1.xls', [Sheet1$])
but got the below error Ad hoc access to OLE DB provider 'Microsoft.Jet.OLEDB.4.0' has been denied. You must access this provider through a linked server.
If DTS package is used then I am not sure how I can place 10 excels at a time so that they can be picked one by one and data is imported in to table.
I asked a similar question like this yesterday but i didnt work the way I wanted it to work. So I will ask in a diffrent way:
I have an Excel list which i want to import into a table in my SQL Server 2005. The ServerTable has one more column (for the the primary key which is created automatically) than the ExcelList. How can i import the ExcelList in a way so that I the first column of my ServerTable is not filled by a column of my ExcelList??
That is how far I came:
select * into SQLServerTable FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0', 'Excel 8.0;Database=D: esting.xls;HDR=YES', 'SELECT * FROM [Sheet1$]')
Firstly, i'm new to integration services and have only done a little with DTS jobs.
I'm trying to create an integration services project which will import data from an two worksheets in an Excel spreadsheet to two different tables in a database. I'm looking at only one table at present to make things a little more understandable.
One stipulation i have is that i need to be able to specify a variable value and insert that as an additional column in the database. I have and Excel source and a SQL destination both of which have been set up with there specific connection managers. I also have a variable which i add in using the derived column task.
When i try to debug this i am getting a few problems. I think these may be to do with the fact that although the worksheet in Excel has 20 rows (1st column shows these numbers) i only want those rows with data in them. If i preview the excel table it shows all the rows including those with null columns. Is there some sort of way that i can only get the rows that have data in the columns after the row number. I.e. can i select rows that do not have a second column value = to NULL.
I hope this makes sense and that someone can help me out with this problem.
All help is greatly appreciated.
Cheers,
Grant
P.S.
Apologies. I have this resolved now. I didn't see the option to use a SQL command as apposed to a table or view when setting up the Excel source.
I am still however getting the following errors which i'd appreciate some help on:
Error: 0xC0202009 at Data Flow Task, Excel Source [1]: An OLE DB error has occurred. Error code: 0x80040E21. Error: 0xC0208265 at Data Flow Task, Excel Source [1]: Failed to retrieve long data for column "Rework Entry Information (BE SPECIFIC)". Error: 0xC020901C at Data Flow Task, Excel Source [1]: There was an error with output column "Rework Entry Information" (170) on output "Excel Source Output" (9). The column status returned was: "DBSTATUS_UNAVAILABLE". Error: 0xC0209029 at Data Flow Task, Excel Source [1]: The "output column "Rework Entry Information" (170)" failed because error code 0xC0209071 occurred, and the error row disposition on "output column "Rework Entry Information" (170)" specifies failure on error. An error occurred on the specified object of the specified component. Error: 0xC0047038 at Data Flow Task, DTS.Pipeline: The PrimeOutput method on component "Excel Source" (1) returned error code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. Error: 0xC0047021 at Data Flow Task, DTS.Pipeline: Thread "SourceThread0" has exited with error code 0xC0047038. Error: 0xC0047039 at Data Flow Task, DTS.Pipeline: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. Error: 0xC0047021 at Data Flow Task, DTS.Pipeline: Thread "WorkThread0" has exited with error code 0xC0047039.
Hi, recently I encountered the following problem: I tried to execute a stored procedure on the newly installed SQL 2005 Server (now on x64 Win Server 2003) which imports an Excel-File into a DB table. We use OPENROWSET to access the Excel data. But I recognized this is dependent on Jet OLE DB which seems is not available for x64 windows.
Is there another way to import excel data using a stored procedure.
l've some excel files controlled by Vendor which changing frequently. The only thing does not change is the header name of each column.
So my question is, is there any way to create a new table based on the excel file selected including the column name in SSIS? So that l can use the data reader as source to select those columns l am interested on and start the integration.
I would like to import an Excel file into my SQL Server 2005 Express db. I'm using SQL Server Management Studio Express to manage my db. I appreciate your suggestions,
I try to import data from excel file, my excel file have column called Name, the value of this column is text: ex: Binh Chanh, 1,2,3,4,5.....When i import into sqlserver 2005, these rows which have value 1,2,3,4,5 (number) , now have Null value in SQL server 2005.
i am handling a project where user can choose the excel file and the field in the excel file to export into sql server 2005. which mean there will be dropdownlist where the user can choose the field and so on. anyone know how to do this?
I have an .xlsx file where I need to import the data into a table. If there is not a way to do this, is there a way to import either a tab del file or different type of .csv file into the database?
Do not want to use the SSIS or import feature from SQL2008 as I tried to save the steps and running it wont work either.
I was using linked servers to import Excel spreadsheets into SQL Server Express 2005. This worked fine with Windows XP and Office 2003.
I have just migrated all my stuff to Vista and Office 2007. Linked servers just can't be created:
TITLE: Microsoft SQL Server Management Studio Express ------------------------------ "The linked server has been created but failed a connection test. Do you want to keep the linked server?" ------------------------------ ADDITIONAL INFORMATION: An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.Express.ConnectionInfo) ------------------------------ Cannot initialize the data source object of OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "XXX". OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "XXX" returned message "Unspecified error". (Microsoft SQL Server, Error: 7303) For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=09.00.3042&EvtSrc=MSSQLServer&EvtID=7303&LinkId=20476 ------------------------------ BUTTONS: &Yes &No ------------------------------
The parameter values I used are:
Provider: Microsoft Jet 4.0 OLE DB Provider Product name: Excel Data source: D:...somelocalfile.xls Provider string: Excel 8.0
I gave full access rights to the Data source file and folder to NETWORK SERVICE and SQLServer2005MSSQLUser$MOZART$SQLEXPRESS where mozart is my PC name.
If I change Data source or Provider string to some garbage string, the exact same error message appears. So it appears the error might be in the Provider?
I used sql server 2012 express Import and Export Data (32-bit) wizard to import data from excel 2010 to a given table. But I got the following error message: Error 0xc0202009: Data Flow Task 1: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Unspecified error". (SQL Server Import and Export Wizard)  Error 0xc020901c: Data Flow Task 1: There was an error with Destination - MPRecord.Inputs[Destination Input].Columns[Top1] on Destination - MPRecord.Inputs[Destination Input]. The column status returned was: "The value violated the integrity constraints for the column.". (SQL Server Import and Export Wizard)  Error 0xc0209029: Data Flow Task 1: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "Destination - MPRecord.Inputs[Destination Input]" failed because error code 0xC020907D occurred, and the error row disposition on "Destination - MPRecord.Inputs[Destination Input]" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. (SQL Server Import and Export Wizard)  Error 0xc0047022: Data Flow Task 1: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Destination - MPRecord" (35) failed with error code 0xC0209029 while processing input "Destination Input" (48). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.