HiI'm rather new to ASP.NET (was working with win forms before) and having a few problems with my first app. To the point - this is an internal system - every week about 50 branches send financial reports to the headquaters where those excel files are validated against data in sql server. after file is uploaded to the server I'm opening it using Excel object model and iterating through it line by line. Apart from format and datatype checking, for each line there are ~2 stored procedures invokedEverything is working actually but my concern is performance.. After about 800 rows it is getting really slow.. Appreciate any comments on this.thanksAM
Our ETL process involves some pre-load validation, and I'm wondering how best to implement it in SSIS.
Some details on my situation: I need to import 30 flat files with different data formats into 30 destination tables. In addition, these files share a common header and footer row format, and I need to validate these headers and footers before using the imported data downstream. (For example, the footer contains a record count, and fields in the header and footer should match some user variables.) My first approach was to write a Perl script that splits each file into three (header, data, and footer), but while that makes it easy to import the data section, it's more complicated to validate the header and footer and work them into the control flow. I think I'd also have to copy the same logic for all 30 data flows, which is less than ideal.
It looks like implementing this logic directly in SSIS is a little ugly (though that could be my lack of experience speaking). As I thought about this some more, I came up with a couple other solutions -- any critiques or comments?
1) Write a custom source adapter (which will probably contain the default flat file adapter) that knows how to validate my header and footer. I'd be able to read the file formats from an XML file, which might make my scripts more generic, and I might even be able to handle some custom data conversions more elegantly than I'm doing right now. (These files represent null numerics as whitespace rather than an empty field.)
2) Beef up the Perl splitter to validate the header and footer. If the cleanest approach is to say "assume that SSIS is only loading pre-validated data", this makes the problem entirely external.
Or am I entirely missing the mark here? Any thoughts?
I created a package loading Excel file. I developed the package using test excel file c:TestTest.xls and use a config file to assign actual folder (the file names will be got from a foreach loop container) when run it.
However, the package cannot run with a configure file which specify a different source folder and still ask for test.xls. It's not possible that it require reset the folder and file every time when it's used for a new folder.
Information: 0x40016041 at Package1: The package is attempting to configure from the XML file "C:workingFinal3.dtsConfig". SSIS package "Package1.dtsx" starting.
Information: 0x4004300A at Load Raw1, DTS.Pipeline: Validation phase is beginning.
Error: 0xC0202009 at Package1, Connection manager "Excel Connection Manager": SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft JET Database Engine" Hresult: 0x80004005 Description: "The Microsoft Jet database engine could not find the object '\SourceFolderTest.xls'. Make sure the object exists and that you spell its name and the path name correctly.".
Error: 0xC020801C at Load Raw1, Excel Source [1188]: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
Error: 0xC0047017 at Load Raw1, DTS.Pipeline: component "Excel Source" (1188) failed validation and returned error code 0xC020801C.
Error: 0xC004700C at Load Raw1, DTS.Pipeline: One or more component failed validation.
Error: 0xC0024107 at Load Raw1: There were errors during task validation.
Various excel files with the same structure will reside in a folder. These excel files should be exported daily in a Sql server table. If the data within the excel file according to an id is new then all data should be appended to the table, otherwise if the id exists in the sql server table then it must update the remaining corresponding fields.
I know how to export an excel file to a sql server table, i need some help in comparing the excel file data with the sql table so to make the appropriate updates.
I'm having a tough time importing some of my legacy database into sql. I have a number of dbase (IV) files I need to get into SQL. I have tried building a SSIS package with either an foxpro oledb connection or a jet 4.0 one, none of them work bec. of inconsistencies in the data format in my tables (e.g. date fields, etc).
I have tried to save the .dbfs as excel 2007 files, taking advantage of the larger space that comes with '07. Problem is you can't use the import/export wizard with 2007 for some reason and I haven't been able to create a package with the access 12 oledb as I have read.
I have to get some crucial data out of that old system and into the new one and I can't seem to be able to import them properly.
Any hints on what I should do ? (maybe I'm doing something awfully wrong)
Thank you for taking the time to answer my question, Val
I have a SSIS package which reads an excel file and loads data into a table using script component(C#) as a source. The package runs without any errors when I manually run it on my machine and on the server. But the package fails when run as a SQL Server Agent job.
I tried all the possible fixes I found on the web but still can't get it to work.
Hello, I have a table T1 with ID and Desc fields. I want to export this table to excel files based on the ID field. So my stored procedure select all the same T1.ID and export it to ID.xls files. How can I do this?
I have data in excel files I want to import it to existing SQL Server database. I can use SQL Server built in import tool but before Importing, the data needs to be validated. What is the best way to do this?
Hello, I have a table T1 with ID and Desc fields. I want to export this table to excel files based on the ID field. So my stored procedure select all the same T1.ID and export it to ID.xls files. How can I do this?
I am writing a SSIS package to load a lot of Excel files. I use SQL statement to select the Excel data. However, I found it's hard to dynamical set the table name (Excel Tab name) - the user name the Tab differently.
I have a client who is sending me 800+ excel files each month with sales data. Each of the files is identical in structure, but has sales data for different stores. I receive all these files at the same time.
Is there a method with Data Transformation Services where I can have it work off of all the files in a given directory. I can set up DTS to work off of specific Excel files with no problem, but what I would like to do is set up a DTS so it could pull from each of the 800+ files.
Is this possible, or do I need to look at a solution outside of SQL to consolidate the Excel files first?
The Excel file would have columns similar to the following: store_id, zip_code, sales, transactions.
What other software, besides MS Excel, can I use to view *.cub files? We plan on giving OLAP access to some people. However, buying MS Office just for to get Excel is too expensive.
Hi, Using SSIS, I am successfully importing data from excel files into a table in sql server 2005. These excel files are ALWAYS open and are being updated from external sources i.e. third party tools. After doing alot of investigations, I have reached the conclusions as follows: When the excel files are NOT being updated, then the ssis package works BUT when the excel files are being updated, then the ssis package does not work. The error is:
[Excel Source [749]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
Please note that the excel files MUST be open so that they get updated by the third party tool. AND, every few minutes i.e. 5 mins, my ssis package should import these excel files.
Question: How is it possible to load the excel data while it is being updated by external third party.
It's very simple import, without any data modifications; from severalexcel files to one table.I tried the wisard and it gives me selection for only one file.I am not used to create DTS packages and schedule jobs, so I would needsome help.Thank youInna
HiI have multiple excel files of the same format in a directory.They are called book1.xls, book2.xls, book3.xls and so on.What is the easiest way to import the tab named sheet1 from each ofthe excel files to a databse using SQL server 2000 enterprise edition?Regards,Ciarán
Hi, I have just started learning SSIS. Could someone please tell me if where can I find step by step instructions on how to simply extract data from two excel files and populate the relevant table. What I simply want to do is:
Excel File 1 (With Columns FirstName, DateJoined) + Excel File 2 (with column Summary) | | ->Add these three columns to a new table called CustSummary
Any thoughts and suggestions will be really appreciated.
Hi, Using SSIS, I am successfully importing data from excel files into a table in sql server 2005. These excel files are ALWAYS open and are being updated from external sources i.e. third party tools. After doing alot of investigations, I have reached the conclusions as follows: When the excel files are NOT being updated, then the ssis package works BUT when the excel files are being updated, then the ssis package does not work. The error is:
[Excel Source [749]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
Please note that the excel files MUST be open so that they get updated by the third party tool. AND, every few minutes i.e. 5 mins, my ssis package should import these excel files.
Question: How is it possible to load the excel data while it is being updated by external third party.
Hi, Trying to dynamically set the connectionstring property of the excel source. This is what I enter instead of the hardcoded excel file paths: "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + @[User::FileName] + ";Extended Properties=Excel 8.0;HDR=YES"
I get this error, every time I set the delay validation property of the dataflow tab to true. Cannot detach from one or more processes. The object invoked has disconnected from its clients Do you want to terminate them instead?
I need to import around 200 excel file data into one table. Is there a way of doing this using SSIS or DTS? I know how to import single excel file into table but i need to automate this process for many files. All help appreciated
I know from experience that the Excel connection mechanisms are somewhat limited. I was wondering if anyone has come across a solution to this problem.
We receive an excel file from a business line. Basically a list of account mods for a given month. Due to the tabular nature of Excel, sometimes the amount of mods exceeds the limits of one tab and has to roll over into a second tab. The names of the tabs reflect the file creation date, which often is the last day of the month, but not always. Here is a question related to this-
Is there a way to "query" a the list of tabs in an Excel file, so that I could store that record set in an SSIS variable, use it to loop through all the tabs? If no, can you think of a way to somehow get the value of the tab name or names so that I can use it to dynamically set the OpenRowset value of the Excel Data Source?
Hi everybody, i'm new to SSIS, so it's possible that mine is a very stupid question
I have to develop a simple ETL package that reads data from a csv file and writes them to an xls file; the problem is that when the number of rows exceeds the maximum number of rows allowed for an xls file i get an error.
There is a way to solve this problem? for example adding a new sheet or creating a new file?
i need to transfer (migrate ) the data from xl sheet to sqlserver but actually the thing is if the source excel file has different sheets, in each sheet i have the data
and i need to move the entire data( all the data that is present in all sheets of the excel file) to a single table into sql server
like wise i have many xl files ( which have many sheets ) .
for eg:
excel file 1:
-> sheet 1
-> sheet 2
-> sheet 3
excel file 2:
-> sheet 1
-> sheet 2
-> sheet 3
excel file 3:
-> sheet 1
-> sheet 2
-> sheet 3
now i need to get the data from all of the files and i need to insert into a single table ( sql server) in ssis package
I am trying to import multiple .csv files to excel sheets using Script task in SSIS. I have trouble importing the reference that allows us to read and write to excel sheets. Can anyone help me create a script task that will import multiple .csv files to excel sheets.
i have a MSsql file wich i want to export information from. Every post in the database should be exportet to a own single file with all the information about the post. So from a single question to the database i would like to have a lot of files named by a speciel filed i the post. I also want to be able to design what the excel file should look like.
I'm having a problem getting the for loop container to process all excel files in a folder. I set the collection folder to where my .xls files are, and i set a variable in the for loop container to the FileName. I then changed my source connection and added expressions for
I'm new to SSIS and have a question about the best way to generate multiple Excel files with my current package design. I have a stored procedure that I run from an Execute SQL Task in a foreach loop container, and it generates results as appropriate for each of the parameters it loops through.
What's the best way to take the result set from each of those executions and generate an Excel file from each set? How do I map that result set variable to be the input for creating a new Excel file? Is this best done as a script task somehow?
Should I not be using SSIS for this task? I thought I would just create a package and schedule it to run daily with SQL Agent, and it would autogenerate the Excel files as needed.
I have a package that loops over ~60 Excel files in a directory. Each file has three named ranges in it, which I import into different tables. Sometimes the package runs without a hitch, sometimes it chokes. But it is intermittent.
If I pull the control flow components out of the foreach loop and point the Excel connection manager to the specific Excel file that has caused the package to choke, I get a message in the dataflow component pointing to the named range that "the metadata of the following output columns does not match the metadata of the external columns......Do you want to replace the metadata of the output columns with the metadata of the external columns?" When I choose 'yes', then the file will be loaded. then I can put the control flow components back into the foreach loop and the file will run again, successfully, along with some more, until it chokes again....
So, first of all, does anyone have any insight into this? Sometimes, somedays, these files will load with no problems. These exact files; I am having to reload constantly... Other times, like today, it is a battle.
Otherwise, is there a way to get Integration Svcs to handle the metadata issue on the fly???
Any ideas, resources, references, war stories, or good clean jokes would be appreciated, Kathryn
I have created a package which import data from excel file and do some technical & business validation on the data. My package has about 20 control flow items. Now I'm asked to handle a second (and probably more in the future) excel file format (columns name are different, some fields are murged in one single column...).
I definitely don't want to create a different package for each excel file format. But I can't find a way in the control flow to execute a particular DataFlow in one case and another DataFlow in other cases. Typically I would like to evaluate an expression an depending on the result execute a DataFlow or another one. Even in a given DataFlow I cant find a way to have a condition and process different Excel Source depending on an expression result. Or it would be good if I could say to my Excel Source to discover the columns name and types at runtime and let me manage the columns manually in the data flow. Is that possible ? I know SSIS manage metadata on the columns based on the data source is there any way to manage the metadata manually ? I coulnd't find anything about that in BOL.
I guess an easy workaround is to have a different package just to import the different excel files in a common staging table and each package calls a single package which contains all technical & business validation.
I need to create about 1,000 (literately) Excel files that each contain 5 tabs. The data being placed on the tabs will always be the same (meaning the columns are static). I am fairly advanced at Excel VBA so I can write code that does all the following in Excel (looped 1,000 times):
Open an Excel template
Bring data in from the tables
Filter, then copy-paste the appropriate rows into each tab.
Save the new Excel file.
Email the file to appropriate individual (it is a Microsoft Exchange Server). As I started this in VBA, I thought that I might be able to do it with SSIS. My concern is I need to have the rows formatted (font, border, etc.) and the number of rows change.
My questions are: Is it possible to format Excel with SSIS? Can I email the files even if it is not with an SMTP protocol? Would SSIS process this data faster then Excel? Does this approach even make sense? Am I better just doing it with VBA?
Hi I am trying to load a bunch of excel files into a table and running into tables
I create a seq container--> add a foreach loop control task configure a variable thats ok as I am displaying the files names through script
I am stuck as how to configure the next data flow task to load into a destination I create a source excel and then map the expression to the variable but the destination SQL connection is not able to see this one
What I'm trying to achieve is a SSIS package that will pick up 1 or more excel files, process the data in them via the conditional splitter, pushing the good data into a table, and all other rows into an error table.
I'm having some issues using the ForEach container to process multiple excel spreadsheets into tables. The excel import into the tables is more or less working (imports data for good cases, but uses a null if the Excel Source when it gets an unexpected value - but that's a seperate problem).
I found something related to this when searching, but it related to CTPs (June and September) and trying to reuse the connection strings they built up there (using my own variable names, naturally) causes a 'Property Value failure':
The connection string format is not valid. It must consist of one or more components of the form X=Y, separated by semicolons. This error occurs when a connection string with zero components is set on database connection manager.
The excel importer works fine as a stand-alone component. Trying to use the process defined in 'Profession SQL Server Integration Services' pp140, I tried to use an expression to assign the variable value to the connection string. I get a validation error:
SSIS breakpoint 'Break when the container receives the OnPreExecute event' at executable 'Excel Importer' has been hit
SSIS breakpoint 'Break when the container receives the OnPreExecute event' at executable 'Foreach Loop Container' has been hit
SSIS breakpoint 'Break when the container receives the OnQueryCancel event' at executable 'Excel Importer' has been hit
Information: 0x4004300A at Import TPNB Ranking Excel spreadsheets, DTS.Pipeline: Validation phase is beginning.
Warning: 0x802092A7 at Import TPNB Ranking Excel spreadsheets, ProductSalesRank Table [278]: Truncation may occur due to inserting data from data flow column "Rank" with a length of 1000 to database column "SalesRank" with a length of 50.
Error: 0xC0202009 at Excel Importer, Connection manager "Excel Connection Manager": An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft JET Database Engine" Hresult: 0x80004005 Description: "Unrecognized database format 'D:TestingTestRanking.xls'.".
Error: 0xC020801C at Import TPNB Ranking Excel spreadsheets, Excel Source [1]: The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009.
Error: 0xC0047017 at Import TPNB Ranking Excel spreadsheets, DTS.Pipeline: component "Excel Source" (1) failed validation and returned error code 0xC020801C.
Error: 0xC004700C at Import TPNB Ranking Excel spreadsheets, DTS.Pipeline: One or more component failed validation.
Error: 0xC0024107 at Import TPNB Ranking Excel spreadsheets: There were errors during task validation.
SSIS breakpoint 'Break when the container receives the OnQueryCancel event' at executable 'Excel Importer' has been hit
Warning: 0x80019002 at Foreach Loop Container: The Execution method succeeded, but the number of errors raised (5) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS breakpoint 'Break when the container receives the OnQueryCancel event' at executable 'Excel Importer' has been hit
SSIS breakpoint 'Break when the container receives the OnQueryCancel event' at executable 'Excel Importer' has been hit
SSIS breakpoint 'Break when the container receives the OnWarning event' at executable 'Excel Importer' has been hit
Warning: 0x80019002 at Excel Importer: The Execution method succeeded, but the number of errors raised (5) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS breakpoint 'Break when the container receives the OnPostExecute event' at executable 'Excel Importer' has been hit
The program '[2460] Excel Importer.dtsx: DTS' has exited with code 0 (0x0).--------------------------------------------------------------------------------------------
We are having some problems over here after our SQL2005 upgrade.
If we try to use an excel file that previously accessed the SQL2000 database (same DSN name and credentials), and then try to refresh the data after the 2005 upgrade, we get the errormessage "[Microsoft][SQL Native Client][SQL Server]User 'DOMAINusername' does not have permission to run DBCC TRACEON."
We have tried to delete and recreate the DSN file.. Tried to use SA user instead (with offcourse full access)..
An even bigger problem is that we can not even get into msquery to copy the SQL query to make a new file, the same problem appears.
Please note that if we try to make a new external database query it works fine!
Using the below script task I am checking for the excel file existence and upon file existence using the data flow task will load the excel data into sql table. After the data is loaded from one file or however number of excel files present, I want to move those into a archieve folder with datetimestamp to the filenames,please let me know how I can move those files with datetimestamp to the filenames, any help is greatly appreciated. Thanks!!
Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Runtime Imports System.IO Public Class ScriptMain Public Sub Main() If File.Exists(ReadVariable("FileNameVariable").ToString()) Then Dts.TaskResult = Dts.Results.Success Else Dts.TaskResult = Dts.Results.Failure End If End Sub
'From Daniel Read's Blog - http://www.developerdotstar.com/community/node/512/ Private Function ReadVariable(ByVal varName As String) As Object Dim result As Object Try Dim vars As Variables Dts.VariableDispenser.LockForRead(varName) Dts.VariableDispenser.GetVariables(vars) Try result = vars(varName).Value Catch ex As Exception Throw ex Finally vars.Unlock() End Try Catch ex As Exception Throw ex End Try Return result End Function End Class
