Changes To Shared Flat File Connection Manager Adversely Affect Package
Jul 14, 2007
I am having frustration with issues I'm seeing within Development Studio. Here's the problem:
A fixed-width flat file is to be imported into SQL 2005. I am running from within BIDS on my workstation and send the data remotely to the server. It has over 200 fields, and I am attempting to import them into dimension tables, and one fact table. I had to manually create the columns on the flat file connection manager for this process, since the data file is delimited.
The SSIS solution has one project, and several independent data flow tasks within it. After I set up the fcm, I mapped fields using a flat file source, hooked up a derived column and sort transform. The idea was to use a 3rd party tool, TableDifference, to implement a Type 2 dimension. I was able to successfully build three data flow tasks that ran correctly. Later, as was building new data flow tasks, our project people decided to change the data types of several of the fields from Datetime to character, and also rename another 40+ fields. I dutifully kept my fcm synchronized, since column mappings are automatic when the names match; data type changes were also unavoidable on some of the columns.
The problem was noticed after I changed some of the columns in the fcm. My flat file source adapters on all the tasks needed to resolve changed metadata so I had to 'edit' and save them again. The derived column, sort, and downstream transforms also had to be updated. I saw strange behavior on previously tested data flow tasks. Some of them gave me errors even connecting to their sources. Other times I saw erroneous paths being taken which led to data being changed incorrectly on the dimension tables (new records being added even if there was an existing match, records being marked as expired incorrectly, etc.).
Are there known issues with editing a shared file connection manager after it has already been referred to within data flow tasks? It appears that the only way to avoid the problems I've seen is to either:
1. Create separate fcm's for each data flow task. Use 'filler' fields for the ranges of all the columns in between each referenced column. This way if you have to change anything with it, the only rework involved is limited to the data flow task using it.
2. Create one fcm for the entire package and NEVER change anything with it. If a column name is wrong, use a derived column to rename it in the data flow. If a data type changes, try to use a data conversion transform, but be sure to rename its output column so that it is not called 'data conversion.<field name>'. I think that's done in the Advanced Editor for the sort task.
My experience using BIDS has been hot and cold. It's nice to be able to quickly build the tasks you need, but it's as if the metadata (pipeline) is much too sensitive to change and requires a lot of touching up, either upstream at the source adapter level, or across data flow tasks when you make changes to the FCM. Maybe the problem is just limited to flat files...
Are there any tips or lessons learned from anyone on this? I've got another developer willing to create the FCM for other data feeds so I can copy/paste them into other projects. It would just be nice to know there was a bug fix or a good method to avoid all this 'rework' when you have to make changes to the FCM after the fact.
I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
Problem: ColA (Source) Rounding error to PARTY_NO (Destination) I have a field of text of in a flat file that the flat file connection manager Source picks up correctly €ś70000893€? However when it gets the OLE DB Connection Destination the data has changed to 70000896. That€™s before its even Written to the database. The only clue that something is wrong in the middle is the great Data viewer shows the number as 7.000009E+07 Other clues looking at the data it appears there is a rounding error on only the number that dont end in 00 ColA (Source) PARTY_NO (Destination) 71167300 71167296 70329000 70329000 70410000 70410000 Any ideas people? Thanks in advance Dave
I would like to set up the flat file connection manager that would take any file name that starts with "test" and then it could be anything after that. Something like test_*.txt.
Hi All, I am using flat file connection manager to create a text file. I could not find any way to set the file location (folder) dynamically such as using variable, expression etc, Is there any alternative I can use to achieve this? I really don't want to hard-coded the file location as it may differ in production environment. Thanks in advance.
As I have a file whose delimiters can be different. I have need to change the delimiters and filename programatically.
I have come across a way to change the filename using the inbuilt expressions setter. Howver the columns are not listed, possibly because they are on the advanced tab and theoretically each column delimiter could be different.
Will I have to do this through a script or is there an easier method?
Hello Experts, I am createing one task (user control) in SSIS. I have property grid in my GUI and 2 buttons (OK & Cancle). PropertyGrid has Properties like SourceConnection, OutputConnection etc....right now I am able to populate Connections in list box next to Source and Output Property.
Now my question to you guys is depending on Source Connection it should read that text file associated with connection manager. After validation it should pick header (first line of text file bases on record type) and write it into new file when task is executed. I have following code for your reference. Please let me know I am going in right direction or not.. What should go here ? ->Under Class A
I have a package A which is copied from another existing package B as most of the data structure and ETL mappings are same.
What I need to change in Package A is to change the file in Flat File Connection Manager. I can change it in the conneciton manager editor. However, it is automatically changed back to the previous one everytime when I try to Save All or run the package.
I also tried to copy/paste this Flat File Connection Manager in the same package but samething happen. The package file is not set 'Read Only" so anything else can Saved well except for the File name in connection manager.
Is this a bug? It would be very appreciated if anyone can give me any idea about this.
Hi, I want to read only the first row in flatfile. I do not see this option on "Flatfile connection manager editor" setting wizard. Any work around for this?
I've tried all of the different column delimiters, but apparently this file does not use any of the built-in delimiters, such as tab. I think there are just blank spaces between the columns.
I've requested that the file be comma-delimited instead, but in the event that this is not possible, how should I handle this situation?
I need to know how I can programmatically set a Flat File Connection Manager's Column Delimiter value.
The Data Warehouse project I am working on, receives daily information feeds that could contain one of two delimiters. Which is just dumb...anyways, as it is now we have two seperate Data Flow Tasks which handle these two delimiters. Currently we have a script taks that "sneak previews" each incoming flat file to determine which delimiter it has, and direct our flow to the correct Data Flow Task to handle it.
I do not want to have to maintain 2 DFTs. How can I get around this problem?
Even if there is a way to do this by passing variables/setting expressions in the Flat File Connection manager, I would do that. Does not necessarily HAVE to be a pure programmatic approach.
ANY help would be greatly appreciated!
Feel free to email me at ccorbin@topcoder.com with any questions, or leave me some good news here :)
I have a Dataflow with three componnets: 1. script component 2. script component 3. flat file destination
After I connected second script to the flat file destination and tryed to opend Flat File Destination Editor, a dialog with this message showed:
TITLE: Editing Component ------------------------------ The component is not in a valid state. The validation errors are: Error at Data Flow Task [Flat File Destination [76]]: The component locale ID has not been set. Flat file adapters need to have the locale ID on the flat file connection manager set.
Do you want the component to fix these errors automatically? ------------------------------ BUTTONS: &Yes &No Cancel ------------------------------
No matter which button I pressed, I was not able to work with flat file connection manager.
When I selected a file path in connection manager editor, I got an error "A valid file name must be selected". I tryed different files in differnet folders, but got the same error.
So I tryed to open some older working packages. On flat file destination componnent , when I clicked on tab "Mapping", recieved error " [Flat File Destination [51]]: Unable to access the acquired connections.",
Flat File connection manager showed "A valid file name must be selected". But althought these old packages seem to be broken in design, they a running OK.
I thought, that the problem could be in wrong expression for connectionString, but it still remains after removing it.
Please, could you give me advice, where could be a problem or what should I check?
I have a flat file that I'm trying to bring in. I've mapped all the columns, and the first record comes out perfect. The problem is that its not recognizing end of line. When I preview it line one looks great, but line two has two square boxes before the actual next record begins and those boxes move over into the next column on the next record and so forth. The Format is Fixed Width, I have no text qualifier, header row delimeter is set to default {CR}-{LF}, and we're starting on row 2 as row 1 is garbage. Any ideas?
The Import wizard was used to create a package. The package simply reads a flat file and writes the columns to a SQL table. The package was added to and opened in an SSIS project. I would like to make several iterations of changing the flat file columns and their names. Most commonly the wizard would be run with the column names defaulted to [Column 0],[Column 1] ... Later the name will changed when the proper name is known.
I have been able to change the name of a column, but don't know how to synchronize the SQL table creation and the column mappings. Help please.
The standard flat file connection component does not handle flat files with different column counts on different rows. Since that's the type of file I have to read, I like to create a modified version of the flat file connection manager that can do this. I have found some info in how to create a connection manager. But I can't find anything that tels me how to build one that would be recognized by the flat file source dataflow component.
So how do you build an connection manager that can used by the standard flat file source component? Anyone ever did this or knows were to get the docs?
I know my problem is also solvable by reading the row into a single column and then using a script component to split them but that is not the solution I want.
I am having some issues with dts packages i have migrated using the package migration wizard.
One of the connection managers points to an execl flat file source.
I have not altered anything after the package migration, the conenction manager is using Native OLE DBMicrosoft Jet 4.0 OLE DB Provider.
When i test the connection i get the following error :
"Test connection failed becuase of an error in initializing provider. Cannot start your application. The workgroup information file is missing or opened exclusively by anotehr user."
Ok i am 99% sure this file is not being accessed by another user, the rest of the error is greek to me. Looking on the net for this error it has to do with access database security and having to sertup htm pages for IIS, i cannot findanything on this error relating to ssis connection manager.
I´m using the Flat File Connection Manager to access a flat file, tab delimited. The flat file has 200 columns, and when I'm editing the columns, I only preview columns from 0 to 97.
Does the flat file connection have a column number limit? How could I increase it?
Another question regarding the frustrating SSIS stuff. Basically when I used to use a DTS task to import a flat file (comma delimited) into the database it worked fine no problems, however, for some reason the same approach in SSIS means that any string values in the file are automatically delimited with double quotes, even though I am specifying <none> as the delimitation. I have also changed the fields so that the the TextQualified is set to false.
This causes a problem because all of the information stored in the database (done by a stored procedure) contains these stupid quotes.
Is there something I am missing or haven't done correctly to get rid of the quotes
Can we disable a flat file connection manager in a ssis package just as how we can disable a OLE db task? When I try to disable the flat file by clicking on work offline its gets disabled which is what i expected , but when i close the package and reopen it again its again enabled , is this the way that a flat file connection manager works.
I have been working with flat file connection managers for a while now, and I have found it very useful to set properties on all types of SSIS objects using expressions that are set to SSIS package variables.
We have recently moved towards using a centralized application configuration infrastructure, so we are now loading up all application configuration information from our own system -- it's actually based off of the 2.0 ConfigurationManager.
I have developed a custom control flow task that will retrieve all of the SSIS variable information from our own application configuration system and set all of the SSIS variables as the first step in a package.
When I have been using this to set the SSIS Variables which are used in expressions for the HeaderRowsToSkip and DataRowsToSkip (there may be others, but I have not tested them yet) properties of flat file connection managers, these values are not being used -- instead it is using the values for the variables that are defined in the package itself (i.e. the default values that I set up in BIDS when developing the package). The ConnectionString property on the flat file connection manager does not exhibit this behavior.
Can anyone provide any insight on this? Is this a bug in SSIS, or is there something that I am missing here?
I'm doing a simple ETL that reads a database table and dumps the content to a text file. The text file will be named Employee.txt. This file name will remain the same across my environments, but I may want to vary the directory location to where I want this file dumped.
So, I defined an environment variable called "DataTargetDir" in all my environments. Now, I want to utilize this variable in the "File name:" box within the Flat File Connection Manager Editor. How do I do this? I'm thinking I can write something like "%DataTargetDir%Employee.txt" in the "File name:" box, but it's not working.
Hi, I got this error in a Flat File connection manager, it was saying that the last column caused the disk I/O Buffer overflow, whereas the last column contains 188 characters at max.
I'm very new to SSIS and seemed to have stumbled into an expected problem. I'm working on a package that was converted from a DTS package and I have a problem with the Flat File Connection manager. I want to change the file name in my connection, but I do not seem to be able to save the change. After clicking "OK" to save the name change I can then save the package, close BIDS, open everything up again and for some reason the "old" filename is still present in the connection. Note that the "new" file has exactly the same layout as the "old" file, just a different name. And I can oreview the data without any difficulty.
URL....I'm having a problem with Integration Services Tutorials SSIS Tutorial: Creating a Simple ETL Package Lesson 2: Adding Looping (step 3).I can't add a variable to Sample Flat File Source Data.Right-clicking on the connection manager just shows two items of the property: File Name and File Path and nothing else.I am using MS Visual Studio Ultimate 2012. Does it have limitations? I've see screenshots where properties of flat file managers have many more options.
I am new with SSIS and I have a problem that I don€™t know solving it. I have a simple package with a delimited flat file source which is loaded into a table in a SQL Server database. Below is the import format: - Row delimiter: carriage return and line feed {CR/LF} - Column delimiter: Comma {,} - Text qualifier: €ś
In the source file, the data looks like this: €śstring1 €?, 34, €ś€?, , ,€? string2 €śtext1€? string2€? , €ś €ś,
This package with DTS works, but now with SSIS does not, when I see the €śprevious rows€? in the Flat File Connection Manager Editor, the last column has incorrect information and when I create a new file without spaces and without twice €ś€? in the same string, it works. I don€™t know what I am missing with SSIS.
I have a requirement wherein I have to setup the flat file connection manager to accept columns on fly. Meaning I want to retrieve list of columns/column count from the database when the package is run and set the connection manager with those many columns.
How to create a text file in UNIX format using Flat File connection manager. By default when we create a connection manager for flat files it is taking "CRLF" as the delimeter.
I beleive the format of the file will be decided based on the control line feed character at the end of each row. There are different control line feeds for different operating systems.
CR - Mac OS (Carraige return) LF - UNIX (Line Feed) CRLF - Windows. (Carriage return Line Feed)
