Column Metadata From Connection Manager Programmatically
May 9, 2006
Hi all!
My problem I've been struggling with is the following. I have a set of text files (around 70), each with different column numbers and types. I define Flat File Connection Managers for each of them where I can nicely rename, set data types and omit certain columns. I do this once and this will be the basis for the rest of the data process (would be nice programmatically too actually).
I would like to pump each of these text files into SQL Server tables using CREATE TABLE and BULK INSERT (because do it one-by-one is really a pain). The question is:
is there a way to obtain column information (Script Task) from a Connection Manager so I can run CREATE TABLE-s? I just need the names, data type for each nothing fancy...
(I bumped into interfaces like IDTSConnectionManagerFlatFileColumns90, which I cannot handle from the Script Task.)
Any help appreciated!
View 6 Replies
ADVERTISEMENT
Mar 2, 2007
We have a Main package and which is calling 2 more other packages.
The first package contains a connection and we are using a Dataflow task. The data flow task has OleDB Data source which is taking getting columns using a Stored Procedure. And the output we need to write in a Flat File.
The second Package also contains the same(The same Tasks, Database and Stored Procedure Calling)
The difference is in the stored procedure Parameters. Based on the different parameters Stored procedures returns the different Columns and Rows output.
When we are trying to Get the second package output in OleDb Data source it shows all the columns which is the output of the First Package because it stores External Meta Data.
So My understanding is the Connection to the same database keeps the External metadata information with the connection and because of that it is always getting the same output columns in Ole DB Data source task in the second Package also.
How to Get my correct output from the second package in this case?
Or If we dont want to store external Meta data with the Connection then is that possible? If yes then How?
Thanks in advance.
View 4 Replies
View Related
Oct 6, 2006
Hello,
I need to know how I can programmatically set a Flat File Connection Manager's Column Delimiter value.
The Data Warehouse project I am working on, receives daily information feeds that could contain one of two delimiters. Which is just dumb...anyways, as it is now we have two seperate Data Flow Tasks which handle these two delimiters. Currently we have a script taks that "sneak previews" each incoming flat file to determine which delimiter it has, and direct our flow to the correct Data Flow Task to handle it.
I do not want to have to maintain 2 DFTs. How can I get around this problem?
Even if there is a way to do this by passing variables/setting expressions in the Flat File Connection manager, I would do that. Does not necessarily HAVE to be a pure programmatic approach.
ANY help would be greatly appreciated!
Feel free to email me at ccorbin@topcoder.com with any questions, or leave me some good news here :)
THANKS!
Chris Corbin
Software Developer
TopCoder Inc.
View 5 Replies
View Related
Sep 18, 2007
I am following the samples in the online books. I've got an OLEDB connection manager and source component defined. I now want to create an Excel connection manager and destination component (see below). How can I do this?
Dim conMgr As ConnectionManager = package.Connections.Add("OLEDB")
conMgr.ConnectionString = "Provider=SQLOLEDB.1;" "Data Source=FLL-EIS;Initial Catalog=DataLoad;" & _
"Integrated Security=SSPI;"
conMgr.Name = "SSIS Connection Manager for OLE DB"
conMgr.Description = "OLE DB connection to FLL-EIS."
' Create and configure an OLE DB source component.
Dim source As IDTSComponentMetaData90 = dataFlowTask.ComponentMetaDataCollection.New
source.ComponentClassID = "DTSAdapter.OleDbSource"
' Create the design-time instance of the source.
Dim srcDesignTime As CManagedComponentWrapper = source.Instantiate
srcDesignTime.ProvideComponentProperties()
' Assign the connection manager.
source.RuntimeConnectionCollection(0).ConnectionManager = _
DtsConvert.ToConnectionManager90(conMgr)
' Set the custom properties of the source.
srcDesignTime.SetComponentProperty("AccessMode", 2)
srcDesignTime.SetComponentProperty("SqlCommand", Dts.Variables("User::gsQuery").Value.ToString)
' Connect to the data source,
' and then update the metadata for the source.
srcDesignTime.AcquireConnections(Nothing)
srcDesignTime.ReinitializeMetaData()
srcDesignTime.ReleaseConnections()
' Create and configure an OLE DB destination. (This is where I need help, as this code does not work)
Dim destination As IDTSComponentMetaData90 = dataFlowTask.ComponentMetaDataCollection.New
destination.ComponentClassID = "DTSAdapter.ExcelDestination"
' Create the design-time instance of the destination.
Dim destDesignTime As CManagedComponentWrapper = destination.Instantiate
' Where can I find documentation about the various ComponentClassIds and Properties???
destDesignTime.ProvideComponentProperties()
destDesignTime.SetComponentProperty("ExcelFilePath", Dts.Variables("User::gsExcelFile").Value.ToString)
destDesignTime.SetComponentProperty("TableName", Dts.Variables("User::gsSheetName").Value.ToString)
View 6 Replies
View Related
Jul 9, 2006
hi all,
I m new to this forum and hope i ll get warm welcome from all of you.... thank you
I m Praveen kumar Dayanithi... a master student doin my assistantship in a Company. Kindly help me with this...
Here is my issue.... i would like to know how can i pass arguments to connection manager programmatically. In other words how can i acquire connection manger through SCRIPT task(Vb script). I know it is very easy to manually select n specify database name, table name by right clicking in dataflow task but in my company if i do that it will be very cumbersome for production people to change all the database and table names manually when the project is moved to production. Rather if i programmatically acquire connection manager using global variables it will make production people's job very easy. They have to just change the value of the variables. So can any one help me with this issue.
Thanks n regards
View 1 Replies
View Related
Mar 24, 2008
I am using SSIS 2005 on Windows 2003 server. Using Excel Source to dump the data for staging database.
I am getting following error while I execute it through BI studio's execute button.
Please help.
- Sachin
View 2 Replies
View Related
Mar 11, 2008
I have deployed my packages into Sql Server and I am using Configuration File. As my Data Source is Excel, I have changed the connection string during deployment with Server Path. But I am getting the following errors. Actually the File Exist in Path. May I know What is cause of the issue? Do I need to give any permission to execute the package.
SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
component "Excel Source Service Contract Upload" (1) failed validation and returned error code 0xC020801C.
One or more component failed validation.
There were errors during task validation.
DTS_E_OLEDBERROR, Error Code: 0x80004005 Source: "MS JET DB Engine" Description : Path is not valid
View 27 Replies
View Related
Feb 27, 2007
Good afternoon,
I'm having the following error when I run my mining model.
Errors in the metadata manager. The dimension with the ID of 'TMP ~MC-ID' cannot be found in the '' database.
I think the error is in caseprocessor.cpp at the ProcessCase method. There's a code snippet below:
The intention of the following code snippet is to run over the attributes of a case and put them on a vector that holds all the cases.
for (UINT iAttribute = 0; iAttribute < _cAttribute; iAttribute++) {
if(iAttribute == 0)
continue;
if(iAttribute == 1)
continue;
if(iAttribute == 2)
continue;
if(iAttribute == 3) {
vet_termos.push_back(rgdmstatevalue[iAttribute].ch);
continue;
if(iAttribute == 4)
vet_dados.push_back(rgdmstatevalue[iAttribute].r8);
DBL dblValue = 0;
if (_vbCategorical[iAttribute]) {
dblValue = (_bMissingAtRandom)
? ::DblCategoricalMAR(rgdmstatevalue[iAttribute])
: ::DblCategorical(rgdmstatevalue[iAttribute]);
}
if (_viewtype == CASEREADER::viewtypeSparse) {
if (dblValue != dblMissing) {
CHECK_STL_MEM(_vdblValue.push_back(dblValue));
CHECK_STL_MEM(_viAttribute.push_back(iAttribute));
}
}
else {
_vdblValue[iAttribute] = dblValue;
}
The attributes of the table are like following:
Index (iAttribute) == 0 -> This is an auto-incremental ID and is the primary key. Integer.
Index (iAttribute) == 1 -> Another ID. Non-key on this table. It is Integer.
Index (iAttribute) == 2 -> Another ID. Non-key on this table. It is integer.
Index (iAttribute) == 3 -> A term. It's a word. This is non-key and the type is varchar.
Index (iAttribute) == 4 -> Weight of the word. Non-key and type double.
As you can notice I also changed the union DM_STATE_VALUE definition on dmalgo.h to accept the reading of a string (if (iAttribute == 3)) :
typedef /* [public][public][public][public][public][public][public] */ union __MIDL___MIDL_itf_dmalgo_0000_0002 {
DM_STATE_INDEX si;
DOUBLE r8;
char* ch;
} DM_STATE_VALUE;
So... Does anyone can help me with the error I discribed on the metadata manager ? I would be very pleased with any help.
Thanks a lot for your attention.
-Renan Souza
View 12 Replies
View Related
Oct 11, 2007
I have excel files where the column headers I care about are on line 5, and the actual data doesn't begin until line 6. Other than deleting the first 4 lines, which is impractical, how can I get the Excel Connection Manager to import the data correctly? I was able to do this under DTS, so I have to imagine it's possible.
Thanks!
View 1 Replies
View Related
Dec 27, 2007
Hi,
I am working on SQL Server 2005 (x64) with Windows Server 2003 (x64) operating system. I am having a major issue in SSIS. Here is the detailed explanation of the issue :
I have an EXCEL file in 2003 / 2007 version. It contains some data. I want to import the data using SSIS into SQL Server 2005 (x64) database table. I have taken "EXCEL FILE SOURCE" and "SQL Server DESTINATION". It was failed on importing data. Surprisingly it works fine in SQL Server 2005 (x32). Can you please explain why it is NOT woking on (x64) ?
Here is the error code i am getting:
[Excel Source [1]] Error: The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0202009.
Appreciate your time and patience !!
Thanks
View 3 Replies
View Related
Oct 22, 2007
Hi,
I'm having an issue running the clustering algorithm in the data mining view of Visual Studio. The databases connect properly and the data subsequently loads. However, upon clicking on the "Mining Model Viewer" tab, I receive the following error message:
Errors in the metadata manager. The D Msample ~MC cube has no measure groups. Errors in the metadata manager. An error occurred when loading the D Msample ~MC cube, from the file, '\?C:Program FilesMicrosoft SQL ServerMSSQL.2OLAPDataewDM_sample.0.dbD Msample ~MC.2.cub.xml'.
We are not using a data cube, so I am assuming that this file is being called through the clustering algorithm. Furthermore, I have run the same process on different systems successfully. The only difference I can detect is that this error resulted on a 64bit system.
Any help would be greatly appreciated.
Thanks
View 8 Replies
View Related
Jul 15, 2015
I've developed a package that is working well at development machine from VS 2013 for a sample flat file. Also, over development machine, I've deployed it to SSISDB catalogue and even from there also it is running well for the same file.When the same package is deployed to production server's SSISDB catalogue database, it throws following error while processing the same sample flat file, “Unable to retrieve column information from the flat file connection manager”
View 5 Replies
View Related
Apr 3, 2008
Hi,
Is there a way of uploading column definitions from the Flat File Connection Manager into a SQL Server table definition. Since I have over a dozen data sources to process each with around 200 columns and of course like many BI techies I have little immediate influence over the structure of these flat files. I just know that these data sources are business critical.
Judging by looking at similar threads I can't be the only one who would greatly benefit from being able to upload column definitions from the Flat File Connection Manager into a SQL Server table definition as opposed to doing this manually.
Thanks in advance,
Kieran.
View 1 Replies
View Related
Mar 17, 2008
Greetings,
I have an SSIS package that I need to modify. It was developed in Visual Studio 2005 and runs faithfully in production on SQL Server 2005. Suddenly, I can't open the package on my own workstation. I see a big red "X" image and a message stating "Microsoft Visual Studio is unable to load this document." The real reason the document can't be loaded appears to be:
Error loading DataWarehouseLoader.dtsx: The connection type "OLEDB" specified for connection manager "Warehouse Logging OLE DB" is not recognized as a valid connection manager type. This error is returned when an attempt is made to create a connection manager for an unknown connection type. Check the spelling in the connection type name.
Does anyone know what causes this and what I can do? The package has been in production for 9 months and I've never seen this problem before.
Thanks,
BCB
View 10 Replies
View Related
Mar 25, 2008
I've tried looking in sys.syscolums and sys.syscomments, but I can't seem to find where the Description information is retain for a Field in the system tables -- any hints?
Thanks, Rob.
View 2 Replies
View Related
Jun 21, 2006
SQL table and column metadata
Is there a way to add or update the column or table (using the extended properties) description metadata via T-SQL (from within a stored procedure) or via a program (such as VB.NET using ADO)?
These metadata properties are available via the SSMS interface:
Columns via the Column Properties/Table Designer/Description
Tables via the Table Properties/Extended Properties/[Extended Property Name]
Thanks in advance,
Mark
View 6 Replies
View Related
Feb 6, 2008
I have a package that uses a for loop to iterate through an unknown amount of excel files and pull their data into a table. However, there will be cases when the file is corrupted or has some sort of problem so that either the transformation will fail or the excel data source will fail with an oledb connection error.
Could anyone suggest a clean way to trap these errors? Specifically, the "Cannot Aquire Connection from Connection Manager", which is the excel connection.
Thanks,
John T
View 3 Replies
View Related
Oct 25, 2007
I keep getting the following error in SSIS. Also, I don't get the error on every server the package is run on, but less than 5 (the package is run on over 100).
"The external metadata column collection is out of synchronization with the data source columns. The column "Timestamp" needs to be added to the external metadata column collection"
Please tell me where I need to remove Timestamp from. Thanks
-Kyle
View 1 Replies
View Related
Oct 24, 2007
I keep getting the following error in SSIS. Also, I don't get the error on every server the package is run on, but less than 5 (the package is run on over 100).
"The external metadata column collection is out of synchronization with the data source columns. The column "Timestamp" needs to be added to the external metadata column collection"
Please tell me where I need to remove Timestamp from. Thanks
-Kyle
View 1 Replies
View Related
Sep 13, 2007
I'm getting really frustrated with using the web.config to store all database information. It seems like you really don't have any real flexibilty when moving an application from server to server (between dev, test and prod). Seems like you have to create separate config files when you move it every time. Seems like you always run into a risk of over writing one of them if you are not careful.
Is there a good way to design the connection object to be able to tell which server it is on and determine if it's a test, dev or prod type of connection. I have used a basic three connection strings that it will compare the computer name with the key. But again, the connection string is hard coded in the web config. At least this way I am able to move all code together without the worry of overwritting the config file. I just have to beleive there is a better way to build something like this. I've heard of setting up enviroment variables on the server, but not sure how safe that is.
I'm looking everywhere on the web, but get really lame examples. If anyone knows a good method or a sample I can check out, that would be great.
Thanks!
View 1 Replies
View Related
Apr 10, 2015
I am having one store procedure which use to load data from flat file to staging table dynamically.everything is working fine. staging_temp table have single column.all the data stored in that single column below is the sample row.
AB¯ALBERTA ¯93¯AI
AI¯ALBERTA INDIRECT ¯94¯AI
AL¯ALABAMA ¯30¯
after the staging_temp data gets inserted into main table.my probelm is to handle such a file where number of columns are more than the actual table.if you see the sample rows there are 4 column separated by "¯".but actual I am having only 3 columns in my main table.so how can I get only first 3 column from the satging_temp table.output should be like below.
AB¯ALBERTA ¯93
AI¯ALBERTA INDIRECT ¯94
AL¯ALABAMA ¯30
View 45 Replies
View Related
Mar 12, 2008
Hi,
I have an Excel file source. I keep getting this error when running the package:
"The external metadata column collection is out of synchronization with the data source columns. The column "x" needs to be updated in the external metadata column collection."
When I get this error with regular flat files, it's because I've changed the data type of a given column in the flat file connection manager. And I resolve it simply by double-clicking on the flat file source task, and viola - it corrects it for me.
How do I correct this with an Excel file? Help.
Thanks
View 10 Replies
View Related
Aug 2, 2007
Hi!
I have created a view and one of the columns in the view has an alias assigned to it.
I'm able to read the metadata from INFORMATION_SCHEMA.VIEW_COLUMN_USAGE and also lookup
from which table each column in the view orginated from except for the column that has an alias assigned to it.
Is there any other way to lookup a column that has an alias assigned to it?
Thanks alot!
Adam
View 2 Replies
View Related
Mar 18, 2008
Have many views based on legacy tables that have different table and column names. Want to create a table that shows view table / column and underlying table column, e.g.
CREATE VIEW [dbo].[Branch]
AS
SELECT
divbra_id BranchID,
cmpny_id CompanyID,
divbra_cde BranchCode,
divbra_nme BranchName
FROM MyDatabase.dbo.divbranc
GO
is an existing view. I want to pull out the following metadata:
divbranc divbra_id Branch BranchID
divbranc cmpny_id Branch CompanyID
divbranc divbra_cde Branch BranchCode
divbranc divbra_nme Branch BranchName
Is there anyway to get this from SQL metadata without actually parsing the view SELECT statement in code?
View 7 Replies
View Related
Mar 13, 2007
I am programmatically creating a package in c#. I need this package to contain a flat file connection manager. I have been successfully able to create the connection manager with the following code:
Package p = new Package(); //New package
ConnectionManager cm = p.Connections.Add("FLATFILE"); //Add a flat file connection manager
cm.ConnectionString = "C:DevmyTestFile.txt"; //Set up the connection string
At this point, I want to configure the flat file specific properties of the connection manager (RowDelimiter, HeaderRowDelimiter, etc.). The problem is that the properties collection of the ConnectionManager object is read only. I think I could access the inner object of the connection manager and set the properties in the following way:
ConnectionManagerFlatFileClass ffClass = (ConnectionManagerFlatFileClass) cm.InnerObject; //Get the inner object
ffClass .RowDelimiter = "{LF}"; //Set the properties here
Even if this would work, I do not like the solution b/c the BOL states that the ConnectionManagerFlatFileClass "supports the SQL Server 2005 infrastructure and is not intended to be used directly from your code". Does anyone know the "right" way to set these properties?
Many thanks in advance!
David
View 1 Replies
View Related
Mar 28, 2006
I'm having some issues getting OLE DB Data Sources to work w stored procs in SSIS. Here's the situation.
I have an OLE DB Data Source set up to call a stored proc w no parameters. The stored procedure loops through a set of databases and inserts data from each database into a results table. I'm attempting to return the results table to SSIS, but the Available External Columns are not populating. However, previewing the query in SSIS does show results. The insert in to the results table is done by a call to sp_executesql.
I've tried setting the results table up as a temp table, table variable, and static table. I have NOCOUNT set ON and am only returning one recordset. I've seen the other threads in here about similar problems, but none of their solutions seem to work for me.
Any help would be much appreciated....
View 2 Replies
View Related
Apr 7, 2008
When running the project in debug mode or non-debug mode, I get the following error from MS Visual Studio:
TITLE: Package Validation Error
------------------------------
Package Validation Error
------------------------------
ADDITIONAL INFORMATION:
Error at Package [Connection manager "SYBASE_CONNECTION"]: The connection manager failed to defect from the transaction.
(Microsoft.DataTransformationServices.VsIntegration)
------------------------------
BUTTONS:
OK
------------------------------
Any ideas???
View 8 Replies
View Related
Apr 15, 2015
I use SQL Server 2012 and visual studio 2010.I created SSIS Project with task "Execute Package". Control flow view as: Package1 (execute package) -> Package2 (data flow).Data flow in Package2 view as: ADO.NET source -> ADO.NET destination.
When I started Package2, it's work. I havn't errors.But when I started Package 1 I have error "Unable to get managed connection from the Connection Manager runtime". In execution log I see that ADO.NET source produced this error on verification stage. Package failed on verification stage, not on execution stage.Why when I started Package 2 it work, but when I started Package1 (and Package1 started Package2) it failed?
View 14 Replies
View Related
Feb 14, 2008
Hi All,
I am getting the following error if I am using the package "Transaction Option=Required" while running through Sql Job:
The AcquireConnection method call to the connection manager "<connection name>" failed with error code 0xC0202009.
while I running the SSIS package on BI environment, I am getting the following error:
[Connection manager "<connection name>"] Error: The SSIS Runtime has failed to enlist the OLE DB connection in a distributed transaction with error 0x8004D00A "Unable to enlist in the transaction.".
I know the alternative solution is to make the "Transaction Option=Supported", but in my case I have run the whole flow in a single transaction. I came to know that this has been fixed in the service pack1(ref. to http://support.microsoft.com/kb/914375). FYI.. some time it was running successful.
I have taken all the necessary step to run the SSIS package in a distributed transaction(like the steps for MSDTC) and also created the package flow in a sequence.
I was going through the link - http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=160340&SiteID=1 but all those didn't solve my problem.
If anyone can help me it will be great. or it is a bug in SSIS?
Thanks.
Jena
View 5 Replies
View Related
Jun 22, 2006
I have a Rounding error: Between flat file connection manager Source & OLE DB Connection Destination (SQL Server 2005) in my Dataflow.
File looks like this lets call column names Col A,B,C,D
70410000 RD1 1223631.92 196042.42
70329000 ICD 11025.84 3353.88
71167300 COL 104270.59 24676.96
flat file connection manager settings: first row Column names then Advanced tab Col A float , Col B float , Col C string ,Col D float ,
OLE DB Connection Destination (SQL Server 2005)
CREATE TABLE [dbo].[PT_CUST_ABR](
[PARTY_NO] [float] NULL,
[PARTY_NAME] [varchar](75) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
[TELECOMABR] [float] NULL,
[GENIABR] [float] NULL,
Problem: ColA (Source) Rounding error to PARTY_NO (Destination)
I have a field of text of in a flat file that the flat file connection manager Source picks up correctly 70000893?
However when it gets the OLE DB Connection Destination the data has changed to 70000896. Thats before its even Written to the database.
The only clue that something is wrong in the middle is the great Data viewer shows the number as 7.000009E+07
Other clues looking at the data it appears there is a rounding error on only the number that dont end in 00
ColA (Source) PARTY_NO (Destination)
71167300 71167296
70329000 70329000
70410000 70410000
Any ideas people?
Thanks in advance
Dave
View 3 Replies
View Related
Dec 17, 2007
hi all,
i got the error, when i run my package after deploy into the server machine;
i can able to run that package in my local machine, if i deploy it to the server or some remote machine, its not running, and rises error messages, that says cant accquire connection from the connection manager;
the problem might be with the login name and password; i could not see any password in any of my config file, or connection manager while running the package.
how to resolve this problem?? please help me!
View 6 Replies
View Related
Dec 27, 2006
Hi,
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
View 5 Replies
View Related
Feb 19, 2015
I have a child package where the ConnectionString property of a Connection Manager is set by a Parent Package Variable Configuration. I set up a script task that brings up a message box with the value of the ConnectionString property right before the dataflow task.
MessageBox.Show(Dts.Connections["CPU_*"].ConnectionString.ToString());When I run the parent package, the message box shows that the connection string is changing with every iteration, but in the dataflow it always draws the data from the same source.
The connection manager is an ADO.Net type, RetainSameConnection is set to False, and I've been researching this for days.
(Update 2/23/2015): To make this stranger, when I look at the diagnostic logs, they tell me that when the new connections are being opened they are using the new connection strings.
View 2 Replies
View Related
