I was intended to write a program that will create a SSIS package which will import data from a CSV file to the SQL server 2005. But I did not find any good example for this into the internet. I found some example which exports data from SQL server 2005 to CSV files. And following those examples I have tried to write my own. But I am facing some problem with that. What I am doing here is creating two connection manager objects, one for Flat file and another for OLEDB. And create a data flow task that has two data flow component, one for reading source and another for writing to destination. While debugging I can see that after invoking the ReinitializedMetaData() for the flat file source data flow component, there is not output column found. Why it is not fetching the output columns from the CSV file? And after that when it invokes the ReinitializedMetaData() for the destination data flow component it simply throws exception.
Can any body help me to get around this problem? Even can anyone give me any link where I can find some useful article to accomplish this goal?
I am giving my code here too.
I will appreciate any kind of suggestion on this.
Code snippet:
public void CreatePackage()
{
string executeSqlTask = typeof(ExecuteSQLTask).AssemblyQualifiedName;
Package pkg = new Package();
pkg.PackageType = DTSPackageType.DTSDesigner90;
ConnectionManager oledbConnectionManager = CreateOLEDBConnection(pkg);
ConnectionManager flatfileConnectionManager =
CreateFileConnection(pkg);
// creating the SQL Task for table creation
Executable sqlTaskExecutable = pkg.Executables.Add(executeSqlTask);
ExecuteSQLTask execSqlTask = (sqlTaskExecutable as Microsoft.SqlServer.Dts.Runtime.TaskHost).InnerObject as ExecuteSQLTask;
execSqlTask.Connection = oledbConnectionManager.Name;
execSqlTask.SqlStatementSource =
"CREATE TABLE [MYDATABASE].[dbo].[MYTABLE]
([NAME] NVARCHAR(50),[AGE] NVARCHAR(50),[GENDER] NVARCHAR(50))
GO";
// creating the Data flow task
Executable dataFlowExecutable = pkg.Executables.Add("DTS.Pipeline.1");
TaskHost pipeLineTaskHost = (TaskHost)dataFlowExecutable;
MainPipe dataFlowTask = (MainPipe)pipeLineTaskHost.InnerObject;
// Put a precedence constraint between the tasks.
PrecedenceConstraint pcTasks = pkg.PrecedenceConstraints.Add(sqlTaskExecutable, dataFlowExecutable);
pcTasks.Value = DTSExecResult.Success;
pcTasks.EvalOp = DTSPrecedenceEvalOp.Constraint;
// Now adding the data flow components
IDTSComponentMetaData90 sourceDataFlowComponent = dataFlowTask.ComponentMetaDataCollection.New();
sourceDataFlowComponent.Name = "Source Data from Flat file";
// Here is the component class id for flat file source data
sourceDataFlowComponent.ComponentClassID = "{90C7770B-DE7C-435E-880E-E718C92C0573}";
CManagedComponentWrapper managedInstance = sourceDataFlowComponent.Instantiate();
managedInstance.ProvideComponentProperties();
sourceDataFlowComponent.
RuntimeConnectionCollection[0].ConnectionManagerID = flatfileConnectionManager.ID;
sourceDataFlowComponent.
RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.ToConnectionManager90(flatfileConnectionManager);
managedInstance.AcquireConnections(null);
managedInstance.ReinitializeMetaData();
managedInstance.ReleaseConnections();
// Get the destination's default input and virtual input.
IDTSOutput90 output = sourceDataFlowComponent.OutputCollection[0];
// Here I dont find any columns at all..why??
// Now adding the data flow components
IDTSComponentMetaData90 destinationDataFlowComponent = dataFlowTask.ComponentMetaDataCollection.New();
destinationDataFlowComponent.Name =
"Destination Oledb compoenent";
// Here is the component class id for Oledvb data
destinationDataFlowComponent.ComponentClassID = "{E2568105-9550-4F71-A638-B7FE42E66922}";
CManagedComponentWrapper managedOleInstance = destinationDataFlowComponent.Instantiate();
managedOleInstance.ProvideComponentProperties();
destinationDataFlowComponent.
RuntimeConnectionCollection[0].ConnectionManagerID = oledbConnectionManager.ID;
destinationDataFlowComponent.
RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.ToConnectionManager90(oledbConnectionManager);
// Set the custom properties.
managedOleInstance.SetComponentProperty("AccessMode", 2);
managedOleInstance.SetComponentProperty("OpenRowset", "[MYDATABASE].[dbo].[MYTABLE]");
managedOleInstance.AcquireConnections(null);
managedOleInstance.ReinitializeMetaData(); // Throws exception
managedOleInstance.ReleaseConnections();
// Create the path.
IDTSPath90 path = dataFlowTask.PathCollection.New(); path.AttachPathAndPropagateNotifications(sourceDataFlowComponent.OutputCollection[0],
destinationDataFlowComponent.InputCollection[0]);
// Get the destination's default input and virtual input.
IDTSInput90 input = destinationDataFlowComponent.InputCollection[0];
IDTSVirtualInput90 vInput = input.GetVirtualInput();
// Iterate through the virtual input column collection.
foreach (IDTSVirtualInputColumn90 vColumn in vInput.VirtualInputColumnCollection)
{
managedOleInstance.SetUsageType(
input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY);
}
DTSExecResult res = pkg.Execute();
}
public ConnectionManager CreateOLEDBConnection(Package p)
{
ConnectionManager ConMgr;
ConMgr = p.Connections.Add("OLEDB");
ConMgr.ConnectionString =
"Data Source=VSTS;Initial Catalog=MYDATABASE;Provider=SQLNCLI;Integrated Security=SSPI;Auto Translate=false;";
ConMgr.Name = "SSIS Connection Manager for Oledb";
ConMgr.Description = "OLE DB connection to the Test database.";
return ConMgr;
}
public ConnectionManager CreateFileConnection(Package p)
{
ConnectionManager connMgr;
connMgr = p.Connections.Add("FLATFILE");
connMgr.ConnectionString = @"D:MyCSVFile.csv";
connMgr.Name = "SSIS Connection Manager for Files";
connMgr.Description = "Flat File connection";
connMgr.Properties["Format"].SetValue(connMgr, "Delimited");
connMgr.Properties["HeaderRowDelimiter"].SetValue(connMgr, Environment.NewLine);
return connMgr;
}
Below is C# code used to create a FuzzyLookup SSIS package programmatically. It does 95% of what I need it to. The only thing missing that I cannot figure out is how to take a Fuzzy Lookup Input column (OLE DB Output Column) and make it "pass through" the fuzzy lookup component to the OLE DB Destination. In the example below, that means I need the QuarantinedEmployeeId to make it into the destination.
Look in the "Test Dependencies" region below to get instructions and scripts used to set assembly references, create the sample tables used for this example, and insert test data.
Can anyone help me get past this last hurdle? You will see at the end of my Fuzzy Lookup region a bunch of commented out code that I've used to try to accomplish this last problem.
Code Block using Microsoft.SqlServer.Dts.Runtime; using Microsoft.SqlServer.Dts.Pipeline.Wrapper; namespace CreateSsisPackage { public class TestFuzzyLookup { public static void Test() { #region Test Dependencies // Assembly references: // Microsoft.SqlServer.DTSPipelineWrap // Microsoft.SQLServer.DTSRuntimeWrap // Microsoft.SQLServer.ManagedDTS // First create a database called TestFuzzyLookup // Next, create tables: //SET ANSI_NULLS ON //GO //SET QUOTED_IDENTIFIER ON //GO //CREATE TABLE [dbo].[EmployeeMatch]( // [RecordId] [int] IDENTITY(1,1) NOT NULL, // [EmployeeId] [int] NOT NULL, // [QuarantinedEmployeeId] [int] NOT NULL, // [_Similarity] [real] NOT NULL, // [_Confidence] [real] NOT NULL, // CONSTRAINT [PK_EmployeeMatch] PRIMARY KEY CLUSTERED //( // [RecordId] ASC //)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY] //) ON [PRIMARY] //GO //SET ANSI_NULLS ON //GO //SET QUOTED_IDENTIFIER ON //GO //CREATE TABLE [dbo].[QuarantinedEmployee]( // [QuarantinedEmployeeId] [int] IDENTITY(1,1) NOT NULL, // [QuarantinedEmployeeName] [varchar](50) NOT NULL, // CONSTRAINT [PK_QuarantinedEmployee] PRIMARY KEY CLUSTERED //( // [QuarantinedEmployeeId] ASC //)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY] //) ON [PRIMARY] //GO //SET ANSI_NULLS ON //GO //SET QUOTED_IDENTIFIER ON //GO //CREATE TABLE [dbo].[Employee]( // [EmployeeId] [int] IDENTITY(1,1) NOT NULL, // [EmployeeName] [varchar](50) NOT NULL, // CONSTRAINT [PK_Employee] PRIMARY KEY CLUSTERED //( // [EmployeeId] ASC //)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY] //) ON [PRIMARY] // Next, insert test data //insert into employee values ('John Doe') //insert into employee values ('Jane Smith') //insert into employee values ('Ryan Johnson') //insert into quarantinedemployee values ('John Dole') #endregion Test Dependencies #region Create Package // Create a new package Package package = new Package(); package.Name = "FuzzyLookupTest"; // Add a Data Flow task TaskHost taskHost = package.Executables.Add("DTS.Pipeline") as TaskHost; taskHost.Name = "Fuzzy Lookup"; IDTSPipeline90 pipeline = taskHost.InnerObject as MainPipe; // Get the pipeline's component metadata collection IDTSComponentMetaDataCollection90 componentMetadataCollection = pipeline.ComponentMetaDataCollection; #endregion Create Package #region Source // Add a new component metadata object to the data flow IDTSComponentMetaData90 oledbSourceMetadata = componentMetadataCollection.New(); // Associate the component metadata object with the OLE DB Source Adapter oledbSourceMetadata.ComponentClassID = "DTSAdapter.OLEDBSource"; // Instantiate the OLE DB Source adapter IDTSDesigntimeComponent90 oledbSourceComponent = oledbSourceMetadata.Instantiate(); // Ask the component to set up its component metadata object oledbSourceComponent.ProvideComponentProperties(); // Add an OLE DB connection manager ConnectionManager connectionManagerSource = package.Connections.Add("OLEDB"); connectionManagerSource.Name = "OLEDBSource"; // Set the connection string connectionManagerSource.ConnectionString = "Data Source=localhost;Initial Catalog=TestFuzzyLookup;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;"; // Set the connection manager as the OLE DB Source adapter's runtime connection IDTSRuntimeConnection90 runtimeConnectionSource = oledbSourceMetadata.RuntimeConnectionCollection["OleDbConnection"]; runtimeConnectionSource.ConnectionManagerID = connectionManagerSource.ID; // Tell the OLE DB Source adapter to use the source table oledbSourceComponent.SetComponentProperty("OpenRowset", "QuarantinedEmployee"); oledbSourceComponent.SetComponentProperty("AccessMode", 0); // Set up the connection manager object runtimeConnectionSource.ConnectionManager = DtsConvert.ToConnectionManager90(connectionManagerSource); // Establish the database connection oledbSourceComponent.AcquireConnections(null); // Set up the column metadata oledbSourceComponent.ReinitializeMetaData(); // Release the database connection oledbSourceComponent.ReleaseConnections(); // Release the connection manager runtimeConnectionSource.ReleaseConnectionManager(); #endregion Source #region Fuzzy Lookup // Add a new component metadata object to the data flow IDTSComponentMetaData90 fuzzyLookupMetadata = componentMetadataCollection.New(); // Associate the component metadata object with the Fuzzy Lookup object fuzzyLookupMetadata.ComponentClassID = "DTSTransform.BestMatch.1"; // Instantiate IDTSDesigntimeComponent90 fuzzyLookupComponent = fuzzyLookupMetadata.Instantiate(); // Ask the component to set up its component metadata object fuzzyLookupComponent.ProvideComponentProperties(); // Add an OLE DB connection manager ConnectionManager connectionManagerFuzzy = package.Connections.Add("OLEDB"); connectionManagerFuzzy.Name = "OLEDBFuzzy"; // Set the connection string connectionManagerFuzzy.ConnectionString = "Data Source=localhost;Initial Catalog=TestFuzzyLookup;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;"; // Set the connection manager as the fuzzy lookup component's runtime connection IDTSRuntimeConnection90 runtimeConnectionFuzzy = fuzzyLookupMetadata.RuntimeConnectionCollection["OleDbConnection"]; runtimeConnectionFuzzy.ConnectionManagerID = connectionManagerFuzzy.ID; // Set up the connection manager object runtimeConnectionFuzzy.ConnectionManager = DtsConvert.ToConnectionManager90(connectionManagerFuzzy); // Establish the database connection fuzzyLookupComponent.AcquireConnections(null); // Set up the external metadata column fuzzyLookupComponent.ReinitializeMetaData(); // Release the database connection fuzzyLookupComponent.ReleaseConnections(); // Release the connection manager runtimeConnectionFuzzy.ReleaseConnectionManager(); // Get the standard output of the OLE DB Source adapter IDTSOutput90 oledbSourceOutput = oledbSourceMetadata.OutputCollection["OLE DB Source Output"]; // Get the input of the Fuzzy Lookup component IDTSInput90 fuzzyInput = fuzzyLookupMetadata.InputCollection["Fuzzy Lookup Input"]; // Create a new path object IDTSPath90 path = pipeline.PathCollection.New(); // Connect the source to Fuzzy Lookup path.AttachPathAndPropagateNotifications(oledbSourceOutput, fuzzyInput); // Get the output column collection for the OLE DB Source adapter IDTSOutputColumnCollection90 oledbSourceOutputColumns = oledbSourceOutput.OutputColumnCollection; // Get the external metadata column collection for the fuzzy lookup component IDTSExternalMetadataColumnCollection90 externalMetadataColumns = fuzzyInput.ExternalMetadataColumnCollection; // Get the virtual input for the fuzzy lookup component IDTSVirtualInput90 virtualInput = fuzzyInput.GetVirtualInput(); // Loop through output columns and relate columns that will be fuzzy matched on foreach (IDTSOutputColumn90 outputColumn in oledbSourceOutputColumns) { IDTSInputColumn90 col = fuzzyLookupComponent.SetUsageType(fuzzyInput.ID, virtualInput, outputColumn.LineageID, DTSUsageType.UT_READONLY); if (outputColumn.Name == "QuarantinedEmployeeName") { // column name is one of the columns we'll match with fuzzyLookupComponent.SetInputColumnProperty(fuzzyInput.ID, col.ID, "JoinToReferenceColumn", "EmployeeName"); fuzzyLookupComponent.SetInputColumnProperty(fuzzyInput.ID, col.ID, "MinSimilarity", 0.6m); // set to be fuzzy match (not exact match) fuzzyLookupComponent.SetInputColumnProperty(fuzzyInput.ID, col.ID, "JoinType", 2); } } fuzzyLookupComponent.SetComponentProperty("MatchIndexOptions", 1); fuzzyLookupComponent.SetComponentProperty("MaxOutputMatchesPerInput", 100); fuzzyLookupComponent.SetComponentProperty("ReferenceTableName", "Employee"); fuzzyLookupComponent.SetComponentProperty("WarmCaches", true); fuzzyLookupComponent.SetComponentProperty("MinSimilarity", 0.6); IDTSOutput90 fuzzyLookupOutput = fuzzyLookupMetadata.OutputCollection["Fuzzy Lookup Output"]; // add output columns that will simply pass through from the reference table (Employee) IDTSOutputColumn90 outCol = fuzzyLookupComponent.InsertOutputColumnAt(fuzzyLookupOutput.ID, 0, "EmployeeId", ""); outCol.SetDataTypeProperties(Microsoft.SqlServer.Dts.Runtime.Wrapper.DataType.DT_I4, 0, 0, 0, 0); fuzzyLookupComponent.SetOutputColumnProperty(fuzzyLookupOutput.ID, outCol.ID, "CopyFromReferenceColumn", "EmployeeId");
// add output columns that will simply pass through from the oledb source (QuarantinedEmployeeId) //IDTSOutput90 sourceOutputCollection = oledbSourceMetadata.OutputCollection["OLE DB Source Output"]; //IDTSOutputColumnCollection90 sourceOutputCols = sourceOutputCollection.OutputColumnCollection; //foreach (IDTSOutputColumn90 outputColumn in sourceOutputCols) //{ // if (outputColumn.Name == "QuarantinedEmployeeId") // { // IDTSOutputColumn90 col = fuzzyLookupComponent.InsertOutputColumnAt(fuzzyLookupOutput.ID, 0, outputColumn.Name, ""); // col.SetDataTypeProperties( // outputColumn.DataType, outputColumn.Length, outputColumn.Precision, outputColumn.Scale, outputColumn.CodePage); // //fuzzyLookupComponent.SetOutputColumnProperty( // // fuzzyLookupOutput.ID, col.ID, "SourceInputColumnLineageId", outputColumn.LineageID); // } //}
// add output columns that will simply pass through from the oledb source (QuarantinedEmployeeId) //IDTSInput90 fuzzyInputCollection = fuzzyLookupMetadata.InputCollection["Fuzzy Lookup Input"]; //IDTSInputColumnCollection90 fuzzyInputCols = fuzzyInputCollection.InputColumnCollection; //foreach (IDTSInputColumn90 inputColumn in fuzzyInputCols) //{ // if (inputColumn.Name == "QuarantinedEmployeeId") // { // IDTSOutputColumn90 col = fuzzyLookupComponent.InsertOutputColumnAt(fuzzyLookupOutput.ID, 0, inputColumn.Name, ""); // col.SetDataTypeProperties( // inputColumn.DataType, inputColumn.Length, inputColumn.Precision, inputColumn.Scale, inputColumn.CodePage); // fuzzyLookupComponent.SetOutputColumnProperty( // fuzzyLookupOutput.ID, col.ID, "SourceInputColumnLineageId", inputColumn.LineageID); // } //} #endregion Fuzzy Lookup #region Destination // Add a new component metadata object to the data flow IDTSComponentMetaData90 oledbDestinationMetadata = componentMetadataCollection.New(); // Associate the component metadata object with the OLE DB Destination Adapter oledbDestinationMetadata.ComponentClassID = "DTSAdapter.OLEDBDestination"; // Instantiate the OLE DB Destination adapter IDTSDesigntimeComponent90 oledbDestinationComponent = oledbDestinationMetadata.Instantiate(); // Ask the component to set up its component metadata object oledbDestinationComponent.ProvideComponentProperties(); // Add an OLE DB connection manager ConnectionManager connectionManagerDestination = package.Connections.Add("OLEDB"); connectionManagerDestination.Name = "OLEDBDestination"; // Set the connection string connectionManagerDestination.ConnectionString = "Data Source=localhost;Initial Catalog=TestFuzzyLookup;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;"; // Set the connection manager as the OLE DBDestination adapter's runtime connection IDTSRuntimeConnection90 runtimeConnectionDestination = oledbDestinationMetadata.RuntimeConnectionCollection["OleDbConnection"]; runtimeConnectionDestination.ConnectionManagerID = connectionManagerDestination.ID; // Tell the OLE DB Destination adapter to use the destination table oledbDestinationComponent.SetComponentProperty("OpenRowset", "EmployeeMatch"); oledbDestinationComponent.SetComponentProperty("AccessMode", 0); // Set up the connection manager object runtimeConnectionDestination.ConnectionManager = DtsConvert.ToConnectionManager90(connectionManagerDestination); // Establish the database connection oledbDestinationComponent.AcquireConnections(null); // Set up the external metadata column oledbDestinationComponent.ReinitializeMetaData(); // Release the database connection oledbDestinationComponent.ReleaseConnections(); // Release the connection manager runtimeConnectionDestination.ReleaseConnectionManager(); // Get the standard output of the fuzzy lookup componenet IDTSOutput90 fuzzyLookupOutputCollection = fuzzyLookupMetadata.OutputCollection["Fuzzy Lookup Output"]; // Get the input of the OLE DB Destination adapter IDTSInput90 oledbDestinationInput = oledbDestinationMetadata.InputCollection["OLE DB Destination Input"]; // Create a new path object IDTSPath90 ssisPath = pipeline.PathCollection.New(); // Connect the source and destination adapters ssisPath.AttachPathAndPropagateNotifications(fuzzyLookupOutputCollection, oledbDestinationInput); // Get the output column collection for the OLE DB Source adapter IDTSOutputColumnCollection90 fuzzyLookupOutputColumns = fuzzyLookupOutputCollection.OutputColumnCollection; // Get the external metadata column collection for the OLE DB Destination adapter IDTSExternalMetadataColumnCollection90 externalMetadataCols = oledbDestinationInput.ExternalMetadataColumnCollection; // Get the virtual input for the OLE DB Destination adapter. IDTSVirtualInput90 vInput = oledbDestinationInput.GetVirtualInput(); // Loop through our output columns foreach (IDTSOutputColumn90 outputColumn in fuzzyLookupOutputColumns) { // Add a new input column IDTSInputColumn90 inputColumn = oledbDestinationComponent.SetUsageType(oledbDestinationInput.ID, vInput, outputColumn.LineageID, DTSUsageType.UT_READONLY); // Get the external metadata column from the OLE DB Destination // using the output column's name IDTSExternalMetadataColumn90 externalMetadataColumn = externalMetadataCols[outputColumn.Name]; // Map the new input column to its corresponding external metadata column. oledbDestinationComponent.MapInputColumn(oledbDestinationInput.ID, inputColumn.ID, externalMetadataColumn.ID); } #endregion Destination // Save the package Application application = new Application(); application.SaveToXml(@"c:TempTestFuzzyLookup.dtsx", package, null); } } }
I am working on modifying a VB6 app that dynamically creates DTS packages to copy data from one database to another depending on the selections made in UI. The project currently uses DTSPackage object library and DTSDataDump Scripting object library. We are in the process of upgrading the server to SQL 2005. I am exploring the possibility of replacing code that generates DTS packages on the fly with SSIS packages.
Is it feasible to do this in VB6 ? I have referred to similar posts which focus mainly on VB.NET or C#. Any help with white paper or sample code would be appreciated.
I have created an application that loads a package and executes it using DTS runtime classes. But when I run the application on a machine where only SQL Express edition is installed it's throwing
"Retreiving COM class factory for component with CLSID E44847F1-FD8C-4251-B5DA-B04BB22E236E failed due to the following error : 80040154"
I wanted to thank everyone for posting a ton of valuable information in these forums. I also want to thank all the moderators that have been replying with really insightful help!
I am trying to programmatically create an SSIS package to take .CSV data and put it into a SQL Server 2005. I am assuming that this is pretty common scenario.
I have used many of the examples in this forum as well as heavily borrowing from this example http://www.codeproject.com/csharp/Digging_SSIS_object_model.asp written by Moim Hossain.
I can get my package to create and execute properly but no data is being written to the SQL Server table. This has puzzled me for the last 2 days!
I know the issue isnt with the server itself because I tested it by graphically creating a test SSIS package and it transfers the .CSV data to the table perfectly.
Would anyone know why this would happen? The Execution results are returning success but no data is written to the table!
Could anyone please provide insight as to what my issue may be?
Thanks in advance!
Code Snippet
using System; using System.IO; using System.Data.SqlClient; using System.Collections.Generic; using System.Text; using Microsoft.SqlServer.Dts.Runtime; using PipeLineWrapper = Microsoft.SqlServer.Dts.Pipeline.Wrapper; using RuntimeWrapper = Microsoft.SqlServer.Dts.Runtime.Wrapper;
// Set some common properties of the connection manager object. //flatFileConnectionManager.Properties["ColumnNamesInFirstRow"].SetValue(flatFileConnectionManager, false); flatFileConnectionManager.Properties["Format"].SetValue(flatFileConnectionManager, "Delimited"); flatFileConnectionManager.Properties["TextQualifier"].SetValue(flatFileConnectionManager, """); flatFileConnectionManager.Properties["RowDelimiter"].SetValue(flatFileConnectionManager, " "); flatFileConnectionManager.Properties["DataRowsToSkip"].SetValue(flatFileConnectionManager, 0);
// Create the source columns into the connection manager. CreateSourceColumns(); }
private void CreateSourceColumns() { // Get the actual connection manager instance RuntimeWrapper.IDTSConnectionManagerFlatFile90 flatFileConnection = flatFileConnectionManager.InnerObject as RuntimeWrapper.IDTSConnectionManagerFlatFile90;
public class Column { private String name; private Microsoft.SqlServer.Dts.Runtime.Wrapper.DataType dataType; private int length; private int precision; private int scale; private int codePage = 0;
public String Name { get { return name; } set { name = value; } }
public Microsoft.SqlServer.Dts.Runtime.Wrapper.DataType DataType { get { return dataType; } set { dataType = value; } }
public int Length { get { return length; } set { length = value; } }
public int Precision { get { return precision; } set { precision = value; } }
public int Scale { get { return scale; } set { scale = value; } }
public int CodePage { get { return codePage; } set { codePage = value; } } }
String a = sourceComponent.RuntimeConnectionCollection[0].Name.ToString(); String b = sourceComponent.OutputCollection[0].Name; String c = sourceComponent.OutputCollection[0].Description; String d = sourceComponent.OutputCollection[0].OutputColumnCollection.Count.ToString();
// Create a path and attach the output of the source to the input of the destination. PipeLineWrapper.IDTSPath90 path = ((dataFlowTask as TaskHost).InnerObject as PipeLineWrapper.MainPipe).PathCollection.New(); path.AttachPathAndPropagateNotifications(sourceComponent.OutputCollection[0], destinationComponent.InputCollection[0]);
I am in the process of moving from a 32-bit SQL Server 2005 Enterprise (9.0.3054) to a 64-bit SQL Server 2005 Enterprise (9.0.3054 with 4 CPUs and 8GB of memory on Win 2003 SP2) and the process has been very frustrating to say the least. I am having a problem with packages that I created on my 64-bit SQL Server. I am importing a few tables from the 32-SQL Server into the 64-bit SQL Server using the Task --> Import to create the package.
Sometimes when I am creating a package I get the following error in a message box:
SQL Server Import and Export Wizard
The SSIS Runtime object could not be created. Verify that DTS.dll is available and registered. The wizard cannot continue and it will terminate.
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (System.Windows.Forms)
Other times when I run a package that has run successfully before I get the following error:
Faulting application dtexecui.exe, version 9.0.3042.0, stamp 45cd726d, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x025d23f0.
The package appears to hang when running. By this I mean that the Package Execution Progress shows progress up to a point then it just stops. (The package takes about 17 seconds to run normally) CPU usage is at 1% and the package cannot be stopped.
I have deleted and re-created the package several times and I have also re-installed the service pack on the SQL Server (9.0.3054) but that did not help.
Can anyone give a newbie a general script for creating an SSIS page to import data nightly from an Access database to sql 2005. They are not on the same server. The Access database us 366,968 kb and grows everyday from input from web content. It has 5 tables and 100's of queries, which is how it is updated.
Please give me a starting point, do not understand the info in books on line.
Following is what I would like to do, so I can keep updating my central SQL Server database with latest updates from the field. I like to use SSIS 2005 to create a package that could do this. Any help to get me started would be appreciated. I need some help soon, pls give me something to get started. APpreciate it. Thanks.
Open connection and read client location table on the local SQL Server database called PODO For each location id in the table do the following: Store locationid/clientid in a variable called CLLOC_ID Construct file name with mdb extension and store in a variable MDB_FILE Establish connection to the data import folder Search for that MDB_FILE in the folder on the file system If there is a file where match = true then do this: 1) Open the access database 2) Read and import the data from the customer experience table 3) Write that data to the SQL Server tables where location = CLLOC_ID 4) Exit process IF there is no match, exit process Keep looping until all the client ids/loc ids are read from the SQL Server client location table.
I'm trying to create a SSIS package. I have used OLEDBSource adapter to get the source table's data and transferring the data to an OLEDBDestination adapter. I tried but I'm facing the problem in mapping the metadata contents.
My code: //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
static void Main(string[] args)
{
// Create a new package
Package package = new Package();
package.Name = "OLE DB Transfer";
// Add a Data Flow task
TaskHost taskHost = package.Executables.Add("DTS.Pipeline") as TaskHost;
taskHost.Name = "Transfer Table";
IDTSPipeline90 pipeline = taskHost.InnerObject as MainPipe;
// Get the pipeline's component metadata collection
I'm very new to ssis, so I hope I don't sound too ignorant. I was told there might be a way to create User interfaces for ssis projects. ie, I am trying to create an import package that will let a user import an excel file to a sql table. What I would like to have is some sort of web interface that lets the user select the source file and destination table.
The more I search for a way to do this the more it seems impossible without knowing a ton about coding. If anyoen can help I'd appreciate it.
I would like to create an event handler that would catch any errors that result from a sys.<table> not existing. The package is designed to run on both SQL Server 2000 and SQL Server 2005 and when I query sys.<tables> there is an error when the query is run on SQL Server 2000. I just need a good starting point...I would like something that when the server isn't 2005 it just skips the server and doesn't fail the package and doesn't get counted towards the max error count. Thanks for any help. -Kyle
I need to run a make-table query against an Access database out of an SSIS package. I tried to do this with an OLE DB Command Task but it fails to create the table even though the task execution comes back successful. Any thoughts???
I want to create a local temporary table in execute sql task and and want to use the same in Data flow task as source table.
I follow the following steps to achieve this:
01. Created a new SSIS package 02. Create a connection string to "(local)/." server, "tempdb" database 03. Set the "RetainSameConnection" property value to "TRUE" 04. Set the "DelayValidation" to "TRUE", where ever I found this property 04. In Control Flow I added to items a. Execute SQL Task b. Data Flow Task 05. For "Execute SQL task" I set the connection to "tempdb" 06. I written the following query Create table #transfer_CompaniesToProcess_tbl ( companyID int not null ) GO 07. In Data Flow task I added "OLE DB Source" and "OLE DB Destination" 08. In "OLE DB Source" I changed the "Data access mode:" to "SQL command" 09. In "SQL command text:" I entered "select * from #transfer_CompaniesToProcess_tbl" 10. When I clicked on the "OK" button; I ended with following error:
TITLE: Microsoft Visual Studio ------------------------------ Error at Data Flow Task [OLE DB Source [1]]: An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Statement(s) could not be prepared.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Invalid object name '#transfer_CompaniesToProcess_tbl'.".
------------------------------ ADDITIONAL INFORMATION: Exception from HRESULT: 0xC0202009 (Microsoft.SqlServer.DTSPipelineWrap) ------------------------------ BUTTONS: OK ------------------------------
I gone through the following article and it seems I missed some thing. http://blogs.conchango.com/jamiethomson/archive/2006/11/19/SSIS_3A00_-Using-temporary-tables.aspx
Trying to figure out the best method of reading in a number of flat files, all with different number of columns and data types and outputting them to a database.
Here's the problem: They are EBCDIC encoded and some of the columns are packed decimal. I've set up one package that takes the flat file, unpacks the decimal (Using UnpackDecimal component) and then sending the rest through a second component to go from EBCDIC -> ASCII.
What I need is a way to do this for every flat file based on the schema for that flat file. One current solution is to write a script/app to create the .dtsx XML file and then execute that for each flat file. It appears like this may be possible, but I haven't gotten far enough to know for sure. So my questions are this:
1) Is there an easier way to do this (ie somehow feed the schema to the package and use it to dynamically set up the column makers and determine which columns get fed to the unpack decimal component.
2) If there isn't a better way, will dynamically creating the .dtsx XML file based on the necessary input/output columns for each flat file work? If so, what is a good source of information on this (information about how the .dtsx XML file is set up, what needs to be changed/what doesn't, etc).
Hi All, I have created fact tables and dimension tables in datawarehouse database, and i created a olap cube from those tables. I want to run SSIS Package which populates these fact and dimension tables from datasources.
I have a problem i receive the following error message when i try to add an new step into a SQL Server Agent job :
Failed to retrieve data for this request. (Microsoft.SqlServer.SmoEnum) Additional information: An exception occured while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo) The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction The current transaction cannot be committed and cannot support operations that write to the log file. Roll back the transaction.(Microsoft SQL Server, Error: 3930)
This error pops up right after i change the type of the step to "SQL Server Intergration Services Package"
I have made the following configurations:
The user group (windows group) that the user belongs has the following roles in msdb :
i have made a proxy to sql server agent which has the following subsystems :
"SQL Server Integration Services Provider" the proxy is tied to the same login which has those SQLagent and dts roles in msdb database.
Im using windows authentication and the user that logs into the sql server is in the same group that i have set all of the rights.
Ps. Clearly im missing some role or right somewhere because as soon as i give the group sysadmin role then all the users in that group can create SSIS steps in the agent.
Ps. Ps. I have been living under the impression that i dont have to give sysadmin rights to people that create ssis packages and schedule then with the agent.
I'm a newbie DBA and i'm trying to create a package that would extract data from MySQL and inserts them to a SQL 2005 Server. I'm quite new to this SSIS and would like to ask help from you to help me go through with this.
I've seen several post asking for that possibility, but all 've read, didn't help me.Some sing SQLDMO, other suggest to use SQLSMO, others only explaining to connect to a server and then do "CREATE DATABASE".I will do this within .Net. Connecting to a SQL Server 2005 and execute "CREATE DATABASE" I could realize.But how doing this with SQLExpress? Trying to do SqlConnection.Open with a non existing DB does not work., says "file not exists".Or do I only have the wrong connection string? Can someone post here an excample connection string, which works with a non existing database?Some hints I've read make me considering to use SQLSMO. But I don't have it on my computer. Where do I get it from? Any links would be nice.
Hello. Im trying to create an SQLDataSource control programmatically. I need to do this because I want to do some stuff on my MasterPage's 'Page_Init' event. heres my code (Master.master.vb): Protected Sub Page_Init(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Init lblUser.Text = Page.User.Identity.Name
Dim PUser As New ControlParameter PUser.ControlID = "lblUser" PUser.Name = "LoginName" PUser.PropertyName = "Text" PUser.Type = TypeCode.String PUser.DefaultValue = Page.User.Identity.Name
Dim SQLDS_Login As New SqlDataSource SQLDS_Login.ID = "SQLDS_Login" SQLDS_Login.ConnectionString = "I put conection string here. How do I use the one on my web.config?" SQLDS_Login.SelectCommand = "SELECT [LoginID], [LoginName], [Role], [Status] FROM [myLogin] WHERE ([LoginName] = @LoginName)" SQLDS_Login.SelectParameters.Add(PUser) SQLDS_Login.SelectCommandType = SqlDataSourceCommandType.Text
When i run, i get this error message: The SqlDataSource control 'SQLDS_Login' does not have a naming container. Ensure that the control is added to the page before calling DataBind. I never had any problem with Inserts, Updates and Deleting, but I have never made it work for Select when doing it programmatically. Can you help me with this?
I got a user who is requesting a weekly report to be exported in csv (comma delimited) format. But this process will run weekly using schedule job and he wants the file to save to a certain directory on the network. Two part questions...
1. Is there a way to create a .csv file programmatically after runing the query?
2. How would I save the .csv file to a specified directory on the network?
I am trying to write a script in VB.NET that will run a report that already exists in the system and export the results to my local machine. We are using MS Reporting Services to manage and manipulate the reports. So here's my question:
Is it possible to programmatically create a report in VB.NET based on an existing report? I noticed that crystal reports has a nice export method, but I have not been able to find anything similar for my situation. Basically I believe I would need some sort of reporting services object in .NET that would allow me to run the report and export the results. Does anyone know of such a structure, or if this is even possible? Thanks!!
I build my SQL statement with these values like so: SELECT * FROM POO WHERE COMPANY = 'foo' AND DAY = 2
The problem I am having is when there are multiple values of the same type in the list box. Say: lstCriteria.items(1).value = "COMPANY = 'foo'" lstCriteria.items(2).value = "DAY= 2" lstCriteria.items(1).value = "COMPANY = 'moo'"
My employer wants this to be valid, but I am having a tough time coming up with a solution.
I know that my SQL statement needs to now read: SELECT * FROM POO WHERE COMPANY = 'foo' AND DAY = 2 OR COMPANY = 'poo' AND DAY = 2
I have code set up to read the values of each list box item up to the "=". And I know that I need to compair this value with the others in the list box...but I am not running into any good solutions.
Hi all this is my code and i find it in microsoft's site if i run it with sql server connection it works but if i try to use it with sql express it give me this error: CREATE FILE encountered operating system error 5(access denied) while attempting to open or create the physical file 'c://mydatabase.mdf' it seems as a permission error but it isn't. I have to set something in sql express while in sql server it is already setted?
static void WriteDB()
{
String str;
//sql server connection
SqlConnection myConn = new SqlConnection("Server=localhost;Integrated security=SSPI;database=master");
//sql express connection SqlConnection myConn = new SqlConnection("Server=localhost;Integrated security=SSPI;database=master");
Does anyone have any examples of programmatically creating a Transformation Script Component (or Source/Destination) in the dataflow? I have been able to create other Transforms for the dataflow like Derived Column, Sort, etc. but for some reason the Script Component doesn't seem to work the same way.
I have done it as below trying many ways to get the componentClassId including the AssemblyQualifiedname & the GUID as well. No matter, what I do, when it hits the ProvideComponentProperties, it get Exception from HRESULT: 0xC0048021
Does anyone know how to create a Source Script Component programmatically. I can only seem to create a Transformation Script Component. I have this:
PipeLineWrapper.IDTSComponentMetaData90 sourceComponent = ((dataflowTask as TaskHost).InnerObject as PipeLineWrapper.MainPipe).ComponentMetaDataCollection.New();
I have been stuck with this problem since few days, need help regarding the same. I am enclosing the problem description and possible solutions that I have found.
Can anyone please help me out here?
Thanks and regards, Virat
Problem Description:
I have a requirement for which I have created a data driven subscription in SQL Server 2005, the whole thing works like this:
I have a report on Report Server which executes a stored procedure to get its parameters; then it calls another stored procedure to get data for the report; then it creates the report and copies it to a file share. This is done using data driven subscription and the time set for repeating this process is 5 minutes.
You can assume that following are working fine:
1. I have deployed the report on the Report Manager (Uploaded the report, created a data source, linked the report to data source) - manually, the report works fine.
2. Created a data driven subscription.
3. The data driven subscription calls a stored procedure, say GetReportParameters which returns all the parameters required for the report to execute.
4. The Report Manager executes the report by calling a stored procedure, say GetReportData with the parameters provided by GetReportParameters stored procedure; after it has generated the report file (PDF) is copied to a file share.
For each row that GetReportParameters stored procedure returns a report (PDF file) will be created and copied to file share.
Now, my question is
1. How to I get a notification that this file was successfully created or an error occurred? 2. The only message that reporting service shows on 'Report Manager > My Subscriptions' is something like "Done: 5 processed of 10 total; 2 errors." How do I find out which record was processed successfully and which ones resulted in an error?
Based on above results (success or failure), I have to perform further operations.
Solutions or Work around that I have found:
1. Create a windows service which will monitor the file share folder and look for the file name (each record has a unique file name) for the reports that were picked up for PDF creation. If the file is not found, this service will report an error. Now, there's a glitch there; if a report takes very long time to execute it will also be reported as error (i.e. when this service checks for the PDF file, the report was currently being generated). So, I can't go with this solution.
2. I have also looked at following tables on ReportServer database:
a. Catalog - information regarding all the reports, folders, data source information, etc. b. Subscriptions - all the subscriptions information. c. ExecutionLog - information regarding execution of the subscriptions and the also manual execution of reports. d. Notifications - information regarding the errors that occurred during subscription execution.
For this solution, I was thinking of doing a windows service which will monitor these tables and do further operations as required.
This looks like most feasible solution so far.
3. Third option is to look at DeliveryExtensions but in that case I will have to manually call SSRS APIs and will have to manage report invocation and subscription information. What is your opinion on this?
My environment details:
Windows XP SP2
SQL Server 2005
Reporting Services 2005
Please let me know if I am missing something somewhere...
Can a package executed from code be aborted in code? The Package class has an Execute() method but no Abort() method. Clearly the debugger can stop a package at any point, so it must be possible somehow.
I built a small application that helps me create new packages by loading a template and modifying some parameters, and eventually save the updated package to a user defined location. This application uses the SSIS programming API.
I know all the packages created this way from the same template will have the same package ID. Is there a way to regenerate the package ID programmatically? I don't want to use dtutil.exe, but rather a nice package level method in the API.
Hi, I was wondering if it is possible to change the password on a SQL Server 2000 DTS package programatically instead of justmanually using the the interface? I have over 150 DTS packages used for building BI data cubes and our password policy requires a password change every 60 days. I would like to write a windows vb.net application that loops through all the DTS packages and changes the password. Please advise. Thanks Chris
I am trying to write a program that creates packages on the fly depending on the variables you pass. for eg. It should create connection managers on the fly specific to a certain file in the variable (eg. sample.csv). the package has a dataflow task and it has flat file source and oledb destination.The problem I am facing is the flat file source when assigned to a flat file connection manager(dynamically), it is not giving any source output columns. i.e, the value for DFSource.OutputCollection(0).OutputColumnCollection.Count is Zero. But when I use the same code and reverse the source and destination(oledb as source and flatfile as destination), it is working fine. I searched everywhere for resources on how to develop packages programmatically, but could not find any except one example on msdn. If anyone knows about this prob or any useful resources on this subject, it would be really helpful.