I wrote a C# class that has a couple of methods that can be called to grab data out of a flat file and import it to a DB. This class was written for a web applicaiton originally. (class has many complex business rules in it)
Now I would like to use this same class in a SSIS package.
Is it possible in a script task to create a new object with my class and use it's methods?
In the end all I want SSIS to do is create my object, call a method in that object and be done.
If I was doing this in DTS, I would have to create an .Net EXE that used my class but I'm hoping to avoid that in SSIS.
In visual studio 2005, I create a new Integration Services Project. It tries to create the first package by default "Package.dtsx". The "Package.dtsx[Design]" tab displays
Microsoft Visual Studio is unable to load this document
Object reference not set to an instance of an object
I try to create new SSIS package or edit an existing one (from tutorial), I get the same error in the SSIS graphical user interface tab.
Can we create custom object wide roles? In the same manner that db_datareader in effect grants SELECT on all tables, can we create roles that affect all objects without having to explicitly grant the permission on every object?
Hi,Using VB.net I have created a custom object (hope my terminology is correct here), it’s just a class that defines a few string, integer, and hash table variables. After creating an instance of this class and populating it with data I need to be able to store this instance of my object in a sql server data base table. How do I do this? I saw an article that used the image data type to achieve this (code was in java unfortunately) is this the correct approach. Could you please give me some code examples Thanks
Hi All,Is it possible to create SSIS custom component with ATL COM dll with MFC support?I'm not familar with C# & VB# languages :-(Regards,Svilen Varbanov
Hi there, I'm still pretty new to SSIS, what I am really battling with is getting a custom task to work. I have developed a custom task, digitally signed it added it to the Global Assembly Cache, written the UI for it - Pretty much as it is done here: http://technet.microsoft.com/en-us/library/ms136080.aspx but my problem is that when I drag it from the tool box into the control flow pane, I get this message: "The task user interface specified by type name 'XXXXXXXXXXXXXXXXXXX' could not be loaded. (Microsoft.DatatransformationServices.Design).
Does anybody have an idea of what I am doing wrong here?
I have finished the development of an SSIS package but I would like to make it a bit more flexible by adding a configuration file. The script migrates a database from a source to a destination and syncs with another source.
So there is the source db and the destination db. However, I have noticed that standard SSIS configurations can get a little tricky. For instance, I have two connection managers for the destination db (one that points to the schema and one that doesnt). So my question is: Is there any way to create a config file thta you can input a few attributes like sourceDB, destinationDB, schemaName and get the package itself to parse and process it?
I am creating a custom transformation component, and a custom user interface for that component.
In my custom UI, I want to show the custom properties, and allow users to edit these properties similar to how the advanced editor shows the properties.
I know in my UI I need to create a "Property Grid". In the properties of this grid, I can select the object I want to display data for, however, the only objects that appear are the objects that I have already created within this UI, and not the actual component object with the custom properties.
How do I go about getting the properties for my transformation component listed in this property grid?
I created a SSIS log provider derived from LogProviderBase, it's appear in my VS2005 and work fine, the only one problem is, there is not a dropdown list of connections in the SSIS Log Configuration column, how can I add it?
Hi, As a hobby, I am creating custom SQL Server Integration Services components, which are installed as an add-on to out of the box components. They are mainly used to reduce the development time and minimize the maintance overhead. I would like to know, how I should market them? What are the mediums to use? Since this is the first time, I am creating a software for mass markets, I would really appreciate your help while marketing this baby.
Just FYI, the first commercially available package that I develop is for creating text based files. My package connects to any OLE DB datasource. Executes the query and stores the output in a text file. It is very useful while developing database driven custom feeds. Instead of working with multiple dataflow tasks, data flow sources, you just need to configure one single component and the rest is taken care for you. Since, the package implements the standard Microsoft Task Editor, it is very easy to learn.
I've written several custom Control Flow and Data Flow components for SSIS. I'm trying to deploy a new Control Flow Task, however it will not show up in the "Choose Toolbox Items" window.
I have my dll's all in the right place. Because of other painful issues we have not used the GAC, and instead have put our DLL's that are referenced in the following directory: C:program filesMicrosoft SQL Server90DTSBinn. This has worked fine for other custom tasks, but not for this one.
The only things that are different about this transform are the following:
1. Not only does the task inherit from Task, it also implements two interfaces that I wrote.
2. One of the assembly references is actualy an executable instead of a dll. This is unusual, but I've done it with other .NET projects and not had any trouble.
I could move the code I need from the EXE out to a DLL. If somebody knows what could be causing this please respond?
Is it possible to use versioning at all with SSIS components?
My experience so far is that I have not figured out how version them at all, because if I change the version number, then apparently the <property name="UserComponentTypeName" ...> has to be changed in every single reference to the component, in every dataflow, in every single dtsx file that uses the component at all -- this will be a tremendous headache as some of this stuff is in source control. It seemsl clearly infeasible to check out and substitute and change all these references, and check them all back in, except very infrequently -- surely not feasible to do it for each small change.
I am in the habit of versioning dlls and assemblies, and actually changing at least a minor build number for each actual change, so I am a bit unhappy to find that apparently I cannot feasibly do this for a custom SSIS component :(
(I just find a dislike at apparently being forced to leave the version number invariant when I make code changes.)
What I would be very happy with, is if there were a way to adjust all the references in all the dtsx files, to refer to only the major and minor build numbers say, specifying that I am not picky about the revision numbers below those, so that it will find the latest component in the GAC matching the major and minor numbers.
That way I could still update the component version with each code change, which would make me happy, and not have to edit a zillion dtsx occurrences (not having to do that would also make me happy).
Anyone know if that is possible?
I'm hoping it is, and that it is only my ignorance at component references preventing me from doing so.
Please suggest me the approach for handling the below custom validation in SSIS
Source: A Table in the database Destination: One or More Tables in the database Transformation:For each record in the source table, I have to do custom validation and redirect to one of the destination tables
Custom validation for each record include
1. Checking a set of fields are not matching with main record data in database - If so move to a duplicatelist table 2. Checking for set of fields have got any new values - If so make an entry in a newvaluesfound table 3. If no new/duplicate found move to success table
Approach 1
An OLEdb Source Task to point the source table An OLEdb Transformation to call an SP for each record in the source Concerns
The Transformation is implemented in SP(Custom validation & loading to destination)
Approach 2
An Script task from the control flow items - To call an SP in the database Concerns
The Source, Transformation, Destination - everything managed in the SP
Suggestions needed
Is there any better approach of handling this type of requirement in SSIS If I write all the code in SP, I may need to use cursors for looping through each record..
I saw some thing called custom properties for the "Derived transformation" in the msdn site. I tried to use them in a simple package, but I am getting an error as "can't write to derivedoutputcolumnname.friendlyexpression". Friendly expression is one of the custom properties available for the derived transformation output columns.
The steps I followed to get to this error are as follows:
1) Get data from a table using OLEDB Source. Suppose I am getting firstName, LastName etc.
2) Derived column input is values from the above OLEDB Source.
3) I have added a new column called "Concatenated name" which is concatenated value of first and last names.
4) Then in the properties editor of this data flow task in expressions option I clicked on ellipse available. I got an editor for property expression, which contained two columns called "Property" and "Expression". Property column contains dropdown with friendly expressions propety for the derived columns and expression column is a text box, where in we can enter expression to be evaluated for the corresponding friendly expression property.
5) Now when I click on OK and try to debug it gives an error as "Can't write to concatenatedname.friendlyexpresiion".
If anybody has already faced this problem and solved it please let me know, because I am struck here a long time.
Is there a way of disabling a custom property in a component so that during design-time the property is grayed out? I looked around the properties of IDTSCustomProperty90 and nothing sticks out.
I can't find information on how you are suppose to handle the TransactionOption setting inside of a custom task.
I have a custom task that I have developed, and it basically calls a COM+ object that writes data to a database. When I have the task inside of a container that has the transactionoption set to required, and my custom task is set to supported, if one of the X items fail to execute in my custom task I am telling my task to fail the parent, which I thought would rollback everything. But it does not.
Is there someplace that I need to write rollback code in custom SSIS tasks? If there is I can't find any mention of it anywhere. Any examples out there on how to build custom SSIS tasks that support the TransactionOption parameter?
I have been working with a client on a legacy application migration to SSIS and I found SSIS to be very helpful, as this is my first project using integration services I found myself in a situation where multiple solutions can work but I cannot assess the risk of using one or the other and the effort required for the implementation.
The situation is as following, we cannot touch the input files that come from the other legacy systems so I need to get information that is packaged following the next multipart schema:
&Header with context information from the source &BatchHeader with context information from the batch (including format N) - Data in format N &BatchHeader with context information from the batch (including format K) +Data in format K &BatchHeader with context information from the batch (including format P) =Data in format P
Etc... Needless to say that they can come in whatever order and they need different processing depending on the format. I have been tempted to use the techniq shown in "Handling Different Row Types In The Same File" from http://www.sqlis.com/default.aspx?54.
However, there are multiple issues that I have to resolve to do that: First is how to feed the pipeline with the Data in format N and then reparse them in an easy way cause the data that comes from the other system is very wide (aprox 120 columns), so using a parsing script is by no means a viable solution (I have multiple formats to handle). So I do not see a way to treat the data without using a Custom Connection Manager. Here comes my first question: Is there anyway someone can use the already provided managers to handle an scenario like this one?
Next comes the separation, which one is the wisest thing to do: Write again that to disk in as easier format or pass along that information to the next task in memory? The volume of data is in the order of hundreds of megabytes (100 to 200), maybe half a Gigabyte in the extreme cases. How about uploading that to SQL Server and work from there?
BTW, the production servers will have 6 to 8 Gb of memory and 4 to 8 processors (more or less).
Near the end of 2005 Microsoft made available some sample C# apps that implemented custom SSIS tasks and components. I think Doug Ladenshlager may have had a heavy hand in building them.
Does anyone know where they are? I can't darned well find them!
Does anyone here have a favorite site or set of sites with resources on SSIS custom component (tasks, transforms, log providers, etc.) development? I've been searching around to try to find this on my own, but so far I've had little luck.
Darren - I've noticed that you always seem to have the most insightful responses to questions related to SSIS .NET development (such as your recent response to evaluating expressions in a custom component) so I am particularly interested in anything that you have to share.
I have a SQL Server 2005 Package (32 bit, 4 processors/16 GB of RAM, 3 GB switch for SSIS) with 1 DataFlow and 5 Custom Components:
> An OLEDB Source that reads a stoed proc from Sybase > 3 Transform Components that perform: 6-required fields check, 4-lookups (3 cached, 1 non-cached), and a calculation that is performed(called) from a .NET Library. > A SQL Destination that uses the .NET BulkCopy Class to insert the data.
It takes about 3 hours to load 1 million records that contain 46 columns (the largest being a varchar(100)). I have tried a good number of tuning options as well as indexing/SQL techniques to improve perfromance.
Some pieces of info: 1) We dynamically create the buffer during runtime based on an SSIS variable. 2) The actions of the components are determined at runtime through config files. We do these two things so we can have multiple data flows run through the same package for better scalability and flexibility. Since we are dynamically creating the buffer I can NOT use any of the out of the box components because of lineage and metadata inconsistency issues.
3) I noticed on PerfMon that I am spooling buffers to disk after 100K rows. I realize this is bad, but what are my alternatives? Should I go to 64 bit? Is there anything else I can do?
Can anyone point me to some sample source codes or any articles that describes how I can programmatically create a package which will import data from a flatfile (csv) to Sql server database.
I know there is some example that describes exporting data from sql server to flatfiles. Anyway I have failed to accomplish my goal by following those examples.
If anyone have a code snippet to do that please help me with that.
I have been writing a custom source adapter that uses a file connection within the connection manager. If I hard-code a specific file then the component works. However if I use a file connection that has an expression defined which updates the connection, for example when you have a for-each loop looping over a set files. The file connection doesn€™t seem to re-evaluate expression each time you access the file connection via the code. Is there something that I am missing or is there a bug?
I am using a Data Flow task which copies data from an Excel Source to a SQL Database Table Destination. From 15 columns I require only 10 columns to be imported to the DB Table. So I have mapped those colums. In SQL DB there is a colum called say X, whose value should be the "Remedy" for all the columns which are imported. Is there any task that can achieve it.
dear experts, i'm trying to build a package programmatically from client c# application. I'm working to create three dataflow components: - OleDB Source - Derived Column Transformations - OleDb Destination.
My package works good, but i have several problems when insert an expression as Value of an IDTSCustomPropriety90 object, like this one: [LEN](#firsname.lineageID) > 5 ? <<if true>> : <<if false>>
Simple expressions like concatenate strings, for example "#firstname.lineageID" + "#lastname.lineageID" seem to work good.
Ive been using SSIS for a month or two and now find I need to create some custom tasks to perform some performance logging. in the the overloaded ProviderComponentProperties section I am trying to create a property which has the same look as the Expressions properties you find elsewhere (Little + on the left and a group of sub properties when expanded).
Ive have played with creating a IDTSCustomPropertyCollection90 collection then adding my sub properties to it but I cant seem to then add my new collection to the ComponentMetaData.CustomPropertiesCollection.
Im assuming the Expressions parameter is a collection added to the properties collection but I cant figure out how. Any help would be much appreciated.
I have created one custom task with PropertyGrid Control and two button on it. I have everything under one class library project. Problem I am facing is when i load task and clik on Edit I can not see those properties into that GUI and even functionlity of those two buttons (OK and Cancel) not working but I am able to see those properties in default property window.
If I create this GUI as a seperate window application then I am able to see those properties in GUI and buttons also working but in SSIS I am not able to load the task.
After reading on internet about SSIS they suggest to create everything under one project which I did.
Basically I am trying to populate connection managers like Source Connection and Destination Connection when I load this task and there are much more backend functionlity but at first step i m stuck and not able to see those properties in GUI.
Please help and give your input on it. I was following "Increment Task" example given by MSDN. If you need more info let me know.
This is a question to the SSIS development team. I would like to know what are the requirements to implement custom SSIS Control Flow task in C++ . There is a documentation describing the process when implementing a managed task, but no such documentation exists for implementing a task in C++.
I created a SSIS custom component, transformation (Asynchronous) with one Input collection and 2 output collections.
The SSIS Package which includes the Component I created works well in the Business Intelligence Studio, but when the same Package is run in the 'Execute Package utility' It fails to run. ( when you Double click on the dtsx file)
The cause of the failiure is
public override void PrimeOutput(int outputs, int[] outputIDs, PipelineBuffer[] buffers)
method receives only one output buffer when executed using the 'Execute Package Utility' { outputs = 1 , buffer.Length = 1 } ( when executed in the BI studio, the method receives parameters of both the output buffers that I expect { outputs = 2 , buffer.Length = 2 } )
The property ComponentMetaData.OutputCollection.Count = 2 as well. Yet the PrimeOutput method provides only 1 buffer.
The Validation Succeeds on both instances, which I assume means that Meta Data is Provided Properly.
What would be the reason for the same pakage to run in 2 different ways like this,
What might I have missed out to do, to make the package run in different ways on 'Business Intelligence Studio' and 'Execute Package Utility'
Thanks a lot
Below are some of the lines from the ProvideComonentProperties Method which deals with the output Collection, Isn't this sufficient for the PrimeOutput to provide 2 output buffers?
ProvideComponentProperties()
public override void ProvideComponentProperties() {
I have a SSIS package that has a script component. In the script I reference a custom dll that contains some functions.
The dll is registerd with a strong name in the GAC on my machine and the SQL Server 2005 box.
The package works fine in Visual Studio. The problem lies when I deploy the package to the server. (note: several other packages created without the dll work fine on the server).
Is there something else that has to be configured on the server to be able to reference and use the custom dll?
