Developing Custom Components By Extending The SSIS Stock Data Flow Components
Sep 7, 2006
Everything I've read says that custom data flow components are built by inheriting from the Microsoft.SqlServer.Dts.Pipeline.PipelineComponent class.
But the stock components such as the Derived Column data flow transformation must each be implemented by their own class. So how do I base my custom components on those classes? The documentation for the PipelineComponent class doesn't list any such subclasses.
View 1 Replies
ADVERTISEMENT
Aug 29, 2007
Hello,
Is it possible to use existing data flow components (Merge Join, aggregation,...) in a custom data flow component?
Thanks,
Yoann
View 15 Replies
View Related
Mar 29, 2007
I've been having an issue with the integration of a third-party DLL into a custom data flow component.
The company sent me a C# project that generates a simple console application. The project includes a class that calls their DLL with DllImport. The console application runs fine.
I created a stand-alone class using the C# class they sent to expose the methods of their DLL. In my custom component, I'm referencing this class to pass data to and from their DLL.
The first method from that stand-alone class that my component encounters simply gets their installation path from the registry and does not use DllImport. That path retrieval works fine. The next method calls a function that is declared with DllImport. Each time the call fails with "System.DllNotFoundException = {"Unable to load DLL AMZip.dll': Exception from HRESULT: 0xE06D7363"}".
I've copied this DLL to countless locations (e.g., the PipelineComponents directory, the project/solution bin directory) and included these paths in all manner of path variables.
What am I missing here? Their DLL is not strong named (does this matter since I'm using DllImport?), my stand-alone class is, and of course, the custom component itself is. I appreciate the help.
View 9 Replies
View Related
Sep 29, 2007
Hi,
I have a SSIS Package which I would like to modify using SSIS API. I need to put new component between some two existing data flow's components. During this process I need to disconnect two data flow's components using SSIS API. How can I do that?
Thanks,
Rafal
View 1 Replies
View Related
Mar 29, 2007
I created a SSIS package and several data flow componenets for this package.
What does strategy exist to deploy SSIS package and data flow components into a enterparise server?
Thanks in advance.
View 2 Replies
View Related
Nov 2, 2006
We're experiencing a problem where intermittently our SSIS packages will hang. There are no log errors or events in the event viewer. It will happen whether the package is executed from the SQL Job Agent or run from BIDs. When running from BIDs it appears to hang inside one of the data flows (several parallel pipes with sorts, merge joins etc...). It appears to hang in multiple pipes within the data flow component. The problem is reproducable, we just kill it and re-run, and it appears to hang in the same places.
Now here's the odd thing: as we simply open and close some of the components in the pipe line after the place it hangs, a subsequent run will go further in the pipeline before hanging. If we open and close all the components after the point it initially hung, the data flow will run fine, from there on out. When I say "open and close" I mean no changes are made, we simply double-click the component, like a merge join, then click 'close.'
To me this does not seem like a memory problem but likely something is wrong with the metadata, where opening a component and closing it somehow alters the metadata to "right it".
This seems to occur intermittently after we make modifications to the package. It's like if you make any mod, even unrelated to the data flow, you then have to go through and open and close every component in your package to ensure it will work. Again, no errors or warnings are fired.
Has anyone seen this type of problem?
View 10 Replies
View Related
Apr 29, 2008
How can I create a source extension that reads a text file and returns a XML file? I would like to have an xml File as output, because the information is already normalized.
If I extend the class €śPipelineComponent€? I can just define IDTSExternalMetadataColumn90 as output. And I do not want to have columns. I would like to return a xml file
Is it possible to achieve this?
View 1 Replies
View Related
Jul 18, 2006
Can anyone provide any direction in extending existing SSIS components with a custom component, if it's possible at all. In some cases there are just slight bits of functionality missing that I think I could add in myself. But, I'm not much of a programmer and need a bit of help in the declaration of the component in my own custom component.
Thanks.
View 1 Replies
View Related
Oct 10, 2007
Hi,
As a hobby, I am creating custom SQL Server Integration Services components, which are installed as an add-on to out of the box components. They are mainly used to reduce the development time and minimize the maintance overhead. I would like to know, how I should market them? What are the mediums to use? Since this is the first time, I am creating a software for mass markets, I would really appreciate your help while marketing this baby.
Just FYI, the first commercially available package that I develop is for creating text based files. My package connects to any OLE DB datasource. Executes the query and stores the output in a text file. It is very useful while developing database driven custom feeds. Instead of working with multiple dataflow tasks, data flow sources, you just need to configure one single component and the rest is taken care for you. Since, the package implements the standard Microsoft Task Editor, it is very easy to learn.
Thanks for your help.
Cem
View 3 Replies
View Related
Jul 17, 2006
Is it possible to use versioning at all with SSIS components?
My experience so far is that I have not figured out how version them at all, because if I change the version number, then apparently the <property name="UserComponentTypeName" ...> has to be changed in every single reference to the component, in every dataflow, in every single dtsx file that uses the component at all -- this will be a tremendous headache as some of this stuff is in source control. It seemsl clearly infeasible to check out and substitute and change all these references, and check them all back in, except very infrequently -- surely not feasible to do it for each small change.
I am in the habit of versioning dlls and assemblies, and actually changing at least a minor build number for each actual change, so I am a bit unhappy to find that apparently I cannot feasibly do this for a custom SSIS component :(
(I just find a dislike at apparently being forced to leave the version number invariant when I make code changes.)
What I would be very happy with, is if there were a way to adjust all the references in all the dtsx files, to refer to only the major and minor build numbers say, specifying that I am not picky about the revision numbers below those, so that it will find the latest component in the GAC matching the major and minor numbers.
That way I could still update the component version with each code change, which would make me happy, and not have to edit a zillion dtsx occurrences (not having to do that would also make me happy).
Anyone know if that is possible?
I'm hoping it is, and that it is only my ignorance at component references preventing me from doing so.
View 4 Replies
View Related
Nov 1, 2006
Does anyone know a good place to download (or even purchase) custom data flow components?
Some ideas I've had (and I'm sure I'm not the first):
Transformations:
Filter
Sometimes I just want to filter out certain rows.
Unicode Converter
Convert all unicode columns in place.
Sequence Column
Adds a column with a sequence of numbers. Or perhaps even random numbers.
GUID Column
Adds a column with a GUID.
Destinations:
Garbage
Would act as a visual indicator in the designer that these rows are being discarded. You could add a data viewer right before it too.
Log File
Good for redirected error rows, writes them to various log files.
Email
Along the same lines as Log File, but sends off all the rows in a single email. Great for day to day things. Nothing too critical.
View 1 Replies
View Related
Nov 15, 2007
I have a SQL Server 2005 Package (32 bit, 4 processors/16 GB of RAM, 3 GB switch for SSIS) with 1 DataFlow and 5 Custom Components:
> An OLEDB Source that reads a stoed proc from Sybase
> 3 Transform Components that perform: 6-required fields check, 4-lookups (3 cached, 1 non-cached), and a calculation that is performed(called) from a .NET Library.
> A SQL Destination that uses the .NET BulkCopy Class to insert the data.
It takes about 3 hours to load 1 million records that contain 46 columns (the largest being a varchar(100)). I have tried a good number of tuning options as well as indexing/SQL techniques to improve perfromance.
Some pieces of info:
1) We dynamically create the buffer during runtime based on an SSIS variable.
2) The actions of the components are determined at runtime through config files.
We do these two things so we can have multiple data flows run through the same package for better scalability and flexibility. Since we are dynamically creating the buffer I can NOT use any of the out of the box components because of lineage and metadata inconsistency issues.
3) I noticed on PerfMon that I am spooling buffers to disk after 100K rows.
I realize this is bad, but what are my alternatives?
Should I go to 64 bit? Is there anything else I can do?
Thanks in advance,
Anthony
View 10 Replies
View Related
Nov 15, 2007
Hello everyone,
I am developing an SSIS solution where the Data Flow task extracts data from a source csv file, then performs several transformations on the source data and then starts inserting the cleansed data into several destination tables.
The Data Flow task is getting too large!
Question:
Is there a way (best practice) for grouping components in the Data Flow - similar to the Container concept in the Control Flow?
I know this question sounds too luxurious, but I really loose the overall picture, when the Data Flow canvas gets too crowded.
Thank you in advance.
Samar
View 7 Replies
View Related
Jun 30, 2006
Hi,
Can anyone tell me where I might find the Class and Sequence Diagrams for the SSIS framework (for Custom Components)?
I've just started trying to create some Custom Transform Components and
I'm really struggling to get my head around the component lifecycle
(i.e what methods are called when, with what arguments, and why) with
just the BOL documentation to guide me.
Thanks in advance,
Lawrie
View 1 Replies
View Related
Mar 6, 2007
HI All,
In several threads there has been discussion regarding adding connection managers to a package's data flow, etc. My challenge is that I have a large solution that contains many packages, and I need to change the connection manager linked to the data flow in all of the packages. When the solution was initially designed, data sources were used, and it has become a tedious maintenance issue to keep those in sync. We want to use a standard OLEDB connection manager, but adding a connection manager to each package and editing the corresponding data flow tasks in each package to use that new connection manager is a daunting task. I've coded a .Net module to access the packages, remove the old connection manager (data source) and add the new OLEDB data source. However, as I traverse the objects in the package hierarchy, when I come to the data flow object, the innerobject is not a dts object, but rather a _com object.. I can't seem to find any documentation/examples as to how to iterate the tasks within a data flow and change the connection manager. If you have any information, that would be quite helpful. If you reply with a code sample, if you would be so kind as to relate it to one of the sample packages provided with SSIS so I can run it, that would be great.
Thank you.
Steve.
View 1 Replies
View Related
Jul 25, 2007
I have Data Flow task that contains 50 components.
My computer configuration: 1 GB RAM Microsoft Windows Server 2003
Periodicaly when i try to save package after making some changes Out of memory ... exceptions message box appears , and soon after this Not fatal error occurs ... message box shows . If i close solution and open it again all my 50 components disappears --instead I see clear list, and all my work losen.
Such "Not fatal errors" making hell out of job -- every time I need to change package i must add package to archive!!!
View 4 Replies
View Related
Jun 25, 2006
Hi All,
I am not sure if this is a correct forum to discuss on the document posted @ http://www.microsoft.com/downloads/details.aspx?familyid=1c2a7dd2-3ec3-4641-9407-a5a337bea7d3&displaylang=en on SQL Server Integration Services (SSIS) Hands on Training - Creating Custom Components.
I am assuming Microsoft Developers are constantly monitoring this forum.
In the document - SSIS Creating a Custom Transformation Component .doc on Page 2 -
Exercise 1 - Writing the no-op data flow transformation component -
Task 1 - Create a new C# Class Library Project
The textual description talks about creating a new Visual C# Class Library project in VS 2005 but the screenshot accompanying it shows the creation of new "Integration Service Project" in VS 2005.
Please change the screenshot appropriately to avoid confusions.
Thanks,
Loonysan
View 1 Replies
View Related
Jun 30, 2006
Hi,
In another thread Jamie Thomson very informatively said "The components in SSIS
are deliberately atomic (i.e. they do something very specific) so that
its easy to put them together to build something greater than the sum
of the parts". Which does make a lot of sense. However, I've been finding that I end up having to create exactly the same "pattern" of combined transform components again and again in order to solve the same problem but in different dataflows (or even within the same dataflow). Cut-and-paste-tastic! In order to obtain real re-use, it seems to me like SSIS is crying out for an easy way to create new components by using composition - i.e. the ability to take commonly-used combinations of existing components and create new "super" components (without having to write Custom Transform Components in C#/VB.Net and handle everything in code).
Does anyone know if this sort of functionality is likely to make it into SSIS in the forseeable future?
Regards,
Lawrie
View 6 Replies
View Related
Jan 22, 2008
Hi again, All!!
I'm having a problem with problems with some custom components I built. I have a custom Connection Manager and Data Flow Source that I have built.
My local PC is 64-bit, but I followed the MS instructions on building and deploying custom components. They run fine inside of VS2005, as long as I have 'Run64BitRuntime" set to 'False'. When I try to run packages using the custom components in 64-bit mode, I get:
Error: 0xC0014005 at : The connection type "<MyConnectionManagerType>" specified for connection manager "PRI" is not recognized as a valid connection manager type. This error is returned when an attempt is made to create a connection manager for an unknown connection type. Check the spelling in the connection type name.
I suspect I need to either change the way I build or deploy the custom components. Can someone shed some light on this?
Thanks,
Frank
View 1 Replies
View Related
Jun 15, 2007
Is it possible to create a custom SSRS .NET component that can view the report parameters and values contained within?
Thanks!
View 3 Replies
View Related
Feb 12, 2007
Hi,
In my case I have to log the errors raised by any task in a package to either windows event log, text file or SQL server. Also I need to send an email notifications to a group of people telling them about the error.
Now can I use SSIS package logging for logging the errors into the required destinations. I mean right clicking on the package and selecting Logging, then adding the required log providers and enabling the events for logging into those. I think I have to upfront select the log providers to log the error, I will not have the liberty to log the error to the destination, the name of which is passed as a variable to the package. This is okay with me though.
Now what will a custom log provider help me to do in this case. Also can I somehow configure my package to call the send mail task everytime an error is raised.
Also, one more option can be developing a package that only does the error handling. It will take in the paramters or the error codes and descriptions, the destination to write to and a flag to send mail or not for that particular type of error.
What do you think? Kindly advise.
Thanks in advance for your help and time.
Regards,
$wapnil
View 3 Replies
View Related
May 10, 2007
The following is a list of questions that I have not been able to obtain concrete answers. I am probably missing something:
1) ReadWriteVariables -- can the updated value for a ReadWriteVariable be accessed within the same data flow? It appears not as I think the PostExecute() fires at the completion of the data flow not the end of the Script Component. Secondarily, the Script Component is a non-blocking transformation so the component does not "see" the end of the pipeline prior to sending data down stream.
2) Record Count -- Because of #1 above, How could you calculate a record count for a data stream? It does not appear that one can calculate the number of records for a data stream within a data flow and then access the count from within the same data flow.
3) FinishOutputs() -- Is the concept of FinishOutputs() applicable to Script Component Destinations? Asked another way, is FinishOutputs() executed at the end of the data stream regardless of whether there are "real" outputs for the component? I can create a "Dummy" output to create FinishOutputs() but is this ok?
4) Script Component -- It appears that the Script Component Source, Transformation or Destination are really defined based on the columns defined in "Inputs and Outputs". Can you convert an Source script component to a transformation script component by simply adding an Output?
Sorry for these basic questions but I am not getting it completely. As you can tell...
View 12 Replies
View Related
Mar 13, 2006
Hello,
I had an idea that it would be nice to be able to extend the functionality of an existing SSIS component or task by inheriting from it. Perhaps in a similar way to how it is possible to extend user controls in .Net.
e.g. The rowcount is a useful component but it would be good to create a new component that inherits from it and then override the PostExecute() method to fire an Information event containing the number of rows. That's a very simple example but I think you get the gist.
Does anyone think that would be useful? Or even made possible?
I was also wondering whether it would be possible for the SSI team to make all the icons used within SSIS available as .ico files so that we could modify them for tasks/components that might do similar things.
Thanks
-Jamie
[Microsoft follow-up]
View 1 Replies
View Related
Jun 30, 2006
Hi,
Is it possible to make use of VB.Net for Applications and the VBA editor from your own Custom Transform Components (like the script component does)?
I would like to create a component that generates boiler-plate VBA code and that then allows developers to add their own specialised code in certain methods using the editor (just like the script component)
If so, is there any documentation anywhere on how to go about this?
Thanks in advance,
Lawrie
View 4 Replies
View Related
Dec 4, 2007
I've been trying to find a solution to adding javascript and/or HTML to a report (not reportviewer). After a search I found that you can add javascript to a textbox under action properties and it works fine. Is there a way to add custom code anywhere on the page.
The problem is the charts that come with SRS is limited so I'd like to add my own. I'd also like to add some ajax functionality as well. I know it's possible because a company called Dundas is doing it but not exactly sure how they implemented it. We considered Dundas but they are just too expensive for the solution we need.
Any help would be much appreciated.
View 8 Replies
View Related
Sep 7, 2007
Is there a way to use a custom component programmatically (I'm using C# to create and run my packages) without installing a compiled .dll of the class into the SQL folders and the GAC? For instance, can I instantiate my custom component and somehow pass it to be used in a task in my code.
View 1 Replies
View Related
May 21, 2006
I need VB.NET source code showing how to create a Custom Component. in SSIS (All the ones I've seen are written in C#.)
TIA,
barkingdog
View 1 Replies
View Related
Apr 11, 2008
Hi all,
I'm wondering if anyone's accomplished this before - I've been unable to find a whiff of info on how to do this so far.
I'm creating a custom component that I'd like to give a "Derived Column" type of ability to. By that, I mean I'd like to populate a property of my component with an expression (including references to input columns, package variables and functions) and be able to evaluate it at runtime - per row processed by the component.
I would also appreciate any information as to how to provide the interface to allow the user to build such an expression as well - is there a UI function in SSIS I can call to pop up the "expression builder"?
Thanks!
View 6 Replies
View Related
Feb 14, 2006
Hi,
I have a 2 custom components - source and destination.
I want to create an error output for each, to allow the users of my component to handle errors the way they choose.
I only found a property in IDTSOuptut90 named isErrorOut - a boolean property indicating whether this output is an error output or not.
Does anyone have additional documentation / articles / code samples regarding how to really populate the rows in the error output?
Thanks
View 1 Replies
View Related
May 28, 2008
Whenever I make a breaking change to a custom SSIS component/tasks and update the Assembly Version, it seems to break my packages beyond repair, telling me it can't load the task:
Error loading Package1.dtsx: Error loading a task. The contact information for the task is "". This happens when loading a task fails.
All of the properties of said task now show:
Could not get value for property 'c-155-designer-name'. Specified cast is not valid.
Typically, a "breaking" change when it comes to code just means that you need to update your components to adhere to the new contract of the updated signatures. But with SSIS, it seems the only solution to this is to completely remove the component, and re-add the new version, and re-enter all of the property values/expressions. If I have a package containing 10 instances of a task that only had one property removed, for example, this results in a very time-consuming process of fixing my package.
So my questions:
1) Am I doing something wrong in my versioning/deployment that is causing my packages to unnecessarily break?
2) If this is just "by design" and the way it's meant to behave, what is the best practice for making breaking changes to custom tasks/components used by many packages? Should I just never change the assembly version, even when it is a breaking change (this seems to be less disastrous)?
3) As a last resort, if I'm stuck with having to fix the broken tasks, is there a better way to fix them rather than having to completely remove them, re-add them, and re-set all of their properties/expressions?
Thanks in advance.
Jerad
View 3 Replies
View Related
Jan 2, 2008
Hi
I am looking for a function or so to format a number to two digits plus one decimal point.
Some examples:
- if input=4 then output= 04.0
- if input=40 then output= 40.0
- if input=4.1 then output= 04.1
- if input=4.32 then output= 04.3
Any idea?
View 1 Replies
View Related
Oct 10, 2007
hello all -- I'm an SSIS newbie who is exploring (custom) programming functionality through third party components, to be plugged into SSIS, for SMTP, FTP, PGP, TAR, WinZIP, basically for a scheduling & automation utility.
Is there such a concept of third party components (and marketplace for this?) or is VB a much better choice for this vis-a-vis components variety, $$ and robustness ?
thanks much,
Cos
View 1 Replies
View Related
Feb 10, 2006
Especially interested in the CodePageConvert.
Installing the .msi just creates a project folder in my Visual Studio directory. I'm unclear how to get from this point to being able to choose this component from my Toolbox items in SSIS. There was a readme file that talked about gacutil.exe and .snk files that was a above my head.
Can anyone dumb it down for me?
View 6 Replies
View Related
