I'm trying to create a script component which produces several rows for each row in the input stream on the output stream. I'm testing it with an OLE Source -> Script Component -> Flat File in a dataflow. Everything works fine for small input, but I keeping getting an exception from the call to AddRow() on the output buffer with large input sets. For example, I can reproduce the failure with this minimal testcase:
Code Block
Public Overrides Sub ProcessInput(ByVal InputID As Integer, ByVal Buffer As Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer)
If Not Buffer.EndOfRowset Then
While Buffer.NextRow()
Output0Buffer.AddRow()
End While
End If
Output0Buffer.SetEndOfRowset()
End Sub
I can increase the number of rows processed before failing by playing with the DefaultBufferMaxRows, but it still fails. I've pasted the error message below. Any help would be appreciated!
at Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSBuffer90.AddRow(IntPtr ppRowStart)
at Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.AddRow()
at Microsoft.SqlServer.Dts.Pipeline.ScriptBuffer.AddRow()
at ScriptComponent_1b906150cbf147c69f8fe3967301ca55.Output0Buffer.AddRow()
at ScriptComponent_1b906150cbf147c69f8fe3967301ca55.ScriptMain.ProcessInput(Int32 InputID, PipelineBuffer Buffer)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.ProcessInput(Int32 inputID, PipelineBuffer buffer)
I am new to SSIS. Heree is the transformation I need to do. In database one, I have tables:
A
B
C
Where B is a detailed table of A, and C is a "derived" table of B. There is a one to one relationship between B and C, but there may be more than one record in B for each record in A.
In database two, we have the table structure:
AA
CC
And there is a one to one relationship between AA and CC. And I need to design a transformation to migrate data from database one to database two.
Table A(->AA) and B(->BB) will be easy, just one to one migration.
The mapping rule for table C(->CC) they decided was: I need to concate each record in B and C and for a record in CC. For example, suppose we have:
A1
B1(C1)
B2(C2)
A2
B3(C3)
B4(C4)
B5(C5)
Then we will have the following records in database two:
A_1
(B1+C1+B2+C2)
A2
(B3+C3 + B4+C4 + B5+C5)
I looked through all the stock data flow components, and it seems to me that none of them can perform this task, so I am thinking to design a Script transform component to do the task.
I have written the script:
in each ProcessInputRow() sub, I check record's foreign key to A, and if they are the same I concate the records, then I put them into a VB.NET collection. Once all the records are processed, in the PostExecute() function, I count the number of new rows, (in the above example 2), then I call AddRow to add the rows, by:
OutputBuffer.AddRow()
But this does not work, I got "Object reference not set to an instance of an object". It seems that in PostExecute, the OutputBuffer is not longer valid?
Help please! :)
Is there a better way of doing what I am trying to do?
I have a package that has a data lfow task. this task imports data from a db2 database (using the IBM Ole DB provider fro db2) and adds it to sql server database table. This package was created on the server. then though version control (using TFS source control) I check out the package on my local machine. and when I open the package I get the foll 3 errors.
Error 1 Validation error. Import Account Num from BMGP_BDR: DTS.Pipeline: The component metadata for "component "DataReader Source" (1113)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
Error 2 Error loading BMAG Download Xref Tables - bmag.dtsx: Microsoft.SqlServer.Dts.Pipeline.ComponentVersionMismatchException: The version of component "DataReader Source" (1113) is not compatible with this version of the DataFlow. [[The version or pipeline version or both for the specified component is higher than the current version. This package was probably created on a new version of DTS or the component than is installed on the current PC.]] at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper90 wrapper, Int32 lPipelineVersion)
Error 3 Error loading BMAG Download Xref Tables - bmag.dtsx: The component metadata for "component "DataReader Source" (1113)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
I have a package which reads an Access file from a folder. My connection manager to this file is .NET providers for OledbMicrosoft Jet 4.0 OLE DB Provider.
Package works from my computer. But when I execute it on the server as a SQL Agent job, I get
The component metadata for "component "DataReader Source" (1) could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
I copied the mdb file to a folder on the server which my packages have no problem reading data from.
My packages run under the same domain account as defined in proxies.
In a Data Flow, I have the necessity to use a SSIS variable of type €œObject€? inside Script Component and assign to it the content of 'n' variables of string type. On exiting from the script the variable of type object should contain something like in the following lines: AAAAAAAAAAAAAAAAAAAAAAAAAAAAA BBBBBBBBBBBBBBBBBBBBBBBBBBBBB CCCCCCCCCCCCCCCCCCCCCCCCCCCCC DDDDDDDDDDDDDDDDDDDDDDDDDDDDD €¦€¦€¦€¦€¦€¦€¦. €¦€¦€¦€¦€¦€¦€¦. On exiting from the data flow I will use the variable of type Object in a Script Task, by reading each element in a cyclic fashion. Is there anyone who have experienced something like this? Could anyone provide any example of that? Thanks in advance!
Hi all I'm into a project which uses a lot of views for joining 2 or more tables. Using the MERGE component in SSIS will be a huge effort coz it only has 2 inputs and I gotta SORT the input too. Isnt it possible to have a VIEW like component that joins more than 2 tables and DOESNT need sorting?? (I've thought about creating views in database engine but it breaks my data floe in SSIS and is'nt a practical solution)
I am writing a custom dataflow transformation component and I need to get the name of the preceeding component.
I have been trying to find a way to get a reference to the Package object, MainPipe object or IDTSPath90 object (connecting to the IDTSInput90 of my component) from my component because I think from there I can get to the information I want.
No idea where this bug crept in from. Have been using SSIS for 1.5 years now without hitting this problem.
I had a script component opening an XML document and parsing it using XPATH. I added some code that uses StreamReader / Streamwriter (closing one stream before starting the other). The code works without issue in my C# app.
And it ran without issue 2-3 times in SSIS. Then suddenly after running my package again, the script component says it completes successfully, yet nothing happens. I set a breakpoint on the first line of code - it never hits it. I add a msgbox as the first line of code - and it never displays.
I then close my package / exit out of ssis ... and then re-open it. When i open my script component, all of my code is GONE. All references that I added are gone.
I tried adding the streamreader/writer process to a dll I created from my c# app ... and added the DLL to the package -- same result.
I can reproduce this on 2 different computers.
Anyone experience this problem ? Any idea how to stop it ? Or debug it ?
Here is a slimmed down code sample of what causes the error :
Public Class ScriptMain Public Sub Main() Try Dim xmlDoc As New XmlDocument xmlDoc.Load("c:ulkasync_86281519_20070628045850225_4.xml") MsgBox("xmlLoaded") --this doesn't display once the package starts "acting up" Catch ex As Exception MsgBox(ex.Message) UpdateXML("c:ulkasync_86281519_20070628045850225_4.xml", ex.Message) End Try Dts.TaskResult = Dts.Results.Success End Sub Private Sub UpdateXML(ByVal fileName As String, ByVal message As String) Try Dim invalidChar As String = message.Trim().Substring(message.Trim().IndexOf("0x"), 4) Dim rd As StreamReader = New StreamReader(fileName) Dim xml As String = rd.ReadToEnd() Xml = Xml.Replace(invalidChar, String.Empty) xml = xml.Replace("", String.Empty) xml = xml.Replace("<![CDATA[<![CDATA[", "<![CDATA[") xml = xml.Replace("]]>]]>", "]]>") MsgBox("replaced") rd.Close() Dim wr As StreamWriter = New StreamWriter(fileName) wr.Write(xml) wr.Close() Dim xdoc As XmlDocument = New XmlDocument() xdoc.Load(fileName) Catch ex As Exception UpdateXML(fileName, ex.Message) End Try End Sub End Class
How can we get the name of the component inside the Data Flow Task . What I want is to log error stating which component in the data flow task has failed. Package and Data Flow names I am getting from system variables. I want to log like the Execution Result screen with Name of the component and [its id]. Like "Derived Column[216]" has failed with some error It is possible?
I'd like to incorporate the "package component tree" UI component that is used within the Visual Studio designers into an SSIS utility I'm building. This is the one I'm talking about:
Edit: Apparently using the IMG tag on these forums works in the editor preview, but not in the actual posts, so I've replaced images with links...
Example 1 Example 2
I've done some searching online, but have not found any information about where this UI is implemented, or if it is reusable. Does anyone here know if it is possible to re-use this component in a .NET application?
Ok. I have 2005 standard with sp2 installed. Went to create a maintenance plan and low and behold the component is missing. Ok.. no big deal, i'll just add it. Of course you know this is the error I get.
To change an existing instance of Microsoft SQL Server 2005 to a different edition of SQL Server 2005, you must run SQL Server 2005 Setup from the command prompt and include the SKUUPGRADE=1 parameter.
Ok.. doesn't seem too bad. So I go to a dos prompt. d: oolssetup.exe skuupgrade=1
Hi, I'm trying to add an icon to a custom component but its not happening and I can't figure out why. I'm sure I've managed to do this in the past. Here's my DtsPipelineComponent attribute:
MyComponent.ico is stored in the same folder as my project (i.e. at the same level as the .cs file and the C# project file). In the application property page I've pointed the icon resource at MyComponent.ico
Everything compiles OK. But I see no icon in my component when I deploy it to the toolbox or use it in a package.
I'm creating a custom interface for reporting services but I am having a few problems. This is my environment:
Using the ReportViewer component in remote mode to show my reports of the report server. There are 2 user levels of which any user can be apart of, Admin & user. In every report, there is a parameter called MERCHANTID, when the user who logs in (via custom login interface) is of group admin, then the merchantid parameter gets prompted before the report is run, if the group of the user is "user" then the merchantid parameter is hidden, and is passed programatically to the report. The report path also
My Questions:
1. When my reportviewer component loads up the report and prompts for a parameter, when you select a parameter value or type it in, it just posts back with no results, just the parameter prompt again with no data... How can this be corrected?
2. How do I pass report parameters programatically... so that when my user id of group "user" then "merchantid" gets passed automatically, but if there is any more parameters, then that gets prompted...
i have 2 source columns testsource and testsource1 and 2 output columns
test and test2
i am passing them through a script component to check if the columns are numeric or not along with some more logic...I am able to get the first column evaluated based on the logic but not the second column
Is it that the script component can only look at one column?
this is the code i wrote
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
Public Class ScriptMain
Inherits UserComponent
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
I have an XML data file and an associated XSD file with properly defined datatypes. However, the datatype of all the data elements are always "string" datatype. For example, in my current xml file, all the data elements are of Decimal datatype which is properly defined in XSD file. However, datatype of all the output columns are of string datatype.
I have several SCD components in my project. As I have to process millions of records, SCD's are taking a lot of time. Is there a way to speed them up? Work arounds?
when loading the transformed data into OLE DB destination, there is no options to truncate destination table first. Have to insert a middle step to run script to truncate the destination table first.
I'm very confused. We even has the options of keeping or deleting the data in destination table in SQL2000 DTS package. Why we don't have this option in SQL2005??
I'm having some trouble getting a script component to work in one of my data flows. I'm loading data from a flat file and essentially, all I want to do is delete any data in the table that has the same period and fiscal year as the data in the flatfile. I was doing this with an OLE DB Command, and it worked, but very slowly. So I thought I'd try using a Script component to call a stored proc to handle things for me. The problem is, while the script component calls the proc and deletes everything exactly as it should (I've checked the counts in the table before and after), it then just sits there forever, and the package never completes. I imagine I must be doing something wrong with the script, since I don't really use VB all that much, but I can't seem to figure out where. Here's what I'm using
Code Snippet Public Class ScriptMain Inherits UserComponent Dim strFY As String Dim strPer As String
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer) With Row strPer = Row.DataConversionPeriod.ToString() strFY = Row.FiscalYear.ToString() End With Dim strConn As String = "data source=server; initial catalog=database; integrated security=sspi;" Dim sqlConn As New SqlConnection(strConn) sqlConn.Open() Dim cm As SqlCommand = New SqlCommand("FinancialHE_DeleteExisting", sqlConn) cm.CommandType = CommandType.StoredProcedure cm.Parameters.Add("@FiscalYear", Integer.Parse(strFY)) cm.Parameters.Add("@Period", Integer.Parse(strPer)) cm.ExecuteNonQuery()
I have a text file which is fixed length. I have a header, trailer and detail records. I need to validate thel records and load only the detail records to the SQL server database. So now in the data flow, I have input file transformation(1 column called "Line"), then i have the script component(parses the column "Line" - identifies header,detail and trailer and do validations. Then I have conditional split, which splits the detail records and the output transformation is SQL server.
My question here is I am basically a database developer, so coding in vb is new to me. So in the script component, how do i split the columns
something like this:
If Row.Line.Substring(1, 1).ToString().Trim() = "H" Then Row.RecType = "H"
If Row.Line.Substring(15, 2).ToUpper().ToString().Trim() <> "" Then Row.OutTranCode = Row.Line.Substring(15, 2).ToUpper().ToString().Trim()
End If etc etc.
. elseif Row.Line.Substring(1, 1).ToString().Trim() = "D" Then Row.RecType = "D"
else
endif
Is that how it works or do you have some better idea to acieve this. Hope I am clear.
i am not very experienced with the SSIS. I am just wondering if there is something like a "if-then-else"-componente like the foreach-component in SSIS.
I want to delete the values of all tables in one database. So I took a foreach-component and selected the smo-enumeration with all tables. I store the tablename in a variable and execute a sql-task with "delete table.." with the variable tablename as parameter. Now I want to delete all except one certain table. I would like to add a selection where the variable tablename is checked. If the tablename is this certain table, I don't want to execute the sql-command, else I want to excecute the delete-command.
I am using a Union All component in a Data Flow Task. I have 3 Input OLE DB Sources connecting to the 1 Union All component. And then the Union All component connects to OLE DB Destination component. Everything looks good, but for some reason, for every time the Data Flow Task runs, 3 rows are inserted into my table. Why is that? Isnt Union All joining all fields, and then just inserting 1 final row to my table. What am I doing wrong or what should i do to just insert 1 row in my table. Thanks and I am really anxious to see responses on this post. Thanks again.
I need to generate a file using script component. I'm taking some data from an oledb source and then i would need to modify the fields, generate new fields and then generate the flat file. I don't have any problems with this but the file i need to replicate would need to have a head record and a trail record which doesn't need to be duplicated. Now when i generate the file and put anything in "Public Overrides Sub Input0_ProcessInputRow" it will work but it will generate the header record again and again as well. Any quick help would be really appreciated.
I need to build a generic text search component for MS SQL server 7.0. It is basically one of those Database searches that appear on web sites. Is there any stored procedure or service that MS SQL server provides for text based search on a database. Mind u my requirement is not to search a table but aan entire database.... anyone done this before.....
