I've written an asynchronous script component and I have created the output buffer. 500,000 rows go into the input but only 1500 to 2000 rows come out berfore I get an SSIS Object reference not set to an instance of an object Error. The error occurs at the AddRow method of the outputbuffer (that's how I know it's gone). Why does this happen? Is there a way to sync up the output buffer with the input buffer?
View 25 RepliesGood day everyone,
I'm experiencing a completely random warning from any given row count component within any given data flow task. It occurs sporadically. Whilst distracting, I don't see any adverse effects to the data after the packages complete. Can someone weigh in on this warning and let me know if it is indeed benign or what I maybe able to do to fix it?
Here's the warning:
"A call to the ProcessInput method for input 75997 on component "CNT Rows sent for STG table" (75995) unexpectedly kept a reference to the buffer it was passed. The refcount on that buffer was 4 before the call, and 5 after the call returned."
Thanks,
Langston
When overriding the PrimeOutput method in a custom component, you get as parameters the outputIDs and the output buffers (of type PipelineBuffer). using the outputIDs you can get IDTSOutput90 outputs.
View 5 Replies View RelatedHi
I have written a custom data flow component that counts the rows coming through the pipeline and adds the counts to an object stored in a package variable.
The component sometimes generates warnings with the following message - "A call to the ProcessInput method for input 358 on component "<Component Name>" (354) unexpectedly kept a reference to the buffer it was passed. The refcount on that buffer was 4 before the call, and 5 after the call returned.".
The component is used several times throughout ~60 packages. It does not always occur on the same instance, or even in the same package, from one occasion to another. When all the packages are executed (from a master package - some in parallel and some in series) the warning is generated several times (around 10), but the warning is not generated at all when the packages are executed individually from BIDS. The values of the refcounts vary from message to message, but the difference is always 1 more after the call than it was before. The server has SP1 and the hotfix package installed, but the warning was being generated before SP1.
I do not understand what could be keeping a reference to the buffer. The ProcessInput method is very basic:
public override void ProcessInput(int inputID, PipelineBuffer buffer)
{
try
{
if (!buffer.EndOfRowset)
{
totalRowCount += buffer.RowCount;
}
}
catch (Exception ex)
{
bool cancel = false;
ComponentMetaData.FireError(0, ComponentMetaData.Name, ex.Message, string.Empty, 0, out cancel);
throw new Exception("Could not process input.", ex);
}
}
Does anyone have any idea what could be causing this, or ideas for things I should be investigating?
TIA . . . Ed
I am using a script component to create the output buffer dynamically. I use the Outputbuffer.AddRow() call. I then set all the fields I want, and its added to the output and later inserted into the database. If a field value fails it causes an error, but the record is partially inserted upto the point where the set field command caused the error. So if I set 10 fields, and it fails on field 5 it inserts data for the 5 fields that worked and nulls into the others.
As a result I have a try catch clause, and if it fails I want to cancell the addition of the new row. Is there a command like RemoveRow(), rollback, etc that can be used to not insert the record in error?
Sample code..
Try
PaymentOutputBuffer.AddRow()
PaymentOutputBuffer.Sequence = pi + 1
PaymentOutputBuffer.RecordID = Row.RecordID
PaymentOutputBuffer.PaymentMethod = PaymentArray(pi)
Catch e As Exception
PaymentErrorOutputBuffer.removecurrentrow(??)
End Try
Hi Guys,
I created a SSIS custom component, transformation (Asynchronous) with one Input collection and 2 output collections.
The SSIS Package which includes the Component I created works well in the Business Intelligence Studio, but when the same Package is run in the 'Execute Package utility' It fails to run. ( when you Double click on the dtsx file)
The cause of the failiure is
public override void PrimeOutput(int outputs, int[] outputIDs, PipelineBuffer[] buffers)
method receives only one output buffer when executed using the 'Execute Package Utility' { outputs = 1 , buffer.Length = 1 } ( when executed in the BI studio, the method receives parameters of both the output buffers that I expect { outputs = 2 , buffer.Length = 2 } )
The property ComponentMetaData.OutputCollection.Count = 2 as well. Yet the PrimeOutput method provides only 1 buffer.
The Validation Succeeds on both instances, which I assume means that Meta Data is Provided Properly.
What would be the reason for the same pakage to run in 2 different ways like this,
What might I have missed out to do, to make the package run in different ways on 'Business Intelligence Studio' and 'Execute Package Utility'
Thanks a lot
Below are some of the lines from the ProvideComonentProperties Method which deals with the output Collection, Isn't this sufficient for the PrimeOutput to provide 2 output buffers?
ProvideComponentProperties()
public override void ProvideComponentProperties()
{
RemoveAllInputsOutputsAndCustomProperties();
base.RemoveAllInputsOutputsAndCustomProperties();
base.ProvideComponentProperties();
//other function calls
IDTSOutput90 output1 = ComponentMetaData.OutputCollection[0];
output1.Name = "Output1";
output1.Description = ".......................";
extracted.SynchronousInputID =0;
IDTSOutput90 output2 = ComponentMetaData.OutputCollection.New();
output2.Name = "Output2";
output2.Description = "..........................";
output2.SynchronousInputID = 0;
//other function calls
}
Stored procedure works:
PROCEDURE dbo.VerifyZipCode@CZip nvarchar(5),@CZipVerified nvarchar(5) outputAS IF (EXISTS(SELECT ZIPCode FROM ZipCensus11202006 WHERE ZIPCode = @CZip)) SET @CZipVerified = 'Yes' ELSE SET @CZipVerified = 'Not Valid Zip'RETURN
Need help calling and processing sp information:
protected void C_Search_btn_Click(object sender, EventArgs e){ SqlConnection con = new SqlConnection(ConfigurationManager.ConnectionStrings["localhomeexpoConnectionString2"].ConnectionString); SqlCommand cmd = new SqlCommand("VerifyZipCode", con); cmd.CommandType = CommandType.StoredProcedure; cmd.Parameters.AddWithValue("@CZip", UserZipCode_tbx.Text);
. . . how do you set up output parameter, call the SP and capture output. . . ? if (@CZipVerified == "Not Valid Zip") { TextBox5.Text = "Zip code not valid please re-enter zip"; } else { Continue processing page }
}
While using Aggregate Transformation to group one column,the rows of output sometimes larger than the rows returned by a T-SQL statement via SSMS.
For example,the output of the Aggregate Transformation may be 960216 ,but the
'Select Count(Orderid) From ... Group By ***' T-SQL Statement returns 96018*.
I'm sure the Group By of the Aggregate Transformation is right!
But ,when I set the "keyscale" property of the transformation,the results match!
In my opinion,the "keyscale" property will jsut affects the performance of the transformaiton,but not the result of the transformation.
Thanks for your advice.
Hi,
I am having problem with the CRecorset::Update() function.
I have declared a CRecordset object in client Application in VC++6.0.
I open the recordset in dynamic mode.
Before opening the recordset , I have obtained the UPDLOCK on the base table.( Obviously this UPDLOCK is in transaction).
When I try to call CRecordset::Update() it gives me exception 16931(There are no rows in the current fetch buffer).
I have define a clustered index on this table.
But I still get the exception 16931 on calling CRecorset::Update().
Any help is much appreciated.
Thanks in advance.
I cannot seem to get my flat file to write columns in error when inserting into a SQL table. I have tried a few examples from MS and did not get anything written to my flat file output. I have set the Source Error Output on this flat file and it uses a script task to created the error description and then write it to a Flat File Destination.
I am new to SSIS and have not had any formal training on it. However, I am very familiar with VS.Net/c# and SQL 2000 DTS - I need to get this working ASAP as there are 45 total flat files that need to be processed. Once I have this solved for one, the rest will follow suit.
If more details would help, I can provide them.
Kind Regards,
Ron
Hi there,
I'm getting an error in ascript transform shape.
When I try to write a value to an output column I'm getting an error saying the object reference cannot be set.
So my code looks like this - Row.OutputColumn = True
Any ideas??
I'm new to ssis (in case that isn't obvious!)
I want to set up a package to automate a transfer of data from a relational database to a datawarehouse. I would like to transfer rows daily, but I only want to transfer those which have not previously been transferred.
How/where do I tell the package/data flow task/ whatever to just transfer the daily differential from the last time it ran?
Thanks
I encountered the following error while attempting to preview an RDL report I was developing in VS2010 using SSDT:"The size necessary to buffer the XML content exceeded the buffer quota"
View 3 Replies View RelatedWe have a set of reports with same header section in all the reports. So while developing a new report i used to copy that header section to the new report with same dataset names (without any change) , but while rendering the report it is throwing error " The size necessary to buffer the XML content exceeded the buffer quota".
View 2 Replies View Related
Hi! I have a problem with cube processing. When i am processing a cube with 5 million rows it will stop responding and the process on the SQL server has been suspended with the waittype ASYNC_NETWORK_IO. The waittime changes both up and down.
However if i change the the view(the cube is getting the data from a view that gets the data from ONE table) to only return up to 35 rows it works. However, 40 and it will go into suspended again. And i can let it run for several hours without it finishing. Other cubes in the same database with more rows works fine.
If i just run the query that is used when processing in Management Studio it works fine.
I have SP2 both on the AS and DB server.
The DB server looks like this:
4GB ram
2 x 2.80 dualcore intel xeon
The AS server:
16 GB ram
2 x 3.00 intel xeon X5450 quadcore
Any ideas?
Edit:
And the AS service runs at 13%, in task manager it has one processor at 100 %.
SQL Server Profiler shows no activity on the AS server.
Hi
I have a master package that executes a series of sub packages run from a SQL Agent job. One of those sub packages has been stable for a week, running at least once per day, but it just failed despite having been run once already today with the same set of input data.
There were a series of errors showing in the event log for the Execute Package Task starting with "Buffer Type 15 had a size of 0 bytes.", then "The buffer manager failed to create a new buffer type.", then "The Data Flow task cannot register a buffer type. The type had 32 columns and was for execution tree 3.", then "The layout failed validation." and finally "Error 0xC0012050 while loading package file "C:[Package].dtsx". Package failed validation from the ExecutePackage task. The package cannot run.".
SQLIS.com reports the constant for the error code as DTS_E_REMOTEPACKAGEVALIDATION ( http://wiki.sqlis.com/default.aspx/SQLISWiki/0xC0012050.html ).
I then ran the package on my dev machine in BIDS and it worked fine, so I re-ran the job on the server and this time that package executed ok, but another one fell over but did not put anything in the event log.
Does any one have any idea what happened?
TIA . . . Ed
I am doing some calculations with a table and would like to know how to reference a previous row without having to alter the dates in a query at the start of each month.
For example I have to find a percentage change for a series of values. The table has three columns, is based on end of month dates, and the "return" column is the calculated column where the value is. In the example the return value for date 2006-04-30 is Index of (2006-04-30/ Index 2006-03-31)-1.
Date Index return
2006-03-31 6535 .05130
2006-04-30 6949.29 .06333
Hi;
I have built packages on SQL Server 2005 using the Import/Export wizard. The basic aim is to import a table from an Oracle database into my SQL database. It was running fine until recently when they Oracle database was upgraded to 10g (i think), and now something is wrong. If I run the saved package as administrator, it runs fine, but when I run it as part of a Agent schedule job, it seems to loose the password. The error message is not very descriptive ("The job has failed."), so i might be mistaken.
Help anyone ?!
Jaco
i have a few ssis packages which are scheduled as sql server agent jobs. now after a reboot of the server all the packages jobs don't work anymore. i had to reenter all the passwords for the connection manager in order to make them run again. ist there a way to store the passwords so that the keep available even after a reboot? because it's necessary to restart the server for maintenance resasons.
View 3 Replies View Relatedi have result set from such query :
resultSetOptions = System.Data.SqlServerCe.ResultSetOptions.Scrollable | System.Data.SqlServerCe.ResultSetOptions.Updatable;
queryText =
SELECT agvb_id as ID
,agvb_cat_id as Category
,agv_brn_id as Brand
,agvb_brn_name as BrandName
,agv_brn_area as Area
,agv_brn_sku_qty as SkuQuantity
,agv_brn_qty as Quantity
,agv_orderliness as Orderliness
,agvb_importance as Importance
FROM AGENTS_VISITS_BRANDS
WHERE (AGENTS_VISITS_BRANDS.agvb_agv_id = @agv_id)
AND (AGENTS_VISITS_BRANDS.agvb_cat_id = @dad_id)
ORDER BY agvb_brn_name
if i open resultSet with such qury it looses its updatable option.
If i remove from where 1 param (no mater which) everything runs ok.
I have a job which executes hourly.
Essentially:
Begin
Truncate table A
Insert into A
(Col1,
Col2,
Col3...
)
Select Value1,
Value2,
Value3...
From Table B
End
The insert operation query takes approximately 3.5 minutes to execute. What's occurring is the Table is immediately truncated, and there are no rows in the table for those 3.5 minutes.
How can I avoid having this gap - where there are no rows in the table for that period of time during the job execution ? The table could be locked, but that doesn't seem like the best solution.
Hi,
If you have two synchronous transformation components and the input of the second is connected to the output of the first, does the first transformation process (loop through) all rows in the buffer before outputting these rows to the second transformation? Or does the first transformation output each individual row to the second transormation as soon as it has finished processing it?
Thanks in advance,
Lawrie.
I inherited a database with Street, CrossStreet1 and CrossStreet2. I wish to read the row once and output 3 rows.
I know I can run 3 separate query's but why read through 1 million rows 3 times, if you can read through it once.
I can do this in SAS using the output statement.
Hoping someone knows how to do this in SQL.
:confused:
Hello,
I want two different set of rows in a single output. For example - the query gets records from the same tables, but first condition is a date range of 60 days and value = '1'
then the second condition is a date range of 180 days and value = '2'
Is it possible?
Thanks
I'll show my schema first, then I'll explain what I'm doing:
--------------------------------------------------
Events
--------------------------------------------------
ID | E_Title
--------------------------------------------------
--------------------------------------------------
EventOptionGroups
--------------------------------------------------
ID | EOG_EventID | EOG_OptionGroupID
--------------------------------------------------
--------------------------------------------------
OptionGroups
--------------------------------------------------
ID | OG_Title
--------------------------------------------------
--------------------------------------------------
Options
--------------------------------------------------
ID | O_OptionGroupID | O_Description
--------------------------------------------------
--------------------------------------------------
EventRegistration
--------------------------------------------------
ID | ER_EventID | ER_Name
--------------------------------------------------
--------------------------------------------------
RegistrantOptions
--------------------------------------------------
ID | RO_EventRegistrationID | RO_OptionGroupID | RO_Selection
--------------------------------------------------
There are several events. Each event has several different sessions (stored in EventOptionGroups), and each session has a certain number of options (stored in Options).
A user can sign up for an event, and their information is stored in EventRegistration. They can choose an option for each session in the event. For each option they choose, a new row is added to RegistrantOptions.
For each row in EventRegistration, I want to output the user's information, and then the option they chose for each session in the event. Like this:
----------------------------------------------------------------------
E_Title | ER_Name | OG_Title1 | OG_Title2 | OG_Title3
----------------------------------------------------------------------
Event | Bob | O_Description1 | O_Description2 | O_Description3
So in that example, that event had 3 sessions.
Right now, I can only output E_Title and ER_Name, I don't know how to output the session information
Hi,
I have a problem with a Merge Join providing no output (when it should have 1890 rows). My Data Flow Task has 4 OLE Data Sources, 3 Multicasts, and 1 OLE Data Destination. I am experiencing the problem near the end of my data flow where two Multicasts create two parallel flows of data (see Level 1 below). I have two Merge Joins which join one leg from each multicast with a leg from the other multicast (see Level 2 below). Then the two remaining legs use a Merge to get my destination output (see Level 3 below).
I am experiencing my problem with the Merge Join (input A2, B2) --> (output C2) transformation. The Merge Join providing output C1 appropriately outputs 1890 rows, but C2 outputs 0 rows. Both Merge Joins are identical. The data is identically sorted prior to entering the problematic Merge Join and a DataViewer (Grid) verified that the data is appropriately entering in. Merge Join (input A2, B2) --> (output C2) has 667 rows as input A2 and 1890 rows as input B2 (using an inner join, just like the other merge join), but C2 baffles me with 0 rows of output (when it too should have 1890). I receive no Ouput errors and the execution completes showing all green.
Level 1
Multicast (output A1, A2) [667 rows]
Multicast (ouput B1, B2) [1890 rows]
Level 2
Merge Join (input A1, B1) --> (output C1) [1890 rows]
Merge Join (input A2, B2) --> (output C2) [0 rows]
Level 3
Merge (input C1, C2) --> (output D1) [1890 rows]*
I read about mysterious behavior with Merge Joins and have attempted modifying my EngineThreads property to values between 2 and 10, with no luck. Any help/ideas would be appreciated.
Thanks,
Devin
* Should be 3780 rows
I'm using a simple data flow to extract rows from a table (using a SQL query) and put them in a flat file. If the query returns no rows, I don't want a file to be created. Right now it creates a file with the headers (since I do want the headers if there is data).
Any one know how to do this?
Kevin
I've got a Derived Columns component as part of a data flow. On this I've set the error output for my columns to Redirect Row in all cases. I've set a data watcher on the error output and then ran the package.
There are several rows which I'm expecting to fail - about 3 of them. These fail but there are also another 697 which seem to have no problem. So I fixed one of the problem columns in the source data and then re-ran the package. I only updated one row in the source so this row no longer appeared in the error output, but neither did several hundred of the other rows.
Is it possible that the error output has been tripped for that one row but for some reason it sends several hundred more rows? The ids on these additional rows follow on from the erroneous row, and when I fix that row the rows following it no longer appear in the error output.
I am currently working on a project where I want to load information from a web service into a table in SQL Server. I wrote a web service to returns a dataset. Next, I went into SSIS and added a Web Service Task. I had the web service task call the method and output the result into an .xml file. I then wrote a data flow to load the XML file, using the Web Source, into the database, using the OLE DB Dest. I then set the connection manager on the XML Source to use inline schema, because the returned dataset from the web service has the schema information included. Next, I picked the columns I wanted and proceded to pick the table and mappings I wanted from the OLE DB Dest. However, when I execute the package the .xml file gets generated correctly but none of the rows get added in the database. How can everything be set up "correctly", but no rows gets added? Is there a specfic format that an XML source must be for SSIS to use it correctly?I would think a dataset that is generated from a .NET web service should just work in SSIS. Any help would be greatly appreciated. Thanks.
I have a table that is returning rows from a table query. It seems I have done it before but I cannot seem to get the right procedure to obtain the values. I will paste in the code below in which you will see my bad attempts at accomplishing what I need.
Dim uid As String
Dim pw As String
Dim em As String, fn, ln, mi As String
Dim par As String
Dim Field, n, j As Integer
Dim JJ As Integer
[code]...
I have a stored procedure that I use to return Purchase Orders from our PO system. It returns the data rows for PO's that match the criteria passed in (including the page to show etc.) + it returns two output params, Number of rows and Number of Pages.
Using query analyzer I can confirm the query works exactly as we want. I cannot however seem to get the data out to our ASP.net app.
Here is a function that I use in one of my classes:
<code>
Function fnListPOsByCoordinatorIDPaged(ByVal strCoordinatorID As String, ByVal intPOStatusID As Int16, _
ByVal intUserTypeID As Int16, ByVal intArchived As Int16, _ByVal intPageNum As Int32, ByVal intPerPage As Int32, _ByVal strConn As String) As SqlClient.SqlDataReader
Dim dr As SqlClient.SqlDataReader
SqlConnection1.ConnectionString = strConn
prListPOByCoordinatorPaged.Parameters("@CoordinatorID").Value = strCoordinatorIDprListPOByCoordinatorPaged.Parameters("@POStatusID").Value = intPOStatusIDprListPOByCoordinatorPaged.Parameters("@UserTypeID").Value = intUserTypeIDprListPOByCoordinatorPaged.Parameters("@Archived").Value = intArchivedprListPOByCoordinatorPaged.Parameters("@PageNum").Value = intPageNumprListPOByCoordinatorPaged.Parameters("@PerPage").Value = intPerPage
SqlConnection1.Open()dr = prListPOByCoordinatorPaged.ExecuteReader(CommandBehavior.CloseConnection)
Me.Pages = prListPOByCoordinatorPaged.Parameters("@Pages").ValueMe.Rows = prListPOByCoordinatorPaged.Parameters("@Rows").Value
If Me.Rows / intPerPage > Me.Pages Then Me.Pages = Me.Pages + 1End If
Return dr
prListPOByCoordinatorPaged.Dispose()SqlConnection1.Close()SqlConnection1.Dispose()
End Function
</code>
It does not crash, it returns my data reader with the correct records. Unfortunately my property values are returned as 0. They should have values.
Anyone know how to do this?
Thanks.
Hi,
Im trying to output the number of rows in a table returned in a report, into a textbox for a total record count
How would I do this ?
thanks
How to summarise the data in this table to a single row output per site (2 records for every SiteID). I am based in the UK so the data copied from SQL is in the US format. I can convert this to UK date without any issues.
CREATE
TABLE [dbo].[MRMReadings](
[SiteIncomeID] [int] IDENTITY(1,1)
NOT NULL,
[SiteID] [nchar](5)
NOT NULL,
[Code] ....
Is it possible to return the data in the following format merging 2 lines of data into one output:
SiteID ReadStartDate ReadEndDate ReadStartIncome ReadEndIncome
L0020 19/05/2015 05:00 20/05/2015 05:00 85.98 145.98
L0101 19/05/2015 22:07 20/05/2015 22:14 1,936.08 1,438.89
L0102 20/05/2015 21:16 19/05/2015 21:48 143.65 243.5
I am using SQL 2008 R2.