Custom Property For Remove Duplicates Transform Input Row

Oct 15, 2006

Im working through the MS example of "removeDuplicates". I cant seem to figure out how to add custom property for input column.

I added the helper method:
private static void AddIsKeyCustomPropertyToInput(IDTSInput90 input, object value)
{
IDTSCustomProperty90 isKey = input.CustomPropertyCollection.New();
isKey.Name = "IsKey";
isKey.Value = value;
}
I call it from:
public override void ProvideComponentProperties()
{
//...
AddIsKeyCustomPropertyToInput(input, false);
//...
}
public override void ReinitializeMetaData()
{
IDTSInput90 input = ComponentMetaData.InputCollection[0];
if (input.CustomPropertyCollection.Count == 0)
{
AddIsKeyCustomPropertyToInput(input, false);
}
// ...
}

However when I deployed it and added the component to SSIS package - I cant see the Custom Column "IsKey" in the input column properties window.
What am I missing - please help

View 3 Replies


ADVERTISEMENT

Value Of A Readonly Property Of Custom Task Is Not Updated In Property Window

Apr 17, 2008

Hi,

I developed a simple custom control flow component which has several read/write properties and one readonly property (lets call it ROP) whichs Get method simple returns the value of a private variable (VAR as string). In the Execute method the VAR has a value assigened. When I put the value of ROP or VAR into MsgBox I can see the correct value. However when I execute the component I can not see the value of the ROP in the property window. I see the property but its value is empty string. For example when I put a breakpoint to postexecute or check the property before click OK in a MsgBox I would expect that the property value would be updated in SSIS as well. Is there a way how to display correct values of custom tasks properties in property window?

Thanks for any hints.

View 3 Replies View Related

Expression Issue With Custom Data Flow Component And Custom Property

Apr 2, 2007

Hi,



I'm trying to enable Expression for a custom property in my custom data flow component.

Here is the code I wrote to declare the custom property:



public override void ProvideComponentProperties()

{


ComponentMetaData.RuntimeConnectionCollection.RemoveAll();

RemoveAllInputsOutputsAndCustomProperties();



IDTSCustomProperty90 prop = ComponentMetaData.CustomPropertyCollection.New();

prop.Name = "MyProperty";

prop.Description = "My property description";

prop.Value = string.Empty;

prop.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY;



...

}



In design mode, I can assign an expression to my custom property, but it get evaluated in design mode and not in runtime

Here is my expression (a file name based on a date contained in a user variable):



"DB" + (DT_WSTR, 4)YEAR( @[User::varCurrentDate] ) + RIGHT( "0" + (DT_WSTR, 2)MONTH( @[User::varCurrentDate] ), 2 ) + "\" + (DT_WSTR, 4)YEAR( @[User::varCurrentDate] ) + RIGHT( "0" + (DT_WSTR, 2)MONTH( @[User::varCurrentDate] ), 2 ) + ".VER"



@[User::varCurrentDate] is a DateTime variable and is assign to 0 at design time

So the expression is evaluated as: "DB189912189912.VER".



My package contains 2 data flow.

At runtime,

The first one is responsible to set a valid date in @[User::varCurrentDate] variable. (the date is 2007-01-15)

The second one contains my custom data flow component with my custom property that was set to an expression at design time



When my component get executed, my custom property value is still "DB189912189912.VER" and I expected "DB200701200701.VER"



Any idea ?



View 5 Replies View Related

Adding Custom Property To Custom Component

Aug 17, 2005

What I want to accomplish is that at design time the designer can enter a value for some custom property on my custom task and that this value is accessed at executing time.

View 10 Replies View Related

Custom Task - Custom Property Expression

Aug 16, 2006

I am writing a custom task that has some custom properties. I would like to parameterize these properties i.e. read from a varaible, so I can change these variables from a config file during runtime.

I read the documentation and it says if we set the ExpressionType to CPET_NOTIFY, it should work, but it does not seem to work. Not sure if I am missing anything. Can someone please help me?

This is what I did in the custom task

customProperty.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY;

In the Editor of my custom task, under custom properties section, I expected a button with 3 dots, to click & pop-up so we can specify the expression or at least so it evaluates the variables if we give @[User::VaraibleName]

Any help on this will be very much appreciated.

Thanks

View 3 Replies View Related

DTS 'Transform Data Task Property' GUI Window

Apr 15, 2005

I am importing data from xls file to a db table with a dts. In the time of the dts creation I am using 'Transform Data Task Properties' GUI window to map incoming xls fields (source) to the table columns (destination).
Question: Is there any way to invoke the 'Transform Data Task Property' GUI window in dts runtime and use it to change the mapping dynamically in the run time?
Thanks, Vadim.

View 3 Replies View Related

Lookup Transform Issuing One Select Statement Per Input Row

Oct 20, 2007

I am using the lookup transform with the following settings:

Reference table: Use results of an SQL query. The query retrieves the surrogate key and four business key columns from a dimension table which contains a few thousand rows.

Columns: business keys in the incoming data are mapped to the business keys in the reference table, and the surrogate key is looked up from the reference table.

Advanced: Enable memory restriction is OFF (and the other items on the Advanced tab are greyed out).

I assumed that this means that the lookup transform would cache all of the rows in the SQL query, and then perform the lookups against this cache. This is the behaviour that I saw when I was running the package in my local environment in the BIDS debugger.


However, a colleague was doing some profiling on our production database server, and noticed that the lookup transform is instead issuing a single SQL query for each row of input. Our production ETL server has many GBs of free RAM available (way more than enough to cache the contents of the lookup table in memory), and given that memory restriction is disabled, I don't understand why the lookup transform is behaving in this fashion. Does anyone have an explanation for this? I'm probably misunderstanding a key concept here.

Thanks,

Greg

View 8 Replies View Related

Check Db Input For Duplicates

Jun 4, 2008

I was just thinking about a situation...
 Let's say hypothetically, I have a textbox, that i would like someone to input their email address to be added to a mailing list. I would like to first check to see if that email address exists in the database, rather than run a sql statement to check, and then run the update command, is it better to run an IFEXIST() type thing in sql and do the code there?

View 4 Replies View Related

Remove Duplicates

Jan 9, 2008

I have a query which gives the following output, How can i get a output like this:

QUERY

COL1COL2COL3
A1AAGG
A1BBHH
A1CCJJ
B1DDKK
B1EELL
B1FFMM

OUTPUT

COL1COL2COL3
A1AAGG
BBHH
CCJJ
B1DDKK
EELL
FFMM

View 5 Replies View Related

Remove Duplicates On A Table

Jan 29, 2008

Hi All
I have the dbo.OperatingHour It has many duplicates and I want to remove duplicates permanently
The statement below works but when I open the table there are no changes

Insert into OperatingHour(Weekdays, Wednesdays, Fridays,Saturdays, [Sundays/Public Holidays])
(SELECT DISTINCT Weekdays, Wednesdays, Fridays,Saturdays, [Sundays/Public Holidays] FROM OperatingHour)

View 2 Replies View Related

Remove Neighbouring Duplicates

May 24, 2007

Welcome,how can I alter following table in order to reduce neighbouringduplicates (symbol, position, quantity, price).Nr Symbol Position QuantityPrice Date1. wz9999b 1 1.02500.0 2007-05-09 08:09:42.6532. wz9999b 2 12.02500.0 2007-05-09 08:09:42.6533. wz9999b 1 100.02590.0 2007-05-10 15:47:04.1404. PZ0008VX 1 2280.8842090.55000000000022007-05-1612:43:12.4035. PZ0008VX 1 2280.8842102.05000000000022007-05-1612:45:27.4206. wz9999b 1 0.0012500.0 2007-05-18 09:47:16.0337. wz9999b 1 0.0012500.0 2007-05-18 09:47:53.2708. wz9999b 1 1.01.0 2007-05-22 12:35:07.8939. PZ0008VX 1 2280.8842102.05000000000022007-05-2409:38:26.16010. PZ0008VX 1 2280.8842102.05000000000022007-05-2409:38:38.80011. wz9999b 1 0.001 2500.02007-05-24 12:35:07.20712 wz9999b 1 0.002 2500.02007-05-24 12:35:14.98713. wz9999b 1 0.001 2500.02007-05-24 12:38:07.207In the result set I would like to get the rows number 6 and 10.Any suggestions??

View 2 Replies View Related

Remove Duplicates Within Pipeline

Sep 27, 2006

I have a situation where we get XML files sent daily that need uploading into SQL Server tables, but the source system producing these files sometimes generates duplicate records in the file. The tricky part is, that the record isn't entirely duplicated. What I mean, is that if I look for duplicates by grouping the key columns, having count(*) > 1, I find which ones are duplicates, but when I inspect the data on these duplicates, the other details in the remaining columns may differ. So our rule is: pick the first record, toss the rest of the duplicates.

Because we don't sort on any columns during the import, the first record kept of the duplicates is arbitrary. Again, we can't tell at this point which of the duplicated records is more correct. Someday down the road, we will do this research.

Now, I need to know the most efficient way to accomplish this in SSIS. If it makes it easier, I could just discard all the duplicates, since the number of them is so small.

If the source were a relational table, I could use a SQL statement to filter the records to remove the duplicates, but since the source is an XML file, I don't know how to filter these out in the pipeline, since the file has to be aggregated to search for dups.

Thanks

Kory

View 5 Replies View Related

Can Someone Proofread My 'remove Duplicates' Script?

Oct 2, 2006

DELETE
FROM tblContacts
WHERE tblContacts.ID IN(
SELECT F.ID
FROM tblContacts AS F
WHERE Exists (
SELECT email, Count(ID)
FROM tblContacts
WHERE tblContacts.email = F.email
GROUP BY tblContacts.email
HAVING Count(tblContacts.ID) > 1
)
)
AND tblContacts.ID NOT IN(
SELECT Min(ID)
FROM tblContacts AS F
WHERE Exists (
SELECT email, Count(ID)
FROM tblContacts
WHERE tblContacts.email = F.email
GROUP BY tblContacts.email
HAVING Count(tblContacts.ID) > 1
)
GROUP BY email
)



I readily admit that I've shamelessly copied 'n pasted this from a tutorial and then taken a stab at tweaking it for my own ends. But I really don't understand what it's doing.

Really, all I want to know is that it will remove records with duplicate email fields. But I could also do with confirming - looking at the "SELECT Min(ID)" bit - does that mean that if it finds a duplicate, it'll delete the latest-added one? And if so, that changing it to remove the earliest-added one is simply a case of changing MIN to MAX?


Thanks :)

View 11 Replies View Related

Is There A Way To Remove Table Rows Duplicates

Dec 3, 2006

If we want to remove the duplicate row and leave only one row instead of 2 or 3 rows for example with the same column values.

2/ The same question but when all the columns of the row are duplicate except the id field.
Thanks a lot.

View 3 Replies View Related

Custom Dataflow Transform In Programmatically Built Package

Mar 4, 2006

I'm building packages programmatically and all is well. I have a new custom transform that I developed. It also works fine. Now I'm trying to add my new component to my packages when I programmatically build them, and I'm unable to do that.

Has anyone added their own custom components to a programmatically built package successfully?

I get a COM error on the line that calls ProvideComponentProperties. I've attempted various modifications including not overriding ProvideComponentProperties or just having it do nothing. I always get the same result. What I don't understand is that the custom transform works and handles ProvideComponentProperties fine when it is added to a package in BIDS.

Thanks.




'Add new dataflow task

combitronics = dataFlow.ComponentMetaDataCollection.New()

'Set assembly to my component

combitronics.ComponentClassID = "Ewrap.SSIS.Combitronics"

'Get instance of component

Dim instance As CManagedComponentWrapper = combitronics.Instantiate()

'Initialize component ***Error Occurs Here***

instance.ProvideComponentProperties()

...



View 2 Replies View Related

SQL Server 2008 :: Remove Duplicates From Query?

Oct 6, 2015

I am working with a bunch of records that have duplicates on the Persid and the intPercentID where there are duplicates I want to remove when I stick them in the temp table, I tried join on tempo table and doing not exists but still inserts, so now I am trying a merge but same thing. how can I keep duplicates from being inserted in the temp table. I made a cursor as well but its slow as heck, but it does work. trying better ways.

Create table #TempStr (STRId int not null Identity(1,1) primary key, Persid int, percentId int, dtCreated datetime, CreatedBy int)

Create table #NewStr (STRId int, Persid int, percentId int, dtCreated datetime, CreatedBy int)

INSERT #TempStr (Persid, percentId, dtCreated, CreatedBy)
select intPersonnelID, intPercentID, dtSubmitted, intSubmittedBy from tblSTR
whereintpercentId in (61,62) group by intPercentID, intPersonnelID, dtSubmitted, intSubmittedBy
UNION ALL

[code]....

View 3 Replies View Related

Transact SQL :: Remove Reverse Duplicates From Table

Sep 1, 2015

I have table with columns as ID, DupeID1, DupeID2. ID column is unique. DupeID1 and DupeID2 -- the combination should only be there once. I don't want reverse combination of duplicates, i.e. DupeID2, DupeID1 in the table. How can I delete the reverse duplicates from this table?

View 10 Replies View Related

Power Pivot :: 2 Tables - Remove Duplicates

Jul 13, 2015

I have 2 tables below:

Table 1:

Product No Quantity
A 1
B 2
C 3

Table 2:

Product No Grade Quantity
A Good
  A Normal
  A Bad
  B Good
  B Bad
  C Good
C Normal
C Bad

In Table 2, Product No divided by Grade. I want to lookup the Quantity from Table 1 to Table 2. The same Product No will have 1 value, the other value is 0. The result for Column Quantity should be like this:

Table 2:

Product No Grade Quantity
A Good 1
A Normal 0
A Bad 0
B Good 2
B Bad 0
C Good 3
C Normal 0
C Bad 0

View 8 Replies View Related

Remove Identify Property From A Column

Mar 1, 2002

Hi, I want to know how to remove identify property from a column without recreating the whole table...
When I do it in Enterprise Manager, it actually drop and recreate the table
in background. I just like to know if there is other way without recreating the tables. Thanks!
Xiao Tan

View 4 Replies View Related

Is There Any Way To Remove IDENTITY Property On Table?

Aug 1, 2007



Is there any way to remove IDENTITY property on particular table? I tried removing IDENTITY property using Manangement studio, but this operation behind the scene use migration concept that is by creating tmp table and then populating with data; droping the orginal and renaming the tmp back to original.

Second, i want some kind of generic solution using certain system table like aya.sysobjects, sys.identitycolumn etc such a way that i should be able to remove the idenity property from all of the table accross a database.

Mandip

View 5 Replies View Related

Extending Derived Column Transform With Custom Function Library

Feb 16, 2007

When data is imported from our legacy system, the same functions need to be applied to several columns on different tables. I want to build a kind of "Function Library", so that the functions I define can be re-used for columns in several packages.



The "Derived Column" transform seems ideal, if only I could add my list of user-defined functions to it. Basically I want to inherit from it, and add my own list of functions for the users to select.

Is this possible ?

What other approaches could I take to building about 30 re-usable functions?

View 7 Replies View Related

Custom Transform Component, Change Type Or Add Output Column

Jun 26, 2006

Would anyone happen to have any pointers or know of any good code examples to either programmatically change the type of an input column when it is passed through the component, or add a new column to the output? I am extracting data from an Oracle database which is in Julian date format (represented within SSIS as a DT_NUMERIC column) and I need to to either transform the input column holding it into a date column, or to dynamically add a new output column holding the transformed data.

Many thanks

View 1 Replies View Related

SQL Server 2014 :: Remove Duplicates In Opposite Columns

Jan 22, 2015

I have a table containing the following data:

LinkingIDID1 ID2
166202180659253178
166202253178180659
166334180380253179
166334253179180380
166342180380180659
166342180659180380
166582253179258643
166582258643253179
264052258642258643
264052258643258642
264502258643258663
264502258643259562

Within the LinkingID, there are duplicates in ID1 and ID2 but just in opposite columns. I have been trying to figure out a way to remove these set based. It doesn't matter which duplicate is removed. Essentially these are just endpoints and I don't care which side they are on. The solution must recognize the duplicates and not just remove based on every 2nd row.

View 8 Replies View Related

T-SQL (SS2K8) :: Multiple Contact / Possibilities - How To Remove Duplicates

Aug 11, 2015

I have a bunch of contacts that I've scored how well their names match to other contacts in the same business. I can programmatically figure out how to parse the results, but would like to know how to do this via SQL. My problem is for Business_fk 968976 I have 7 contacts. In the end I should have 4 contacts based on name match. For the business key listed Gerardo Lopez is in the ContactScore table twice for Contact keys 7355719 and 57028145. I then have two rows like so:

PossibleBusinessContactMatch_pk BusinessContact_fk Business_fk BusinessContactMatch_fk MatchTypeCode MatchScore MatchRank FirstName LastName Phone Email
------------------------------- ------------------ ----------- ----------------------- ------------- ----------- ----------- -------------------------------------------------- -------------------------------------------------- ---------- --------------------------------------------------------------------------------------------------------------------------------
1772960 57028145 968976 7355719 C 46 1 GERARDO I LOPEZ 8162214000
838834 7355719 968976 57028145 C 50 1 GERARDO

Each reference each other, and 2 is a good case, a more difficult case would have key 1 listed 10 times showing a ContactMatch_fk of 2 - 11, and then Contact_fk 2 listed 10 times with a ContactMatch_fk of 1, 3-11.I know 57028145 maps to 7355719 from the first row in the ContactScore table, so when Contact_fk of 7355719 comes up I should be able to skip it and not process that match. Hopefully that makes sense. Anyway here is the test data:

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[ContactScore]') AND type in (N'U'))
DROP TABLE [dbo].[ContactScore];
GO
CREATE TABLE [dbo].[ContactScore]
(
[ContactScore_pk]INT NOT NULL,
[Contact_fk]INT NOT NULL,

[code]..

View 9 Replies View Related

Integration Services :: Remove Duplicates In String And Take The Correct One

Apr 23, 2015

How can i perform this task with ssis  OR TRANSACT SQL? I HAVE THESE ROWS WITH THE NEXT DATA, I want to take just the valid one, BUT I HAVE A LOT OF COMBINATIONS AS following names, it can be animals, things or personal names

GABRIEL OBANDO --CORRECT
GABRIEL OVANDO
Gavriel OVANDO
gAbriel OBANDO
GABRIE OBANDO
Gabri OBONDA
MANAGUA --CORRECT
NANAGUA
NAMAGUA

View 5 Replies View Related

Duplicates Again! UNION Join - Remove Records With Column Diff.

Sep 9, 2004

Hello All,

We all were new at one point.... any help is appreciated.

Objective:

Combining two 49,000 row tables and remove records where there is only 1 column difference. (keeping the specified column value removing the one with a blank.)

Reason:

I have 2 people going through a list, coding a specific column with a single letter value. They both have different progress on each sheet. Hence I am trying to UNION them and have a result of their combined efforts without duplicates.

My progress/where I'm stuck:

Here is my first query/union:

SELECT * FROM [Eds table]
UNION SELECT * FROM [Vickis table];

As shown above, I have unioned these 2 tables and my results removed th obvious whole record duplicates, but since 1 column is different on these, a union without criteria considers them unique.....

an example of duplicates that I must remove are as follows:

142301 - Product 5000 - 150# - S (Keep)
142031 - Product 5000 - 150# - "" <--- Blank (Remove)

I am trying to run another query on my first query results so I don't mess my first query up. Here it is:

SELECT DISTINCT [Prod #], [Prod Name], [Prod Description], [Product Type]
FROM [Combined Tables]
WHERE [Product Type]<>" ";


Please Help! Thank you in advance.

--------------------

5 minutes away from pulling my last one!

BaldNAskewed

View 7 Replies View Related

Integration Services :: Remove Duplicates And Maintain Dupe Log By Using SSIS

Jun 23, 2015

I had Excel file input & import to DB Table by using Data flow in SSIS.but it had duplicates so I dont use the Dupe Records

So I planned like below:

Method 1:
Here OLEDB Destination are Good Records(Without Duplicates)
        OLEDB Destination are Not Good Records(only Duplicates)
                    or
 Method :2
If I add a column(GOOD_RECORD) in DB Table and Should I update '1' for  top 1 record (for Good Record)  and remaining as '0' for other Records (for Dups)latter I utilize Through flag of GOOD_RECORD

 i.e.,,  select * from DB_TABLE where GOOD_RECORD='1' .

I think that Method :2 Advisable for Performance/flexible but Here How can I update by using SSIS(Data flow)    ????

View 4 Replies View Related

Reporting Services :: Remove Duplicates Based On A Specific Column

Jun 22, 2015

I have some duplicate values for my query results, about 200 duplicates out of 30000 rows.  Of these 200 duplicates I want to keep the ones that have a higher value for... 'UpdatedBatchID'. 

SELECT
    IR.Id                            as 'ID'
  , CAST(IR.Priority as varchar)    as 'Priority'
  , IRSupportGroupDN.DisplayName    as 'Support Group'
  , DATEADD(MI,DATEDIFF(mi,GETUTCDATE(),GETDATE()),IR.CreatedDate)    as 'Created Date'
  , DATEADD(MI,DATEDIFF(mi,GETUTCDATE(),GETDATE()),IR.ResolvedDate)    as 'Resolved Date'
  , SLOConfig.DisplayName            as 'SLO'
  , DATEADD(MI,DATEDIFF(mi,GETUTCDATE(),GETDATE()),SLOFact.TargetEndDate)    as 'SLO Target'
  , SLOStatusDN.DisplayName            as 'SLO Status'
  , SLOMetric.DisplayName            as 'SLO Metric'
  , SLOFact.UpdatedBatchId            as 'UpdatedBatchID'

View 11 Replies View Related

Custom Property Question

Oct 18, 2006

I have a custom transform, with a custom property of value 1. My question is, can I change that value during runtime with the value from a variable?



Thanks

View 6 Replies View Related

Readonly Custom Property

May 20, 2007

Is there any way to provide a readonly custom property in the ProvideComponentProperties() method of a custom PipelineComponent implementation?

View 7 Replies View Related

Sort Component, Remove Duplicates, Comparison Flags - Ignore Case

Feb 29, 2008

What happens when you add the Ignore Case flag into the mix?

I'm having a hell of a time - I'm dealing with an SCD situation using TableDifference component and I have both existing dimensions and new data coming in, each go through identical Case-Insensitive/Sort with remove duplicates, but I'm getting identical new and deleted records detected - I think because of ordering issues. I'm still trying to whittle the test case down, but I think data from all around the records I'm investigating seems to get sorted in between them, so I'm having trouble getting a small test case built.

I think the mixed case data is the root of the problem, and I think the design is bad, but before I go back to the technical lead, I need to understand enough to show that you cannot take two pipelines sorted and de-duped case-insensitively and then do a case-sensitive table difference operation.

View 4 Replies View Related

Errors With Custom Task (Could Not Get Value For Property ...)

Jul 5, 2007

I wrote a custom task following the outline on MSDN. I signed it and installed it into the Tasks folder and in the GAC.

When I go to an SSIS project and add my task, the properties window shows "Could not get value for property 'd61935d9-430b-4c93-9f3e-a29f720d8659'. Specified cast is not valid." (where the guid is different obviously) for many of the properties.

What have I done wrong?

Update: I know this isn't my code because I tried a simple task that just returns success and doesn't do anything. I get the exact same errors, so I must be installing it incorrectly.

View 3 Replies View Related

Custom Components - How To Evaluate A Property As An Expression?

Apr 11, 2008



Hi all,

I'm wondering if anyone's accomplished this before - I've been unable to find a whiff of info on how to do this so far.

I'm creating a custom component that I'd like to give a "Derived Column" type of ability to. By that, I mean I'd like to populate a property of my component with an expression (including references to input columns, package variables and functions) and be able to evaluate it at runtime - per row processed by the component.

I would also appreciate any information as to how to provide the interface to allow the user to build such an expression as well - is there a UI function in SSIS I can call to pop up the "expression builder"?


Thanks!

View 6 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved