Dynamic Column Sorting After Merge Join
Feb 25, 2008
Hello,
I have data coming in from two sources, one being SQL and the other being Oracle. The end result needs to be a CSV file with the columns in a specific order. I have a Data Flow task setup that takes both sources and does a Merge Join on them. I can add a Sort Transformation and manually set the sorting of all 156 columns that end up going to a CSV file destination. However, I have a table setup that holds the names of the 156 columns and the order that the CSV file expects them to be in. I would much rather do this step dynamically as the column names and order may change in the future. Anyone who has used the Sort Transformation for a large number of columns knows how tedious it can be and how adding a column in the middle will cause you to change the sort # for each of the columns that come after it.
So I added a Script Component between the Merge Join and the Flat File Destination hoping that I could alter the order of the columns there. However I added the following code and found that the SortKeyPosition is ReadOnly.
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim column As IDTSInputColumn90
For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
column.SortKeyPosition = 1
Next
End Sub
I was hoping to add some code to get the sort index from my table for each of the columns and set it to the SortKeyPosition. Has anyone out there done this before or seen an example that might point me in the right direction? I've searched for 2 days without coming up with much.
Thanks!
View 5 Replies
ADVERTISEMENT
Aug 7, 2015
I have a tablix that looks like this...
Â
30 Jul
31 Jul
01 Aug
02 Aug
03 Aug
04 Aug
05 Aug
Region1
0.00
0.00
0.00
0.00
100.00Â
100.00
0.00
[code]....
I would like to sort the last column which is a dynamic date (always the most recent from the last 7 days) from 0 to 100.
View 11 Replies
View Related
May 4, 2007
Hi,
I have three tables.
The "Master table has a recordid, a masterID, a "IsSubField" and other stuff.
I need to do a join to a second table based on the MasterID...
However for each record, if the "IsSubField" has a True value then it has to use table A to JOIN to, where as if it's False, then it uses table B.
Make sense?
Anyone got any pointers?
Thanks in advance
James
View 8 Replies
View Related
Jun 23, 2015
Not sure if this is ever going to be possible, but I'm trying to do a dynamic join on EXECUTE, but the execution string has to come from a table column.... i.e
select * from (select table1.theSql,table1.userid from table1 ) as a
inner join (execute a.thesql) as b
on a.userId=b.userid
View 9 Replies
View Related
May 7, 2015
How do I pass a single column of values from a successful merge join to an EXECUTE SQL statement so it can be used with an "IN" criteria of the WHERE clause? Â Here's an example of my update statement with two random key values:
UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK IN ("XYZ123", "DEF890")
Is this even possible in SSIS, or am I better off using a loop and running the update EXECUTE SQL Statement for each individual key value, as in the following example?
UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK = "XYZ123"
UPDATE dbo.MyTable SET MyStatus = 1 WHERE MyPK = "DEF890"
View 6 Replies
View Related
May 2, 2007
If anyone could confirm...
SQL Server 2000 SP4 to multiple SQL Server 2005 Mobile Edition on PDAs. My DB on SQL2k is published with a single dynamic row filter using host_name() on my 'parent' table and also join filters from parent to child tables. The row filter uses joins to other tables elsewhere that are not published to evaluate what data is allowed through the filter.
E.g. Published parent table that contains suppliers names, etc. while child table is suppliers' products. The filter queries host_name(s) linked to suppliers in unpublished table elsewhere.
First initial sync with snapshot is correct and as I expected - PDA receives only the data from parent (and thus child tables) that matches the row filter for the host_name provided.
However - in my scenario host_name <--> suppliers may later be updated E.g. more suppliers assigned to a PDA for use or vice versa. But when I merge the mobile DB, the new data is not downloaded? Tried re-running snapshot, etc., no change.
Question: I thought the filters would remain dynamic and be applied on each sync?
I run a 'harmless' update on parent table using TSQL e.g. "update table set 'X' = 'X'" and re-sync. Now the new parent records are downloaded - but the child records are not!
Question: I wonder why if parent records are supplied, why not child records?
If I delete existing DB and sync new, I get the updated snapshot and all is well - until more data added back at server...
Any help would be greatly appreciated. Is it possible (or not) to have dynamic filters run during second or subsequent merge?
View 4 Replies
View Related
May 22, 2015
I have two xml source and i need only left restricted data.
how can i perform left restricted join?
View 2 Replies
View Related
Aug 10, 2007
Scenario:
OLEDB source 1
SELECT ...
,[MANUAL DCD ID] <-- this column set to sort order = 1
...
FROM [dbo].[XLSDCI] ORDER BY [MANUAL DCD ID] ASC
OLEDB source 2
SELECT ...
,[Bo Tkt Num] <-- this column set to sort order = 1
...
FROM ....[dbo].[FFFenics] ORDER BY [Bo Tkt Num] ASC
These two tasks are followed immediately by a MERGE JOIN
All columns in source1 are ticked, all column in source2 are ticked, join key is shown above.
join type is left outer join (source 1 -> source 2)
result of source1 (..dcd column)
...
4-400-8000119
4-400-8000120
4-400-8000121
4-400-8000122 <--row not joining
4-400-8000123
4-400-8000124
...
result of source2 (..tkt num column)
...
4-400-1000118
4-400-1000119
4-400-1000120
4-400-1000121
4-400-1000122 <--row not joining
4-400-1000123
4-400-1000124
4-400-1000125
...
All other rows are joining as expected.
Why is it failing for this one row?
View 1 Replies
View Related
Sep 4, 2007
Hi,
I needed a way to sort dynamically in a Reporting Services report by selecting a Column Name dynamically from a dropdown list.
Thanks
Prajith
View 7 Replies
View Related
Jun 5, 2006
I have a merge join (full outer join) task in a data flow. The left input comes from a flat file source and then a script transformation which does some custom grouping. The right input comes from an oledb source. The script transformation output is asynchronous (SynchronousInputID=0). The left input has many more rows (200,000+) than the right input (2,500). I run it from VS 2005 by right-click/execute on the data flow task. The merge join remains yellow and the task never finishes. I do see a row count above the flat file destination that reaches a certain number and seems to get stuck there. When I test with a smaller file on the left it works OK. Any suggestions?
View 3 Replies
View Related
Oct 6, 2006
Hello everybody!I have a small table "ABC" like this:id_position | value---------------------------1 | 112 | 223 | 33I try to use a dynamic cursor as below.When the statement "order by id_position" in declare part of the cursor_abcis omitted - cursor work as it should.But when the statement "order by id_position" is used, cursor behave asstatic one.What's the matter, does anybody know?Code:declare @id_position as int, @value as intDECLARE cursor_abc CURSORFORselect id_position, value from abcorder by id_positionset nocount onopen cursor_abcFETCH NEXT FROM cursor_abcINTO @id_position, @valueWHILE @@FETCH_STATUS = 0BEGINprint @id_positionprint @valueprint '----------------------------'update abc set value=666 --next reading should give value=666FETCH NEXT FROM cursor_abcINTO @id_position, @valueENDCLOSE cursor_abcDEALLOCATE cursor_abcGORegardsLucas
View 2 Replies
View Related
Jul 20, 2005
Hello,I need to have a stored procedure, which performs sorting. Something likethis:CREATE PROCEDURE procname@sortby varchar(30)ASBEGINSELECT some, columnsFROM some_tableORDER BY @sortbyEND(of course, i know this won't work, but it gives the idea of what i mean)Is there a possibility to write a procedure which behaves like that? It isimportant for me not to have multiple procedures just for different sortingcriteria...Thanks,Mike
View 3 Replies
View Related
May 12, 2006
Hi,
In one of my packages, I have join data from 2 data sets. The problem is 'Join' requires both dataset to be sorted and I can't sort one dataset without 'sort' transformation. The real problem is since I have to sort 65million records, the sort operation sucks always. The other option is using 'Lookup', but since I have to do 'Range Lookup', it is taking hours and hours to process less # of records.
My question is, is there any other way, I can do join of 2 datasets without sorting?
View 9 Replies
View Related
Nov 7, 2006
Is there a way to do a super-table join ie two table join with no matching criteria? I am pulling in a sheet from XL and joining to a table in SQLServer. The join should read something like €œfor every row in the sheet I need that row and a code from a table. 100 rows in the sheet merged with 10 codes from the table = 1000 result rows.
This is the simple sql (no join on the tables):
select 1.code, 2.rowdetail
from tblcodes 1, tblelements 2
But how to do this in SSIS?
Thanks - Ken
View 2 Replies
View Related
May 6, 2008
I read that merge joins work a lot faster than hash joins. How would you convert a hash join into a merge join? (Referring to output on Execution Plan diagrams.)
THANKS
View 3 Replies
View Related
Apr 30, 2008
Hi,
We have a matrix report which displays columns in a default sorting order. This report columns vary dynamically depending on the user input.
e.g. If user wants to see the report for column Alfa, Beta , Gama then a report will be genarted with column Alfa, Beta , Gama sorted in alphabetical order.
Site
%Risk
Alfa
Beta
Gama
X
2
1
2
3
Y
10
4
5
6
However the users want the Columns to be sorted in the order which they provide the inputs e.g. if the user entered Gama, Alfa, Beta the report should display the columns in the same order instead of applying the default sorting order.
Site
%Risk
Gama
Alfa
Beta
X
2
3
1
2
Y
10
6
4
5
Any thoughts on ways to achieve this in SSRS matrix report would really help.
Cheers,
Viv
View 3 Replies
View Related
Aug 25, 2007
Hi Craig/Kamal,
I got your email address from your web cast. I really enjoyed the web cast and found it to be
very informative.
Our company is planning to use SSIS (VS 2005 / SQL Server 2005). I have a quick question
regarding the product. I have looked for the information on the web, but was not able to find
relevant information.
We are getting Source data from two of our client in the form of Excel Sheet. These Excel sheets
Are generated using reporting services. On examining the excel sheet, I found out that the name
Of the columns contain data itself, so the names are not static such as Jan 2007 Sales, Feb 2007 Sales etc etc.
And even the number of columns are not static. It depends upon the range of date selected by the user.
I wanted to know, if there is a way to import Excel sheet using Integration Services by defining the position
Of column, instead of column name and I am not sure if there is a way for me to import excel with dynamic
Number of columns.
Your help in this respect is highly appreciated!
Thanks,
Hi Anthony, I am glad the Web cast was helpful.
Kamal and I have both moved on to other teams in MSFT and I am a little rusty in that area, though in general dynamic numbers of columns in any format is always tricky. I am just assuming its not feasible for you to try and get the source for SSIS a little closer to home, e.g. rather than using Excel output from Reporting Services, use the same/some form of the query/data source that RS is using.
I suggest you post a question on the SSIS forum on MSDN and you should get some good answers.
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
Thanks
Craig Guyer
SQL Server Reporting Services
View 12 Replies
View Related
May 14, 2008
Hi all,
i have this column name "Effective Date". the data inside it is like this:
01-Aug-2006
01-Jan-2007
01-Jun-2006
01-Mar-2004
01-Nov-2003
01-Oct-2005
02-Oct-2007
05-May-2006
08-Oct-2003
1-Agustus-2005
10-Feb-2002
10-Jan-2005
10-Mar-2003
11-Feb-2008
12-Dec-2006
12-May-2008
14-Nov-2006
15-Jun-2004
19-Jun-2007
22-Mar-2007
23-Apr-2007
23-Jan-2008
26-Feb-2008
27-Jan-2005
27-Mar-2007
27-Sep-2007
How to sort the column? i've using Order by [Effective Date], but it doesnt work.
please help.
thanks,
Addin
View 8 Replies
View Related
Jan 5, 2005
Hi,
I would like to setup a merge replication from a main database (publisher), to around 100 client databases (subscribers), that contain only subset information for each distinct client: each client has then its own database to view and modify its own data (Filtered on rows and columns). The client databases will initially be hosted on the same SQL server Instance.
I don't want to setup manually 100 publications with static filters, and more may come in the future: a pain to setup and maintain because I have to configure the tables, the columns and the row filters (and joins) each time.
I would like then to setup one publication with a dynamic filter, to filter in a way or another on a specific client. Creating subscribers becomes then a piece of cake. The dynamic filter would apply based on a property specific to the client, but... HOW?
Filtering on HOST_NAME() will not work because several subscribers are on the same server.
Filtering on SUSER_SNAME() will not work because merge agent will always use the same user name for connecting to the publisher (using push subcription, all merge agents are on the same server), and I have not find out how it can be configured by merge agent: even if the merge agent jobs have different owner, it is always the SQL Server Agent login that is used to connect to the publisher (I am using windows authentication).
I was thinking about using DB_NAME(), and have specific db name for each client DB, but DB_NAME() provides the name of the publisher DB, not the subscriber DB. etc ...
What could I use in this case to dynamically filter on client data without having to fall into heavy replication administration and setup.
Thanks for your help,
Best regards
View 2 Replies
View Related
Jun 14, 2006
Brand new to SQL Server 2005 and replication, I was able to use online books info to get a replication topology set up (using wizards) and successfully replicate an article (a single table) to a SQL Server Express database.
I then cleared the subscription table and added a simple dynamic filter (colx = SUSER_SNAME()), which I can't get to work. Based on what I read, I'm thinking that SUSER_SNAME() is not returning the proper value because the merge agent is set to connect to the publisher and distributor by 'impersonating the process account'. This is on the subscription merge agent security form. However, 'using a SQL Server login' is greyed out and nothing I tried will enable it, including setting up logins and SQL user ids on the publisher and playing with entries on the publication properties data partitions page, etc.
I think that I have discovered only some of the pieces that I need and not enough details to find the rest and put them together. Any info on what to have set up where would be appreciated. Any suggestions for 'step by step' information that would help in addition to SQL Server 2005 Books Online would be great.
Thanks in advance.
View 8 Replies
View Related
Feb 2, 2007
Hi!
I'm a merge newbie and have a couple of questions. I'm about to setup a merge replication with Sql Server 2005 and Sql Server CE as a subscriber. Situation is like this, we have 10 service technicians using pda.
I want to each pda user have their own data. What I understand I need to use dynamic filter and SUSER_NAME()?? Do I need to create a "translation" table to map my system's UserId against SUSER_NAME? How have you solved this problem?
/Magnus
View 1 Replies
View Related
Jan 27, 2014
I need to sort on a column that has a mix of alpha/numeric...
oActual Sort: A1, A100, A14, A2, A222, A25 …
oFavored Sort: A1, A14, A100, A2, A25, A222 …
View 4 Replies
View Related
Mar 7, 2008
I have the column of type string in the database
Following is the data in that column
1
2
11
12
21
abc
If i sort the table with the help of this column then the output come in the following manner
1
11
12
2
21
abc
can i improve the order actually i want the output like 1 2 11 12 21 abc
View 2 Replies
View Related
Jul 20, 2005
We've just installed SQL Server 2000 on one of our servers and havenoticed a strange behavior. When clicking the column headings in thejob display in Enterprise Manager, the list is sorted (first clickascending, then descending) on the column clicked except for "Next RunDate." When we click on "Next Run Date," the list is reordered, butrandomly. Each time we click it we get a different order but neverascending or descending. This is true whether we're at the server oron a remote machine. We never saw this behavior with SQL Server 7.0.Any ideas? Thanks!
View 4 Replies
View Related
Apr 26, 2004
I have got a query in which a merge join is 99% of the cost .... and I am confused ... is not merge join supposed to be the fastest ??? Anyone seen this before ???
Any ideas why this could be happening ... and sorry ... do not ask me to post the code coz I will not be able to ...
View 14 Replies
View Related
Mar 3, 2008
Hi, all experts here,
Any advices for when will be a better way of using Merge join instead of other options?
Thank you very much and I am looking forward to hearing from you shortly.
Best regards,
View 2 Replies
View Related
May 5, 2006
All,
I need to use Merge Join transformation to join two sources. One is from a PIVOT transformation and one of the output columns is ISSORTED, the other is from an OLE DB Source using a query. The Merge Join transformation requires both input source have to be sorted. I cannot find the ISSORTED property on the OLE DB Source!!
I tried to use Derived/ copy transformations but cannot find the property also. How can set the OLE query sorted in order to use the MergeJoin?
Thanks a lot
View 4 Replies
View Related
Jun 15, 2007
Hello all,
I have a package where I use merge join for two sorted inputs and the output is stored in a raw file.
In another package, the raw file from above package is again merge joined with another sorted input. Now my question is....do we need to sort again the raw file from first package? or is it OK to set the isSorted property to True and define the sort keys?
Thank you.
View 5 Replies
View Related
Dec 17, 2007
I am new to this SSIS.
I have a simple join query like this
select a.id from tbl_a a, tbl_b b where a.id = b.id and I want insert the result to my temp table.
the query results is 1500 rows.
but when I use merge join in SSIS, it only inserts to my temp table 4 rows.
I use inner join and I already set the IsSorted to true and specify the sort position for the columns in both source tables
In tbl_a, there are one million rows, in tbl_b, there are 2000 rows.
I don't know why the merge join cannot work out my task.Is there other way that I can just run this simple join query in SSIS to copy the data?
Please help, thanks in advance.
View 6 Replies
View Related
Apr 30, 2008
Hello,
I have a Merge Join transformation and when i sort values in OLEDB source the merge join fails, but if i use a sort transformation it works! Why??
Best regards,
Fred
View 5 Replies
View Related
Sep 29, 2006
Hi,
I have a SQL Statatment:
SELECT * FROM TABLE1 AS A
JOIN TABLE2 AS B
ON A.X= B.X
AND A.Y= B.Y
When i execute this code in sql server returns 549 lines. I created a package with two oledb sources one for each table, sorted the tables with fields X and Y after placed a Merge Join with the fields:
A.Y join B.Y order 1
A.X join B.X order 2
both fields with the Join Key checked
But my package return 411 lines.
What's happened?? :(
When a i have the code:
SELECT A.X, A.Y, B.X, B.Y
FROM TABLE1 AS A
JOIN TABLE2 AS B
ON A.X= B.X
When i did the join only one field SSIS worked fine, sql server returns 622 and SSIS returns 622 lines.
Please help-me...
Thanks,
André
View 8 Replies
View Related
Aug 12, 2006
Hi ...
I am working on a project where the server version of application has vouchers from different entities. I have created a publication manually. My next step was to create a client subscription using rmo and to execute a pull. This part works fine. Code samples from http://msdn2.microsoft.com/en-us/library/ms147314.aspx
My next step would be to implement dynamic filtering using the guid of the entity as a parameter.
I dont want to use suser_sname() or host_name() as I want to use a fixed login for the replication for all users, and a client could have several host dbs (sql express, sql mobile)
My goal would be to pass a guid-value to the HostName Property of the MergePullSubscription class and convert it to an uniquidentifier and use it as a filter as I have not found any other way to pass a guid as a filter.
RMO-Code:
subscription.HostName = "4bb0e468-c68a-4253-ba82-f71c3a6e302d"
Filter:
SELECT <published_columns> FROM [dbo].[voucher] WHERE [entity_ID] = dbo.fx_ConvertHostToEntity()
Function:
create function fx_ConvertHostToEntity()
returns uniqueidentifier
as
Begin
declare @host nvarchar(50)
set @host = host_name()
declare @entity uniqueidentifier
set @entity = cast( @host as uniqueidentifier)
return @entity
End
When trying to set the filter sql server complains that a character string cannot be casted to a uniqueidentifier - so i can not set this filter. Is there a way to pass a parameter other then the username or the hostname as a filter?
SELECT <published_columns> FROM [dbo].[voucher] WHERE [entity_ID] =@entity, where @entity is a guid
Thanks for your support
Alex
View 6 Replies
View Related