Consolidating Multiple Lookups
Feb 18, 2007
In many of my packages I have to translate an organizational code into a surrogate key. The method for translating is rather convoluted and involves a few lookup tables. (For example, lookup in the OrgKey table. If there is a match, use that key; if not, do a lookup of the first 5 characters in the BUKey table. If there is a match, use that key; if not, do a lookup of the first 2 characters... You get the idea.)
Since many of my packages use this same logic, I would like to consolidate it all into one custom transformation. I assume I can do this with a script transform, but then I'd lose all the caching built into the lookup transforms.
Should I just bite the bullet, and copy and paste the whole Rube Goldberg contraption of cascading lookup transforms into each package? Or is there a better solution I'm overlooking?
View 4 Replies
ADVERTISEMENT
Feb 19, 2008
I am having a bit of a problem over here. I am trying to consolidate dates from multiple records into a time line that has no date overlap. I'll give you an example to make things clear.Let's say I have 3 data records:RowID BeginDate EndDate Price ($)----------------------------------------------------1 01/01/2008 01/10/2008 1.002 01/05/2008 01/15/2008 2.003 12/20/2007 02/01/2008 1.50 The result I would like to see should look like this: 12/20/2007 - 12/31/2007 the price was 1.5001/01/2008 - 01/10/2008 the price was 2.50 because row 1 and 3 overlap.01/05/2008 - 01/15/2008 the price was 3.50 because row 2 and 3 overlap.01/16/2008 - 02/01/2008 the price was 1.50 because of the row 3. Any idea on how I can automate generation of this data?I have a lot of code written for that but I can't get the result I want.I don't know if someone wants to see my code, I got around 500 lines of it.I would appreciate any help with this.Thanks!
View 3 Replies
View Related
Jan 4, 2015
I was messing around with stored procedures and I was wondering if creating a SP that populates a single table for reporting is a good idea?
I have ~10 queries that I have to currently run manually and was hoping to drop them into a physical table and then leverage that single table to pull into excel.
Some of my queries use virtual tables or CTE's, this is to get the aggregate set correctly.
Essentially I am working out of a data warehouse and would like to eventually get all my queries in one SP or something similar and then call that query for a insert.
Speaking of which could you create a SP that has several selects than with that output drops the records into a single table by using an insert into query so the data from the all the queries would line up into the right columns?
View 1 Replies
View Related
Nov 16, 2006
Let's say I have 4 columns coming from my OLE DB source.
Column1
Column2
Column3
Column4
I also have a table that I'll be using in a lookup, LUPTable. In LUPTable, I have two fields, LUPField, ReplaceField.
In my data flow, I need to take columns, 2-4, and look them up against LUPField in LUPTable. I then need to add the value of ReplaceField (when a match is found) into the data flow.
The problem that I'm running into is that I don't want to sequentially do the lookups in the dataflow, because that's just a waste of time/memory. I only need to build the in-memory lookup table once, because that exact same data (it is static, for the most part) will be used for the remaining lookups.
What is the best way to achieve this?
The goal is to have the following columns remaining in the dataflow:
Column1
NewColumn2 (containing value from ReplaceField)
NewColumn3 (containing value from ReplaceField)
NewColumn4 (containing value from ReplaceField)
Column2-4 can be dropped from the dataflow after the lookups.
Thanks,
Phil
View 7 Replies
View Related
Jan 9, 2008
I have implemented a single lookup and would like to know the optimal approach to implement multiple lookups within a single €śdata flow task€? i.e. my question was if I had to look up multiple reference tables to obtain surrogate keys. I am oversimplifying for illustration purposes€¦
Source Table
Destination Table
Customer
Customer
Customer Name
City
State
Name Key
City Key
State Key
ABC
Los Angeles
CA
1
2
1
MNO
Boston
MA
2
1
2
Reference Tables
Customer Name
Customer City
Customer State
PK Key
Customer Name
PK Key
City
PK Key
State
1
ABC
1
Boston
1
CA
2
MNO
2
Los Angeles
2
MA
View 10 Replies
View Related
Jul 31, 2006
I have a table (thanks to r937 and others) that list the 3 closest stores to each customer. I now want to take this data and 'group by' the customer so it shows only one record with the 3 closest stores in separate fields within the record. I am still new with SQL so if this is not possible or extremely easy, i apologize. Thanks in advance.
What it looks like now:
Code:
Cust# ClosestStore
customer1 508
customer1 604
customer1 414
customer2 579
customer2 987
customer2 417
customer3 751
customer3 487
customer3 113
What I want it to look like:
Code:
Cust # closestStore1 closestStore2 closestStore3
Customer1 508 604 414
Customer2 579 987 417
Customer3 751 487 113
View 1 Replies
View Related
Feb 1, 2008
I need help figuring out how to consolidate duplicate records. For example We have a termed client same info until you get to the number of live because they are different. How can I take a both these lines and have them combined. There are other clients that are dups too. Please help if you can.
25-001610270-00000-00013 WEXFORD HEALTH SOURCES 12/31/200747
25-001610270-00000-00013 WEXFORD HEALTH SOURCES 12/31/200775
View 7 Replies
View Related
Sep 26, 2005
Let's say I have two tables:CREATE TABLE dbo.OldTable(OldID int NOT NULL,OldNote varchar(100) NULL) ON [PRIMARY]GOANDCREATE TABLE dbo.NewTable(NewID int NOT NULL IDENTITY (1, 1),OldID int NULL,ComboNote varchar(255) NULL) ON [PRIMARY]GOALTER TABLE dbo.NewTable ADD CONSTRAINTPK_NewTable PRIMARY KEY CLUSTERED(NewID) ON [PRIMARY]GOOldTable's data looks like this:OldID OldNote----- -------1 aaa2 bbb3 ccc2 ddd4 eeeNewTable's data (which is derived from the OldTable) should look likethis:NewID OldID ComboNote----- ----- ---------1 1 aaa2 2 bbb + char(13) + ddd3 3 ccc4 4 dddHow can I combine the notes from OldTable where two (or more) recordshave the same OldID into the NewTable's ComboNote?
View 6 Replies
View Related
Apr 4, 2007
I am about to move 8 SQL 2000 clusters instances residing on 2 seperate MCS clusters (4 instances each with 2 nodes each active/passive) to one single MCS cluster (2 nodes active/passive). The SQL Cluster VMs will have the same DNS names and IPs, however the MCS VM and nodes will have different names.
The planned method for moving the DBs is just to stop all SQL services copy the system and user MDF and LDF files and then restart SQL. From everything I have read this should be fine. However here are my concerns on a cluster platform:
Will the change in node names on the target cluster be a problem when moving the master DBs over from source SQL VMs? Are the cluster nodes listed somewhere in the Master DB?
A couple of the SQL instance VMs are invovled in transactional replication. If all of the SQL files are copied over and the target VM has the same DNS and IP name, will there be a problem with the transactional replication when SQL is restarted?
Thanks,
Chris
View 1 Replies
View Related
Dec 4, 2007
I'm using a query to see how many times an action was recorded on a person. The query works, it returns this:
John Smith 1
John Smith 1
John Smith 1
Jane Doh 1
Jane Doh 1
Al Johnson 1
but I need it to return totals like this
John Smith 3
Jane Doh 2
Al Johnson 1
This is the query I am using:
Select Player.First_Name, Player.Last_Name, COUNT(Action.Employee_ID)
from Player INNER JOIN
PlayerVisit on PlayerVisit.Player_ID = Player.Player_ID
join Treatment on Treatment.Visit_ID = PlayerVisit.Visit_ID
join Action on Treatment.Action_ID = Action.Action_ID
group by Player.First_Name, Player.Last_Name, Action.Employee_Id;
View 4 Replies
View Related
Jun 5, 2007
Hello All,
I have some groups set up in a matrix that basically group by transaction names. I am trying to consolidate all but one into the same group. Right now I have the expression of...
Code Snippet
=iif(Fields!TestName.Value="Name Search","Name Search","Logon Function")
this consolidates the aggregate results and group fine but it does not label the groups as expected. "Name Search" comes up as "Name Search" but instead instead of "Logon Function", It displays the rolled up group as the name of the fist group field. Is there a way to make an alias inside of an expression?
View 1 Replies
View Related
Jul 8, 2015
Code:
SELECT DISTINCT LEFT([REPORTING_MONTH], 4)+'-'+SUBSTRING([REPORTING_MONTH],5,6) as REPORTING_MONTH, t.EMPLOYEE,
'' as COUNT_FTP,
CASE WHEN [MEDIUM_RCVD] = 'EMAIL' THEN COUNT(MEDIUM_RCVD) ELSE '' END AS COUNT_EMAIL
FROM [GPO].[dbo].[DW_SUBMISSION] as s
JOIN #TEMPActivity as t
[Code] ....
I'm trying to get the set to come out all on one line. REPORTING_MONTH, EMPLOYEE, COUNT_FTP, COUNT_EMAIL
But when I try null or '' it creates a second record and doesn't merge the two results.
View 3 Replies
View Related
Mar 15, 2007
I'm not quite sure if this is the correct forum to post this, if not please advise where should I post.
I have 7 databases with same structure, but different data in it, I need to have a query to consolidade some info from all of them in one report, is it possible just in onw script? how should I do it?
thanks,
Marcus
PS: I'm a beguinner in this so I apologize if the question seems stupid, or wrong.
View 1 Replies
View Related
Aug 24, 2001
I saw a DTS Lookup sample here on SWYNK, but I still have no idea what they're talking about.
Can someone dissect this piece of code and tell me what it does?
DTSLookups("TransformSpecialty").Execute(DTSSource ("LocalSpecialty").Value)
You can reference those tables they have in the example. http://www.swynk.com/friends/green/dtslookups.asp
Thanks!
Lost, ttlai
View 1 Replies
View Related
Feb 28, 2008
Hi,
My process passing 1,000,000 rows to a data flow with about 20 lookups to get the keys that I wantted .
Most lookups have small number of rows except one with over 5,000,000 rows. I cannot get the process to run (the process hanged) probably because of memory issue. Any clue where/how I can tune it.
Thanks
View 1 Replies
View Related
Nov 13, 2006
I want to do a lookup on date column. My lookup date is of type smalldatetime, and my date is of type datetime (date with time component). My lookup is failing because of incompatible data types.
How do I perform the lookup with date columns having date and time components?
View 1 Replies
View Related
Mar 17, 2008
Im having a little diff speeding up the below process
I have two tables table a contains 300,000 rows which are unique however the identifier can appear more than once. The fields im interested in are the identifier and a date/time field.
Table two also contains an identifier and a date field and again can contain multiple instances of the identifier with a variyity of dates.
For each row in table a i want to review the date and look up table b for the first date greater then or equal to the date linking by the identifier.
i have managed to do this via the code below however it takes 45 mins and i want to speed this up.
Select
a.*,
(select Min(DateB) as DateB From #tableB b where a.identifier = b.identifier and b.DateB >= a.DateA) asDATE
From #TableA a
View 2 Replies
View Related
Apr 28, 2008
Hello,
I have a source table with few million rows in it. As a part of transformation, I need around 10 lookups in 10+ different tables, all of them having few million rows each. I am looking for an approach that would be reasonably speedy and easy to manage future changes.
Here are some of the things that I have tried...
(1) If I implement as lookup components, they cause developer machine to go really slow and takes forever to run.
(2) I tried having OLE DB Source query to fetch required data up front. But the source query becomes very complicated which will be even harder for future changes. And this big query cause SQL Server to become unresponsive.
(3) Update queries on target table are also causing server to be unresponsive.
What would you guys suggest for this type of implementation?
Kapil
View 5 Replies
View Related
Sep 12, 2014
I have some SQL experience, but nothing past basic commands. I'm trying to take some data held by an application to use as CSV import into another application.I have two tables from an application, one holds references made in another.The first tables holds details about a person:
field1=name field2=age field3=country
Joe,50,1
Country is held as a number, then there is another table that holds all the countries:
field1=id field2=description
1,USA
2,France
3,Germany
I want to do a lookup where it returns:
Joe,50,USA
View 1 Replies
View Related
Mar 26, 2008
Hi,
We use lookups to join a few huge tables in SSIS (each has more than 40 million rows). The process took almost two days to complete when we select partial load on lookups. It stops/locks if we select full load on lookups.
We have a 32bit server so SSIS uses only 2-3GB of available memory no matter how big RAM we have. It seems the best solution for my problem is to move to 64bit server so SSIS uses up to 16GB of Ram.
For now I am researching for a remedy solution to get better performance from our current environment while we are waiting for the big server.
I’d like to hear your thoughts and options that may improve the performance of our package. Dose partitioning help? What else could be helpful?
View 4 Replies
View Related
Jul 6, 2006
I have a dimension table for Retail Order Size. Each row in the dimension has a Starting Value and Ending Value column pair. In TSQL, the correct RetailOrderSize key is found by using the BETWEEN statement, like so:
SELECT RetailLevelKEY
FROM dbo.DimRetailOrderSize
WHERE @Sec1Retail BETWEEN StartingValue AND EndingValue
Is there a Data Flow Task Transformation in SSIS that replicates this functionality, or some other way of getting to the same answer in SSIS?
Thanks in advance for your help
View 1 Replies
View Related
Apr 16, 2008
Whenever we open a task and choose the lookups tab ellypsis button we get this error
Microsoft SQL Server Management Studio has encountered a problem and needs to close. We are sorry for the inconvenience.
If you were in the middle of something, the information you were working on might be lost.
Any idea what might be causing this? Am I missing something?
View 3 Replies
View Related
Apr 25, 2006
When do a fact table load...I have to perform lookups against the dimension
tables. The dimensions tables I have support slow changes, however, and thus
have multiple rows for a single legacy key under different effective start
and end dates. In order to do this lookup, I have to not just join on the
legacy key, but also validate that the date of the transaction I'm loading is
between the effective date range of the dimension item.
It seems the Lookup task only supports equijoins. Am I missing something
here? How is this accomplished if you can use greater than or equal to and
less than type join conditions?
View 3 Replies
View Related
Dec 22, 2007
Ive got an ETL process I have written which takes about 10 million rows from a staging database and loads it into production database with an INSERT statement. The INSERT statement makes a function call to retrieve the surrogate key for each row. The function looks in a replicated copy of our production database so no load is on our production environment during this time.
So: INSERT INTO foo(...) SELECT name, address, zip, dbo.fnGetSurrKey( name, address)
It took about 12hrs to insert 6 million rows last night and Im wondering if there is a better way of doing this. Maybe a multithreaded way like SSIS might have.
Assuming my function is optimized as much as possible, does anyone have any tips for speeding this up?
Also, the machine this ran on has 16gb of RAM but was setup to use only 2GB during this process. I have already changed it to 12gb and restarted the process a week ago, but the change doesnt take affect until you reboot. Would I see a significant performance increase from that?
View 7 Replies
View Related
Feb 2, 2006
Is there an easy way to get the Lookup component to ignore case?
View 1 Replies
View Related
Nov 13, 2006
So I have three lookups in a row in my data flow. Basically they are doing data quality checks for me using a reference table.
I want to be able to take the error flows of the three lookups and merge them together (union all) so that I can insert the "errors" (or non matches) into a table.
Can't do it. Because SSIS deems non-matches as "errors" you automatically get the errorCode and errorColumn fields. When you try to union a lookup error output with another lookup's error output, you can't do it.
What I would like to see is a lookup act more like a conditional statment where you have three outputs of a lookup table: match found, no match found, and error. Either that, or I'd like to be able to edit the names of the errorCode and errorColumn fields.
Am I missing something here, or do I need to just add an OLE destination for each lookup error flow when I only want one? 'Course the problem then is that I want to count the number of rows that are in "error" across all of the lookups.
View 4 Replies
View Related
Jun 1, 2007
I am trying to make a SSIS package..
In the dataflow of my package, I must check from one table whether a row exists, and if that row exists, I should get some other row from another table, and update that..
I think to check whether a row exists, i should use "Look Up"
But cant we pass parameters to LookUP?
I am trying to use this SQL:
SELECT count(*) FROM ServicePackets where ID = ? and CHANGEDATE > ? and status = 1
I should get if that row exists or not only... (true or false)
How can i use parameters in LookUps?
View 7 Replies
View Related
Apr 10, 2008
OK I have this table I am grbbing from Oracle and I need to take selected columns and do a value lookup against another table: IE Here is a list of fields I get from Oracle:
Code SnippetENTRY_ID
SUBMITTER
CREATE_DATE
ASSIGNEE
LAST_MODIFIED_BY
MODIFIED_DATE
STATUS
SHORT_DESCRIPTION
CATEGORY
TYPE
ITEM
SCHEMA_NAME
SCHEMA_KEYWORD
ID
FULL_NAME
USERNAME
PAGER
PAGERPIN
PAGING_METHOD
RECORD_TYPE
GROUP_ID
GROUP_ALIAS
APPLICATION
NOTIFICATION_SERVICE
ARS_GROUP_NAME
GROUP_TYPE
AUDIT_TRAIL
FULL_NAME_
RESPONSIBILITY
PAGER_EMAIL_ADDRESS
Now I need to do a value lookup on these fields:
Code SnippetSTATUS
RESPONSIBILITY
In this example it is only 2 but in other conversions it could be 20 or more... Now here is my select statement for each Field (The ? being the FIELD from before IE Status or Responsibility):
The is for Status:
Code Snippet
SELECT VALUE AS STATUS_VALUE
FROM Field_Values
WHERE (NAME = 'Project Name') AND (ENUMID = ?) AND (FIELDNAME = 'Status')
This is for Responsibility:
Code SnippetSELECT VALUE AS RESPONSIBILITY_VALUE
FROM Field_Values
WHERE (NAME = 'Project Name') AND (ENUMID = ?) AND (FIELDNAME = 'Responsibility')
So for this the way I am doing it now is I have 2 "Lookup" Components setup... It works fine... However as I said when I get say 20 or so it gets really tiresome. I was wondering if I could feed it a EXCEL or XML file saying these are the fields that need a value lookup where:
FIELD - Project - FIELDNAME VALUE - OUTPUT VALUE
So with this example I would have a file saying something like this:
STATUS - Project Name - Status - STATUS_VALUE
RESPONSIBILITY - Project Name - Responsibility - RESPONSIBILITY_VALUE
Then it runs whatever and returns the *_VALUE for each row it goes through... Any suggestions?
View 12 Replies
View Related
Jul 20, 2006
Why can you not turn off the caching in a Lookup against Oracle?
I have an exceedingly complicated SQL statement like this -
SELECT OBJECT_ID, OBJECT_CODE FROM OBJECT_TABLE
If I turn off the cache for a lookup I get bombarded with this rubbish-
Error 8 Validation error. DFT Load STATUS: LKP Get RESULT_NO [128]: An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Oracle" Hresult: 0x80040E14 Description: "ORA-00933: SQL command not properly ended ". Update.dtsx 0 0
Error 9 Validation error. DFT Load DAY_STATUS: LKP Get RESULT_NO [128]: OLE DB error occurred while loading column metadata. Check SQLCommand and SqlCommandParam properties. Update.dtsx 0 0
I have tried modifying the Cache SQL Statement as well, but to no avail. I am using the MSDAORA.1 provider against "Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production".
Any ideas ?
View 7 Replies
View Related
Jun 19, 2007
Hi,
Please help me out in loading the fact tables
I had used lookup on DIM table to get my SUK and if I use union transformation to get the out put from each lookup and then loading the data with some condition the data in my fact is not loading in a proper format.
The union transformation is splitting the out put in to different records
Please do inform me about which transformation should be used to get the data from lookup tables.
Or please do inform me the approach to load the fact table in SSIS.
I€™m basically INFORMATICA resource and I€™m implementing in terms of INFORMATICA
View 6 Replies
View Related
Sep 28, 2006
Hello,
I'm trying to clean my data using fuzzy lookup algorithm though SSIS, but i get null values everywhere. This is what i did:
I applied the fuzzy lookup in a table (tblValues). As source table i have the tblValues, and as reference table in Fuzzy Lookup i have the tblValues as well, resulting null values in all fields/columns.
Do i have to create my own reference table? If yes, how do i do that and what values will i have in this table?I didn't understand how the reference table must be in order the algorithm to work. Any suggestions?
Thank you in advance!
View 2 Replies
View Related
Apr 8, 2008
Our existing DW's ETL was written in a very complex fashion by the previous team. They use DTS package lookups to read a row in the Source SQL Server database see if that row exists in the taget SQL Server database. If the row does not exist, they use ActiveX scripts to INSERT the row in the target SQL Server database. If it exists, they update the row on the target side. How would you do this in SSIS? Apologize if this sounds like a basic question, however, I would have done this via Stored Procedures or SQL Scripts especially since it involves SQL Servers alone. Appreciate any help.
View 6 Replies
View Related
Feb 4, 2006
I notice that SQL Server 2005 creates worktables where SQL 2000 does not. Often these work tables appear in STATISTICS IO, but they show a 0 scan count and 0 logical reads. These worktables often appear to be substituted for bookmark lookups.
Has the optimizer decided to use worktables instead of bookmark lookups (often resulting in a higher cost plan)?
Sharon
View 7 Replies
View Related
