Update Logic In Warehouse ...
Feb 5, 2008
Hi,
I am facing a small problem while updating the warehouse using SSIS package.
Here is an sample scenario.
I have a staging table T1 with some important columns as stateid,ename etc ..
and target table as Trg with only two columns stateid and statecnt.
The Rule is, each state will be sending an excel file which basically contains state and employees belonging to the state information.
For the first time when i execute the SSIS package, i need to load statetid and count_of_employees for that particular state.
say for example.
SELECT stateid,
count(ename)
from T1
where stateid = ?
For the next time if the same state information is received then we need to update the count to the existing record in the Dataware house rather than creating a brand new record once again.
I am little bit confused of implementig this logic.
Can anyone help me out in accomplishing this task.
What can done or what steps is involved in achiving the task.
Thanks in advance.
View 2 Replies
ADVERTISEMENT
Feb 5, 2008
Hi All,
I am facing a small problem while updating the warehouse using SSIS package.
Here is an sample scenario.
I have a staging table T1 with some important columns as stateid,ename etc ..
and target table as Trg with only two columns stateid and statecnt.
The Rule is, each state will be sending an excel file which basically contains state and employees belonging to the state information.
For the first time when i execute the SSIS package, i need to load statetid and count_of_employees for that particular state.
say for example.
SELECT stateid,
count(ename)
from T1
where stateid = ?
For the next time if the same state information is received then we need to update the count to the existing record in the Dataware house rather than creating a brand new record once again.
I am little bit confused of implementig this logic.
Can anyone help me out in accomplishing this task.
What can done or what steps is involved in achiving the task.
Thanks in advance.
View 3 Replies
View Related
Nov 8, 2004
Posts: 20
Joined: Aug 2003
Mon November 08, 2004 9:42 AM
I have a table where from 1 to 3 fields will be update (lastname,firstname,birthdate). Any combination of the 3 can be updated. I need a trigger to handle the updates which update some other db tables. I can handle the update trigger part, the ? is how to best approach the logic. I can do thee separate statements to handle an update for each of the columns, but that could cause 3 separate updates to fire. I was looking at doing something like
If UPDATE(last_name)
and UPDATE(first_name)
and update(date_of_birth)
BEGIN
...
end
else if
If UPDATE(last_name)
and UPDATE(first_name)
begin
...
end
View 1 Replies
View Related
Jan 25, 2006
I am dealing with two tables and I am trying to take one column from a table and match the records with another table and append the data of that column.
I used an update query that looks like this:
UPDATE Acct_table Set Acct_table.Score =
(Select Score_tbl.Score from Score_tbl
Where Acct_table.Acctnb = Score_tbl.Acctnb
This process has been running for over an hour and a half and is building a large log file. I am curious to know if there is a better command that I can use in order to join the tables and then just drop the column from one to the other. Both tables are indexed on Acctnb.
Any insight would truly help.
Thanks!
View 4 Replies
View Related
May 9, 2008
Hi guys
We have a table X with a gender column taking values M(male) or F(Female).There are 52 rows in the table X.We want to update all the rows with Gender M as F and update all remaining F as males (M) in a single Update Query.Im looking for the exact logic we would write in the single update statement.
Please help me out.
Thanks in advance
Regards
ARvind L
View 2 Replies
View Related
May 1, 2006
I have a stored procedure that I need to either perform an update if a record exists or an insert if the record doesn't exist.
Here is my procedure:
CREATE PROCEDURE dbo.Insert_Temp_ContactInfo
@sessionid varchar(50),
@FirstName varchar(50),
@LastName varchar(50),
@SchoolName varchar(50),
@address varchar(50),
@City varchar(50),
@State int,
@Zip varchar(5),
@Phone varchar(10),
@Email varchar(50),
@CurrentCustomer varchar(3),
@ImplementationType int,
@ProductType int = null,
@Comment varchar(500)
AS
--check if a current record exists
SET NOCOUNT ON
SELECT FirstName, LastName, SchoolName, Address, City, State, Zip, Phone, Email, CurrentCustomer, ImplementationType, ProductType, Comment
FROM dbo.Temp_ContactInfo
WHERE sessionid = @sessionid
SET NOCOUNT OFF
--if exists update the record
UPDATE dbo.Temp_ContactInfo
SET
FirstName = @FirstName,
LastName = @LastName,
SchoolName = @SchoolName,
Address = @address,
City = @City,
State = @State,
Zip = @Zip,
Phone = @Phone,
Email = @Email,
CurrentCustomer = @CurrentCustomer,
ImplementationType = @ImplementationType,
ProductType = @ProductType,
Comment = @Comment
WHERE sessionid = @sessionid
--if the record does not exist, then insert a new record
INSERT INTO dbo.Temp_ContactInfo (sessionid, FirstName, LastName, SchoolName, address, City, State, Zip, Phone, Email, CurrentCustomer, ImplementationType, ProductType, Comment)
VALUES (@sessionid, @FirstName, @LastName, @SchoolName, @address, @City, @State, @Zip, @Phone, @Email, @CurrentCustomer, @ImplementationType, @ProductType, @Comment)
GO
I think I can use an if statement. Can anyone help me out with this?
Thanks.
View 3 Replies
View Related
Nov 8, 2006
hi.I am having probelms with an update statement. every timei run it, "every" row updates, not just the one(s) intended.so, here is what i have. i have tried this with both AND and ORand neither seem to work.i dont know why this is elluding me, but i'd appreciate help with thesolution.thanks.UPDATE addSET add_s = 1WHERE add.add_status = 0 and add.add_email = 'mags23@rice.edu'or add_s in(SELECT a.add_sFROM add a, edit eWHERE a.email_address = e.email_addressand e.public_name = 'professor')
View 22 Replies
View Related
May 14, 2008
please need rescue- complex update logic
this is my table
1
2
3
4
5
EMPID
fld1
fld11
fld111
fld2
fld22
fld222
fld3
fld33
fld4
fld44
fld444
fld5
fld55
fld555
1111
A
B
C
7
8
9
G
H
I
J
K
L
M
N
2222
N
M
L
K
J
I
H
G
F
E
D
C
B
A
3333
1
2
3
A
B
C
C
E
Y
I
O
W
Y
P
i need to update for example the eployee 1111 with employee 3333
but with swap ( take the value of employee 1111 in field- fld2,fld22,fld222 and swap value between employee 3333
in field- fld2,fld22,fld222 )
Code Snippet
---update eployee 1111 with employee 3333
-so
if i put the value 2
than ------------------ swap value between 2 employee
set empid1= 1111
set empid2=3333
value_swap=2
if value_swap=2
than
update fld2,fld22,fld222
with fld2,fld22,fld222
------------------- take the value of employee 1111 in field- fld2,fld22,fld222 and swap value between employee 3333
--------------------in field- fld2,fld22,fld222
value_swap
=1
=2
=3
=4
=5
EMPID
fld1
fld11
fld111
fld2
fld22
fld222
fld3
fld33
fld4
fld44
fld444
fld5
fld55
fld555
1111
A
B
C
A
B
C
G
H
I
J
K
L
M
N
2222
N
M
L
K
J
I
H
G
F
E
D
C
B
A
3333
1
2
3
7
8
9
C
E
Y
I
O
W
Y
P
Code Snippet
---update eployee 2222 with employee 1111
-so
if i put the value 5
than ------------------ swap value between 2 employees
set empid1= 1111
set empid2=2222
value_swap=5
if value_swap=5
than
update fld5,fld55,fld555
with fld5,fld55,fld555
------------------- take the value of employee 1111 in field- fld5,fld55,fld555 and swap value between employee 3333
--------------------in field- fld5,fld55,fld555
=1
=2
=3
=4
=5
EMPID
fld1
fld11
fld111
fld2
fld22
fld222
fld3
fld33
fld4
fld44
fld444
fld5
fld55
fld555
1111
A
B
C
7
8
9
G
H
I
J
K
W
Y
P
2222
N
M
L
K
J
I
H
G
F
E
D
C
B
A
3333
1
2
3
A
B
C
C
E
Y
I
O
L
M
N
TNX FOR ALL THE HELP I GET IN THIS Forum
View 7 Replies
View Related
Nov 17, 2006
I have got a business logic update conflict handler working, but I have had to work round what appears to be a bug.
Please can someone confirm if this is indeed a bug €“ and if it is a known bug?
My conflict handler needs to take some columns from the publisher row and some from the subscriber row in the event of conflict.
I can quite happily generate a custom dataset which contains the winning row that I want €“ I can see that because I can step through the conflict handler with debug when a conflict occurs.
However, just returning ActionOnUpdateConflict.AcceptCustomConflictData from the UpdateConflictsHandler method does not set the publisher and subscriber columns correctly. I end up with different values on the two databases.
I have found that the only way to get the correct rows on both publisher and subscriber is to create a new ADO connection to the publisher and actually perform an update €“ updating all the modified columns. This now works reliably in my testing.
Fortunately, due to business rules the frequency of update conflicts are likely to be very infrequent, but I would very much like to avoid having to do the €˜unnecessary€™ update.
Notes:
I am using column level tracking €“ but I have seen the problem with row level tracking too
I have mainly been using SP1 but have repeated the test on a configuration using the SP2 CTP €“ and the problem occurs there too
The problem is not due to complex logic in my code. If the method just sets customDataSet = publisherDataSet.Copy and then returns ActionOnUpdateConflict.AcceptCustomConflictData, the changed and winning publisher values are not sent to the subscriber
Any thoughts would be much appreciated
View 5 Replies
View Related
Oct 20, 2014
I have a stored proc that contains an update which utilizes a case statement to populate values in a particular column in a table, based on values found in other columns within the same table. The existing update looks like this (object names and values have been changed to protect the innocent):
UPDATE dbo.target_table
set target_column =
case
when source_column_1= 'ABC'then 'XYZ'
when source_column_2= '123'then 'PDQ'
[Code] ....
The powers that be would like to replace this case statement with some sort of table-driven structure, so that the mapping rules defined above can be maintained in the database by the business owner, rather than having it embedded in code and thus requiring developer intervention to perform changes/additions to the rules.
The rules defined in the case statement are in a pre-defined sequence which reflects the order of precedence in which the rules are to be applied (in other words, if a matching value in source_column_1 is found, this trumps a conflicting matching value in source_column_2, etc). A case statement handles this nicely, of course, because the case statement will stop when it finds the first "hit" amongst the WHEN clauses, testing each in the order in which they are coded in the proc logic.
What I'm struggling with is how to replicate this using a lookup table of some sort and joins from the target table to the lookup to replace the above case statement. I'm thinking that I would need a lookup table that has column name/value pairings, with a sequence number on each row that designates the row's placement in the precedence hierarchy. I'd then join to the lookup table somehow based on column names and values and return the match with the lowest sequence number, or something to that effect.
View 9 Replies
View Related
Apr 21, 2015
I am currently working with C and SQL Server 2012. My requirement is to Bulk fetch the records and Insert/Update the same in the other table with some business logic? How do i do this?
View 14 Replies
View Related
Feb 19, 2008
This might not be the right place to post this but I'm going crazy on this....
Has anyone seen this error or know what this error is and how to fix it?
Error:
A connection cannot be made to the database.
Set and test the connection string.
Additional Information:
Either the user, (me), does not have access to the TFSWarehouse database, or the database does not exist. (Microsoft SQL Server 2005 Analysis Services)
Thanks,
Nici
View 2 Replies
View Related
Apr 8, 1999
hi, Can anyone list the type of error that will make @@error =1
I created a procedure to update a table based on a cutomer id, Id 7 doest not exist in table A, and I am suppose to have Not valid id, but in this case nothing happen I always get table a updated
thanks
Ali
begin tran
update table a
set title =' manager '
where id =7
if (@error <> 0)
begin
rollback tran
print 'not valid id '
return
end
ELSE
begin
commit tran
print 'table A updated'
end
View 3 Replies
View Related
Jan 15, 2008
Hello,
I'm here in my first attempt at using SSIS, and I'm creating a data warehouse for my company. I made a huge oversight as I was building my facts and dimensions, which is that while I created a new integer primary key for each table, I destroyed the referentiality between them. I hadn't planned that part out. When I went to go connect the tables at the end, I realized that even though I could create the FK's in each table, they wouldn't mean anything.
So I'm a little confused as to how to maintain the referentiality between the tables during the creation of the new warehouse tables. My only idea right now is to create a new table for each original entity in the relational system -- I'll call it a "LOOKUP" table -- that table would have 2 columns, their original GUID column, and a new integer id column. Then, when I create a new warehouse table, I can do a lookup to the proper LOOKUP table and assign the new PK or FK values so they'll continue to be consistent across the new warehouse tables, maintaining the references to each other.
Am I approaching this right?
Thanks,
Andy
View 1 Replies
View Related
May 7, 2003
Hi,
I am trying to learn the MDX tutorial(http://www.databasejournal.com/features/mssql/article.php/1550061) by william Pearson.It uses the example of "warehouse" cube.I am unable to proceed with the tutorial as I don't have this cube.Any links or articles on how to build this "warehouse" cube being used in the tutorial.
Thanks,
Copernicus
View 2 Replies
View Related
Dec 22, 2004
I am going to use Microsoft SQL Server to develop my data warehouse, but one thing makes me confused. Since Analysis Service can create a Star schema database, do I have to pre-set up a Star schema database for ETLed data? Basically, I am wondering what's the relationship between an ETLed database and the one created through the Analysis Services.
Can any one give me an explanation from the implementation perspective?
Many thanks!
View 4 Replies
View Related
Sep 1, 2004
I am starting to load a data warehouse for a retention period of 10 years. My database backup plan is as follows -
1. Perform full back on Sunday.
2. Perform differential backup everyday from Mon - Sat.
3. Perform transaction log back-ups every hour on all days.
My recovery mode is going to be BULK-LOGGED at all times. I had a few questions / comments on the Maintenance Plan that I would be creating for the back-ups. My database name is Warehouse.
1. Differential Backups cannot be created via a Maintenance Plan. Only a full-backup gets created. Am I correct?
2. I shall be running Optimizations and Integrity checks prior to full-backup. Is this ok?
3. Remove files (both .BAK and .TRN) older than - I am thinking of having 6 days. I want only one full back-up at a time in the server. What settings can I use? I think the old back-up gets deleted when the new one is successful. What settings in the Maintainence Plan do I have to use to overwrite the previous back-up with the current one?
System Databases -
Should the settings for System Databases be the same as my Warehouse database?
The Maintenance Plan takes care of full-back up and TLOG back-ups. For Differential Backups I have to use the All Tasks from EM and specify the Differential Backup job. Correct?
All kinds of back-ups can occur in the database when it is active. Meaning, I have a job that loads data in the warehouse when a back-up is occurring simultaneously. Am I correct?
I do not intend to shrink the Transaction Log at any time, since it gets backed up every hour I do not expect it to grow to a large size. If I do have to shrink it, then I change the recover mode to Simple, shrink the log and then immediately do a full-backup and after that set the mode back to Bulk-Logged. Is the sequence of steps correct?
Please let me know your inputs.
Thanks,
Vivek
View 1 Replies
View Related
Feb 27, 2008
Hi all,
I am trying to restore my data warehouse from a January 2008 backup under a new name to recover a table that I accidentally deleted. It is taking a long time for the restore to get done. Here is the command I am running as sa in QA
---
RESTORE DATABASE Warehouse_new FROM DISK = 'H:MSSQLDataMSSQLBACKUPDBBackupsWarehouseWa rehouse_db_200801050600.BAK'
WITH
MOVE 'Warehouse_Data' TO 'G:MSSQLDataMSSQLDataWarehouse_New_Data.MDF',
MOVE 'Warehouse_Log' TO 'H:MSSQLDataMSSQLLogsWarehouse_New_Log.ldf'
----
There Warehouse_New_Data.MDF is 375 GB and the log is 12 GB.
There is still 169 GB of free space on the drive I am restoring to after the presence of Warehouse_Data.MDF and Warehouse_New_Data.MDF (each 375 GB).
Its been 4.5 hrs and the restore is still running. Backups take about 3.5 hrs to complete. Can I do any checks on the restore to see what point it is at? I stopped the restore using EM earlier after it took 8 hours and still no progress.
Please advise.
Thanks.
View 1 Replies
View Related
Jul 23, 2005
Has anyone seen a 25 TB data warehouse on SQL Server? I do not know ofsuch a thing, but maybe it exists out there.
View 6 Replies
View Related
Mar 15, 2006
Hello..I was wondering if anyone out there could tell me how they deal withNULL values in a data warehouse? I am looking to implement a warehousein SQL 2005 and have some fields which will have NULL values and Iwould like some further ideas on how to deal with them. At my last jobin dealing with Oracle we were just going to leave the fields NULL, butin SQL how would you best recommend cleaning the data? I greatlyappreicate your help and look forward to your reponses.Thank you
View 2 Replies
View Related
May 21, 2007
I€™m making warehouse for our HMIS (healthcare management information system)by using SSIS. I€™m facing some problems now, could you please help me to solve my problem.
Brief idea about my Warehouse:
Source: oracle 9i
Destination: Sql server 2005
ETL tool: SSIS
Problems:
How to refresh or load the current data to data warehouse.(now i'm using truncate sql task for deleting old/entire data for each packages, i really dont want to use in the production) . For example: The patient admissions data is adding everyday so i want to load the current data into my warehouse.
Could you pls suggest me good solution for this?
Refresh Cycle timings: is there any task available in SSIS?
current status:
First Time load completed, i set one Execute Sql statement ctrl flow task for Truncate the existing loaded data in the sql server 2005. and then again i process one data flow task for loading the data from oracle to sql server.
View 8 Replies
View Related
Jul 6, 2007
Hi, all,
Thanks for your kind attention.
Just have some enquries about some issues of data warehouse design and would like to hear from any of you for any prestigious forum on data warehouse?
Thanks a lot in advance and I am looking forward to hearing from you shortly.
With best regards,
Yours sincerely,
View 3 Replies
View Related
Apr 10, 2008
Is SSIS a tool for extracting realtime data from staging to data warehouse? Realtime in my case can be loading every 15 minutes but no more than 30 minutes. I've a data warehouse which data refresh once a day and it worked fine. The data that I extract into the warehouse is from a Staging database which is realtime replication of multiple production databases. Once a day, I've to have replication pauses on staging for a couple hours to refresh the data warehouse. That's the only way so SSIS can pull the data correctly; if I've replication on while SSIS pull data, it will always copy less rows than it supposed to.
I cannot afford to have replication pauses every 15 minutes just so I can refresh data warehouse. Does anyone every have this problem? or any best practice how to do this?
Thanks,
-Ash
View 2 Replies
View Related
Apr 25, 2008
Hi,
Can any one give some good articles and webcast links for Developing SSRS reports with Data WareHouse.
Regards
Sithender
View 1 Replies
View Related
Jun 26, 2003
Does anyone have a sample RFP for a Data Warehousing project?
My manager hired an outside consultant to draw up a proposal for our company. But it is getting stuck in details and we are way behind schedule.
It will help me greatly if there is an outline of a DW RFP.
View 4 Replies
View Related
Jun 9, 2004
Hello, everyone:
Does any body have the experience to execute data warehouse backup/recover? What I want to know is how to backup/recover database in data warehouse and cubes.
Thanks.
ZYT
View 5 Replies
View Related
Feb 28, 2008
At my office, we've been slowly working on putting together a data warehouse.
We're a financial services company and one of the services that we offer is debt collection. As far as reports go, our clients are interested in knowing how much money we collect over time. In particular, they want to know how many payments we've gotten 5, 10, and 15 months (and so on) after we receive a case. (Obviously, the 5-month payments are also included in the 10 and 15-month calculations).
When I wrote this report using our transactional database, I was completely new to SQL and the ever-resourceful Patron Saint took pity on me, so you can see a good description of the details at http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=78510
Now that I'm no longer a total newbie at SQL, and having been through a relatively extensive seminar on data warehousing, I've been entrusted with researching certain aspects of data warehouse development (rest easy, though, folks - the real DWH work is not being done by the very inexperienced me, but by an actual professional :) ).
My question:
how would you model this kind of "relative time" in a data warehouse? How would you display the 5-month, 10-month, and 15-month payments in a DWH? I can't really imagine that the kinds of joins necessary to do this in a transactional database would be desirable in a data warehouse.
We have the following:
1.) FACT_Payment: A fact table showing each each payment to the most detailed granularity. One attribute of this table is the payment date. Another attribute is a foreign key to the case dimension described below.
2.) DIM_Case: A dimension table showing information on each case, including the case start date. DIM_Case
3.) DIM_Date: A date dimension table.
(For added clarification: The FACT_Payment payment date has to be 5, 10, 15 months etc... after the DIM_Case start date.)
Any ideas, comments, experience with something like this?
Thank you.
View 7 Replies
View Related
Mar 12, 2008
Hi all,
I'm working on my first data warehouse and I'm not sure how I should name the columns in the database.
The first phase of the data warehouse is to store a bunch of data from one third party source. The source contains over 100 pieces of data and the business user doesn't even know what some of the fields are but he wants to store everything. The third party refers to the each field with a somewhat cryptic short name and a longer description. The short name isn't always cryptic.
My question is am I better off naming my columns the same as the source system's short name so that I can easily debug problems later? Should I instead try to shorten their definition into something meaningful? On a side note, I'm 100% positive that we'll never populate the tables in questions with data from an additional source.
Thanks!
View 3 Replies
View Related
Jun 14, 2006
Regarding the code/db from the REAL project that just got released, I have no problem attaching the "REAL Sample V6" database, but the "REAL Warehouse Sample V6" database requires ENTERPRISE edition, because the default copy uses Partitioning (i.e. the PT version, although the document stated that the multi-table (MT) version is the default). I only have the STANDARD edition of SQL 2005, is there a workaround?
Appreciate any help.
View 7 Replies
View Related
Jun 1, 2004
I'm reviewing a data warehouse design schema for a client that is following Kimball's data warehousing principles. One of the first things I noticed was a table of dates with expanded columns giving such information as the year, month, month name, fiscal year, quarter, etc for each date, They also have a surrogate key (int) for the date value. The fact tables store the surrogate key rather than the date value itself.
They were very surprised when I questioned the purpose of this table, assuring me that Kimball was very strong on the concept of having a date dimension for each table.
I don't see the purpose of a table containing nothing by derived date formats. I think they will get a bigger performance hit from having to link through the surrogate key than they would suffer from having to convert datevalues stored in the fact tables.
Has anybody else ever seen this before? Does Kimball really advise this?
View 14 Replies
View Related
Jul 20, 2005
Hi,I would like to know if anyone out there really uses SQLServer 2000 (which edition?) to hold the data for a datawarehouse? How much data does it handle efficiently?TIAFrank
View 1 Replies
View Related
Jul 20, 2005
Hello all,I just started a new job this week and they complain about the length oftime it takes to load data into their data warehouse,which they do once a month.From what I can gather, they rebuild the indexes before the insert with an80% Fillfactor, then insert the data (with theindexes enabled), then rebuild the indexes with a 100% Fillfactor.Most of my RDBMS experience is with a different product. We would havedisabled the indexes and Foreign Keys, loaded the data, thenre-enabled them, moving any records that violated the constraints into anappropriate audit table to be checked after.Can someone share with me what the accepted "best practices" are for loadingdata efficiently into a data warehouse?Any thoughts would be deeply appreciated.Steve
View 2 Replies
View Related
Jan 6, 2006
I have a question about staging design using SSIS. Has anyone come up with an ETL design that would read table names from a generic table and dynamically create the ETL to stage the table.
1. Have a generic table which would have table name and description and whatever else that was required.
2. Have a master ETL that would enumerate through the table and stage all the table names found in the generic table.
This way I wouldn't have to create an ETL which would hardcode the names of 300-500 tables and have the appropriate 300-500 data sources and targets listed.
Not sure if I am making sense but I hope someone understands the attempt.
thanks
View 10 Replies
View Related
