Select Data From Huge Fact Tables

Oct 12, 2006

Hi,

I have a situation where I have 4 tables:

1. 2 Dimensional tables(Parent), DIM1 with 50000 rows, and DIM2 with 1000 rows

2. Fact 1 with 50 columns, 25 Million rows and with FK to DIM1 and DIM2

3. Fact 2 with 40 columns, and 25 Million rows and with FK to DIM1 & DIM2 tables.

Actually the fact 1 and fact 2 have same related data but since our Analysis cube person wanted the fact table not to have more than 50 columns we divided the tables into 2, but they have the same compound key.

Above said, I have a situation where I have to select all the columns, in both fact tables, and do a group by. I wrote the query and ran "Analyze Query in the Database Engine Tuning Advisor" for it. It gave bunch of recomendations about the statistics and indexes which I created. When I executed the query the result came up in matter of seconds, which was good.

In the query I had a condition having MarketName='Bridgeview' and DateID = 344 (FK of today-1).

When I wanted the data for last 30 days I changed to DateID in ( > FK of today -32 and < FK of today), the query responded and worked fine.

But when I changed the query to get MarketName='Aurora' (other than I used when I ran Tuning Advisor), the result returned is empty set. When I removed the MarketName condition, it is supposed to return all markets' data, but it returns only Bridgeview data.

I know the data is in the table for all markets, since reports are rendered from these fact tables for all of these markets(also ran queries to check the fact table data).

I am unable to point out the reason why the query behaves like this. It responds to the date change, but not to the MarketName change.

I really appreciate if anyone can help me point out the problem.

Thanks,

Venkat

View 3 Replies


ADVERTISEMENT

Tranferring Huge Data To Various Tables

Mar 13, 2007

Actually in my transformation i am transferring huge amount of data.

i have been using oledb command finally to dump my incoming data to respective tables.

For Example :

if you have two tables

table 1,table 2

in my incoming data i have a lookup and check for two unique columns with that of the unique columns in the table 1.if the record does not exsist i try inserting a record into table 2 and get the unique filed of the record and store that in particular column of table 1.

the data is very large an is this the better why or any suggesstions do let me know..



View 5 Replies View Related

Loading Data Into Dimensions &&amp; Fact Tables

Jul 10, 2007

Hello Everybody,



Can anybody please let me know the procedure for loading data into Dimensions & Facts? And what is the sequence of loading?



Thanks in Advance

Rajesh

View 1 Replies View Related

Data Warehousing :: Populating Fact Tables With Surrogate Key From Dimension Table?

Sep 11, 2015

How do I correctly populate a fact table with the surrogate key from the dimension table?

View 4 Replies View Related

Analysis :: Relating Multiple Fact Tables To Find Data That Exists In One But Not The Other

Nov 18, 2015

I'm currently setting up a Tabular Model to do some research between several fact tables.  In this example i have two fact tables (table 1 and table 2) which I've created a 1 to 1 relationship on phone number.  Typically I create a relationship between these tables to find common data between the two.  However, in this case I am trying to figure out the best way to model the data so that I can easily surface data from one table that does not exist in the other.  I would liken this to a LEFT JOIN or a WHERE NOT EXISTS in SQL.

Table 1 has all of the data and Table 2 Only has a subset of the data from Table 1.  What I'm trying to do here is display what attributes in Table 1 may play a part in records not existing in Table 2. What is the best way to model this?

View 3 Replies View Related

Using String Col From Fact Tables

May 11, 2005

Hi
I have a fact table in which i have a String column now i want to show it in the mdx queries..when i add it as a measure it shows some numerical values in the cube i cannot even add it as a member propeties due to some datwarehouse design constraints..so can anyone out there help me please....it urgent
Thankx in advance
regards
Hemant

View 3 Replies View Related

How To Compare Two Huge Tables

Aug 15, 2012

I need to compare if two developers did the job correctly and created identical tables.

The problem is more complex, but I will try to solve it somehow if I solve the problem of comparing two tables (let them be in different SQL Server 2008 databases) and their properties. No data needs to be compared.

View 6 Replies View Related

Huge Select Into.. (Can I Choose Not To Log?)

Dec 12, 2000

I have a job that selects alot of data from one database into another.

Can I choose not to log this operation (doesn't need to be and the log fills up before it's done)

Thanks..

type of code:

Insert into database_1
Select * from database_2

View 1 Replies View Related

Dimensions Vs. Factless Fact Tables

Jan 17, 2005

I am building a health care application that marries transaction-level data (health care services provided) with person-level characteristics that have a time-dimension. The person-level characteristics are diseases that the person has (these disease all have a start and some have an end date). The diseases are stored in a table in which the foreign keys are a person-identifier, a time identifier (month/year) and a surrogate for the disease. Persons can have more than one disease at a time (the diseases are NOT mutually-exclusive). There are no measures in this table. The transaction table has a foreign key for person and time (month/day/year), a procedure code (the type of service rendered) and money (the cost of the services).

How do I answer the following questions:

What is the total cost of care (the sum of all service costs) last year for persons with "disease A"?

What is the total cost of care last year for persons with "disease A" AND "disease B"?

What is the total cost of care last year for persons with "disease A" OR "disease B"?

I've tried a factless fact table but can't get it to work. If anyone has the right solution and can communicate to me before I slit my wrists, I would be greatly appreciative!!!

View 1 Replies View Related

Calculated Measure From 2 Fact Tables

Dec 6, 2007

I have a cube with 2 fact tables and I would like to create a calculated measure that uses values from both tables like:

fact1.measure1 / fact2.measure2

I can't see how to do this?

View 1 Replies View Related

From Dimension Tables To Fact Table

Feb 6, 2008

Hi All,

What is the best way to move data from Online system tro data warehouse?
I have created 3 dimension tables(product,date and customer tables) and
I wanna create fact table and get foreign keys from dimension tables.
What is the best method to do that in SSIS?

thanks

View 1 Replies View Related

SSIS (DImension And Fact Tables)

May 16, 2007

Hi,

I am new at SSIS and I am trying to create a Datawarehouse using SSIS. I have the data files as flat files I have the Dimensional Model ready on Paper and Now I need to use the SSIS for the ETL process.

I am trying to figure out how to make dimension tables in SSIS? I mean I want to create the 5 Dimension tables and then create a Fact table out of it but I cant understand where to start? Can any one tell me how we create Dimesion tables in SSIS. Like one of the dimesion tables I need to create uses 2 flat files and is like a flattened dimension, How would I create this in SSIS?

Even if there is any tutorial which shows this step by step do let me know. I would really appreciate any guidance on this.

Thanks,

Sarang

View 11 Replies View Related

Addding A Column To Dim And Fact Tables

Mar 6, 2008

Hi All,
I want you an urgent respones, I want to add a column to a Dim_table and Fact_table, the Dim_table is sourcing from different database table and now the new column is going to source from another DB, thus without changing the structure of the table relationship can i add a column? To make it clear Dim_table has got Clomun1 from table1, Clomun2 from table2, Clomun3 From table3, Clomun4 From table4. Now I want Clomun5 From table5. How do i add and let it source from table5.
Thank you for your qiuck response.

View 4 Replies View Related

Delete Duplicate Records From Huge Tables

Jul 20, 2005

Hi All,I've the following table with a PK defined on an IDENTITY column(INSERT_SEQ):CREATE TABLE MYDATA (MID NUMERIC(19,0) NOT NULL,MYVALUE FLOAT NOT NULL,TIMEKEY INTEGER NOT NULL,TIMEKEY_DTTM DATETIME NULL,IID NUMERIC(19,0) NOT NULL,EID NUMERIC(19,0) NOT NULL,INSERT_SEQ NUMERIC(19,0) IDENTITY(1,1) NOT NULL)GOALTER TABLE MYDATAADD CONSTRAINT PK_MYDATAPRIMARY KEY (INSERT_SEQ)GOThe TIMEKEY_DTTM field is generated, from the value actually insertedinto theTIMEKEY field, by the following trigger:CREATE TRIGGER TIMEKEY1ON MYDATAFOR INSERT ASBEGINDECLARE @M_TIMEKEY_DTTM DATETIMESELECT @M_TIMEKEY_DTTM = DATEADD(SECOND, INS.TIMEKEY +EP.GMT_OFFSET * 0 ,'1970-01-01 00:00:00.000')FROM INSERTED INS, LOCATIONINFO EPWHERE INS.EID = EP.EIDUPDATE MYDATASET TIMEKEY_DTTM = @M_TIMEKEY_DTTMFROM INSERTED INS, MYDATA MDWHERE MD.INSERT_SEQ = INS.INSERT_SEQENDGOThere is also a composite, non unique, index defined on thetuple:(MID,IID,TIMEKEY,EID)CREATE INDEX IX_METDATA ON MYDATA (MID,IID,TIMEKEY,EID)GOAs a consequence of an application design change, I would also changethis index to be UNIQUE, but when I try to drop and create it I get anerror, because the tables stores some duplicated rows...In order to succesfully upgrade the index definition, I wrote some DMLstaementsto lookup and remove the duplicated rows, keeping only the firstrecord inserted, i.e. the one with the lowest INSERT_SEQ:---- This table stores then umber of duplicated records eventuallydiscovered-- into the MYDATA table; the initial value for the NUM_DUPLICATESfield is-- 0 (no duplicated record)--DROP TABLE DUPLICATESGOCREATE TABLE DUPLICATES (TABLENAME VARCHAR(17),NUM_DUPLICATES NUMERIC(19,0) )GOINSERT INTO DUPLICATES VALUES ('MYDATA',0)GOINSERT INTO DUPLICATES VALUES ('CATEGORIESDATA',0)GO---- ///////// CLEAN UP OF MYDATA TABLE--DROP TABLE TMP_MYDATAGOCREATE TABLE TMP_MYDATA (MID NUMERIC(19,0) NOT NULL,TIMEKEY INTEGER NOT NULL,IID NUMERIC(19,0) NOT NULL,EID NUMERIC(19,0) NOT NULL,INSERT_SEQ NUMERIC(19,0) )GO---- Insert into the TMP_MYDATA table all the duplicated records for-- the tuple (MID,IID,TIMEKEY,EID) and NULL for the INSERT_SEQ field--INSERT INTO TMP_MYDATA (MID,IID,TIMEKEY,EID)SELECT MID,IID,TIMEKEY,EIDFROM MYDATAGROUP BY MID,IID,TIMEKEY,EIDHAVING COUNT(*)>1GO---- Updates the INSERT_SEQ field to the lowest value in the group-- of duplicated records--UPDATE TMP_MYDATASET TMP_MYDATA.INSERT_SEQ = (SELECT MIN(INSERT_SEQ)FROM MYDATAWHERE TMP_MYDATA.MID = MYDATA.MID ANDTMP_MYDATA.IID = MYDATA.IID ANDTMP_MYDATA.TIMEKEY = MYDATA.TIMEKEY ANDTMP_MYDATA.EID = MYDATA.EID )GO---- Updates the value of NUM_DUPLICATES for the MYDATA table.--UPDATE DUPLICATESSET NUM_DUPLICATES = (SELECT COUNT(*) FROM TMP_MYDATA)WHERE TABLENAME = 'MYDATA'GO---- Delete from the MYDATA table all the duplicated records,-- keeping only the row with the lowest INSERT_SEQ-- The delete is performed only if there are duplicated recors;-- this is achieved using a "short circuit" AND on the number ofrecords-- stored into the NUM_DUPLICATES field of the DUPLICATES table for-- the MYDATA table...--DELETE FROM MYDATAWHERE ( SELECT NUM_DUPLICATES FROM DUPLICATES WHERE TABLENAME ='MYDATA') > 0 ANDEXISTS ( SELECT 1FROM TMP_MYDATAWHERE MYDATA.MID = TMP_MYDATA.MID ANDMYDATA.IID = TMP_MYDATA.IID ANDMYDATA.TIMEKEY = TMP_MYDATA.TIMEKEY ANDMYDATA.EID = TMP_MYDATA.EID ANDMYDATA.INSERT_SEQ > TMP_MYDATA.INSERT_SEQ )GOThis tecnique works fine on a normal table (1M recs) but is not veryperformanton huge tables (>10M records)!Do you know a better way to achieve the task of removing all theduplicates records, preserving the lowest INSERT_SEQ betwee theduplicates and also preserving the sequence seed, so that a new recordinserted at time t1>t0 is enumerated with an INSERT_SEQ|t1 >max(INSERT_SEQ)|t0 ?Thanks a lot for your help!PatrizioPS. sorry for such a large post!

View 1 Replies View Related

JOIN 2 FACT Tables And ETL With SSIS 2005

Nov 21, 2007


Hello,

I have the following Datawarehouse problem (SQL Server 2005 and SSIS 2005):

2 Fact tables and both with some foreign surrogate keys of dimensions (some of them with common dimensions, other with different dimensions).

I have to create another FACT table that is an aggregation and join table between the 2 ones.

How can I do?

View 6 Replies View Related

How To Implement Incremental Load In Fact Tables.

Feb 19, 2007

Hi all,

i have a fact table which loads through package,when
i m trying to load this table by running the package,i m truncating the
fact table and loading the fresh data,instaed of this without
truncating the fact table i have to implement the incremental logic in
this.


For this i can use SCD or Conditional split,but problem
here is i have many source tables to load this fact table,so its
very difficult to trace the changes in different source tables.

can any one help me out in this??????

View 1 Replies View Related

Copying Dimension And Fact Tables From One Database To Another...

Dec 17, 2007

Hi there, my question is really simple. I want to setup an automatic task in SSIS that drops the tables in the target database and substitutes them with tables from the source database. We are talking about two or three dimension tables and one fact table. The dimension tables are pretty small. The fact table will contain, at maximum, 300,000 rows and 12 columns. I do not use delta or flag historisation btw. What tasks in SSIS would you suggest to use?

BTW I'm new to SSIS... ;-) Thanks in advance!

View 9 Replies View Related

Load Ordering For Dimension And Fact Tables

Sep 22, 2006

Hi ,

I have situation where I get data from SRC Flat file and have to load Dimensional table and also fact table, using same data flow(have no other choice since I have to unpivot some src data). Since I have to load both tables in same data flow, I have to have a way to put load ordering constraint (I know informatica allows that). Does any one have any idea on how this can be done in SSIS?

I would be really grateful.

Thanks

View 3 Replies View Related

How To Select Specific 2 Rows Out Of A Huge Table

Jul 24, 2013

I have a very large table , and from that table I need just 2 records with column1 = 'A' and column1 = 'B' .

Here I don't think if I can not use OR or IN or Case operators because I need exactly 2 records not more.

View 6 Replies View Related

Query:Source From Multiple Tables To A Fact Table

Jan 19, 2006

Greetings,

Iam new to SQLl2005. Iam using DTS to transfer data from my source to the warehouse. I have a couple of tables in my source whein I have to join these to tables fields and insert the same in teh warehouse fact table. I have used a Join query in my Oledb source component, What other component needs to be used to insert the data into the fact table.
I also need to extract same data with aggregation and insert the same into an another Fact table.

Kindly help.

View 21 Replies View Related

SQL Tools :: How To Print Huge Diagram Of Many Tables In SSMS

Nov 19, 2015

I have many tables, and I just want to print the relationships between them.  The ones without foreign keys to primary key relations are irrelevant.  I made a diagram of all tables in sql server management studio, and it shows the key relations, but its a very large diagram horizontally and vertically.  Is there a way to print the whole thing so that it doesn't take endless pages that I don't know how to piece together?

View 3 Replies View Related

Transact SQL :: Huge Performance Difference For Same Select Between Environments

Jun 22, 2015

I have encountered a problem with a specific set of tables. The same select yields slightly differing execution plans in two different environments (instances). But the slight variation seems to contain a huge differences in stats. I don't know the significance of these stats. The two tables have the exact same indices.

This is the selcet statement:

SELECT 'xx' FROM DUKS.dbo.Profiler
WHERE DNA_Løbenummer IN
(SELECT DNA_Løbenummer FROM DUKS.dbo.Effektregister
WHERE Sagsnummer = '2015-00002')

View 17 Replies View Related

Huge Tables: Minimizing Row Size - Remove Null Bitmap

Apr 21, 2007

I am in the process of designing a SQL 2005 database with tables that may hold several hundreds of millions of rows.

Due to various constraints, I am trying to save as much space as possible by optimizing the size of a row. Currently one row contains the following columns:



byte(4), byte(3), byte(3) = 10 bytes.



Adding 4 bytes for the row header, plus 3 bytes for the null bitmap (as described in BOL) I am ending up with 17 bytes/row. In a real world test it was an average of 18.3 bytes/row.



There are no indexes, no primary key and all columns are not NULL. Hence my question: since no column allows a null value, is there a possibility to "remove" the 3 bytes Null Bitmap?

Is there any other way to shave off one or two more bytes by using a clustered index etc?



Thanks

View 4 Replies View Related

Analysis :: Role Playing Dimension Among Multiple Fact Tables

Oct 28, 2015

I am modelling cube in SSAS. Cube has around 20 dimensions and 6 fact tables. Some of the dimensions are common among the fact tables. e.g. Time dimension. Fact_PNL has 3 date columns for those we have 3 role playing dimensions in the dimension usages.

Another fact table has 5 date columns for them as well we have separate role playing dimensions in dimension usage tab. We have a common dimension Company which is foreign key in all fact tables. We might need to combine the data from multiple facts to get final output.

Should i create 6 role playing dimension for each of the fact table or use the same dimension for all fact tables?

Role playing dimensions should be created when we have multiple columns pointing to the same dimension ?

View 2 Replies View Related

Analysis :: Fact Tables Appearing As Dimensions In Tabular Model

Jul 6, 2015

I built my first tabular model and see that my fact tables are also appearing as dimensions. In Multi dimensional mode i could choose which are the dimensions. How do i do that in tabular model.

View 2 Replies View Related

Fact Table With 3 Keys From Dimension Tables - Avoid Populating NULLs

Jun 10, 2014

I created a Fact Table with 3 Keys from dimension tables, like Customer Key, property key and territory key. Since I can ONLY have one Identity key on a table, what do I need to do to avoid populating NULLs on these columns..

View 3 Replies View Related

How To Optimize Data Import With Huge Volumes And Joins Across Data Sources Not All SQL Server Based?

Jun 7, 2006

I need to periodically import a (HUGE) table of data from an external data source (not SQL Server) into SQL Server, with the following scenarios:
Some of the records in the external data source may not exist in SQL.Some of the records in the external data source may have a different value at different imports, but this records are identified univocally by the same primary key in the external datasource and in SQL Server.Some of the records in the external data source may be the same in SQL.

Due to the massive volume of the import, I would like to import only the records which are different from what I have in SQL Server (cases 1 and 2 above). In fact case 2 is the most critical.

I thought of making a query with a left outer join between the data in the external data source table (SOURCE) and the data in the SQL Server table (DESTIN). The join is done on the respective primary keys (composed keys of up to 10 columns) and one of the WHERE conditions will be that the value in SOURCE is different from the value in DESTIN.

The result of this query would be exactly what I need to import.
How to do this in SSIS??? I couldn't figure out how to join tables in different data sources yet.

In fact I cannot write a stored procedure to do that, since one of the sources is in a datasources not SQL Server.
I have seen the Lookup transformation in this article http://www.sqlis.com/default.aspx?311 but this is not exacltly what I want to do.
Another possibility is to use the merge join, but due to the sorting I believe its performances would be terrible!

Thanks in advance for your suggestions!

View 9 Replies View Related

Data Access :: Importing Huge Data From One Database To Another Daily

Jul 7, 2015

We have a daily process, which copies millions of rows of data from one DB to another over Linked Server. Just checking on the best practise, are there more efficient ways than the Linked server to copy millions of rows of data from one DB to another? I checked bulk insert but that transfers the data from the file to DB not DB to DB. 

View 6 Replies View Related

Select Data From Two Tables

Jan 17, 2008

I have numerous tables in my db and would like to do this: (not sql below just simple explanation'

select contacts from wce_contact where idstatus is like 'london%' and select the same contacts where notes are not blank in the table wce_history

Hope it makes sense?

I just don't know how to query two tables at once...

View 3 Replies View Related

Power Pivot :: Creating A Summary Report Using Relationship Between Date Table And Two Fact Tables

Oct 19, 2015

I have below tables in my power pivot.Fact 1 & Fact 2 - connected directly to Mainframes - Data is from the same table broken up due to size of the data.Date Table - Relation ship between both the fact tables.How do i create a summary pivot to get the number of tasks that have been completed in each month.

Month   Count
July
August
September
October.

View 3 Replies View Related

Including Data From Two Tables In One Select

Apr 21, 1999

Hi All

I have a table of students, and a table of lecturers, and I wish to use one select statement to return a list of users.

What I really need to do is something like this:

SELECT Students.UserID AS UserID, Lecturers.UserID AS UserID, Students.FullName AS FullName, Lecturers.FullName AS FullName FROM Students, Lecturers

which would (in my dream world!) produce something like:

UserID FullName
s0002131 Darren Student
s0054242 Richard Student
e13412 Michael Lecturer
x92342 Linda Lecturer

Does anyone know how I should do this? Should I be doing it in an stored proceedure and returning the record set from that? Currently I'm only using ASP and one line SQL statements, and would like to continue doing so.

View 1 Replies View Related

Compare And Select Data Of Two Tables

Dec 4, 2014

I try to compare and select data from table that which not in other table

Select
AAA_Id
,BBB
From dbo.Table1 a
Where AAA_Id not in(Select b.AAA_Id
From dbo.Table2 b, dbo.Table1 a
Where b.AAA_Id = 11)

But when I modified it like:

Select
AAA_Id
,BBB
From dbo.Table1 a
union
Select
0
,'Select'
From dbo.Table1 a
Where AAA_Id not in(Select b.AAA_Id
From dbo.Table2 b, dbo.Table1 a
Where b.AAA_Id = 11)
Order by 2

I cannot to get same result.

View 2 Replies View Related

Select Data From Multiple Tables

Aug 20, 2007

Hi,
I have a number of related tables:
RGData is related to RGCrossReference
RCPPositionData is related to RCPCrossReference
RGCrossReference is also related to RCPCrossReference.
The data is returned correctly from these tables.
However, I also want to return data from another table - RComments.
How do I do this?
RComments is related to either RGData or RCPPositionData only.
Thanks.



Code Snippet
SELECT cm.CommentImage AS ViewComment, gd.PositionID AS GPositionID, cd.UniquePositionID AS CPPositionID
FROM RGData gd
INNER JOIN
RGCrossReference g
ON g.GPositionID = gd.PositionID
INNER JOIN
RCPCrossReference c
ON c.GMatchID = g.GMatchID
INNER JOIN
RCPPositionData cd
ON cd.UniquePositionID = c.CPPositionID
left outer JOIN
RComments cm
ON ((cm.CPPositionID = cd.UniquePositionID) or (cm.GPositionID = gd.PositionID))
AND cm.CommentsDate =
(SELECT MAX(CommentsDate) AS Expr1
FROM RComments
WHERE (GPositionID = g.GPositionID))
WHERE
(cd.Quantity != gd.Quantity
OR
cd.Currency != gd.Currency)
AND g.ForcedMatch = 'no';

View 3 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved