Analysis :: Power BI On Larger Data Sources
Jun 1, 2015
We have been trying for a while to use Power BI tools (Power Query, Power BI Designer (Desktop) and Power BI Designer (Cloud) in line with larger data sources ( 2-7m records in fact table). Unfortunately up to now with unsatisfactory results. It seems that these tools are just not designed to handle this kind of data volumes?
To my understanding it seems that the approach with PowerBI components is to always try to create a local (or cloud based) cache that only has a limited capacity. 2m+ records already seem to be exceeding this limit. So as opposed to firing off a query on demand (for example only based on distinct filter options as opposed to entire fact set) only the intermediary cache of the model can be used.
Our initial focus was to use a normalized table in SQL Server with around 4m records. First problem is that Power Query/Power BI Designer fails to provide a complete list of the distinct filter items:
This I can understand in some way as doing a distinct on a large data set like that is not trivial. As a workaround I could imagine to setup a star scheme dimension table with the distinct "dimension members". Â I.e. that filters are driven by this table which has only a few thousand rows. The filters there are then applied to the fact table. I haven't found a way to do this effectively with Power Query.Â
Another option that we have tried was using the cloud based Power BI service in conjunction with a SQL Cloud Service with the same data set. That  unfortunately didn't work. The setup of the source works fine but as soon as we try to start a query by dragging a value field on the dashboard errors occur:
View 5 Replies
ADVERTISEMENT
Apr 22, 2015
I have uploaded excel report with data model , Â but when i click on ,anage data refresh link it is showing shedular disable. I have check logs then I am getting analytics server connection time out error.
Right now architecture of system  having different analytics server and sharepoint server is also different. Any better solution due to which i can able to schedule my reports.
View 2 Replies
View Related
Jul 1, 2015
I work almost 1 month to try to fix this issue, but still failed.(In my test server, it's working). seem like I have 2 problem.
1. My SQL server analysis service(PowerPivot) service mode is "Multidimensional", in my test server, this is SharePoint mode.
2. After upload PowerPivot worksheet, I check the analysis service, No this sheet DB save in. in "Analysis Server Properties", I add "sp_admin" name that I use this name to install PowerPivot add-in.
View 3 Replies
View Related
Aug 4, 2015
I've
1. SQL Server Data Warehouse
2. OLAP CUBE in Analysis Services
My question is - If my SQL Server Data Warehouse is changed (Having Append Data) - Is that My OLAP Cube will have the Append Data?
It's possible, my OLAP Cube always having Append Data if my Data Warehouse is changed? If yes, how to do it without re-deploy and re-process my Analysis Services Project.
View 3 Replies
View Related
Mar 5, 2015
I have, a SSAS 2012 tabular instance with SP2, there is a database on the instance with a read role with everyone assigned permissions. When configuring the Power BI analysis services connector, at the point where you enter Friendly Name, Description and Friendly error message, when you click next I receive the error "The remote server returned an error (403)." I've tested connecting to the database from Excel on a desktop and connect fine.I don't use a "onmicrosoft" account so don't have that problem to deal with.
We use Power BI Pro with our Office 365. As far as I can tell that part is working ok as I pass that stage of the configuration with a message saying connected to Power BI.The connector is installed on the same server as tabular services, its a Win2012 Standard server. The tabular instance is running a domain account that is the admin account for the instance (this is a dev environment) that account is what I've used in the connector configuration. It's also a local admin account. There is no gateway installed on the server.
View 10 Replies
View Related
Jul 20, 1998
Is it possible to make a text-field or something like that, which can contain more than 255 characters ?
View 1 Replies
View Related
Jun 16, 2015
I have a problem with calculating discount in PowerPivot analysis. I have
dCalendar , dDiscount, fSalesBook .
dDiscount columns:
DiscKey, StartDate, EndDate, Product, DiscountName, DiscountPercent
fSalebook columns:
OrderDate, Product, Qty, Sales, Discount
dCalendar columns:
Date, Holiday, MonthNo, MonthName, QuarterNo, QrtName, Year.
The problem (at least I feel) is I defined some dates in the Startdate and Enddate for a particular year. But these discounts should apply for all of the years in my fSalesBook, not just for that year in the dDiscount table.
So in essence, when you calculate discount for a product in fSalesBook you should consider only the month and day of the transaction, if the sale is in between the discount dates.
I have the following relationships.
Table [Column] ------->Table [Column]
fSalesBook[OrderDate] -----> dCalendar[Date]
I am not sure how to relate the dDiscount table with the dCalendar table and not sure even there is a need to relate them to calculate Row level discount in fSalesBook. You can change the relations/model if there is need be.
PS: One more clause, Sometimes I offer discount only for a particular product in that Discount season, that's why there is a column for Product in the dDiscount table.
What should be my DAX formula to calculate Row level discount in fSalesBook?
Here is the link for the sample workbook.
[URL]
View 6 Replies
View Related
May 12, 2015
getting a complete Microsoft DW/BI (Tabular), Power BI and Office enviroment up running. I am a (Microsoft) DW/BI manager and want to learn about the latest Microsoft DW/BI (SQL Server Tabular) and Power BI (Power Pivot, Power View, ...). To save time & access from anywhere, I thought I could use Windows Azure but quickly got lost in the Azure preview and market place! I even contacted the Windows Azure team and that didn't clarify things.
So, how to get a SQL Server (2014) environment with all DW/BI (SSIS, SSAS, SSRS, Data Mining) technologies together with Power BI and Excel working? If so, how to get started.If this is not possible on Azure and it has to be done on Window, what do I need to install for the Power BI stuff (to work with Tabular)? I already have SQL Server 2014 and Office Pro Plus installed natively on my Windows machine.
View 6 Replies
View Related
Jan 8, 2008
I have written code to combine, delete redundant data in my system. The table structure remains the same, except I changed some INTs to TINYINTs.
When I do sp_spaceused, it tells me that number of rows is smaller(which is correct), but the datasize, and index_size is significantly larger AFTER the deletions.
I tried using shrink, but that doesn't seem to change anything.
When I right-click the database, and choose PROPERTIES, it also confirms that the database got significantly larger.
I am confused about how deleting data and changing to TINYINTs could make my database bigger. What would cause this?
View 9 Replies
View Related
Aug 2, 2015
I am trying to show images in a product listing in power view.I work with an excel 2013 desktop version based on an office 365 pro account.I did the following steps:
import of an excel file with an article list via power query and loading the data to the data model import jpg images from a folder via power query, setting content as binary type and loading the data to the data modellinking both tables in power pivot--> manage via the image namesetting the table behavior for the images table under power pivot --> manage --> Advanced (e.g. Default Image: Content)opening power view and building article cards with article number and imageProblem: only a camera icon shows up in power view
Is there a solution with a desktop version?Can I use my Office 365 Pro account to make it work? How?Why is there no solution showing images in a pivot table?Link to Dropbox with power pivot filesÂ
View 9 Replies
View Related
Jun 9, 2006
Dear,
I created a package getting data from files and database sources, doing some transformations, retrieving dimension id's and then inserting it into a fact table.
Running this package with a limited amount of data (about a couple of 100.000 records) does not result in any errors and everything goes fine.
Now running the same package (still in debug mode) with more data (about 2.000.000 rows) doesn't result in any errors as well, but it just stops running. In fact, it doesn't really stop, but it doesn't continue as well. If I've only been waiting for some minutes or hours, I could think it's still processing, but I waited for about a day and it still is 'processing' the same step.
Any ideas on how to dig further into this in order to find the problem? Or is this a known problem?
Thanks for your ideas,
Jievie
View 4 Replies
View Related
Nov 12, 2015
My Power view reports do not show the grand totals correctly. I think my issue relates perfectly to [URL]. However we have updated service pack 2 on both the sql data warehouse and the ssas box. They are 2012. However the power view is still incorrect. Pivot tables seem to work just fine. In AS I have a 2 separate measure groups that has a total count. I then use mdx to bring those two together. Something I have done many times in past with as 2008. My mdx looks like this below. I then hide Total Count EC (from measure group 1) and Total Count SF (from measure group 2). This way I have 1 total count value and they can use another dimension to see the two separate if needed.Â
CREATE
MEMBERCURRENTCUBE.MEASURES.[Total
Count New] AS
//
([Measures].[Total Count] + [Measures].[Total Count SF]),
DISPLAY_FOLDER
=
'Totals',Â
ASSOCIATED_MEASURE_GROUP=
'Report',
FORMAT_STRING
=
"#,#0",
VISIBLE=
1;
The measure Total Count EC works just fine but the Total Count (MDX) does not . All in Power Pivot works fine but in Power View the grand totals are not correct.
View 3 Replies
View Related
Jun 7, 2006
I need to periodically import a (HUGE) table of data from an external data source (not SQL Server) into SQL Server, with the following scenarios:
Some of the records in the external data source may not exist in SQL.Some of the records in the external data source may have a different value at different imports, but this records are identified univocally by the same primary key in the external datasource and in SQL Server.Some of the records in the external data source may be the same in SQL.
Due to the massive volume of the import, I would like to import only the records which are different from what I have in SQL Server (cases 1 and 2 above). In fact case 2 is the most critical.
I thought of making a query with a left outer join between the data in the external data source table (SOURCE) and the data in the SQL Server table (DESTIN). The join is done on the respective primary keys (composed keys of up to 10 columns) and one of the WHERE conditions will be that the value in SOURCE is different from the value in DESTIN.
The result of this query would be exactly what I need to import.
How to do this in SSIS??? I couldn't figure out how to join tables in different data sources yet.
In fact I cannot write a stored procedure to do that, since one of the sources is in a datasources not SQL Server.
I have seen the Lookup transformation in this article http://www.sqlis.com/default.aspx?311 but this is not exacltly what I want to do.
Another possibility is to use the merge join, but due to the sorting I believe its performances would be terrible!
Thanks in advance for your suggestions!
View 9 Replies
View Related
Aug 4, 2006
Hi,
I am pretty new to SSIS. I am trying to create a package which can accept data in any of several formats. i.e. CSV, Excel, a SQL Server database/table and import the data into my destination database.
So far i've managed to get this working OK. However I am now TOTALLY stuck. I'm currently trying to just concentrate on the data sources being a CSV (using a Flat File Data Source) and/or an Excel Spreadsheet.
I can get the data in and to my destination using a UNION ALL component and mapping the data sources to it so long as both the CSV file and the Excel spreadsheet exist.
My problem is that I need my package to handle the possibility that only the CSV file might exist and there is no Excel spreadsheet. In which case i'd like the package to ignore the Excel datasource completely. Currently either of my data sources do not exist I get errors and the package terminates.
Is there any way in SSIS that I can check all my data sources to see which ones exist (i.e. are valid). If they exist I want to use them. If it doesn't exist i'd like to disgard it (without error - as long as there is a single datasource the package should run)
I've tried using the AcquireConnection method in a script task on each of my connections, hoping that it would error if the file/datasource did not exist. It doesn't though (in the case of an Excel datasource it just creates a empty excel file for me).
The only other option I can come up with are to have seperate packages depending on the type of data we want to import and then run a particular package depending on the format of the source data. This seems a bit long winded. I am pretty sure I must be able to do what I want to achieve but I can't work out how.
I'll be grateful to anyone who can send me any tips/hints/links on how I can achieve this.
Many thanks
Rob Gibson
View 5 Replies
View Related
Feb 18, 2013
It is possible to create Analysis Services project (*.dwproj) in Visual Studio 2012 Data Tools?
View 5 Replies
View Related
Feb 26, 2008
Hi, i'm wondering which is the best way to search data in a SQL Server.
I reach data using Data Sources and Data Views and also with OLE DB Source with a Data access mode: Named query.
I have to write the data into a Flat File. So, does any one knows which is the best practice for this? Or any one of the two are good choices?
Thanks for your help.
Beli
View 6 Replies
View Related
Jan 9, 2007
Hi,
In my project i want a report. In that report data is getting from more than one data sources(systems). While creating data source view i used named query for both primary and secondary data source. But at the time of crating "Report Model" i am getting below error.
An error occurred while executing a command.
Message: Invalid object name 'Table2'.
Command:
SELECT COUNT(*) FROM (SELECT SerialNum, ModelNum AS com_model
FROM Table2) t
Is there any way to create a report with multiple data sources?
View 8 Replies
View Related
Mar 19, 2007
Hi all :)
I'm wondering if SSIS will be the solution to the problem I'm working on.
Some of our customers give us an Excel sheet with data they want to insert or update in the database.
I've created a package that will take an Excel sheet, do some data conversion so the data types match up and after that I use a Slowly Changing Data component to create the insert/update commands.
This works great. If a customer adds a new row to the Excel sheet or updates an existing row changes are nicely reflected in the database.
But now I€™ve got the following problem. The column names and the order of the columns in the Excel sheet are not standard and in the future it could happen a customer doesn't even use an Excel sheet but something totally different.
Can I use SSIS for this? Is it possible to let the user set the mappings trough some sort of user interface? I€™ve looked at programmatically creating the package but I€™ve got to say that€™s quit hard to do€¦ It would be easier to write the whole thing myself than to create the package trough code ;)
If not I thought about transforming the data in code before I pass it on to the SSIS package in something like XML. That way I can use standard column names and data types.
So how should I solve this problem? Use SSIS or not?
Thnx :)
Wouter de Kort
View 6 Replies
View Related
Aug 20, 2015
Given a model in Power Pivot with a number of KPIs that are hidden with the 'hide from client tools' option. After restoring the model in SSAS and connecting to the model from Excel, the KPIs are visible under a separate category/table "KPIs".
Other hidden items in the model like columns and measures remain invisible.
Is this normal behavior and what can I do about this?
View 2 Replies
View Related
Feb 21, 2008
Hi i have data on a Server in a different database which i like to access from within my ssis job.
I just need to look up information from one table on this database so i can references it. Is there a way of doing this is SSIS. Rather then me having to load the data from one database to another as the data may change.
i tried having 2 sources of data feed into a look up but that does not work..
View 1 Replies
View Related
Apr 1, 2008
Hi there,
On my home page I have several different folders to reports which require different data sources. the problem is that within these folders there are multiple copies of the same datasource. is it possible to store all of the datasources in one folder, one location? it would certainly be easier when changes to usernames and passwords need to be modified!
Thanks,
Rhonda
View 4 Replies
View Related
Apr 4, 2008
I searched and read about Data Sources and I'm seeing that there is no advantage in using it, which is what I found from playing around with it.
I expected that you would set a global connection in Data Sources and somehow link this to the things in your Connection Manager, giving you one place to switch from one environment to another. But reading the discussions here and playing around with it, this is not the case.
So, why is it there?
Next question.... another thing I gathered so far is something called "Configurations" that will do what I was describing above. Where do I do this?
View 8 Replies
View Related
Jun 28, 2007
Hi,
how can i use var for Data Sources ,
my project have to be suitable for the QA system and the production system and i want to connect them by var or something else.
if you have another idea about this subject i'll happy to read.
thanks,
gil.
View 8 Replies
View Related
Jun 12, 2007
I'm trying to combine two reports I've created into a form letter. The goal is to have a letter that indicates the students maximum UBSCT score for Math, Reading and Writing. Also in the letter we want to include the student's total number of credits awarded and overall GPA. I not sure if I need to try and combined the two queries or use two different data sources in Reporting Services. I've tried creating one sql statement but the grouping are giving me fits. I fairly new to Reporting Services.
Report 1:
704 Dixie High 11 311880 Student_1 UBSCT Math 176
704 Dixie High 11 311880 Student_1 UBSCT Reading 182
704 Dixie High 11 311880 Student_1 UBSCT Writing 173
704 Dixie High 11 311881 Student_2 UBSCT Math 168
704 Dixie High 11 311881 Student_2 UBSCT Reading 172
704 Dixie High 11 311881 Student_2 UBSCT Writing 165
SELECT
track.schoolc as School_Code,
school.schname as School,
stustat.graden as Grade,
studemo.ident as Ident,
RTRIM(studemo.lastname) + ', ' + RTRIM(studemo.firstname) AS [Student],
stutscors1.testc as Test_Code,
left(zsubtest1.descript,20) as [Subject],
max(stutscors1.testscore) as Score
FROM
stutscors1 INNER JOIN
stutests1 ON stutscors1.ststuniq = stutests1.ststuniq INNER JOIN
studemo ON stutests1.suniq = studemo.suniq INNER JOIN
stustat ON studemo.stuuniq = stustat.stuuniq INNER JOIN
track ON stustat.trkuniq = track.trkuniq INNER JOIN
school ON track.schoolc = school.schoolc INNER JOIN
testdef1 ON stutests1.testuniq = testdef1.testuniq INNER JOIN
zsubtest1 ON stutscors1.subtestc = zsubtest1.subtestc and stutscors1.testc = zsubtest1.testc
WHERE
stutscors1.testscore <> ' ' and
stutscors1.testscore <> '0' and
school.schname = 'Dixie High'and
stutscors1.testc = 'UBSCT' and
stustat.graden = 11
GROUP BY
track.schoolc,
school.schname,
studemo.ident,
RTRIM(studemo.lastname) + ', ' + RTRIM(studemo.firstname),
stustat.graden,
stutscors1.testc,
zsubtest1.descript
---------------------------------------------------------------------------------
Report 2:
704 Dixie High 11 Student_1 311880 23.2500 23.2500 87.50300000 3.763569
704 Dixie High 11 Student_2 311881 20.2500 20.2500 76.84300000 3.794716
select
stugrp_active.schoolc as School_Code,
school.schname as School,
stugrp_active.graden as Grade,
rtrim(stugrp_active.lastname) + ', ' + rtrim(stugrp_active.firstname) as Student,
trnscrpt.suniq as Ident,
sum(trnscrpt.gpacratt) as CreditAtt,
sum(trnscrpt.gradcrawd) as CreditAwd,
round(sum(case when Trnscrpt.GpaCrAtt is null then 0 else Trnscrpt.GpaCrAtt end * gpamarks.gpavallvl0),3) AS TrmGpaPts,
(round(sum(case when Trnscrpt.GpaCrAtt is null then 0 else Trnscrpt.GpaCrAtt end * gpamarks.gpavallvl0),3))/(sum(trnscrpt.gradcrawd)) as GPA
from
dbo.trnscrpt
inner join dbo.stugrp_active on (trnscrpt.suniq = stugrp_active.suniq) INNER JOIN school ON stugrp_active.schoolc = school.schoolc INNER JOIN
gpamarks ON trnscrpt.marksetc1 = gpamarks.marksetc AND trnscrpt.markawd1 = gpamarks.mark
where
school.schname = 'Dixie High' and
stugrp_active.graden = 11 and
trnscrpt.graden >= 9
group by
stugrp_active.schoolc,
school.schname,
stugrp_active.graden,
rtrim(stugrp_active.lastname) + ', ' + rtrim(stugrp_active.firstname),
trnscrpt.suniq
View 2 Replies
View Related
Jul 9, 2007
Hi guys,
I manage to get the SSIS working. Now I would need to do these tasks.
I first want to get data from 2 different sql servers. What would be the best method to accomplish this? Both are in Sql Server 2005.
Secondly I want to make sure if any of the servers couldn't be found on the network or in any case the getting data task failed for any one of them the package won't continue and an email should be send to an email address.
Thirdly If everything is ok then I should combine both and generate one sequence no for them and save them on to another location and then generate a file with modified values.
Can anyone help me regarding these tasks?
Thank you
Gemma
View 17 Replies
View Related
May 15, 2007
In brief, here is my quandary… I have a GridView on a page with some radio buttons, a couple of check boxes, some drop downs etc. And depending on what the user selects the Grid View needs to show different columns from the database…
The easiest way I have found to do this is to have a separate Data Object for each “view�, i.e. each variation of the checkboxes and radio buttons (which determine which dropdowns to use for the variables) and then bind the GridView to the specific Data Source at runtime…
It seemed like a good idea at the time, but now I have over 7 different data connections cluttering up the design view, and more to come and I feel like I am missing a better way to achieve the same results…
Does anyone have some advice for a major novice?
Here’s the beginning of a large set of nested ifs, if this is the way it’s done that’s fine too, I’ll just keep creating data sources, I just get the feeling there must be an easier way to do this..
protected void Button1_Click1(object sender, EventArgs e)
{
if (RadioButtonList1.SelectedValue == "Vendor") {
if (CheckBox1.Checked == true) {
GridView2.DataSourceID = "ObjectDataSource4";
GridView2.DataBind();
TextBox1.Text = "Search";
} else {
GridView2.DataSourceID = "ObjectDataSource1";
GridView2.DataBind();
TextBox1.Text = "Search";
}
} else if (RadioButtonList1.SelectedValue == "Store Number")
{ GridView2.DataSourceID = "ObjectDataSource2";
GridView2.DataBind();}
else {
GridView2.DataSourceID = "ObjectDataSource3";
GridView2.DataBind();
TextBox1.Text = "Search";
}
}
Any advice for a major novice would be greatly appreciated! Thanks!
View 5 Replies
View Related
Feb 27, 2006
I'm just wondering if it is bad programming practice to have many data sources on one page. I need a "Matrix" type of page that grabs data from many tables and so far I have 5 data sources. Is this common or is there something I should be doing to make it more efficient?
View 1 Replies
View Related
Mar 7, 2002
Is it possible to create a recordset from two different datasources?
Such as
SELECT * FROM DataSource1.TableName WHERE ColumnName IN (SELECT Column FROM Datasource2.TableName)
Thanks.
View 1 Replies
View Related
Mar 7, 2002
Is it possible to create a recordset from two different datasources?
Such as
SELECT * FROM DataSource1.TableName WHERE ColumnName IN (SELECT Column FROM Datasource2.TableName)
Thanks.
View 2 Replies
View Related
Apr 21, 2007
Can I import in my query a file from other sources (in this case it's a job that has elaborated server database data, but I could be in the need of using for instance Excel files or others) and compare its fields with the tables in my query?
Thank you.
Anna - Verona (Italy)
View 3 Replies
View Related
Jan 24, 2008
Not sure if this is possible, here is the scope:
I have a vb program that calls a sp. My vb program connects to DataSource1, Catalog1 and to DataSource2, Catalog2. My sp resides in Catalog1 using a table from catalog1 and a table from catalog2.
Is this possible to do?
View 2 Replies
View Related
Dec 14, 2006
If I were to change a data source that many packages use (say, from box "SQL-2" to "SQL-1"), would I have to re-upload the packages that used that data source? Or would the packages already on the server receive that change somehow?
(currently realizing that most of the packages use package-local data connections instead of project-wide data sources...)
Thanks!
-Rob
View 1 Replies
View Related
Jan 3, 2005
Hey all,
I'm coming from ASP and I used to have a global connection string accessible to all of my ASP pages that I'd use for all of my data access. This was really convenient because I could easily switch to a backup or local data source for testing/debug by changing the connection string in one place.
Now, I'm using ASP.NET with Web Matrix and I love the drag and drop functionality but its dropping my connection string all over the place. How can I do this and keep my connection string in one spot so I can have the same convenience of switching data sources.
Thanks in advance!
Larry
View 3 Replies
View Related
