hi all,
i use quite some number of lookups for almost all of my fact tables ETL.
i was wondering if i can create a template or something similar so i can reuse the lookup instead of creating it again in each data flow for the fact table..
Thanks.
We did some "at scale" fuzzy lookup tests today and were rather disappointed with the performance. I'm wanting to know your experience so I can set my performance expectations appropriately.
We were doing a fuzzy lookup against a lookup table with 25 million rows. Each row has 11 columns used in the fuzzy lookup, each between 10-100 chars. We set CopyReferenceTable=0 and MatchIndexOptions=GenerateAndPersistNewIndex and WarmCaches=true. It took about 60 minutes to build that index table, during which, dtexec got up to 4.5GB memory usage. (Is there a way to tell what % of the index table got cached in memory? Memory kept rising as each "Finished building X% of fuzzy index" progress event scrolled by all the way up to 100% progress when it peaked at 4.5GB.) The MaxMemoryUsage setting we left blank so it would use as much as possible on this 64-bit box with 16GB of memory (but only about 4GB was available for SSIS).
After it got done building the index table, it started flowing data through the pipeline. We saw the first buffer of ~9,000 rows get passed from the source to the fuzzy lookup transform. Six hours later it had not finished doing the fuzzy lookup on that first buffer!!! Running profiler showed us it was firing off lots of singelton SQL queries doing lookups as expected. So it was making progress, just very, very slowly.
We had set MinSimilarity=0.45 and Exhaustive=False. Those seemed to be reasonable settings for smaller datasets.
Does that performance seem inline with expectations? Any thoughts to improve performance?
I'm working with an existing package that uses the fuzzy lookup transform. The package is currently working; however, I need to add some columns to the lookup columns from the reference table that is being used.
It seems that I am hitting a memory threshold of some sort, as when I add 3 or 4 columns, the package works, but when I add 5 columns, the fuzzy lookup transform fails pre-execute:
Pre-Execute
Taking a snapshot of the reference table
Taking a snapshot of the reference table
Building Fuzzy Match Index
component "Fuzzy Lookup Existing Member" (8351) failed the pre-execute phase and returned error code 0x8007007A.
These errors occur regardless of what columns I am attempting to add to the lookup list.
I have tried setting the MaxMemoryUsage custom property of the transform to 0, and to explicit values that should be much more than enough to hold the fuzzy match index (the reference table is only about 3000 rows, and the entire table is stored in less than 2MB of disk space.
Any ideas on what else could be causing this?
Say I want to lookup a value in another dataset, but there is a grouping that requires you to know what the values for each level is in order to get to the correct detail record. Can you still use the lookup function with more than one field to compare against? So for example
Department
\___SalesPerson
\___Measure
I want to be able to add a new row at the Measure level, but lookup each field from another dataset. In order to do that I will need the Department AND SalesPerson values to do the lookup, but I dont think the Lookup function will let us do that will.
Hi All,
Actually this is in regard to SCD Type 2 Dimension, Scenario is like that I am moving Fact table from some old source and I have dimensionA description value in fact which I want to replace with appropriate id from Dimension Table and that Dimension table is SCD Type 2 based on StartDate and EndDate and Fact Table doesn't contains direct date value rather there is timeId in Fact so to update the value in Fact table I have to Join Time Dimension table and other Dimension Table to replace fact Description with proper Id.
Lets assume DimensionA Structure
id
Description
StartDate
EndDate
Fact Table
id
measure1
measure2
TimeId
Description
Time Dimension
TimeId
Date
Day
Hour ...
I am doing a lookup that requires mapping 2 columns in the column mapping section. When I do this, I get the error "Row yielded no match during lookup" . The SQL that I captured in SQL profiler does find the record when I run it in Management Studio. I have already tried trimming everything to no avail.
Why is this happening?
I tried enabling memory restrictions but then I my package hangs and I get a SQLDUMPER_ERRORLOG.log file with the following logged:
07/24/07 13:35:48, ERROR , SQLDUMPER_UNKNOWN_APP.EXE, AdjustTokenPrivileges () failed (00000514)
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, Input parameters: 4 supplied
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ProcessID = 5952
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ThreadId = 0
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, Flags = 0x0
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, MiniDumpFlags = 0x0
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, SqlInfoPtr = 0x0100C5D0
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, DumpDir = <NULL>
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ExceptionRecordPtr = 0x00000000
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ContextPtr = 0x00000000
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ExtraFile = <NULL>
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, InstanceName = <NULL>
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, ServiceName = <NULL>
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, Callback type 11 not used
07/24/07 13:35:48, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, Callback type 15 not used
07/24/07 13:35:49, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, Callback type 7 not used
07/24/07 13:35:49, ACTION, SQLDUMPER_UNKNOWN_APP.EXE, MiniDump completed: C:Program FilesMicrosoft SQL Server90SharedErrorDumpsSQLDmpr0033.mdmp
07/24/07 13:35:49, ACTION, DtsDebugHost.exe, Watson Invoke: No
Why am I getting this error with "Enable Memory Restriction"?
Hi all,
I don't understand what's happening here.
I have a Conditional Split with 3 outputs. On the first output I have a lookup, when I execute the package I have 56 rows going through the Conditional Split, all rows are then going to the 2nd and 3rd output but the lookup on the first output generates an error "Row yielded no match during lookup".
I don't understand why the lookup is generating an error while there is no row going through it.
Any idea ?
Sébastien.
I am designing a ssis package,This is intends to mine text data(Data extracted from websites).
Term lookup/Term extraction has been used as tools for mining.
I have lookup terms defined with me for reference table,but the main problem lie in extracting the nearby text/number/charcters to these lookup terms during mining.
For example :
I found noun "Email" 200 (frequency score) times in my text,Now I want to extract nearby email address(this is also true for PhoneNumber,Address attributes also).so how can I achieve this with SSIS.
If u have some idea/suggestion to carry out this challenge with or without Term Extraction/Term Lookup,plz do write here.
Somehow I have saved an sp as a template - how do I delete it ?
View 4 Replies View RelatedI need to import data from a CSV file into a db I'm designing. I figured I'd use DTS (which I understand now uses VS in the form of a 'Business Intelligence' (BI) project). My problem is that my only choices for BI projects are:
Analysis Service Project, Import Analysis Services 9.0 Database, Integration Services Project, Report Server Project Wizard, Report Model Project, Report Server Project.
No "Data Transformation Project".
I have SQL Server 2005 Developer's Edition. Might it be that DTS is not included in that version?
If I can't use DTS, what choices do I have?
Hi! I'm new to SQL Server 7 and I hope I could get some help. I'd like to duplicate an empty database and use the duplicate for the transactions. I want the empty database to somehow act like a template so, whenever I want to, I could create a new instance of it. Problem is I don't know if it is possible to do that on SQL server (it's possible in Access). If it is possible, I hope any of you could tell me how. Thanks.
View 1 Replies View RelatedHello,
I'm having a hard time trying to figure out how to run the logtemp.sql script that is included with IIS4 for ODBC logging. Any help would be appreciated.
In Enterprise Manager when I bring up Trigger Properties there is an active button titled "Save as Template". But when I switch to an existing trigger the button is disabled. Oddly, the Help file for the dialog box doesn't even mention this button.
Anybody know what this does, how it works, or why it would be usefull? Any references to The Holy Book would be helpfull.
HI,
how can I specify the number of records that have to be sent back dynamically in an Xml Template?
I am currently busy developing a client-server application that pulls data from an Sql Server via Xml. If the client application has a bad connection to the server, the program has to get the data in small portions.
When I use:
SELECT TOP @amount *
FROM employees
I get the error: syntax near '@amount'.
The query fails as wel when I tried to use:
SET ROWCOUNT @amount
SELECT *
FROM employees
SET ROWCOUNT 0
This works fine in Sql analyser, but sql doesn't allow the integers to be parameters and casting them to int does not work either.
thanks for your help!
Hi all. How to make a template header for reports please? So that everytime I make a report, ill just call the template in the header.
Thanks.
-Ron-
Hi,
Is there any way to change path for template explorer? Default path for SQL templates is ..Application DataMicrosoftMicrosoft SQL Server90ToolsShellTemplatesSql. I have scripts & would like to share with my teammates. Is there any way to set this path to a common folder so that whenever anybody connects with their login to our TS server, they should be able to see these scripts in template explorer.
Thanks in advance
Sandesh
Hi,Suppose I have a table something like:name (VARCHAR(64)) age (INT) quotation (TEXT)=================================================Donald Trump 50 I am richBugs Bunny 26 What's up doc...and a template string something like:SET @template = 'My name is {name}, my age is {age}, and I always say"{quotation}".'I'd like to be able to dynamically replace the placeholders in thetemplate string with values extracted from the corresponding columns inthe table, so I'd get a set of results like:'My name is Donald Trump, my age is 50, and I always say "I am rich".'The best I've come up with so far is:SET @Query = 'SELECT '''+ REPLACE(REPLACE(@String, '{', '''+CONVERT(varchar,'),'}', ')+''')+ ''' FROM Table'EXEC (@Query)This converts the template string into a query string, castingeverything to avarchar to allow numeric values to work. In this case it would be:SELECT 'My name is '+CONVERT(varchar,name)+', my age is '+CONVERT(varchar,age)+', and I always say "'+CONVERT(varchar,quotation)+'".'The problem with this is that if the length of a varchar isunspecified, it defaults to 30, which truncates long string values.Can anyone figure out a way round this, or perhaps an alternativemethod entirely?--Oli
View 5 Replies View RelatedThere is a base-installed template for a stored procedure called "Create Procedure with CURSOR OUT" which has this snippet:
EXEC <Schema_Name, sysname, Schema_Name>.<Procedure_Name, sysname, Procedure_Name> <@proc_cursor_name, , @sample_procedure_cursor> = <@variable_cursor_name, , @test_cursor_variable> OUTPUT
WHILE (@@FETCH_STATUS = 0)
BEGIN
FETCH NEXT FROM <@variable_cursor_name, , @test_cursor_variable>
PRINT 'put user defined code here'
END
But even after instantiating it - it doesn't work. The "FETCH_STATUS" must be seeded with a prior "FETCH" before it has any meaning. This leads me to ask: (1) how could such a widely circulated template have such a conspicuous error and (2) why doesn't TSQL have a looping mechanism that does not require such awkward FETCH "pre-seeding"?
p.s. How do I insert code into a forum post that is single-spaced, rather than duoble-spaced (like my ugly double-spaced snippet above)
I have created a template like report which is all working correctly. The report shows a order slip for all orders that are in a certain pick, and has a list of all products for each order. (so a list of products is printed for each order in the pick)
The question I have is, when the order is very long ie doesnt fit in 1 page, I need someway of the user easily knowing that there is more than 1 page... ie it should be Page 1 of 2... but in the whole report it may be Page 56 of 60.... is there a way of doing this??? or something similar...
I hope i have made myself clear... and would be VERY garteful for any ideas, im bit lost!...
Thanks
I read the steps on creating a package template and using that template in future Projects.
My question is, how is doing Add -> New Item from the solution different from Add Existing Package from SSIS Packages. Other than the quicker navigation, I see no difference. So, is that it?
I have Visual Studio 2012 (Ultimate) installed (Update 4) with SSDT (version 11.1.50512.0), I can see the template SQL Server Database Project, but where is the server-level project, SQL Server Project?
Previously, in Visual Studio 2008, I had a Category Database Projects with subcategories for SQL Server CLR, SQL Server 2000, SQL Server 2005 and SQL Server 2008. Each of those had a SQL Server 200x Server Project template.These seem to be missing in Visual Studio 2012.
Hi. I'm running VS2005 and trying to use the SQL Database installed template (right-click my project in Solution Explorer / Add / New Item / SQL Database template ("An empty SQL database for local data") to create an MDF file per the "AbsoluteBeginner-Lesson08" video available on MSDN. When I click "Add", I get an error stating that "Connections to SQL Server files require SQL Server Express to function properly. Please verify the installation of the component or download from the URL: http://go.microsoft.com/fwlink/?LinkID=49251". I'm running SQL 2000, SQL 2005 and SQL Express 2005 (all work and can be hit from Management Studio) on 2003 Server. I'm assuming that it's because the Express instance is a named instance, not the default but I can't find any references. Anybody else had this probelm? Thanks.
View 1 Replies View RelatedHI
I want to create a report template in SQL Server 2005 Reporting Services.
The template will have a fixed header and footer.I want to use this template in all my reports.
How to do so?
Hello,I need to be able to CREATE DATABASE by copying an existing
database.I would be doing this inside of a web app during an event.How do I set this up on SQL 2005 ?Thanks!Here's an article on how to do it in PostgreSQL http://www.enterprisedb.com/documentation/manage-ag-templatedbs.html
Hello All,
Can someone tell me how to achieve this template of the report:
Initially when not expanded
+Item number
- - - - - -(these are summed values)
After expnading the fields(+)
-Item number
-item name
[ (details)
[ (details)
[ (details)
- - - -(summed values)
Right now, I can achive the second scenario (expanded one) but initially my report just comes up with
+Item number
and the second line (summed values) are not in there . Any sort of help will be appreciated.
Thanks,
Rashi
I accidentally altered the CREATE TABLE template from SQL Server Management Studio Express. Now I don't have the original. Could somebody please post CREATE TABLE template.
View 4 Replies View RelatedHi All,
I'm working on a web application where the user needs to be able to create and name new databases that are identical in structure to other existing databases (that is, all tables, stored procedures, functions, indexes, etc.). This is so that they can create a new database for each client and need to be able to do this through the web application. Having hunted around a fair bit, I've established that SMO is capable of doing pretty much everything that I want. The only problem is that everything I do seems to be based on the actual SQL Server and associated databases rather than the ones I have created in the App_Data folder.
The relevant code (so far) is:Dim sqlServer As New Server()
With sqlServer.ConnectionContext
.ServerInstance = "(local)"
.Connect()
.Disconnect()
End With
For Each db As Database In sqlServer.Databases
ListView1.Items.Add(db.Name)
Next
Dim newDatabase As New Database(sqlServer, DbName.Text.ToString)
newDatabase.Create()
This does actaully create a new database, just not where I want it! Can anyone point me in the right direction as to how I can create a copy of a database in the App_Data folder?
Thanks & regards,
Paul
When I go to create a new stored procedure, function, etc. I get a default template... is there anyway to replace this template with one of my own? One that will already have the important information pre-populated (the stuff I always end up having to type in, like company name, etc.).
View 1 Replies View RelatedI need some feedback on how this sql is perfroming a cast. I have used CAST before, but I amlooking at a stored procedure, and the CAST looks slightly different. Usually when I do acast, it is of the form CAST(ColumnName,DataType) but this cast is slightly different as theyappear to be using some type of template to cast with. Here is the code. I will put line numbers in for reference.
1 CREATE TABLE #Groups2 (PartyId uniqueidentifier)3 INSERT INTO #Groups (PartyId)4 SELECT PartyId5 FROM ttg_sfGroupsByPartyId(@PartyId)67 IF (@PortalId = CAST('00000000-0000-0000-0000-000000000000' AS uniqueidentifier) AND88 @TabId = CAST('00000000-0000-0000-0000-000000000000' AS uniqueidentifier))0 BEGIN
Also just to let you know, the the values PartyId, and PortalId are GUIDS.
So, I guess my question is, are they using a kind of template to format the data? So look at lines7 and 8. Are they using some type of template to signal the compiler, that the data will be of thatformat and will be seperated by dash symbols?
Does anyone have a template for storing all SQL Server 7 information.
Database, filegroup, drive, ...etc. Any help appreciated
Paul.
I'm dumping data from a table via BCP and when BCPing them back in to another table, it errors out on numeric and date fields. I'd like to place quote marks on the text fields. How do I do this using BCP?
View 4 Replies View RelatedI modified the trigger template in SQL Server Enterprise Manager.
How to restore default trigger template,like follow
--------------------------------------------------------
CREATE TRIGGER [TRIGGER NAME] ON [dbo].[tablename]
FOR INSERT, UPDATE, DELETE
AS
--------------------------------------------------
Does the information of trigger template save in a file ?
Anyone has idea ?
I created a template that I use all the time. Normally I just drag-n-drop from the Template Explorer menu, however I'm wondering if I can assign a shortcut key to have it open? I see you can assign stored procedures and other items, but I can't find where/if I can do this for a template.
View 2 Replies View Related