Lookup With Default Full Cache Option Yields No Results
Nov 20, 2007
I'm either missing something or this is a bug. I have a Lookup that finds no matches if I use the default option of full caching (everything on the Advanced tab unchecked). The lookup table is relatively small (15348 bytes) in only 544 rows. If I check only the Enable Memory Restriction box and eliminate caching, it works fine. I can also check the Enable Caching box and accept the default cache size of 5MB and it works fine. Anyone have any ideas? I'm running on Standard Edition, SP2.
I am using a lookup and full cache, occasionally i get this warning:[Lookup [150]] Warning: The component "Lookup" (150) encountered duplicate reference key values when caching reference data. This error occurs in Full Cache mode only. Either remove the duplicate key values, or change the cache mode to PARTIAL or NO_CACHE. Now I know it is only a warning but it is highlighting a real issue.Is there a way of capturing that this has happened?
I would like to know what happens when a very large reference data set for a lookup transform with full caching enabled is getting loaded during package execution and the computer memory runs out or is very low. Does SSIS a) give an out of memory error of some sort b) resort to a no caching or partial caching mode c) maintain the full caching mode but will switch to using the paging file(virtual memory).
I think it will resort to using the page file in which case the benefits of in memory lookups are lost and performance would suffer. If I cannot upgrade the memory or shrink the reference set somehow, i should switch that lookup task to use partial caching or no caching with an indexed lookup table. Would this make sense?
I'm currently loading a package that does a lookup on a column of data type nvarchar(4).The values itself are (A+, A, B+, B, C, D, /). The strange lookup behaviour is happening for each of the cases, so it's not related to a specific value. After trying to put the cache on NO CACHE, the lookup works perfectly. When using the default FULL CACHE the strange behaviour happens. Could it be related to the data type? I have not yet tried to use a CHAR instead of a NVARCHAR but it looks like people have similar issues using CHAR.
My source has 2.2 million of records. I'm performing the incremental load.In the lookup transformation i used the destination table for the reference using Full cache mode.For the first time package executed successfully but when i executed the package second time, Suddenly Package hangs while running.Than i truncate the data from the destination table and restart the SQL Server Services.After doing all this i executed package again and it worked but when i executed package second time, again package hangs up .I have 8GB RAM and i5 2.5 GHz Processor laptop.
I am getting inconsistent results when BULK INSERTING data from a tab-delimited text file. As part of my testing, I run the same code on the same file again and again, and I get different results every time! I get this on SQL 2005 and SQL 2012 R2.We have an application that imports data from a spreadsheet. The sheet contains section headers with account numbers and detail rows with transactions by date:
AAAA.1234 /* (account number)*/ 1/1/2015Â Â Â Â Â $150Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â First Transaction 1/3/2015Â Â Â Â Â $24.233Â Â Â Â Â Â Â Â Â Â Â Â Â Second Transaction
BBBB.5678 1/1/2015Â Â Â Â Â $350Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Third Transaction 1/3/2015Â Â Â Â Â $24.233Â Â Â Â Â Â Â Â Â Â Â Â Â Fourth Transaction
My Import program saves this spreadsheet at tab-delimited text, then I use BULK INSERT to bring the data into a generic table full of varchar(255) fields. There are about 90,000 rows in each day's data; after the BULK INSERT about half of them are removed for various reasons. Next I add a RowID column to the table with the IDENTITY (1,1) property. This gives my raw data unique row numbers.
I then run a routine that converts and copies those records into another holding table that's a copy of the final destination table. That routine parses though the data, assigning the account number in the section header to each detail row. It ends up looking like this:
AAAA.1234Â Â Â Â 1/1/2015Â Â Â Â Â $150Â Â Â Â Â Â Â Â Â First Purchase AAAA.1234Â Â Â Â 1/3/2015Â Â Â Â Â $24.233Â Â Â Â Â Second Purchase BBBB.5678Â Â Â Â 1/1/2015Â Â Â Â Â $350Â Â Â Â Â Â Â Â Â Â Â Â Third Purchase BBBB.5678Â Â Â Â 1/3/2015Â Â Â Â Â $24.233Â Â Â Â Â Â Â Fourth Purchase
My technique: I use a cursor to get the starting RowID for each Account Number: I then use the upper and lower RowIDs to do an INSERT into the final table. The query looks like this:
SELECT RowID, SUBSTRING(RowHeader, 6,4) + '.UBC1' AS AccountNumber FROM Â GenericTable WHERE RowHeader LIKE '____.____%'
Results look like this:
But every time I run the routine, I get different numbers! my results are not accurate. I get inconsistent results EVERY TIME.Here is my code, with table, field and account names changed for business confidentiality.This is a high profile project at my company;
TRUNCATE TABLE GenericImportTable; ALTER TABLE GenericImportTable DROP COLUMN RowID; BULK INSERT GenericImportTable FROM 'SERVERGeneralAppnameDataFile.2015.05.04.tab.txt' WITH (FIELDTERMINATOR = ' ', ROWTERMINATOR = ' ', FIRSTROW = 6)
I designed an SSIS package about 200 packages in one project. the package extract from live to reporting server. some of my packages are very slow about 10 of them. strage enough the ones with more data number of rows run very fast. I'm using Source->Lookup->Conditional split->OLEDB Destination or OLDB Comand.
Can someone help me to found out what could be the problem. I'm very new in SSIS.
I have to perform a lookup in a table based on a query like:
"... where ? = [RefTable].fieldID and ? between [RefTable].AnotherFieldValue and [RefTable].AThirdFieldValue"
So, SSIS has put the CacheType to none. As I really need to speed up the job I want to set the CacheType to partial (full isn't an option due to the custom query I use here).
But here it comes: when using partial CacheType, one has to set the cache size manually - and I really don't know what value I should assign to it - is there a guideline on this topic?
I work on a Win2003 server platform with sql server 2005 - 2 processors - 2Gb Ram - enough disc space
I searched around and not been able to find any guidance on this question: if I am designing a lookup transformation, how do I decide what I should set the cache size to?
For the transformation on which I am currently working, the size of the lookup table will be small (like a dozen rows), so should I just reduce it to 1MB? Or should I not even bother with caching for a lookup table that small?
Hypothetically speaking, if I am working with much larger lookups in the future (let's say 30,000 rows in the lookup table), is there some methor or formula that I should use to try to figure out the best cache size?
If the cache size is set larger than the actual data being cached, is the entire cache size allocated in memory, or is the cache size managed to only be as large as it needs to be? In other words, is the cache size a maximum to which the cache will grow, or is it a preset that sets the cache for that lookup to exactly the specified size?
I've a simple lookup transform in SSIS 2008 (R2). I've created it with a full cache and it worked fine. When i switch to partial cache, it will give me this error:
-------------------------------------------------------------------------------------------------- TITLE: Package Validation Error ------------------------------ Package Validation Error ------------------------------ ADDITIONAL INFORMATION: Error at DFT_AdventureWorks [Lookup [411]]: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
[Code] ....
I've created a OLE source with the following query :
SELECT SalesOrderID, OrderDate, CustomerID FROM Sales.SalesOrderHeader
And this will flow into the lookup transform and this has the following lookup reference query:
SELECT CustomerID, AccountNumber FROM Sales.Customer WHERE CustomerID % 7 <>0
I am using a lookup component in a SSIS data flow. The lookup is a select to a foxpro table. THe lookup works fine with full cache selected. I cannot get the lookup to work with a partial or no cache. I have the latest Foxpro OLE DB driver installed which I understand to support paramaterized queries. Has anyone had success with using cached lookup to Foxpro? Does anyone know how to set the lookup properties of sqlcommand and sqlcommandparam? I am unable to find any examples in BOL or on the web.
Here are some details. IF I go with "use a table or a view" option with the default cache query I get initialization errors
[lkp_lab_worst_value [6170]] Error: An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Visual FoxPro" Hresult: 0x80040E14 Description: "Command contains unrecognized phrase/keyword.".
In the advanced editor I see
SQLCommand set to
"select * from `kcf`"
and SQLCommandParam set to
"select * from (select * from `kcf`) as refTable where [refTable].[patkey] = ? and [refTable].[dayof_stay] = ? and [refTable].[modifier] = ? and [refTable].[kcf_code] = ? and [refTable].[source] = ? and [refTable].[kcf_time] = ?"
I believe the above error is because Foxpro V7 does not support the inner subselect . In addition the query contains CRLF without a continuation character (";").
If I remove the CRLF in the sqlcommandparam query, using the advanced editor, I get this design time error "OLE D error occurred while loading column metadata. Check the sqlcommand and sqlcommandparam properties". The designer requires both properties to be set, its unclear to me how the interact.
I cannot find any examples in BOL or on the web on how to set these 2 properties. Can someone give me a few guidelines?
I can get past the design errors by changing sqlcommandparam to a plain select that is VFP 7 compatible ( I removed the subselect and the square brackets):
select * from kcf as refTable where refTable.patkey = ? and refTable.dayof_stay = ? and refTable.modifier = ? and refTable.kcf_code = ? and refTable.source = ? and refTable.kcf_time = ?
But then I get a runtime error
[lkp_lab_worst_value [6170]] Error: An OLE DB error has occurred. Error code: 0x80040E46. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Visual FoxPro" Hresult: 0x80040E46 Description: "One or more accessor flags were invalid.".
[lkp_lab_worst_value [6170]] Error: OLE DB error occurred while binding parameters. Check SQLCommand and SqlCommandParam properties.
After converting from SSIS 2008 to SSIS 2012, I am facing major performance slowdown while loading fact data.When we used 2008 - one file used to take around 2 hours average and now after converting to 2012 - it took 17 hours to load one file. This is the current scenario: We load data into Staging and Select everything from Staging (28 million rows) and use a lookups for each dimension. I believe it is taking very long time due to one Dimension table which has (89 million rows).Â
With the lookup, we currently are using partial cache because full cache caused system out of memory.Lookup Transformation Editor - on this lookup -Â how to increase the size on partial Cache size 64-bit? I am being stuck at 4096 MB and can not increase it. In 2008, I had 200,000 MB partial cache size.
I work in the healthcare area, and am handling the survey data ETL's. There are around 8 different survey areas and based on information received from them for the visit they reference, I want to pull in more info from our invoicing database. My idea is this:
1.) Pull in the flat file to an ODBC staging table 2.) Cache all invoice records that fall between the MIN(Date of Service) and MAX(Date of Service) from the staging table. 3.) First lookup the information needed on patientID, providerID, date of service, and billing location. 4.) For the surveys that didn't match on those 4 columns, try looking up based on patientID, date of service, and billing location (since I could be 99% sure this would still return the record I need). 5.) For the remaining surveys, lookup based just on patientID and date of service. These records will be flagged for manual review because clearly, if a patient has multiple appointments in the same day, this will be prone to error.
However, in trying to use only 3 of the columns in the lookup, I get the error saying basically that I need to utilize all 4. Is there a way around this, or is there an entirely different way I should be approaching this? The reason I thought cache transform was the answer is because I will need to run a different package for each lookup, as the data and logic between each survey will vary, but the invoice data "pool" will stay the same regardless.Â
I have an Excel file which contains some data. I want to load that into a SQL server Table. Here are my conditions :
1. If the table doesn't have any matching records from the Excel file, then my DFT should load the data from that Excel to the Dest Table.
2. If the table has even one or more matching records, then the DFT should not process at all, instead I should send an email to the business stating that there are some matching records and hence the package is not process...ed.
P.S. If i use Lookup, I have two matching and non-matching output. which will process the non matching records into the table and matching can be redirected to any flat/Excel file. But i don't want to do this. I just want to lookup the Sql Server table and excel.
It'll be good if there is an additional option in the Lookup "Fail component on matching records".
Hi,I am using SQL Server 2005 32 bit version. Could any one clarify methe caption database option with some examples? I am quite confusedwith the explanation given in MSDN. Kindly help me.Thanks in Advance.Om Prakash
Hello, I have installed SQL Server 2005 with Advance Services and also SQL Server Management Studio.I have a database that I am trying to create a FullTextSearch for in SSMS, but the "use full-text indexing" checkbox is greyed out, or disabled, how can I enable it?I have also tried the "CREATE FULLTEXT CATALOG" command and that doesn't work.What have I done wrong?Thanks,Louis
In TSQL, is there a way to 'Set' the query option results in text as the head of my sql statement?
kind of tired of hit contol+t to get the text result.
also, when I print '@dateStart--' + cast(@dateStart as varchar(30)) I don't get seconds and I even need mis. Is there a simple way instead of a combination of datepart() to get ss, mis?
I have a Managers table in my SQL DB and it has a ManagerID, MgrName and MgrPhoto field base...
I can code a form with a select button that displays a drop down with the managers in it as choices, but am a little confused as to how I would make the PHOTO (MgrPhoto) change to the corresponding ManagerID that is selected from that option drop down since the SELECT CODE in the form only queries the ManagerID / MgrName combo for the choice.. The photo is below that drop down box and how to make it change to whatever photo is assigned to the selected ManagerID / MgrName ?
Here is the form code with query :
<form enctype="multipart/form-data" action="updatemanagerphoto.php" method="POST"> <select name="ManagerID" id="manager" style="width:400px" class="form_textbox"> <? $db_connect= mysql_connect($db_host, $db_username, $db_password); mysql_select_db($db_name, $db_connect) || die(mysql_error()); $sql_query= "SELECT * FROM Managers ORDER BY MgrName ASC";
I have a table "Ratings" in which I store rating categories like the following:
FROM TO Rating 1 10 1 11 20 2 21 30 3
In another table "Results" I have the actual results like:
Employee ID Results Rating 120 6 ? 130 18 ?
Now I would like calculate the rating in the results table in the rating column (where "?" in the above example)
How can I do that with an SQL update command - maybe using a Case statement - but I don't know how to do that by looking up values from another table...
Parameterized queries are only allowed on partial or none cache style lookup transforms, not 'full' ones. Is there some "trick" to parameterizing a full cache lookup, or should the join simply be done at the source, obviating the need for a full cache lookup at all (other suggestion certainly welcome)
More particularly, I'd like to use the lookup transform in a surrogate key pipeline. However, the dimension is large (900 million rows), so its would be useful to restrict the lookup transform's cache by a join to the source.
For example:
Source query is: select a,b,c from t where z=@filter (20,000 rows)
Lookup transform query: select surrogate_key,business_key from dimension (900 M rows, not tenable)
Hi, Is it advisable to use full text search on asp.net live applications? Will live database servers allow full text search?. If so what are the problems we could face in future. My DB server is SQL Server2000 and ASP version is 1.1. Thanks in advance Hamlin Stephen
i'd like to use ssis on a certain project but am concerned that one of my transformations needs lookup results to be based on actions taken on previous lookups and that the toolkit doesnt really offer something like that.
so, i have a dataflow whose first component extracts certain kinds of data from an xml document.
each row returned by the latter needs a lookup but the results of that lookup may dictate a certain kind of update. The next row's lookup may need to be influenced by the previous row's update.
So I think I have two challenges, 1) combining a lookup and update, 2) making sure the buffer architecture completes one lookup and update before the next lookup begins.
Im not too sure if this is the category to post this...but what value can I pass in the CONTAINS method that will behave like the "*" character in a SELECT query?
Server: Msg 1105, Level 17, State 2, Line 2Could not allocate space for object '(SYSTEM table id: -276436763)' indatabase 'TEMPDB' because the 'DEFAULT' filegroup is full.this error occured during dbcc checkdb the total size of the file we aredoing on is [81GB].what is the next step we need to takewe increased about 3699.mb in temp dboverall we have 8 databases in our serveranyone help would me much appreciated.Thanks,pardhi--Message posted via http://www.sqlmonster.com
I got a message that 'Could not allocate space for object '(SYSTEM table id: -732777483)' in database 'TEMPDB' because the 'DEFAULT' filegroup is full. Connection Broken'
I have a DB creation script which starts with the following lines
EXEC sp_addrole 'RAVENROLE' go IF NOT EXISTS (SELECT * FROM sys.database_principals WHERE name = N'USER1') CREATE USER [USER1] FOR LOGIN [USER1] WITH DEFAULT_SCHEMA=[USER1] go IF NOT EXISTS (SELECT * FROM sys.schemas WHERE name = N'USER1') EXEC sys.sp_executesql N'CREATE SCHEMA [USER1] AUTHORIZATION [USER1]' GO EXEC sp_addrolemember 'RAVENROLE', 'WS-NOLANKTFSBUILD2'
Basically what I am doing is creating a role, then adding user USER1 to it. The problem is USER1 has full update, insert etc. permissions on all tables in the database by default. Even when I deny that user permissions they can still update, insert etc.
I am using Windows Authentication, User1 also exists as a windows user.
How do I ensure that User1 has no permissions by default? I want to be able to explicitly add them.
Im getting this error when trying to set up a cache dependency...are there any special permissions etc?From CS:SqlCacheDependency dep = new SqlCacheDependency("MySite-Cache", "Products");Cache.Insert("Products", de.GetAllProductsList(), dep); From connectionStrings.config:<add name="SiteDB" connectionString="Data Source=localhost,[port]SQLEXPRESS;Integrated Security=true;User Instance=true; AttachDBFileName=|DataDirectory|ASPNETDB.MDF" providerName="System.Data.SqlClient" />Also tried this using my machinename<add name="SiteDB" connectionString="Data Source=<machinename>,[port]SQLEXPRESS;Integrated Security=true;User Instance=true; AttachDBFileName=|DataDirectory|ASPNETDB.MDF" providerName="System.Data.SqlClient" /> From web.config: <caching> <sqlCacheDependency enabled="true" pollTime="10000"> <databases> <add name="MySite-Cache" connectionStringName="SiteDB" pollTime="2000"/> </databases> </sqlCacheDependency> </caching> EDIT: So making progress I can't seem to get the table registered for cache dependency:The sample i have says"aspnet_regsql.exe -E -S .SqlExpress -d aspnetdb -t Customers -et"and the command line response is "Enabling the table for SQL cache dependency..An error has happened. Details of the exception:The table 'Customers' cannot be found in the database."Where does this "Customers" table come from? There is obviously not an application specific "Customers" table in aspnetdb I'm confused probably more by the example than anything....
I want to allow a user to only select from certain tables in a SQL 2005 database. Without granting any permissions, the user can select from any table.
All tables are owned by dbo. I am using a Windows login and it is not a member of any groups other than one to allow Terminal Server logins.
SQL login properties: Only server role is Public Mapped to a user with the same name Default schema is dbo No securables listed
User properties: No owned schemas No role memberships No securables listed
Database permissions: Under explicit permissions, this user is only granted Connect Effective permissions lists everything!!!
Where are these effective permissions coming from? TIA
Is there a way to drop clean buffers at the database level instead of the server/instance level like the undocumented €œDBCC FLUSHPROCINDB (@dbid)€?? Is there a workaround for €œdbo€? to be able to flush procedure and data cache without being elevated to €œsysadmin€? server role?
PS: I am aware of the sp_recompile option that can be used to invalidate cached execution plans. Thx.
I need to load a table with 820,000 records from a Sybase db via DTS. It always fail with the error: "Error at destination for row number 820000. Could not allocate space for object in tablespace tempdb . The default filegroup is full.".
There is only the primary filegroup defined in the db. I've increased the size from 1.5GB to 2GB, and specify that it shd grow automatically by 10% and there's no limit to the size. There is still some 28GB in the server, so it should be fine.
It still fail so I added another file to the primary filegroup with size 100MB. Again, it failed with the same error msg.
When the SSISDB database is created, it appears to be using the "Full" recovery model by default. Because of the recovery model, the SSISDB transaction log needs to be regularly backed up or I risk running out of disk space.I would like to set the recovery model to "Simple", so that I do not need to worry about the transaction log consuming too much space. However, I am not sure what the consequences of that action are.features lost by switching the recovery model of the SSISDB database from "Full" to "Simple"?
Hi - I am trying to add paging to my stored procedure. The stored procedure successfully executes a full-text search. Unfortunately, the paging routine below the full-text search operates on the articles table after the search has been conducted. This means that it utilizes the row numbers from the entire table rather than the row numbers from the result set. I somehow need to the paging routine starting at "WITH tempArticles AS" to operate on the search results rather than the articles table. I am too new to SQL and can't figure out how to populate a temporary table storing text search results to use in this paging routine. Can anyone help me with this? Thanks! Peter dbo.Search_Articles @searchText varchar(150), @PageSize int, @PageNumber int AS Declare @RowStart int Declare @RowEnd int if @PageNumber > 0 Begin SET @PageNumber = @PageNumber -1; SET @RowStart = @PageSize * @PageNumber + 1; SET @RowEnd = @RowStart + @PageSize - 1 ; CREATE TABLE #results (ArticleID int) INSERT INTO #results SELECT ArticleID FROM articles WHERE CONTAINS(Description, @searchText) OR CONTAINS(Title, @searchText) UNION SELECT ArticleID FROM article_pages WHERE CONTAINS(Text, @searchText) /*this returns all matching records from the text search*/ SELECT * FROM articles, #results WHERE #results.ArticleID=articles.ArticleID; WITH tempArticles AS ( SELECT Title, PostDate, UserID, City, Country, Tags, StoryID, Approved, ROW_NUMBER() OVER (order by PostDate) as RowNumber FROM articles WHERE Approved = 1) SELECT * FROM tempArticles WHERE RowNumber >= @RowStart and RowNumber <= @RowEnd; END
