Data Warehousing :: How To Increase Partial Cache Size On Lookup Stuck
Apr 20, 2015
After converting from SSIS 2008 to SSIS 2012, I am facing major performance slowdown while loading fact data.When we used 2008 - one file used to take around 2 hours average and now after converting to 2012 - it took 17 hours to load one file. This is the current scenario: We load data into Staging and Select everything from Staging (28 million rows) and use a lookups for each dimension. I believe it is taking very long time due to one Dimension table which has (89 million rows).
With the lookup, we currently are using partial cache because full cache caused system out of memory.Lookup Transformation Editor - on this lookup - how to increase the size on partial Cache size 64-bit? I am being stuck at 4096 MB and can not increase it. In 2008, I had 200,000 MB partial cache size.
I've a simple lookup transform in SSIS 2008 (R2). I've created it with a full cache and it worked fine. When i switch to partial cache, it will give me this error:
-------------------------------------------------------------------------------------------------- TITLE: Package Validation Error ------------------------------ Package Validation Error ------------------------------ ADDITIONAL INFORMATION: Error at DFT_AdventureWorks [Lookup [411]]: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
[Code] ....
I've created a OLE source with the following query :
SELECT SalesOrderID, OrderDate, CustomerID FROM Sales.SalesOrderHeader
And this will flow into the lookup transform and this has the following lookup reference query:
SELECT CustomerID, AccountNumber FROM Sales.Customer WHERE CustomerID % 7 <>0
I have to perform a lookup in a table based on a query like:
"... where ? = [RefTable].fieldID and ? between [RefTable].AnotherFieldValue and [RefTable].AThirdFieldValue"
So, SSIS has put the CacheType to none. As I really need to speed up the job I want to set the CacheType to partial (full isn't an option due to the custom query I use here).
But here it comes: when using partial CacheType, one has to set the cache size manually - and I really don't know what value I should assign to it - is there a guideline on this topic?
I work on a Win2003 server platform with sql server 2005 - 2 processors - 2Gb Ram - enough disc space
I searched around and not been able to find any guidance on this question: if I am designing a lookup transformation, how do I decide what I should set the cache size to?
For the transformation on which I am currently working, the size of the lookup table will be small (like a dozen rows), so should I just reduce it to 1MB? Or should I not even bother with caching for a lookup table that small?
Hypothetically speaking, if I am working with much larger lookups in the future (let's say 30,000 rows in the lookup table), is there some methor or formula that I should use to try to figure out the best cache size?
If the cache size is set larger than the actual data being cached, is the entire cache size allocated in memory, or is the cache size managed to only be as large as it needs to be? In other words, is the cache size a maximum to which the cache will grow, or is it a preset that sets the cache for that lookup to exactly the specified size?
I have table having around 100 million rows.Everyday we have an ETL process in which table will be trucnated and relaoded. Will creating a partition on the table increase the inserting speed?
Hi All, Is it possible to increase the size of data file in SQL Server 2005 Express Edition. I think the licensed limit is 4096MB whcih i am unable to increase. Could anyone let if is it possible to increase the data file size and if yes then how?
I encounter one weird problem, I have a database with around 7 GB ...when I delete a bunch of data from it, it suppose to reduce thedatabase file size, but weirdly, the file size increase to 8 GB.Wondering why. Is it suppose to be like that?Is it the architecture is designed to work like that?Is there any way for me to reduce the database file size?Thanks.Peter CCH
I have a question, I keep getting the warning where the lookup found duplicate reference key values when caching reference data. It also says to switch over to partial or no cache mode. I switched to partial and and it took way longer to run my package. I have around 200,000 records right now but this table will continue to grow. I guess my question is which mode should I be using? I have another table that will have millions of records in it.
Need to confirm if we can add space(increase data file size) for the database which is configured for always on similar to that of mirroring or we need to follow any different procedure.
I have a requirement wherein the datafiles on both the primary and secondary replica got full, if i add space to the primary database will it automatically get added to the secondary replica or not?
Can anyone explain to me (or show me where to get info) in what scenario should I use Full Cache, Partial, or No_Cache mode in the Lookup task? I got below warning message during execute my package. I understand that my lookup data have duplicate reference keys. I want to change the mode to partial or no cache to see how it works, but don't know what to change for the parameter map.
Warning: 0x802090E4 at Data Flow Task, Lookup [37]: The Lookup transformation encountered duplicate reference key values when caching reference data. The Lookup transformation found duplicate key values when caching metadata in PreExecute. This error occurs in Full Cache mode only. Either remove the duplicate key values, or change the cache mode to PARTIAL or NO_CACHE.
I designed an SSIS package about 200 packages in one project. the package extract from live to reporting server. some of my packages are very slow about 10 of them. strage enough the ones with more data number of rows run very fast. I'm using Source->Lookup->Conditional split->OLEDB Destination or OLDB Comand.
Can someone help me to found out what could be the problem. I'm very new in SSIS.
Hi all, I encountered error 1105 (Can`t allocate space for object .....) while running a query. I had dump transaction with no log sucessfully and retry running my query but still encounter error 1105.
I had checked my database size and following are the info: Datasize : 2500MB Database size available : 753.39 Log space : 500MB Log space available : 500MB (this was acheive after after dump trans. twice. it was around 480MB before)
Thus looks like the data segment is full!!! I expanded the size of the disk device D on which the segment device A is on by 50MB. When I did a sp_helpdevice D, the expansion is reflected. Next I run sp_extendsegement, A, D. However running sp_helpsegment A doesn`t show that segment A is expanded. Please help!!! How do I expand the size of the segment? (without creating a new disk device, is it possible) Any other way to resolve this problem?
Also how to check if a particular segment is full? eg. the %usage of data/log segment.
I seem to remember that the font size and type can be controlled in SQL Server 6.5 and the process controlls the fonts displayed throughout the program - Enterprise Manager, Query Tool, etc.
I have just re-installed 6.5 and the fonts are way too small.
i've a db running sql server express sp2. the db size now is 1.1 gb i've a table with a varchar column of size 20 . when i try to increase the column size to 50 i get a timeout exception, and the the cloumn size is unchanged. this table has 2.5 million records
i use sql server management studio express to do the changes
is there a way to increase this timeout or whtever i can do to update this column size?
How can I increase the transaction log size of my sql server 2005 database? i can execute the shrink and backup log which can shrink the .ldf file to 1MB. However, since my dts involves a lot of inserts and updates, the my transaction log grows to 700MB in the middle of my dts execution. How can i increase the transaction log size maximum capacity to probably 1.5GB just so it can accommodate my full dts execution?
A few months ago a customer moved from SQL 2000 to SQL 2005. The db wasbacked up on SQL 2000 and restored to SQL 2005. The application using thisdata works on SQL 2005 but takes no advantage of new features. The db onSQL 2000 was about 2.9GB, now on SQL 2005, it is 16.5GB. The db is set toSimple recovery so trx log is only 2mb. The mdf file is 16.5GB, ManagementStudio shows only 5mb free space. There has not been a huge increase intransactions.One of the largest tables has only added 2,000 rows since the move to SQL2005. Yet the data and index size has jumped from about 400mb to 3.5 GB. Iused the 'BigTables.sql' script found at various SQL sites:www.databasejournal.com/img/BigTables.sqlAny ideas why such a large increase?Thanks
I am using a lookup and full cache, occasionally i get this warning:[Lookup [150]] Warning: The component "Lookup" (150) encountered duplicate reference key values when caching reference data. This error occurs in Full Cache mode only. Either remove the duplicate key values, or change the cache mode to PARTIAL or NO_CACHE. Now I know it is only a warning but it is highlighting a real issue.Is there a way of capturing that this has happened?
I'm trying to query a table where in the data in a cell is 65KB and when i try to do a SELECT I am unable to get the entire data from the cell.
SELECT CAST(Xml_data as XML) from TableName where ID=100 Error Message: Msg 9448, Level 16, State 1, Line 1 XML parsing: line 241, character 76, well formed check: undeclared entity
No transaction log involved, only the table itself.
Use sp_spaceused "table_name" to check the space used.
It seems the table size actually increased from the beginning to the middle of deletion, at the end of deletion, its size decreased.
Recovery mode set to be simple, autoshrink turned on.
The tables tested are about 50MB ~ several GB in size, all have the same behavior. The size increased about 5%~10%.
Since the deletion is called from another software, I want to know if it is possible for SQL Server to have this behavior or it is absolutely the 3rd party software's issue
I am using a lookup component in a SSIS data flow. The lookup is a select to a foxpro table. THe lookup works fine with full cache selected. I cannot get the lookup to work with a partial or no cache. I have the latest Foxpro OLE DB driver installed which I understand to support paramaterized queries. Has anyone had success with using cached lookup to Foxpro? Does anyone know how to set the lookup properties of sqlcommand and sqlcommandparam? I am unable to find any examples in BOL or on the web.
Here are some details. IF I go with "use a table or a view" option with the default cache query I get initialization errors
[lkp_lab_worst_value [6170]] Error: An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Visual FoxPro" Hresult: 0x80040E14 Description: "Command contains unrecognized phrase/keyword.".
In the advanced editor I see
SQLCommand set to
"select * from `kcf`"
and SQLCommandParam set to
"select * from (select * from `kcf`) as refTable where [refTable].[patkey] = ? and [refTable].[dayof_stay] = ? and [refTable].[modifier] = ? and [refTable].[kcf_code] = ? and [refTable].[source] = ? and [refTable].[kcf_time] = ?"
I believe the above error is because Foxpro V7 does not support the inner subselect . In addition the query contains CRLF without a continuation character (";").
If I remove the CRLF in the sqlcommandparam query, using the advanced editor, I get this design time error "OLE D error occurred while loading column metadata. Check the sqlcommand and sqlcommandparam properties". The designer requires both properties to be set, its unclear to me how the interact.
I cannot find any examples in BOL or on the web on how to set these 2 properties. Can someone give me a few guidelines?
I can get past the design errors by changing sqlcommandparam to a plain select that is VFP 7 compatible ( I removed the subselect and the square brackets):
select * from kcf as refTable where refTable.patkey = ? and refTable.dayof_stay = ? and refTable.modifier = ? and refTable.kcf_code = ? and refTable.source = ? and refTable.kcf_time = ?
But then I get a runtime error
[lkp_lab_worst_value [6170]] Error: An OLE DB error has occurred. Error code: 0x80040E46. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Visual FoxPro" Hresult: 0x80040E46 Description: "One or more accessor flags were invalid.".
[lkp_lab_worst_value [6170]] Error: OLE DB error occurred while binding parameters. Check SQLCommand and SqlCommandParam properties.
I'm either missing something or this is a bug. I have a Lookup that finds no matches if I use the default option of full caching (everything on the Advanced tab unchecked). The lookup table is relatively small (15348 bytes) in only 544 rows. If I check only the Enable Memory Restriction box and eliminate caching, it works fine. I can also check the Enable Caching box and accept the default cache size of 5MB and it works fine. Anyone have any ideas? I'm running on Standard Edition, SP2.
I save Table size and recs. no every day. and check it some days.
... insert into @t exec sp_msforeachtable 'exec sp_spaceused ''?''' ...
But Today I saw sudden increase size in a table. about 128 MB in a day. (Average Growth fro this table was 4 or 5 MB in a day)This growth was for Only 4222 Records. While for more number of records (about 7000) in yesterday we had only 2 MB GRowth!
This Table information (Now):
sp_spaceused 'Table1'
Result:
name ---Rows --reserved --data
Table1--1021319--460328 KB --283104 KBI Try to gess The reason. I copy These new records to another table.But The result was more strange : on new table the size of these record was : < 1 MB I copied All records to another table . The size was : 148 MB (while this is 283 MB in my real database)
