I am not an expert in either SSIS or VFP technology but know enough to get my way round. One anomaly I did discover I thought was worth sharing for all those concerned with getting large amounts of data out of VFP in as short a time as possible. When you search for performance tips in relation to SSIS the advice is to never use select table or view from data access mode list in ole db source as this effectively translates to select * from table and I've never come across anything to contradict this €“ well I am and let me explain why:
When you use SQL command as data access mode in ole db source (where ole db source is foxpro dbc) and you write out select column1, column 2 etc etc from table a etc etc and then connect that to a destination (in my case ole db destination) the SSIS task spends ages stuck on Pre-execute before anything happens (the bigger the fox tbl the longer the wait). What is happening behind scenes is that the foxpro engine (assuming its foxpro engine and not sql engine €“ either way don€™t think it matters too much) is executing the sql command and then writing results to a tmp (temp) file on your local temp folder €“ (in my case : C:Documents and SettingsautologinLocal SettingsTemp1). These files take up gigs of space and it is only when this process is complete does the SSIS task actually finish the Pre-execute and start the data transfer process. I couldn€™t understand a) why my packages were stuck on pre-execute for such long times? and b) why were the tmp files being created and why they were soo big?
If you change from SQL command in source to Table or view and then select your table from list the SSIS task when executed kicks off immediately and doesn€™t get stuck on pre-execute nor create any tmp files €“ so you save time and disk space. The difference in time is immense and if like me you were really frustrated with poor performance when extracting from VFP now you know why.
Btw maybe this does not apply to all versions of VFP but it certainly does to v7.
Hi, We have a poorly performing SQL 2000 db. i have just defragged ( the HD, not indexes, these are done daily via SQL Agent) the data files of our server and have not found any improvement in response. I have now got into using SQL profiler to analyse the server performance. in the results that the trace is returning there are some huge (REALLY BIG) values for the duration and cpu values but these rows have no textdata value returned (ie it is null)
why is this? for these rows, the reads and writes columns are also high.
if these rows are what is taking the cpu's time then how can i identify what the server is doing to make any changes?
any thoughts on what other values i might trace or what action i can take to find the slow down cause?
in performance manager the processors (dual Xeons) are rarely dropping below 60%.
Dear Experts, I'm a DBA, Working for a Product based company. We are implementing our product for a certain client of huge OLTP. our reports team is facing problem (error: all the reports are timed out).though the queries are written properly, Each query is taking some minutes of time. I've given the command DBCC DROPCLEANBUFFERS. the time immediately dropped to 10 sec.
now my question is : please suggest me the DBCC commands or any DBA related commands to improve the performance of the application for my reports team.
from your experience in SQL 2005 - do i have any free software that can help in improve performance or can help in identifying performance bottleneck. two examples of performance and help that i use usually use are the maintenance plan that do (check DB > reorganized index > rebuild index > update statics) and the second software is the SQL 2005 DASHBOARD for the reporting help. do you have any other free tools and help that you can give me for performance or any thing that i must have in my SQL 2005 servers.
I have a table called work_order which has over 1 million records and acontractor table which has over 3000 records.When i run this query ,it takes long time since its grouping bycontractor and doing multiple sub SELECTs.is there any way to improve performance of this query ??-------------------------------------------------SELECT ckey,cnam,t1.contractor_id,count(*) as tcnt,(SELECT count(*) FROM work_order t2 WHEREt1.contractor_id=t2.contractor_id and rrstm=1 and rcdt is NULL) as r1,(SELECT count(*) FROM work_order t3 WHEREt1.contractor_id=t3.contractor_id and rrstm=2 and rcdt is NULL) as r2,(SELECT count(*) FROM work_order t4 WHEREt1.contractor_id=t4.contractor_id and rrstm=3 and rcdt is NULL) as r3,SELECT count(*) FROM work_order t5 WHEREt1.contractor_id=t5.contractor_id and rrstm=4 and rcdt is NULL) as r4,(SELECT count(*) FROM work_order t6 WHEREt1.contractor_id=t6.contractor_id and rrstm=5 and rcdt is NULL) as r5,(SELECT count(*) FROM work_order t7 WHEREt1.contractor_id=t7.contractor_id and rrstm=6 and rcdt is NULL) as r6,SELECT count(*) FROM work_order t8 WHEREt1.contractor_id=t8.contractor_id and rcdt is NULL) as open_count,(SELECT count(*) FROM work_order t9 WHEREt1.contractor_id=t9.contractor_id and vendor_rec is not NULL) asAck_count,(SELECT count(*) FROM work_order t10 WHEREt1.contractor_id=t10.contractor_id and (rtyp is NULL or rtyp<>'R') andrcdt is NULL) as open_norwoFROM work_order t1,contractor WHEREt1.contractor_id=contractor.contractor_id andcontractor.tms_user_id is not NULL GROUP BYckey,cnam,t1.contractor_id ORDER BY cnam*** Sent via Developersdex http://www.developersdex.com ***Don't just participate in USENET...get rewarded for it!
Hey guys,Here's my situation:I have a table called lets say 'Tree', as illustred bellow:Tree====TreeId (integer)(identity) not nullL1(integer)L2(integer)L3(integer)....L10(integer)The combination of the values of L1 thru L10 is called a "Path" , andL1 thru L10 values are stored in a second table lets say called'Leaf':Leaf====LeafId (integer)(identity) not nullLeatText varchar(2000)Here's my problem:I need to lookup for a given keyword in each path of the tree table,and return each individual column for the paths that match thecriteria. Here's the main idea of how I have this now.SELECT TreeId,L1,L2,...,L10, GetText(L1) + GetText(L2) as L2text + ...+ GetText(L10) AS PathTextINTO #tmp FROM Tree //GetText is a lookup function for the Leaf tableSELECT L1,GetText(L1),L2,GetText(L2),...,L10,GetText(L10) FROM #tmpWHERECharIndex(@keyword,a.pathtext) > 0Does anyone would know a better,smart, more efficient way toaccomplish this task? :)Thks,
I'm facing a performance issue with the following query... The Output of the following Query is 184 Records and it takes 2 to 3 secs to execute the query.
SELECT DISTINCT Column1 FROM Table1 (NOLOCK) WHERE Column1 NOT IN
(SELECT T1.Column1 FROM Table1 T1(NOLOCK) JOIN Table2 T2 (NOLOCK)
ON T2.Column2 = T1.Column2 WHERE T2.Column3= <Value>)
Data Info.
No of records in Table1 --> 1377366
No. of distinct records of Column1 in Table1 --> 33240
Is there any way the above query can be rewritten to improve the performance, which should take less than 1 sec... (I'm using DISTINCT because there are Duplicate records of Column1 inTable1 )
Any of your help in this regard will be greately appreciated.
Hi, I have database D1 which contains 5 million users and one more database D2 having 95k Users. i wanted to insert common users into new database D3 based on filter which is Phone number and is unique value. Below is the structure of my tables in D1 and D2:
Now userProfiles table contains data in string format as below: User.state AA User.City CC User.Pin 1234 User.phonenumber 987654
so iam parsing for each user using cursor and writing phone numbers into some temp table and wanted to query D2 database to verify whether this phone number exists in Alerts Table of D2 database.
can anyone please suggest on how i can go ahead with this and also help me on how to improve perfomance.
I've been working as a .NET developer for many years. However, my educational background isn't in IT at all, so I occasionally find that there are big gaps in my knowledge when I'm called on to do certain things.
My current role is the first one I've had where the efficiency of my database calls has needed to be top-notch. Most of my previous posts have been in building intranet functionality for medium-sized companies so it's just not been an issue and my fairly basic level of SQL knowledge has been enough to get me by.
However, I'm increasingly finding I'm needing to look stuff up, ask for help and so forth and I'm not at all sure that what I'm putting out is up to scratch. I have three main areas of concern:
1) The developers here seem to like doing a lot of the "work" of data sorting and manipulation in SQL whereas I'd previously have just grabbed the whole table and done the manipulation in .NET. I'm finding building my complex queries is just doing my head in - I'm just not used to "thinking" in the way that SQL requires which seems very different from procedural programming.
2) A lot gets said about properly indexing tables for maximum efficiency. I know what an index is and vaguely what it does, but the means of applying them to ensure the best performance is pretty much beyond me.
3) In the same manner I'm now having to worry a lot more about performance issues in my database design. I now the five normal forms and so on, but is there anything else here I need to do to maximise performance?
What I'm asking, really, is whether anyone can recommend a good source for SQL knowledge (book, site, whatever) that specifically focuses on my first problem - it'll teach me the proper way to structure my queries and think about them properly so that they don't look so damn impenetrable. It then needs to go on and address my other issues.
Hi everyone I need a solution for this query. It is working fine for 2 tables but when there are 1000's of records in each table and query has more than 2 tables. The process never ends. Here is the query (select siqPid= 1007, t1.Gmt909Time as GmtTime,(t1.engValue+t2.engValue+t3.engValue+t4.engValue) as EngValue, t1.Loc1Time as locTime,t1.msgId into #temp5 from #temp1 as t1,#temp2 as t2,#temp3 as t3,#temp4 as t4 where t1.Loc1Time = t2.Loc1Time and t2.Loc1Time = t3.Loc1Time and t3.Loc1Time = t4.Loc1Time) I was trying to do something with this query.
But the engValues cant be summed up. and if I add that in the query, the query isnt compiling. (select siqPid= 1007, t1.Gmt909Time as GmtTime, t1.Loc1Time as locTime,t1.msgId,(t1.engValue+t2.engValue+t3.engValue+t4.engValue) as engValue --into #temp5 from #temp1 as t1 where exists (Select 1 from #temp2 as t2 where t1.Loc1Time = t2.Loc1Time and exists (Select 1 from #temp3 as t3 where t2.Loc1Time = t3.Loc1Time and exists (Select 1 from #temp4 as t4 where t3.Loc1Time = t4.Loc1Time))))
I need immediate help on that, I would appreciate an input on it.
I should add an Identity field (Identity=True) and a row version field(timestamp) to my table, and avoid to arrange tables into different databases, is it true in general?
I have a view which uses UNION of two tables. First table has a 1.5 Million records and the second one has 40,000 records. When I query the view with a column (that is indexed in both tables) in the where clause, it's taking taking 3 Minutes to give the result. The column is of DateTime data Type. Any ideas as to how to improve the query performance ???
Hello, I have the following setup and I would appreciate any help in improving the performance of the query.
BigTable: Column1 (indexed) Column2 (indexed) Column3 (no index) Column4 (no index)
select [time] = CASE when BT.Column3 = 'value1' then DateAdd(...) when BT.Column3 in ('value2', 'value3') then DateAdd(...) END, Duration = CASE when BT.Column3 = 'value1' then DateDiff(...) when BT.Column3 in ('value2', 'value3') then DateDiff(ss, BT.OrigTime, (select TOP 1 X.OrigTime from BigTable X where X.Column1 > BT.Column1 and X.Column3 <> 'value4' order by X.Column1 )) END,
FROM BigTable BT where BT.Column3 = 'value1' OR (BT.Column3 in ('value2', 'value3') and BT.Column4 <> (select X.Column4 from BigTable X where X.Column1 = BT.Column1 and X.Column3 = 'Value1'))
Apart from the above mentioned, there are a few more columns which are just a part of select statement and are not in any condition statments.
The BigTable has around 1 Mil records and the response time is very poor, it takes around 3 mins to retrieve the records (which would be around 500K)
I have this queston that I cannot get a clear answer on. I have searched the internet to find out if using foreign keys have any performane benefits but some articles yes and some say no. So what should I believe here. Does foreign keys have any performance benefits.
I have a table that contains an ntext column for storing values up to a couple of Mb in size.
However, I estimate that 95% of the values stored in this ntext field will fit into an nvarchar(4000) field.
Is it worth me having both fields in the table?
i.e. For rows where the values < 4000 characters I would store the value in the nvarchar column. Otherwise I would use the ntext column.
Can anyone confirm whether this technique would increase performance given that ntext values are sort of stored separately to the rest of the table data?
A colleague of mine is an Oracle DBA and he mentioned this technique is fairly caommonly adopted in the Oracle world.
i have column in table which contains tabs and " i want replace with space...i am using repalce function is thier other way to improve performance with out using replace function.
Dear All, i've tried with indexed views, but because the view is referenceing another view, i was unable to create a clustered index on that view. so please let me know how can i improve the performance of the view.
thank you very much
Vinod Even you learn 1%, Learn it with 100% confidence.
Hello,I have the following setup and I would appreciate any help in improvingthe performance of the query.BigTable:Column1 (indexed)Column2 (indexed)Column3 (no index)Column4 (no index)select[time] =CASEwhen BT.Column3 = 'value1' then DateAdd(...)when BT.Column3 in ('value2', 'value3') then DateAdd(...)END,Duration =CASEwhen BT.Column3 = 'value1' then DateDiff(...)when BT.Column3 in ('value2', 'value3') then DateDiff(ss,BT.OrigTime, (select TOP 1 X.OrigTime from BigTable X where X.Column1 >BT.Column1 and X.Column3 <> 'value4' order by X.Column1 ))END,FROMBigTable BT where BT.Column3 = 'value1' OR (BT.Column3 in ('value2','value3') and BT.Column4 <> (select X.Column4 from BigTable X whereX.Column1 = BT.Column1 and X.Column3 = 'Value1'))Apart from the above mentioned, there are a few more columns which arejust a part of select statement and are not in any condition statments.The BigTable has around 1 Mil records and the response time is verypoor, it takes around 3 mins to retrieve the records (which would bearound 500K)With the Statistics ON,I get the following:Table 'BigTable'. Scan count 2, logical reads 44184, physical reads 0,read-ahead reads 0.Table 'WorkTable'. Scan count 541221, logical reads 4873218, physicalreads 0, read-ahead reads 0.Is there any way to increase the performance, so that I can get theresult under 1 minute?Any help would be appreciated.P.S: I tried indexing the Column3, but no improvement.
I am developing reporting service and using lots of 'LEFT OUTER JOIN',I am worried about the performance and want to use some subquery toimprovethe performance.Could I do that like below,[the origin source]SELECT *FROM TableALEFT OUTER JOIN TableBON TableA.item1 = TableB.item1WHERE TableA.item2 = 'xxxx'TableB.item2 > yyyy AND TableB.item2 < zzzzI add the subquery to query every table before 'LEFT JOIN'--------------------------------------------------------------------------SELECT *FROM(SELECT *FROM TableAWHERE TableA.item2 = 'xxxx') TableCLEFT OUTER JOIN(SELECT *FROM TableBWHERE TableB.item2 > yyyy AND TableB.item2 < zzzz) TableDON TableC.item1 = TableD.item1WHERE TableC.item2 = 'xxxx'TableD.item2 > yyyy AND TableD.item2 < zzzz--------------------------------------------------------------------------Can anyone give me some suggestion?Thanks a lot.Leland Huang
Hi All,I am getting slower performance of select statements in MS SQL. I amfinding select statements in MS SQL are even slower than MS ACCESS. Isthere any way to improve the performance of select statements in MSSQL by tuning the database of anything else??Thanks in advance!Hoque
Dear Sql Server experts:First off, I am no sql server expert :)A few months ago I put a database into a production environment.Recently, It was brought to my attention that a particular query thatexecuted quite quickly in our dev environment was painfully slow inproduction. I analyzed the the plan on the production server (itlooked good), and then tried quite a few tips that I'd gleaned fromreading newsgroups. Nothing worked. Then on a whim I performed anUPDATE STATISTICS on a few of the tables that were being queried. Thequery immediately went from executing in 61 seconds to under 1 second.I checked to make sure that statistics were being "auto updated" andthey were.Why did I need to run UPDATE STATISTICS? Will I need to again?A little more background info:The database started empty, and has grown quite rapidly in the lastfew months. One particular table grows at a rate of about 300,000records per month. I get fast query times due to a few well placedindexes.A quick question:If I add an index, do statistics get automatically updated for thisnew index immediately?Thanks in advance for any help,Felix
We are thinking about buying new harddrives to improve sql server performance. Currently TEMPDB is running on a dedicated RAID 0 with 3 harddrives of 136 GB, 10.000 RPM. When running a large bulk insert within a SSIS package to 15 destination tables we notice high numbers in the Avg. and current Read Queue length (above 3000) of the drive where TEMPB is on. No other programs or swap file is using this RAID 0 drive. Can anyone tell me if it is worth buying 4 harddrives of 15.000 RPM each 33 GB big replacing the current 3 drives? How much impact will it have on the Avg. and Current Read queue length and will it improve the time sql server needs to bulk insert data?
Hi, I am using compact framework 1.1 and SQL CE database for my mobile application. My database has a total of 160000 rows of records and whenever i do a query searching, it will take about 20 seconds to look through the whole database if the record does not exist. Is there any method to improve the searching performance? i am using data reader for the query.
I am making a ASP.NET web application that involves 2 SQL Server(A & B). I created a view in SQL server A pointing to the table in SQL Server B. I found out my application will run REALLY slow when accessing such a view. so I try to avoid using them. But in the case of 2 table joining from 2 different SQL Servers, I have no choice. Can anyone help me with this? Thanks!
Hi,I am using SQL 2000 and has a table that contains more than 2 millionrows of data (and growing). Right now, I have encountered 2 problems:1) Sometimes, when I try to query against this table, I would get sqlcommand time out. Hence, I did more testing with Query Analyser and tofind out that the same queries would not always take about the sametime to be executed. Could anyone please tell me what would affect thespeed of the query and what is the most important thing among all thefactors? (I could think of the opened connections, server'sCPU/Memory...)2) I am not sure if 2 million rows is considered a lot or not, however,it start to take 5~10 seconds for me to finish some simple queries. Iam wondering what is the best practices to handle this amount of datawhile having a decent performance?Thank you,Charlie Chang[Charlies224@hotmail.com]
I have a SQL Server 2005 database where covering indexes had to be used to improve performance for the heavy amounts of retrievals; however, the inserts into the tables are now very slow of course. Is there any way to improve the performance of the inserts without taking away the indexes.
Would changing locking or partitioning the index help the inserts?
Other databases use a concept of "freespace" to set up in the beginning - making pre-existing space for inserts - is there anything like this in SQL Server 2005?
I have a problem of performance with my XML source component. In my XSD file, I have over 300 outputs, for each output of the XML component I generate a Row file (so I generate 300 Row File in the same Data Flow) that I integrate into a table in another Data Flow. When I debug just the data flow of the XML source, it takes much time over 10 hours and the XML source component keeps a yellow color, and it generate empty Row Files, but when I interrupt the debug process it generates the right row files. For information I use a machine with 8 microprocessor and 4GO of Ram, and my XML file has 850KO. So my question is, how can I increase the performance of my package without splitting the XML file. Thank you in advance
i need to select data by using a very complex sql statement. when i use a ole db source componente and choose SQL command as data access mode the process never ends. but when i put the sql statement in an sql task component it works fine and fast. isn't an oledb source always based on an sql statement (select *)? so how is it possible that this component becomes so slow?
Given the following scenario, what kind of performance should be expected in transferring about half a million rows? We are seeing about a 9 minute execution time. Is this reasonable for about 460,000 records moving from source to target, with 3 inner joins from the source?
Source: Server A.OLTPDB
Target: ServerA.DataMartDB
Server A is running SQL Server 2000. SSIS is running on a different machine, Server B.
The reason for this is that we are distributing the SSIS package for use with a BI product built on SSAS 2005 and the requirements are such that the client could very well have the source OLTP database on a different physical machine than the data mart.
My understanding is therefore that:
1. SSIS will do all of the heavy lifting on Server B.
2. Even though OLTPDB and DataMartDB are on the same server, it is expensive for Server B to pull the records from Server A and then send them back to Server B, but with SSIS being on a different machine, this is inevitable.
3. In the OLE DB Source Adapter, specifying table or view has the effect of an OPEN QUERY command, whereas a SQL command with straight SQL will be executed on Server B and the former would be somewhat more performant.
I have the following SQL, which works but I think it can be done simplier. I seem to have to group it by multiple columns, but I am sure there must be a way of grouping the results by a single column. Any Ideas?
Code:
SELECT count(order_items.order_id) as treenum, orders.order_id, orders.order_date, orders.cust_order, orders.del_date, orders.confirmed, orders.del_addr FROM orders, order_items WHERE orders.order_id = order_items.order_id GROUP by orders.order_id, orders.order_date , orders.cust_order, orders.del_date, orders.confirmed, orders.del_addr ORDER BY orders.order_id DESC
I have four different transactions such as below and I do one insert and one update in each transaction and it seem it is slow and creates deadlock with the user interface.
These transactions are performed against the tables that users are accessing with another user interface. I have following two questions:
1. T2.TextField1 and TextField2 = @TextField2 are Ok, Nok fields so I did not put index since only two distinct values. Should I put indexes on these fields?
2. Can I make this transaction let user interface do its task in case accessing the same rows, I can start transaction again but I do not want users get disturbed? .
BEGIN TRANSACTION pTrans BEGIN INSERT INTO T1 (fields) SELECT (fields) FROM T2 INNER JOIN View1 ON T2.TrID = View1.MyTableID WHERE (T2.TextField1 = @TrType AND T2.TextField2 = @TextField2)
UPDATE T2 SET TextField2 = 'Ok', TextField2Date=@MyRunDateTime FROM T2 WHERE (TextField1 = @TrType AND TextField2 = @TextField2)
IF @@ERROR <> 0 BEGIN rollback transaction pTrans return(-1) END ELSE BEGIN commit transaction pTrans END END
