Hi,I'm implementing mass information procedures that is stored in a SQL Database.
What methods and actions i need to take in order for the process to be faster in a case like this (its more then 1000000 rows).
I'm trying also to improve the memory usage since i use the DataTable in C# and I'm looking for a better way to process the retrieved data. is there a better class or method thats improving the speed and preventing any memory leaks ?
please advise....
Thanks for any help,
Lior S ;)
We're about to put together a db solution for storing and searching large quantities of documents, with SAS running on top of it. Ultimately this is going to be a data mining project. We're talking about 2 million docs per year, ranging from 1 to possibly hundreds of pages of text per doc. We'll be pulling down roughly 40,000 docs per week, most likely XML, HTML, or SGML format. The user base is actually pretty small - probably no more than 5.
My gut feeling is to go with DB2, but to be fair I wanted to see if SQL 2005 would be up to the task. I'm not very familiar with the current version, but I udnerstand there have been some big improvements in scalability. It is the policy of SAS not to make recommendations. Does this sound like something SQL 2005 could do well, or should I stick with my initial plan and use DB2?
I'm considering options for a large scale data warehouse. Even though SQL can theorectially scale to 10 Terabytes plus, in practice - will it be able to do it? Has anyone else actually done it? Or should Oracle be used?
My name is Atma SIngh I am have a strange problem here, on my system. I have develop a solution that runs on window xpsp2 and Microsoft SQL Server 2005 Express Edition with Advanced Services SP2. Once in every day the system will prompt a error message "Not enough storage to process this command". The Hard Disk space is avaliable.
I will never get this error in the lab test. So I am forwarding this error to the forum if someone there knows how to resolve this problem.
Good Day, please help me, i have a data driven site that displays computed data to a Complete Gridview, this is my problem, when i run my site, it all displays the informations and the gridview a i intended to. generally my site works fine BUT!!!!! my site have different levels of display, like thisSite 1:it has 3 dropdown menus, dropdown 2 is dependent to dropdown 1 and dropdown 3 is dependent to dropdown 2 and the complete gridview is dependent to dropdown 3, it is a postback process of dependencies, i hope you get what i mean, Site 2:this is the same as the site 1 but this time, is has 2 dropdown menus only, and the complete gridview has a gridview in side of it thru details template. Site 3: almost the same as Site 1 and 2 but this time there will be no dropdown menus, only the complete gridview, but the gridview has 3 levels, the same level sa the level of dropdown menus in site 1, where the main gridview has a gidview below it and a gridview again below it and soon, The Problem:the site works fine but it runs super SLOW!!! specially in the Site 3, where it has to display and compute different gridviews from the complete gridview at the same time, sometimes it cause the TIME OUT EXPIRE.that can i do to speed the process other than upgrading my server? is there an alternative?please helpfor more info this is the SQL codes that i used in each level in the complete gridviewThe highest level in the gridview of the initial display of the complete gridview: select rbu,count(distinct dslamname) as NumberOfDslam, (select count(secode) from dslamdata as a where a.rbu = r.rbu ) as Capacity, (select count(secode) from dslamdata as a where a.rbu = r.rbu and dnum > '1') as Used, (select count(secode) from dslamdata as a where a.rbu = r.rbu and dnum < '1') as Remaining, (select (select count(secode) from dslamdata as a where a.rbu = r.rbu and dnum > '1')*100/(select count(secode) from dslamdata as a where a.rbu = r.rbu )) as Utilization, (select sum(dwlink) from dslamdata as a where a.rbu = r.rbu ) as Sold_Bandwidth from dslamdata as r group by rbu The Second to the highest when you would click on the +/- button in the complete gridview select aco,count(distinct dslamname) as NumberOfDslam, (select count(secode) from dslamdata as a where a.aco = r.aco ) as Capacity, (select count(secode) from dslamdata as a where a.aco = r.aco and dnum > '1') as Used, (select count(secode) from dslamdata as a where a.aco = r.aco and dnum < '1') as Remaining, (select (select count(secode) from dslamdata as a where a.aco = r.aco and dnum > '1')*100/(select count(secode) from dslamdata as a where a.aco = r.aco )) as Utilization, (select sum(dwlink) from dslamdata as a where a.aco = r.aco ) as Sold_Bandwidth from dslamdata as r where rbu = ? group by aco The Last in the display when you would click on the +/- of the second gridview select dslamname, (select count(secode) from dslamdata as a where a.dslamname = r.dslamname ) as Capacity, (select count(secode) from dslamdata as a where a.dslamname = r.dslamname and dnum > '1') as Used, (select count(secode) from dslamdata as a where a.dslamname = r.dslamname and dnum < '1') as Remaining, (select (select count(secode) from dslamdata as a where a.dslamname = r.dslamname and dnum > '1')*100/(select count(secode) from dslamdata as a where a.dslamname = r.dslamname )) as Utilization, (select sum(dwlink) from dslamdata as a where a.dslamname = r.dslamname ) as Sold_Bandwidth from dslamdata as r where aco group by dslamname if you would look at it, all the codes are the same except for the selected field, i hope someone can help me with this, thanks and good day SALAMAT PO.,
I am asking this question on behalf of a friend. I have little knowledge of SQL 2005 but my friend is quite knowledgeable, although this is the first time he is dealing with large database for a client. So here's the story.
His client has a database containing 1.5 million books. Now he is setting up a website which will enable users to search books. Searching by ISBN is no problem as it only takes 1 seconds. The problem is, searching by Title takes more than 20seconds, which is unacceptable. My friend has only done smaller database and he just recently thought of implementing indexing and now looking for other ideas.
Each row contains book details such as Title, Author1, Author2, Author3, Publisher, Publication Date, ISBN, etc.
Can anyone who are more experienced in doing large database share with me some design ideas? His client is aiming for 8seconds or less.
I have a query below which filters detail field in the #TempLogins table. The details field is a text field which contains many types of text strings, some containing urls that have parts like "ResultID=5" which is what is contained in the ResultIDSearch and ResultSetIDSearch fields. The records with entries like "ResultID=5" are the ones I'm trying to filter for.
The problem I have is that the query takes way too long to run. The TempLogin table has around 200 K records and the TempSearch table has around 80 K records.
select * from #TempLogins a where exists (select 1 from #TempSearch t1 where a.detail like '%' + t1.ResultIDSearch + '%' or a.detail like '%' + t1.ResultSetIDSearch + '%')
Hi, I'm running an application on a server which grabs data from a database table on another server using SqlConnection, SqlDataAdapter and DataSet. The application then updates every row in that DataSet's DataTable and the updates are saved back using DataAdapter. The code is pretty much straightforward code that you would find on MSDN documentation for using DataSets. The table contains a little over a million rows. When I run the application, I get an error saying the Server Application is not available. Upon looking into the application event log, I get this message. aspnet_wp.exe was recycled because memory consumption exceeded the 306 MB (60 percent of available RAM) How do I get round this? I thought DataSets were supposed to handle large datatables comfortably without having memory issues. -Thanks
I am attempting to do a rather simple purge task on a very large table. This task will need to take place daily and delete records older than 6 months out of the database. On first pass this will delete well over 130 million rows. I thought the best way to handle this is create a proc and call the proc from a SQL Agent Job that runs nightly. Here is an example of the script:
CREATE PROCEDURE usp_Purge_WCFLogger AS SET NOCOUNT ON EXEC sp_rename 'dbo.logs', 'logs_work' GO SELECT * INTO dbo.Logs_Backup FROM dbo.Logs_Work WHERE TIMESTAMP < DATEADD(month, -6, GETDATE())
I'm running a resource-intensive stored procedure, which reads a filewith about 50,000 lines with a BULK INSERT into a temp table, thengoes through it and inserts a record for each line into another table.While this procedure is running, SQL server stops accepting any otherrequests coming from the website.Question:Is there a way to make SQL server "listen", or emulate an "interrupt"to other requests while in the middle of a long intensive process?I really appreciate your replies.Thank you,Oleg.
We recently installed SQL server 2005 on a couple of our servers. I use Visual Basic 6.0 at the moment and use ADO to connect to our various SQL servers.
I recently discovered on one of the new servers, that every time my programs runs, (every 4 minutes for 12 hours a day) the SQL process shown in task manager grows by 1-10 Megs.
The SQL process was at 776,912K when I rebooted this afternoon. It started back up at 106,120K.
I am not doing anything differently than I did when my programs were talking to SQL 2000, and I have never seen this memory leak issue. Is there something extra I need to do in SQL 2005 to finish/clear these SQL queries and not bog down SQL's memory?
An example of how I would connect and do a SQL transaction:
How to implement distinct storage tiers on SQL Remote BLOB Storage (RBS)?
I want to use this SQL Feature to move files(images, videos, pdf files) from a database to a distinct database dedicated to RBS. Then I want to have several storage tiers, where objects will be saved and moved according access frequency. Old data will be arquived in cheap storage, but it must be always accessible if needed.
Description: - 1st and main tier: new and frequently accessed objects stored in high performance storage; - 2nd tier: automatically move older or less accessed objects to an inexpensive and different storage tier; - in all cases, all objects must be accessible to all users, but accessing to archived objects(2nd tier) will be much slower;
I have several data bases on a server (SQL Server 2000 only, no web server installed) and lately, as the company keeps gowing, my users complain saying the server gets slow, (this dbs are well designed and recieve optimizations and integrity checks, etc) because of this, Im thinking about getting a new server to repleace my old ProLiant ML 330 which was bought 4 years ago but Im concerned about what server arquitecture or characteristic can help me best to improve response performance, is it HD speed? Processor speed? or more Ram? I want to make a good decision, so I´d really appreciate your help...
I am a Windows developer for the IBM Tivoli Storage Manager Server (TSMS) product. Our product installation is built with InstallShield and uses the Windows Installer.
On a new installation of Windows 2003 x64 Storage Server R2, at a customer's site, the TSMS product fails to install. The install of the OS has version 3.01.400.3959 of the Windows Installer and I see no newer version that installs.
Part of our product is 32 bit (console) and another part is x64 (server). When installing I can see that the install's default is being redirected/reset to C:Program Files (x86)TivoliTSM after it is explicitly set by a custom action to ..Program Files.. . I further observe that our custom actions to write 64 bit registry entries are being refused.
REGSAM samMask = KEY_ALL_ACCESS; if ( regIsWow64Process () ) samMask = samMask | KEY_WOW64_64KEY; lStatus = RegCreateKeyEx( hLocalConnectKeyRoot, szSubkey, 0L, NULL, REG_OPTION_NON_VOLATILE, samMask, NULL, hKey, &dw ) ; The above fails to create the key.
We have tried four versions of our TSMS spanning many changes but the install acts the same. This does not happen on any other Windows OS we test on but we do not test on Windows 2003 Storage Server R2 being that it is an OEM product. We did test on Windows server 2003 R2 x64 and do not see this problem.
Do you have any suggestions on how to tackle this problem? I have full installation traces but can only see that the registry work is being refused. I can't see why.
I was trying to extract data from the source server using OLEDB Source and SQL Server Destination when i encountered this error:
"Transaction (Process ID 135) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
What must be done so that even if the table being queried is locked, i wouldn't experience any deadlock?
Hello all, I am running into an interesting scenario on my desktop. I'm running developer edition on Windows XP Professional (9.00.3042.00 SP2 Developer Edition). OS is autopatched via corporate policy and I saw some patches go in last week. This machine is also a hand-me-down so I don't have a clean install of the databases on the machine but I am local admin.
So, starting last week after a forced remote reboot (also a policy) I noticed a few of the databases didn't start back up. I chalked it up to the hard shutdown and went along my merry way. Friday however I know I shut my machine down nicely and this morning when I booted up, I was in the same state I was last Wenesday. 7 of the 18 databases on my machine came up with
FCB:pen: Operating system error 32(The process cannot access the file because it is being used by another process.) occurred while creating or opening file 'C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLDataTest.mdf'. Diagnose and correct the operating system error, and retry the operation. and it also logs FCB:pen failed: Could not open file C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLDataTest.mdf for file number 1. OS error: 32(The process cannot access the file because it is being used by another process.).
I've caught references to the auto close feature being a possible culprit, no dice as the databases in question are set to False. Recovery mode varies on the databases from Simple to Full. If I cycle the SQL Server service, whatever transient issue it was having with those files is gone. As much as I'd love to disable the virus scanner, network security would not be amused. The data and log files appear to have the same permissions as unaffected database files. Nothing's set to read only or archive as I've caught on other forums as possible gremlins. I have sufficient disk space and the databases are set for unrestricted growth.
Any thoughts on what I could look at? If it was everything coming up in RECOVERY_PENDING it's make more sense to me than a hit or miss type of thing I'm experiencing now.
Dear list Im designing a package that uses Microsofts preplog.exe to prepare web log files to be imported into SQL Server
What Im trying to do is convert this cmd that works into an execute process task D:SSIS ProcessPrepweblogProcessLoad>preplog ex.log > out.log the above dos cmd works 100%
However when I use the Execute Process Task I get this error [Execute Process Task] Error: In Executing "D:SSIS ProcessPrepweblogProcessLoadpreplog.exe" "" at "D:SSIS ProcessPrepweblogProcessLoad", The process exit code was "-1" while the expected was "0".
There are two package varaibles User::gsPreplogInput = ex.log User::gsPreplogOutput = out.log
How do I use the execute process task? I am trying to unzip the file using the freeware PZUnzip.exe and I tried to place the entire command in a batch file and specified the working directory as the location of the batch file, but the task fails with the error:
SSIS package "IngramWeeklyPOS.dtsx" starting.
Error: 0xC0029151 at Unzip download file, Execute Process Task: In Executing "C:ETLPOSDataIngramWeeklyUnzip.bat" "" at "C:ETLPOSDataIngramWeekly", The process exit code was "1" while the expected was "0".
Then I tried to specify the exe directly in the Executable property and the agruments as the location of the zip file and the directory to unzip the files in, but this time it fails with the following message:
SSIS package "IngramWeeklyPOS.dtsx" starting.
Error: 0xC002F304 at Unzip download file, Execute Process Task: An error occurred with the following error message: "%1 is not a valid Win32 application".
The command in the batch file when run from the command line works perfectly and unzips the file, so there is absolutely no problem with the command, I believe it is just the set up of the variables on the execute process task editor under Process. Any input on resolving this will be much appreciated.
I am designing a utility which will keep two similar databases in sync. In other words, copying the new data from db1 to db2 and updating the old data from db1 to db2.
For this I am making use of the 'Tablediff' utility which when provided with server name, database, table info will generate .sql file which can be used to keep the target table in sync with the source table.
I am using the Execute Process Task and the process parameters I am providing are:
The customer.bat file will have the following code: tablediff -sourceserver "LV-SQL5" -sourcedatabase "TC_CTI" -sourcetable "CUSTOMER_1" -destinationserver "LV-SQL2" -destinationdatabase "TC_CTI" -destinationtable "CUSTOMER" -f "c:SQL_bat_Filessql5TC_CTIsql_filescustomer1"
the .sql file will be generated at: C:SQL_bat_Filessql5TC_CTIsql_filescustomer1.
The Problem: The Execute Process Task is working fine, ie., the tables are being compared correctly and the .SQL file is being generated as desired. But the task as such is reporting faliure with the following error :
[Execute Process Task] Error: In Executing "C:SQL_bat_FilesSQL5TC_CTIpackage_occurrence.bat" "" at "C:Program Files (x86)Microsoft SQL Server90COM", The process exit code was "2" while the expected was "0". ]
Some of you may suggest to just set the ForceExecutionResult = Success (infact this is what I am doing now just to get the program working), but, this is not what I desire.
I'm pulling data from Oracle db and load into MS-SQL 2008.For my data type checks during the data load process, what are options to ensure that the data being processed wouldn't fail. such that I can verify first in-hand with the target type of data and then if its valid format load it into destination table else mark it with error flag and push into errors table... All this at the row level.One way I can think of is to load into a staging table then get the source & destination table -column data types, compare them and proceed.
should I just try loading the data directly and if it fails try trouble shooting(which could be a difficult task as I wouldn't know what caused error...)
I am having this table locking issue that I need to start paying attention to as its getting more frequent.
The problem is that the data in the tables is live finance data that needs to be changed and viewed almost real time so what I have picked up so far is that using 'table Hints' may not be a good idea.
I have a guy at work telling me that introducing a data access layer is the only way to solve this, I am not convinced but havnt enough knowledge to back my own feeling up. (asp system not .net).
I have been trying to develop an automatic way of programmatically accessing datasources and performing some predefined(-supported) processing on them.
The question I would like to ask you people has to do with numeric fields. What exactly is precision? Is it the maximum length in digits of a field, or is there more to it? What about a "field's scale", what is it and how does it affect a field's value handling?
Can you have different scales on a bar chart. I want to chart sales and quantity. Oracle lets you label the top of the bar chart as money and the bottom as quantity.
I am dividing my sales by one million and quantity by one thousand to make them similar in size on the same chart. I am using the sum of the sales/1000000 as a point label. Is there any way to limit the number of decimal places displayed. Currently it is displaying something like 1.94889312043; 1.95M would be better.
I have set the output columns to decimal and data scale of 2. And have also set the field to be 0.00, and in the csv desination file it always puts .000000, How can I get it to be 0.00?
Hi, I'm trying to upload the ASPNETDB.MDF file to a hosting server via FTP, and everytime when it was uploaded half way(40% or 50%) I would get an error message saying: "550 ASPNETDB.MDF: The process cannot access the file because it is being used by another process" and then the upload failed. I'm using SQL Express. Does anybody know what's the cause? Thanks a lot
Hi. When I try to start a package manually clicking the Start Debugging button I get this after a little while:
Cannot process request because the process (3880) has exited. (Microsoft.DataTransformationServices.VsIntegration)
How can I prevent this from happening? This happens every time I want to start the package and every time the process id is different. Here it is 3880.
This is not a real big deal, cause I worked around it.. but I just tried to INCREASE the scale on a decimal column & got an arithmatic overflow error. I can understand why this would occur if trying to decrease the scale, but not increasing it. This is not a computed column.. why does SQL give an overflow error if all it has to do is add a couple of zeros to the end? Yes, I know SQL considers them to be different data types, but this is still confusing me.
Our topology: Internal corporate network domain utilizing Integrated Windows Authentication between local workstations and the web and database servers. (domain similiar to: http://reports.ourdomain.com)
Machine #1: Windows 2003 Server w/SQL Server 2005 (SP2 and hotfixes applied) running SSRS. Machine #2: Windows 2003 Server w/SSRS only. (pointing to the SQL DB on Machine #1) Machine #3: Windows 2003 Server w/SSRS only. (pointing to the SQL DB on Machine #1) Device #1: Arrowpoint load balancer for Machines #2 & #3 (w/sticky sessions).
We recently added 2 web servers and configured them to load balance the web requests to a seperate SQL Server machine. (the load balancing is done thru an Arrowpoint device not Microsoft NLB)
The installation of reporting services (on Machine #2 & #3) went smoothly and the configuration was a breeze. But we could not access the "http://reports.ourdomain.com/Reports" directory for reporting services on the 2 web server machines. We received the "401.1 - Unauthorized" http error.
Since we were using a domain name to access the report server we made sure to update the <URL> tag in the config files.
We still received the error, so then we added a couple of entries to the HOSTS file on each web server: File location --> C:WINDOWSsystem32driversetcHOSTS
In order for the load balancers to "see" themselves from the domain reference we had to update the HOSTS file on each web server to have the domain name "reports.ourdomain.com" point to the IP Address of the specific web server. (we also added the IP of Machine #1 just to be safe):
Then the web servers were able to authenticate the users using IWA and NTLM and we could access the report servers normally. Even the the scheduled reports were being load balanced and emailed from the web servers which we didn't expect to happen. (pretty cool!)
I hope this helps someone else as it tooks us about 4 days to figure this out.
I am using the statement below to calculate the average scores of the columns. When the result set is returned I would like to have a scale of 2. I am currently returning a scale of 6. What could I do to fix this?
Thanks for you help, John
SELECT ((CONVERT(decimal(4,2),c2_3) + CONVERT(decimal(4,2),c2_15) + CONVERT(decimal(4,2),c2_16) + CONVERT(decimal(4,2),c2_17)) / 4 * 100) AS Score_A FROM dataquestionnaire WHERE confirmation = '10/1/2001-999-1'
