SSB && Scaling Architecture Question
Apr 26, 2007
Hi,
My application is about to scale up significantly, and it seems that SSB could be very useful to help me scale it right, especially I like the multiple readers feature.
So, here's the deal
my application is about to get lots of records (peaks could gets up to 300k records per hour). Each record must be processed separately, but could be processed in parallel no dependency between rows (gut feeling tells me not to overuse parallelism, but Ill do my tests).
It takes about 10ms-15ms to process a single row; I don't mind sliding to non peak hours if needed.
Each record is constructed of about 10 columns.
My questions:
1. Is SSB the right solution as a buffer queue? any famous use case is similar to my problem ?
2. What is the fastest easiest way to serialize / deserialize each record?
I would prefer not using CLR integration due to performance issues and stick with tsql for now. I dont necessarily prefer XML serialization. If binary serialization works faster, its fine with me.
View 1 Replies
ADVERTISEMENT
Jan 27, 2008
Hi,
assuming a data driven web application was written, and it is used by a huge number of customers, we can set up some load balancing infrastructure, where multiple web servers run to process all requests. In this case, the database might run on its own machine. But what can be done if one machine is not enough for the database anymore? Is it possible to build some kind of database cluster? And if yes, how does the application work with such a cluster? Is there still ONE connection string or is it needed to query multiple SQL servers?
Long story short; Can a SQL Server database cluster be handled as ONE SQL Server instance from outside, even it is hosted on multiple machines?And if yes, what edition of SQL Server 2005 is required?
RegardsMarco Buerckel
View 2 Replies
View Related
Mar 17, 2008
Hi Folks,
I have web application for which I need to come up with a scalability plan. The application right now is running on 2 computers, the front end is running Tomcat with JSP (Java Server Pages). The back end is MS SQL 2000
The database is running fine now, but the file size and CPU utilization is growing and I need to figure out a way to distribute this database over several machines.
I looked at MSCE and setting up a cluster, but I think that the solution will NOT work in an intensive enterprise environment. (Am I wrong?). For one thing, there is a hard limit of 16 nodes in a cluster. Also, it requires a shared RAID or FibreChannel Array. These solutions are very expensive. Also, it's not easy to add additional capacity if you reach the maximum of the RAID array. For example, if you have a FibreChannel Array with 8 Terrabytes of capacity, if your DB uses up all 8 terrabytes, you have to buy a whole new enclosure.
One of the major objectives I have is to minimize cost and support unrestricted database growth.
Here is what I would like to achieve. If any of you know of a solution or something that YOU have USED which might give the functionality I require .. I would greatly appreciate your feedback.
Wish List:
1) Run single SQL Instance on regular PCs, which host 1 or More Tables (Table level partitioning) of a very large database.
2) Distributed JOINs on Tables residing on different physical PCs
3) Multiple Machines hosting same table (Fault Tolerance)
4) Single Virtual IP for database to Application Front End.
-----------
I looked at C-JDBC which is an open source middleware that can do some of the things above. I liked it, but it doesn't seem to support Distributed JOINs which means that our application - which requires quite a few tables JOINs - won't be able to achieve Table level partitioning.
Can anyone recommend any clustering middleware which you can recommend?
View 3 Replies
View Related
May 14, 2008
I'm currently developing an ASP.NET site with SQL Server 2005 Standard and I'd like to ask a question about the future of the database. It needs to have continuity and performance. I'm thinking about doing replication or mirroring for continuity and table partitioning for performance. I admit I've never done any of those before and I'll learn about them but they're not needed at this time. The question is, I'm currently designing the database and do I have to anything for consideration for those things I'm thinking of implementing later? For example, I'm using Identity in my tables but I've heard about identity crisis using replication with identity columns, therefore I'm thinking of using Guid's but now I fear the Guid column index itself will be the slowdown factor in the first place.
Any suggestions to consider? I'd appreciate any opinions.
View 5 Replies
View Related
Jul 23, 2005
I have a pivot chart that displays 3 data series.Series 1 values range from 17 to 106.Series 2 values range from 1 to 18.Series 3 values range from 0 to 1.When I display all three series on the chart, the value axis labels run from1 to 120 with major unit lines at intervals of 20.When I remove Series 1, the chart automatically adjusts itself and thevalue axis labels run from 1 to 20 with lines at intervals of 2, thusnicely accommodating the value ranges of Series 1 and Series 2.All this is fine. But when I remove Series 2, leaving only Series 3,the labels run from 1 to 1.2 with interval spacing of .2. Since thesevalues are counts, it makes little sense to show decimal values.Can anyone provide advice on how to keep these axis labels fromshowing decimal values, if a user removes Series 1 and Series 2?Thanks.- Bob
View 2 Replies
View Related
Jun 21, 2006
Read only transacted replication to about 60 clients. 100 or so transactions a day (8 hours), transaction (record) is probably about 8K/transaction. Reliable connection over 100base-T. Latency - 15 minutes would be great.
This doable? -- I couldn't find a timing/ sizing model any place.
View 1 Replies
View Related
Sep 26, 2007
We use timed subscriptions to do almost all of our reporting. Reports are delivered (primarily via e-mail and printer) once they are completed and users don't have to "watch the pot boil" so to speak.
Apparently SSRS has some load balancing capability whereby it lets only a limited number of threads/reports run concurrently. We often reach this max and lock ourselves up on some very long-running reports, causing other important reports to wait a long time.
We've added some operational reports (ie. document prints) to the mix. These reports run off of OLTP data. They are very fast and very high priority. Waiting on them is not an option. Is there some way we can get SSRS to work on these operational reports in preference to other types of reports (eg. "just for kicks" reports)? I think we'd almost like to add another SSRS server and dedicate it to the operational reports. Ideally the new SSRS server would use the same Report Server database but would only work on subscriptions for certain documents.
Has anybody else tried to solve this problem? This MS document does really address subscriptions or load balancing by report: http://www.microsoft.com/technet/prodtechnol/sql/2005/pspsqlrs.mspx
Any help would be very much appreciated.
View 6 Replies
View Related
Apr 14, 2006
We have a 3 month old quad processor/dual core server running SQLServer 2005 and already it is getting close to hitting the CPU wall.An 8 way CPU box is prohibitively expensive and out of the question. Iam looking desperately for a way to TRULY scale out SQL server...in thesame way that IIS can be scaled out via App Center.The "in the box" solution for SQL Server 2005 scaling out is the DMV.Unfortunately this solution makes the system less available rather thanmore (one server outage takes them all out for that table) and requiresserious rearchitecting of the software to use. Contrast this to IISand AppCenter where each added server makes the system more available,and requires no rearchitecting to work.Before someone says "what you want can't be done in adatabase"...Oracle has an application server middleware product thatlets you do both of the above. Just plug a new server with Oracle onit, and you've doubled your capacity. But SQL Server 2005 doesn't yethave a similar capability.So I read with great interest the following article that talks aboutwhy this is the case with SQL Server. There are two issues that makeit very difficult to do:http://www.sql-server-performance.c...ability_availab...You can create a crude pool using replication, but the performancetimes look horrendous.However, the article also talks about the latest developments in thisfield...specifically MIDDLEWARE that can create a scale out solutionthat is more available and that requires simply adding new servers toscale up.I found two companies which seem to offer this new capability:http://www.metaverse.cc/newsevents.asp?cid=17999andhttp://www.pcticorp.com/product.aspxBoth companies appear to have patents or a patent pending on theprocess. I tried to contact metaverse but got no reply, despite theirrecent press release. I just emailed Pcticorp today to see if I couldlearn more about their product.My question for this group is:Does anyone have experience with either of the two products (or anyothers that provide this capability)?Many thanks in advance for your help.Ian Ippolitohttp://www.rentacoder.com
View 17 Replies
View Related
Sep 6, 2004
Hello,
I want to kow if the following architecture is good :
Disque 1&2 ( Raid 1)
c: OS
d: sqlserver + system tables + log files
Disque 3&4&5 (Raid 5)
e: data
View 1 Replies
View Related
Sep 29, 2004
Hi,
Someone can tell me if the following architecture is good :
f:master db
g:soft db + log
View 2 Replies
View Related
Dec 18, 2007
where can we get to know the architecture of sqlserver 2005
View 4 Replies
View Related
Apr 9, 2007
Hi,
I've a report projects where each report has a number of columns and the spec is to have all the columns print-out on the same page. Is there a setting that will auto-scale these columns to fit to the page or will I have to edit the font size and widths manually on each report to fit to the page?
Thanks for any comments.
View 4 Replies
View Related
Jan 22, 2001
We have database thats transaction intensive, so we are trying to sepetrare ldf file from mdf file to a different disk array. what raid should I use for the Transactional log file(.ldf).
Thank You,
John
View 1 Replies
View Related
Apr 2, 1999
I would like to know where I can find a senior database architect. Someone who can develop
and implement the database and its stored procedures. I am looking for an experienced person.
It is a contract position, in San Franicisco. The pay is good. Could anyone help me?
I tried Dice, Monster and it seems all of you are working...
View 2 Replies
View Related
Jun 2, 2004
I am replacing the corporate SQL Server at work. The new server will have 6 striped disks of 160G with about 4G of RAM. The current SQL Server currently has two instances which run web applications and a small database warehouse about 6G. Analysis Services is also installed.
Due to a couple of new apps being added to the server and the SQL Server 2000 enterprise license we acquired, i was thinking of adding 2 more instances so that the applications can be independently managed in terms of restarting the SQL Server. I also would like to permanently fix the memory settings on each instance to give more resources to more important applications. The log and data files would also be spilt onto 2 separate hard disks. i understand there are implications on performance such as CPU etc. Is it normally advisable to have more than 1 or 2 instances ? Most of the applications are not very CPU intensive. What other implications or performance issues would l have ?
View 4 Replies
View Related
Sep 1, 2006
We will be creating a moderately high-volume OLTP database application that needs 24/7 availability. We are planning to offload OLAP processing to a second copy of the system. We will be using SQL Server 2005.
I originally planned to set the second server up with SQL Server 2005 mirroring to cover the 24/7 availability requirement, with the idea that we could also do OLAP reporting off of the mirrored copy of the database. But I've gotten some indications that a mirror database is offline and not available for querying. So I figured I would use transactional replication to keep the OLAP database current. Now I am wondering if I need to use mirroring at all, or if I should just use transactional replication on the entire database and swap to the replicated database if the production server crashes.
What is everyone's opinion?
Replication only, for both OLAP reporting and failover?
Mirroring to one database for failover, with replication to a another database for OLAP reporting?
View 3 Replies
View Related
Feb 21, 2004
Hello, Everyone:
What duties are included within SQL Server Architecture? Thanks a lot.
ZYT
View 1 Replies
View Related
May 13, 2008
Can any one of you please explain the SQL architecture in a short..
Or can you give me the link, so i can refer it..
Its very urgent..
Pls help me..
Thanks In Advance
Ganesh
Solutions are easy. Understanding the problem, now, that's the hard part
View 4 Replies
View Related
Jun 14, 2007
We are designing a SQL Server architecture that will need to handle a lot of inserts and many users.
We are not sure what approach to take if we should build a distributed server architecture with a Single Master SQL and many Slave SQL Servers.
Or if we should get a really powerful machine with many processers to handle all the connections.
We project we will have about 5,000 Inserts per sec coming from 500,000 users.
What do you think or what has your previous experience been with handling many inserts and few reads for many users?
Thank you
View 2 Replies
View Related
Oct 20, 2007
Hi there,
My question is more "architectural" than technical.
One of our standard is to systematically enclose the code of our stored proc within a TRY/CATCH block.
What is your point of view?
Should a TRY/CATCH be included even for simple operations?
Are there any drawbacks?
Thanks!
EDIT: Thanks for your input.
View 4 Replies
View Related
Jun 29, 2007
I've implemented a solution with application, database and report server on seperate machines. The application is a web app and is Internet facing. What is the best method for executing reports on the RS server that are initiated from the web server? Using URL access requires a login or anonymous access neither of which are desired. Web services works but I loose access to the toolbar. Is there some other way to pull this off where I can let the public access reports and give them access to the toolbar?
Thanks.
View 4 Replies
View Related
Jun 5, 2007
In a situation where one may have a single master SQL Server that ultimately needs to communicate information back down to 1000's of downstream servers, what is the recommended architectural approach?
It doesn't feel right to have to add 1K-5K routes to the master SQL Server. Is there a way to have the dowstream servers "broadcast" their existence to the master, so that new servers can be added and updates can happen seamlessly? Does this fall into a pub-sub scenario or is there a better way? And, if so, how to ensure an open conversation (so that one server doesn't miss information that all the other servers received)? Should the master dynamically create routes or better to rely on an open conversation initiated by the downstream server?
View 20 Replies
View Related
Dec 31, 2007
Hi Everybody ,
Could Anybody please explain in detail about the Achitecture of Sql server reporting services?
1) What a Report processor in detail and what it does?
2) What is the exact work of Extensions?
3)What is SOAP and WMI?
Thanks in Advance
Mahasweta
happy new year
View 4 Replies
View Related
May 31, 2007
hi
i am haveing 1.5 years exp in asp.net(c#),i want to improve my knowlege in Database Architecture (datmodeling,uml,normalization,etc..).could anyone suggest me course or any books
View 4 Replies
View Related
May 23, 2007
I have the following stored procedure:
INSERT INTO MyTable ( Value1, Value 2)
VALUES( @Value1, @Value2)
SELECT SCOPE_IDENTITY()
How do I put this sp in the DAL typed dataset, so I can get the Identity value in the Business Layer?
View 2 Replies
View Related
May 22, 2008
When having multiple files for log, do they get filled one after the other (like Oracle) or proportionately fill?
------------------------
I think, therefore I am - Rene Descartes
View 2 Replies
View Related
Aug 23, 2006
Hi all,
Can u explain what exactly happens after index creation .
View 2 Replies
View Related
Sep 10, 2007
Not sure if this is the right forum?
Is it possible to export an ERD on System Architecture to VISIO/WORD?
Thanks
View 7 Replies
View Related
Dec 18, 2007
Don't know if this is the appropriate forum. I am looking for an experienced SS consultant to review our setup, hardware architecture, recovery plan, and to provide high-level advice moving forward. My company is a CRM hosted software provider with a dynamic, metadata-based product built in Visual Studio 2005. Currently we run on SS 2000, but plan to migrate to SS 2005 or 2008. We anticipate quite a bit of growth and want to make sure that we are on the right path. Let me know if you are interested or know someone who is.
If I should post this elsewhere, let me know.
View 2 Replies
View Related
Jul 23, 2005
Hi,Is 'sqlserver.exe' the only windows process does everything for thatinstance of the database?Please explain in details the SQL server process architecture.Thanks*** Sent via Developersdex http://www.developersdex.com ***
View 2 Replies
View Related
Feb 7, 2008
To all the SQL H/A experts, we were wondering if we could have 3 physical nodes and 2 active/passive clusters architecture setup on a SAN as seen in the image below? http://www.geocities.com/juanlieu/CP_Arch.JPGIn case you cannot see the diagram, it would looks something like this:
active/passive Cluster A ---> physical server A (Win2003/SQL2005) ---> HP EVA SAN ---> physical server B (Win2003/SQL2005) ---> HP EVA SANactive/passive Cluster B ---> physical server B (Win2003/SQL2005) ---> HP EVA SAN ---> physical server C (Win2003/SQL2005) ---> HP EVA SAN
In this setup, I understand that Server B cannot be called upon as the active server at the SAME time by both clusters. question: what would happens if it does, would Server B reject the last cluster that calls it?Appreciated in advance.
View 3 Replies
View Related
Jan 5, 2007
Hi All... We're developing an asp.net based project. We plan to deploy it across multiple servers running in an NLB environment. That is NLB via hardware or software - we generally leave that decision up to our customer. Different from prior projects we've done, this application will rely on a SQL Server 2005 database. With the NLB, we essentially install our application many times across multiple servers. As hits come in from clients, they'll get directed to one server or another by whatever NLB technique is being used. The applicaiton generally doesnt care which server is hit.
But what about the database? What's a typical or best architecture to employ? Should each server have an instance of SQL Server and then somehow through replication they all keep each other updated? Or should each of the servers hit one lonely instance of SQL Server and it somehow keeps some other backup instance updated? I think the first approach seems to make more sense from a load balancing point of view, but depending on how many servers there are, it could get quite complicated.
Again, we're in the early stages so we really havent done much research on this yet. Any tips would be appreciated. Any good white papers out there?
Thanks! -- Curt
View 1 Replies
View Related
Jul 27, 2007
I need to load a lot of Excel, CSV, ... etc. files. These files have hundreds of columns and I need to validate the data. Some are simple range type checking, some are more complex checking involve multiple columns.
There may have several hundreds of such rules. And I may need to let the program to automatically correct some invalid data in the future.
Where to implement it in SSIS?
Or just load the files without any checking (all type to text), and checking using T-SQL?
(BTW, I don't have biztalk server).
Thanks in advance.
Read more >> Options >>
View 5 Replies
View Related
