How To Setup A Many-to-many Schema To Favor Retreival Speed.
Oct 30, 2006
I'm am novice in the RDMS world. I would like to set up a database that will scale with a large dataset of over 1,000,000 unique records. In a nutshell, I have about a million+ documents that are stored on a file server as TIFF files. Each document has 2 or more persons (names) associated with that document like "John Smith" or "Mr. John Doe" etc.. A person can be related to many documents (depending on how many transactions they conducted over the years) and any document can be related to many people as most of the transactions were from multiple sellers and multiple buyers. The end users almost always searches for documents by using the first and last name of the person related to the document along with other document fields such as DATE and TYPE to narrow the search. I would like to design and index the database for fast searching as updates are not as time sensitive. I think I need at least 3 tables:
1) NAME_TABLE normalized into NameID(PK) NameFirst, NameMiddle and NameLast
2) DOC_TABLE normalized into DocID(PK), DocTransNum, DocFilePath, DocDate, DocType, (and other attributes)
3) NAME_DOC table to relate the above two tables in a many to many relationship.
My questions are: Where should I put the field that identifies the person as a buyer or seller as it relates to the document? John Smith can be both a buyer and seller on different documents so I don't think it belongs in the NAME table. If I have a another table that splits the names into SELLER_NAMES and BUYER_NAMES it would create some duplicate names and increase the search time right? I want to search for a document using a name and the option to specify if the person name search is limited to buyers, sellers or both. Do I need to do anything special in setting up the tables for fast searching other than indexing all the name fields and the DocID field? What indexing needs to be done on the many-to-many table if any? If fast searching on a mostly static database is a top requirement, should I denormalize the document table to include the names even though there will be duplicate names throughout? How large of a test set of records do I need to load to see any real deltas in performance as I modify and tweak?
Sorry for all the questions but I believe they are related to the overall problem to solve.
I am working on a .NET project for order manangement system. There will be multiple users accessing the application simultaneously. We have primary key columns as identity columns in most of our tables in our database. When we create new records, in our SPs we are doing an insert and to get the value of the Primary Key immediately after the insert, we are retrieving the last inserted identity value and using it in the rest of the procedure.
Which one is the best to retrieve the value of the record just inserted - @@Identity, Scope_Identity OR Ident_Current
We should have the right column value considering there will be multiple inserts in the tables precisely at the same, say, micro second. There should be no overlaps where the id value generated for one user/session etc should not be retrieved for another user or session etc.
Please let us know what is the best to use in our condition.
I am trying to develope a Pocket PC application using MS Visual Studio 2005 (C#)and the database is in the device (.sdf file).
Am trying to retreive a single row from a table containing around 10000 rows. I would like to know the fastest method to retreive the values.
whether it is faster to use an SQLDataAdapter to retreive the values to a dataset or datatable OR retreive the values to a SQLCEResultSet OR is there any other faster retreival method.
I have several data bases on a server (SQL Server 2000 only, no web server installed) and lately, as the company keeps gowing, my users complain saying the server gets slow, (this dbs are well designed and recieve optimizations and integrity checks, etc) because of this, Im thinking about getting a new server to repleace my old ProLiant ML 330 which was bought 4 years ago but Im concerned about what server arquitecture or characteristic can help me best to improve response performance, is it HD speed? Processor speed? or more Ram? I want to make a good decision, so I´d really appreciate your help...
I have created a Web Setup Project for my web application which also contains a database. Currently I am having troubles scripting the database and run it during my Custom Action for the setup project.
I used the publishing wizard to create a sql script that contains both the schema and the data for the database. However when I run the script using
Code Block SqlCommannd.ExecuteNonQuery();
it fails. I believe this is because the ExecuteNonQuery method does support SQL statements like GO etc that are part of the SQL Script. So I now believe my approach is wrong. So my question is how should I script my database (both schema and data) and run this script during the Custom Action of my setup?
Locally I develop in SQL server 2005 enterprise. Recently I recreated my db on the server of my hosting company (in sql server 2005 express).I basically recreated the tables and copied the data in it.I now receive the following error when I hit the DB:The 'System.Web.Security.SqlMembershipProvider' requires a database schema compatible with schema version '1'. However, the current database schema is not compatible with this version. You may need to either install a compatible schema with aspnet_regsql.exe (available in the framework installation directory), or upgrade the provider to a newer version.I heard something about running aspnet_regsql.exe, but I dont have that access to the DB. Also I dont know if this command does anything more than creating the membership tables and filling it with some default data...Any other solutions/thought on what this can be?Thanks!
Hello everybody!I'm using ASP.NET 3.5, MSSQL 2005I bought virtual web hosting .On new user registrations i have an error =(The 'System.Web.Security.SqlMembershipProvider' requires a database schema compatible with schema version '1'. However, the current database schema is not compatible with this version. You may need to either install a compatible schema with aspnet_regsql.exe (available in the framework installation directory), or upgrade the provider to a newer version. On my virtual machine it work fine but on web hosting i have an error =(What can you propose to me?
I would like to use SSIS tool to move the data from one database schema to another database schema.
For example:
Source table has
1. UserName (varchar 20) (no null)
2. Email (varchar 50) (can be null)
Destination table has
1. UserID (uniqueidentifier - GUID)
2. UserName (varchar 50) (no null)
3. EmailAddress (nvarchar 50) (can be null)
4. DateTime
Questions:
1. What controls do I use in my Data Flow to make data move between databases with different data types and include new value in UserID as a new GUID and DateTime as a date (GETDATE)?
OLE DB Source, OLE DB Destination, Data Converson and .....
How do I insert Guid and Date at the same time?
2. I have many tables to do data moving. Any sugestions? How do I architect my project? If I create many data flows for each table - it will look complicated.
I have XP Pro SP2 with MDAC 2.8.1022. It had a problem so I tried to reinstall MDAC and got a Fatal Setup Error. This setup does not support installing on this operating system. I downloaded MDAC 2.8 1177 and get the same error.
I thought of uninstalling/reinstalling SP2, but this is a 2 month old Dell Latitude 610 with factory installed XP. There is no Windows Service Pack 2 option listed in the Control Panel > Add/Remove Programs.
There's some other strange things, so I wonder if they are related.
1) I have Paul set up as an administrator account. Some folders like MSSQL show that account with no permissions. I grant all the permissions to Paul for that folder. I come back later and the permissions are gone.
2) I deleted 20 files in Explorer, but 7 of them did not go away. I deleted those 7 again and they instatnly reappeared. I deleted those 7 again and then they finally went away.
3) I get a slow reaction time for things like Windows Explorer and opening and closing programs. This is suprising since it has 2 gig of RAM and 2.3 Gig processor. Could it be a memory handling problem that's causing OS problems. Probably, the memory didn't handle the OS installation well and the whole system is compromised now.
We're doing upgrades from SQL 2008 R2 to SQL 2014. This is blocked due to RS is installed but not configured. Our desired action is to uninstall RS and proceed with the upgrade. But when setuparp.exe is raised, it does not list all the features on the 'Select Features' page. In fact, it only lists the last 2 shared features (SQ Client Connectivity SDK and Microsoft Sync Framework). However, all items appear to be listed on the 'Select Instance' page including RS. I've seen this issue on 2 of our SQL 2008 R2 Servers already.
I used SSEUtil to add a schema to my database but I am having problems. Used these steps:SSEUtil -c> USE "c:Rich.mdf"> GO>!RUN Resume.SQL//indicates success>SELECT * FROM SYS.XML_SCHEMA_COLLECTIONS>GO//schema not shown in list> USE master>GO>SELECT * FROM SYS.XML_SCHEMA_COLLECTIONS>GO//schema is shown in the queryIt appears that the schema is not added to the desired database, so when I try to use the schema in Visual Studio, the schema does not appear when I connect to the Rich.mdf database. Any ideas on what I am doing wrong or why this might be happening?ThanksKevin
I'm trying to install SQL Server Management studio 2012 on my Windows 7 (x64) standalone laptop. When I click "New SQL stand-alone installation..." it runs a Setup Support Rules check and always fails "Setup Account Privileges". I've looked into the error and I keep getting that I need to change security rules but I don't have that option in window 7. How do I get around this without having to resort to a computer running Windows Server?
I have Visual Studio 2013 premium installed along with Localdb v11. I just want to connect and manage my database engine through SSMS when developing any application.
This forum is intended for users who are new to SQL Server, and have basic usage questions. If you have setup or installation issues or questions, you should check out the Setup forum.
I am running a number of SQL instances on my PC. Within the network, I have think server with various System Center components. For compatibility reasons, some features of System Center 2012 R2 had to be delegated to different SQL databases. My question is, because there is now more than one IP address on my system, and each instance of SQL is assigned to its own IP, is there a way to setup DNS and SQL so the namespace points to the desired IP address? For Instance:
MSSQL2008 instance is set to run on = 11.12.13.1 MSSQL2012 is set to run on = 11.12.13.2 IN DNS: A Record: Mike-PC = 11.12.13.1 A Record: Mike-PC = 11.12.13.2
If I want to use MSSQL2008 by specifying Mike-PC as the DNS name, how would I do that with 100% accuracy? If there is another way to get the job done, I am more than willing to approach this differently.
hello, i need some opinion on how to sum up or group by more than 2k records faster.. eg, how do i optimize this?
SELECT DISTINCT r.ClientID,c.ClientName, r.ItemID, r.StockID,r.StockName, r.ExpectedQty,r.QCQty,r.AVAQty,r.PNDQty as pnd, r.VMIQCQty,r.VMIAVAQty,r.VMIPNDQty as vmipnd,
(Select isnull( SUM(d.HoldQty) ,0) FROM tblItemdetail d WHERE d.itemid=r.itemid AND d.ConsignorID=@ClientID AND d.Ownerstatus='VMI') AS VMIPNDQty,
(Select isnull( SUM(d.HoldQty) ,0) FROM tblItemdetail d WHERE d.itemid=r.itemid AND d.ConsignorID=@ClientID AND d.Ownership= i.Supplier AND d.Ownerstatus='VMI') AS PNDQty,
(Select isnull(SUM(d.OriginQty - d.PickQty -d.HoldQty -d.qcqty),0) FROM tblItemDetail d WHERE d.ConsignorID=@ClientID AND d.Ownership= i.Supplier AND d.Ownerstatus='OWN') AS StockAtCustAVAQty,
(Select isnull(SUM(d.HoldQty),0) FROM tblItemDetail d WHERE d.ConsignorID=@ClientID AND d.Ownership= i.Supplier AND d.Ownerstatus='OWN') AS StockAtCustPNDQty,
(Select isnull(SUM(d.qcqty),0) FROM tblItemDetail d WHERE d.ConsignorID=@ClientID AND d.Ownership= i.Supplier AND d.Ownerstatus='OWN') AS StockAtCustQCQty
FROM tblItemCrossRef r INNER JOIN tblClient c ON c.ClientID=r.ClientID INNER JOIN tblItemClients i on i.Supplier=r.ClientID WHERE r.ClientID=@ClientID AND r.StockID LIKE @StockID+'%'
Hello all-Given the following UDF, in sql 2000 can it be sped up, complied oranything of the like. A query returning 300,000 + rows times out whenran through the udf, inline case statements returns the rows in 5seconds.Thanks!JeffCREATE FUNCTION dbo.TimeFormat(@input datetime,@groupformatvarchar(20) --DAY, WEEK, MONTH)RETURNS datetimeASBEGINdeclare @dtvar as datetimeif @groupformat = 'DAY'set @dtvar = CAST(CONVERT(char(10), @input, 101) AS datetime)else if @groupformat = 'WEEK'set @dtvar = CAST(DATEADD([DAY], 1 - DATEPART(dw, CONVERT(char(10),@input, 101)), CONVERT(char(10), @input, 101)) AS datetime)else if @groupformat = 'MONTH'set @dtvar = CAST(CONVERT(CHAR(6), @input, 112) + '01' AS datetime)return @dtvarEND
This is x-posted in:alt.php.sqlcomp.databases.ms-sqlservermicrosoft.public.sqlserver.programmingI have events that occur during the day. I want to be able to search thoseby a form with checkboxes (multiple select).Let's say for instance an event is happening from 3-10pm. When someonesearches for 4-6 (checkbox option) it needs to show up.I don't need code so much as I just need theory. My theory that I coded outand worked, just a missight in theory is as follows. I did a BETWEEN callthat pulled any event that began BETWEEN 4 AND 6 or ended BETWEEN 4 AND 6.As you can see. The event spans that time, but does not start or stopbetween 4 and 6, thus was not pulled. Ooops.So if someone call tell me of another function or perhaps just a better wayto use BETWEEN that would be great. I don't think that code is necessary atthis juncture, so save the 'Please post code' post :) Thanks.
I am using sp_executesql this to pass parameter to sql string and I am seeing deadlock between sp_prepexec which does UPDATE with another UPDATE done by another process. When it comes to speed and deadlock, would you recomand not using sp_executesql?
Hi, Can anyone tell me a way to speed up these querys? //This is selecting a number of records (sent by user) from a table and randomizing those tempSQL.Text = "select top " + amount.Text + " number from [" + src.Text + "] Where pull='N' order by newID()";
SqlConnection conn2 = new SqlConnection(ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString); conn2.Open(); SqlCommand cmd3 = new SqlCommand(tempSQL.Text, conn2); cmd3.CommandTimeout = 1000; SqlDataReader dr = cmd3.ExecuteReader(); //Then I open a data reader that uses the records SqlConnection conn2a = new SqlConnection(ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString); conn2a.Open(); while (dr.Read()) { //the records are then placed 1 by one into a temp table string fillresultID = "Insert into [" + src.Text + "_Additional_Temp] (number) Values('" + dr["number"] + "')"; SqlCommand cmd4 = new SqlCommand(fillresultID, conn2a); cmd4.CommandTimeout = 0; cmd4.ExecuteNonQuery(); //then the original table that held the numbers is marked as used(again one by one) string update = "Update [" + src.Text + "] set pull='Y' where number='" + dr["number"] + "'"; SqlCommand cmd5 = new SqlCommand(update, conn2a); cmd5.CommandTimeout = 0; cmd5.ExecuteNonQuery(); } dr.Close(); conn2.Close(); conn2a.Close(); Thanks, Doug
Hi, how can i speed up this query, it seems to be taking a very long time to bring back the reults; --This stored procedure retrieves access rights for usersCREATE PROCEDURE wc_User_Access_Right_List ASSELECT dbo.tblRep.Rep_ID, RTRIM(dbo.tblRep.Rep_Forename) + ' ' + RTRIM(dbo.tblRep.Rep_Surname) AS User_Full_Name, dbo.tblAccessRight.Access_Right, dbo.tblAccessRight.Access_Right_IDFROM dbo.tblRep LEFT OUTER JOIN dbo.tblAccessRight ON dbo.tblRep.Access_Right_ID = dbo.tblAccessRight.Access_Right_ID ORDER BY User_Full_Name
--Make sure this has saved, if not return 10 as this is unexpected error IF @@rowcount = 0 return 10 DECLARE @RETURN_VALUE tinyintIF @@error <>0 RETURN @@errorGO
I have a select statement that has many 'ands' where one side uses the same thing for ex.
where t1.column1=r3.other and t1.column1=e5.new and t1.column1=k9.old etc...is there any speed gained by putting the value for column one into a variable and using that variable each time instead of t1.columns1?
I have indexed my SQL Server tables to gain some speed on calling up tables and queries ( using VB and ADO ). It is still very slow...Is there a move I have to make once my tables are indexed or is there any tricks to improve the speed cause I am getting kinda desparate right now :(
I am in the planning stages of a website and want to design my database to allow for optimum performance in case the site becomes popular. In my database I will have around 5 main categories with each category having around 25 or 30 subcategories and each subcategory having around 100 to 300 items. I plan to use MS-SQL Server7. Would it be best to have one large table or have 5 tables for the main categories or have around 100 tables one for each subcategory? Database usage will be simple (no complex queries) but most pages will hit the database and I need to allow for a potential of 4 or 5 million page views per month. The SQL Server will be a shared one.
Any one got any tricks on speeding up storing and retrieving image data? I have a typical 2MB image and I am trying to improve speed... it now takes 8 seconds to store the blob.
I have migrated a database from ms access to ms ms sql. But I am encountering a problem with the website loading speed. It was running much faster with ms access then it is running with ms sql.
We are currently running Backup Exec v7.0 and are backing up two SQL 6.5 servers on our network. The backups have begun taking excessive amounts of time considering the limited amounts of data on the servers. Does anyone have an idea of parameters we could check on the servers or other settings that should be checked in Backup Exec? Also, this problem seems to have surfaced after we loaded Network Associates VirusScan NT onto our NT Servers and Workstations. Has anyone had problems with this product on their SQL servers?
I've recently been given the task to maintain a website that runs off mssql using vb and asp .net. One of the main task is to improve the mssql access time. I am fairly new to mssql, can anyone give me some insight on the procedure to improve access time to mssql. Obviously sql query plays a big part, but what about database config, asp/vb .net commands or design patterns (for large amount of data pulling and join?)
Any suggestions or pointer will be greatly appreciated.
it is working but takes about 3-4 seconds per exec.
CREATE PROCEDURE isp_ap_calc_apt_totals @p_comp char(2), @p_vend char(6), @p_asofdatechar(8) as
if (@p_asofdate <= '00000000') begin set @p_asofdate = '99999999' end
delete from XAPAPTTOT where xapt_comp = @p_comp and xapt_vend = @p_vend and xapt_asof_date = @p_asofdate
insert into XAPAPTTOT select apph_comp, apph_vend, apph_type, apph_id, @p_asofdate, sum(apph_paymnts), sum(apph_discts), sum(apph_adjts), count(apph_paymnts), sum(apph_paymnts)+ sum(apph_discts) + sum(apph_adjts) + b.apt_gross, 0, max(str(yy,4) + replace(str(mm,2),' ','0') + replace(str(dd,2),' ','0')) from APPHISTF.a join APTRANF.b on b.apt_comp = a.apph_comp and b.apt_vend = a.apph_vend and b.apt_type = a.apph_type and b.apt_id = a.apph_id where ((a.apph_comp = @p_comp) and (a.apph_vend = @p_vend) and (a.apph_unpost_dt = 0) and (str(a.yy,4) + replace(str(a.mm,2),' ','0') + replace(str(a.dd,2),' ','0') <= @p_asofdate)) or ((a.apph_unpost_dt > 0 and a.apph_unpost_dt <= @p_asofdate and b.apt_unposted_fg = 1 and b.apt_comp = @p_comp and b.apt_vend = @p_vend and b.apt_type = a.apph_type and b.apt_id = a.apph_id)) or (((str(a.yy,4) + replace(str(a.mm,2),' ','0') + replace(str(a.dd,2),' ','0') <= @p_asofdate) and a.apph_unpost_dt > @p_asofdate and b.apt_comp = @p_comp and b.apt_vend = @p_vend and b.apt_type = a.apph_type and b.apt_id = a.apph_id)) group by apph_comp, apph_vend, apph_type, apph_id
update XAPAPTTOT set xapt_last_payck = (select max(apph_payck) from APPHISTF where apph_comp = xapt_comp and apph_vend = xapt_vend and apph_type = xapt_type and apph_id = xapt_id and str(yy,4) + replace(str(mm,2),' ','0') + replace(str(dd,2),' ','0') = xapt_last_paydt ) where xapt_comp = @p_comp and xapt_vend = @p_vend and xapt_asof_date = @p_asofdate GO
I'm not sure if this is the right forum, but I have a general question about running/storing databases. I have been running a process with 60+ million records in one table and another 16 million in another table and it is taking forever to get everything imported in and run the appropriate queries. I've been doing this all on a desktop and I am anxious to learn of a more efficient, faster method of processing this amount of data.
What solution should I pursue if I am doing this work a few times a year so that it doesn't take three full days of processing to reach an answer with the data?
