Hi there,I'm looking for advices from people with more experience using SQL Server. I have a situation where from my perspective i have 3 ways to accomplish my task, but i dont know which way would be more efficient and performant.Here's the deal, i have a table which holds millions of records. This table will eventually receives between 30-40 insertion a second. So it's pretty busy. To display the data correctly in reports over the web, i need to manipulate the data in such a way that certain transaction are modified and others are completly eliminated depending on a set of preferences choosen at the moment of the request for the report. Obviously i dont see any ways to do this with just one SQL query, so my choices are the following:1. Create a second table, which would hold the manipulated data. Create a trigger and on each insertion manipulate the data and modify the second table.2. Create a stored procedure, from the stored procedure i query my records, create a temp table, manipulate my data and return it to the user3. Simply fetch the data, return it to the IIS server and process it there before returning the result to the client.The manipulation process consists of querying about 1000 to 2000 records, eliminating duplicate results (or similar results within a chosen range) by comparing them to the last one, and to indicate for each record how many other records were similar. In other words, it's a basic loop and i compare each record to the last one to see if it changed.Personally i think choice #2 would be better, but i could be wrong. I'd like to hear your comments and suggestions.Thanks,
I would like to develop an SSIS solution which loads data from large-sized csv files into the database either in REPLACEMENT or UPDATE mode.
In REPLACEMENT mode I delete all the exisitng data (if any) in the corresponding tables and then I do insertions with a few transfomations beforehand. I have accomplished this mission successfully with SSIS as follows: - Control Flow: Execte SQL Task for performing pre-execution tasks --> Data Flow Task --> Execte SQL Task for performing post-execution tasks - Data Flow: Flat File Source --> Some transformation components --> OLE DB Destination with FastLoad options.
In UPDATE mode I would like to update existing records in the database. I have found a great deal of articles talking about updating existing records. Based on my readings, here are the statements, I have concluded: 1. If the expected number of updates is high, use a staging table and perform the update as a batch operation from the Control Flow. 2. If the expected number of updates is low, use an OLE DB Command transformation.
Question: Well, in my particular case all the rows in the source file represent record updates. And I am expecting a large number of update records. Thus, according to the previous conclusions, I should use the staging table alternative. What I do not like about this alternative is that I will have to insert all the source rows in a staging table first and then I will have to update them all. This seems like too many extra work.
Is there a way to avoid this extra insertion work? I am thinking of something like "FastLoad Update" similar to the "FastLoad Insert", where I directly map input columns to output columns after defining a certain WHERE-CLAUSE?
PeopleFields PersonID FieldDefID FieldValue 1 1 John 1 2 Smith 2 1 Alice 2 2 Johnson
Of course, we need to be able to search and display this data in a tabular format like: PersonID PersonType FirstName LastName 1 Consultant John Smith 2 Partner Alice Johnson
We have been building dynamic queries based on which fields are needed (users can select the fields), e.g.:
SELECT p.PersonID, p.PersonType, pf1.FieldValue AS 'First Name', pf2.FieldValue AS 'Last Name' FROM People p LEFT JOIN PeopleFields pf1 ON (p.PersonID=pf1.PersonID AND pf1.FieldDefID=1) LEFT JOIN PeopleFields pf2 ON (p.PersonID=pf2.PersonID AND pf2.FieldDefID=2)
This is very flexible but slow. We've done lots of optimization but can't get this below 5 seconds for common scenarios.
So I'm back to the drawing board now trying to figure out a better way to approach this. I'm wondering if it would be better (if even possible) to break this out into some sort of view or table UDF that would contain a full representation of all person data, pre-joined. Problem is, this is almost certainly going to have to involve dynamic SQL since we can't know anything about what fields are defined. I think that rules out any sort of view or table UDF, no?
Does anyone have a suggestion for a good approach? Thanks.
I have an EDI file with Different Transaction types. I would like to read for a Header, capture some specific info on the header, and read for another specific tran type that comes after and capture additional info. I have a couple of ideas like tagging the records with a key and loading into two temp tables for matching later but that means I would have to do double processing. Maybe a conditional split for the two transaction types and then unioning them downstream, but not sure if the right records would be associated. Possibly tagging sequential key and writng to raw files and matching on the raw file keys downstream.
I have a lot of ideas but I am looking for the best proven practice here so i don't spin my wheels or have to go back and re-engineer later.
Hi everyone In my SqlServer Management Studio Express, on start up it shows the server type option, but greyed.So that value is fixed to database engine. ( I'm trying to work on an SqlServer Compact Edition database through the SSMStudiothat's why I'm trying to get this to change.)Besides, after I connect i go to the Object Explorer, expand the server node, and go to Replication.When i expand replication, i get the "Local Subscription" option, but nothng for Publication.( I want to work on Merge Replication, that's why I desparately need Publication to work)Am i missing something here? I did not install SqlServer separately, I only have what comes bundled with the Visual Studio 2005 Setup.
So I started a new job recently and have noticed a few strange configurations. Typically I would never mess with min memory per query option and index create memory option configuration because i just haven't seen any need to. My typical thought is that if it isn't broke... They have been modified on every single server in my environment.
From Books Online: • This option is an advanced option and should be changed only by an experienced database administrator or certified SQL Server technician. • The index create memory option is self-configuring and usually works without requiring adjustment. However, if you experience difficulties creating indexes, consider increasing the value of this option from its run value.
Hello Which is better and faster?? and WHY????Writing Select Statement with joins in Stored procedure, or creating view and calling it from stored procedure (select * from view)..
I am not entirely ignorant to web technologies, and best practices but i am having a bit of a planning dylema.
My company has a well established SQL 2000 database with windows application which has been created by myself, what i am planning on doing is creating a web site, using asp.net and publishing some of the information, so that our clients may use it, and stop pestering us on the phone. what i would like to know is what would be the best way forward, obviously i don't want to show them all our information, and don't want to put 5Gb worth of data onto a ISP website. What would you suggest i do?
DECLARE @returnDay int SELECT @returnDay = DatePart(day,GetDate()) If @returnDay = 8 BEGIN select * from Hospitals left join Units ON Units.HospitalID = Hospitals.HospitalID where Units.HospitalID is null RETURN END
this is just a part of the procedure I am trying to create, I am getting hospitals that haven't submitted any data and wish to send them an email.
on the other hand I have two tables that have all the data for emailing to hospitals but are not linked to tables giving the list of hospitals
I have been advised to create a cursor(easier said then done) that will go through my list of records that need to receive an email
nothing going very well with that at the moment.
so I was hoping to see if somebody has any other suggestions for me.....
Let me try to explain it...I am getting DEGREEID from one of the SELECT query . I want to OUTPUT (ie , COUNT) from procedure,the number of departments with the degreeid, got from the above query.
With below procedure, Since an employee can have multiple DEGREEID , the cursor is giving OUPTUT ie, COUNT for the LAST Degreeid. Eventhough the previous DEGREEID dont have any DEPARTMENT...but only for the LAST DEGREEID...!
How can I solve this..... whether I can solve this with CURSOR or I have to use someother way...Please advice me !
My first request for help here even if I read this site quite often and got tons of usueful information. Thanx all
I have an application (VB 6) calling store procedure on a SQLServer 2000 DB. One of the table gives me headache. I wrote a simple store procedure to insert a record into that table. If I call the store procedure from query manager it works perfect but if I call it from VB it looks like to work (return from the execute fine) but then I query the table for that record and it'll just take time and then return time out. I have to stop the VB application and then query it again then it'll return no record.
I suspect the table being locked somehow but I check inside the VB app code and that's the only place the table is called. Further more I have hundred of store procedures being used that way and they're all ok. I indexed the table , no use either...
I am just short of ideas how to debug this...I'll need your advice :)
Okay, given my newness to SQL, and the complexity of this query, I thought I'drun this by you for your opinion:SELECT DISTINCT a.WeekEnding, c.lastdate, c.numlate, b.totaldateFROM Accomplishment a LEFT OUTER JOIN(SELECT weekending, COUNT(weekending) AStotaldateFROM AccomplishmentWHERE (EmployeeID = 50)GROUP BY weekending) b ON a.WeekEnding =b.weekending LEFT OUTER JOIN(SELECT weekending, MAX(entrydate) ASlastdate, COUNT(weekending) AS numlateFROM accomplishmentWHERE employeeid = 50 AND entrydate >weekendingGROUP BY weekending) c ON a.WeekEnding =c.weekendingORDER BY a.WeekEndingWhat I'm trying to do is for each pay period find which ones the employeesubmitted a timesheet and which they were late (and if they were late, howmany of them). However, the query takes a good 5 seconds, and it seemsremoving the "entrydate > weekending" clause speeds things up to almostinstant, however it does ruin the count that I really want. No idea whythat makes such a difference..
I have an Accounting system(vb.net 2003, SQL server 2000), every new year data is cleared, but i may use some data from previous years (such as liabilities)??
whats the best way to that ???
-Shall I create programmatically a new clone DB every year (new DB with same structure as previous year) OR -Shall I add a "year field" for tables in DB????
knowing that data will keep growing every year??????
whats the best solution, knowing that i dont want the end user of my application to do anything manually, such as creating DB ......
I have a couple of files that I have that are comma seperated, and am looking for the best way to take those files, but them in a temp location to run some sql up against, and than update or insert a production sql database based on a SP i have written that takes a number of variable. I have played around with using the recordset destination and defining all the variables and than using a for each loop but seem to be stuck. Somewhat new to the whole programming field, and to be honest, seems a little intimidating with the little I know of the programming side. Just looking for some thoughts.
I am new to the CE OS, SQL CE and mobile computing in general. I have been developing database applications using MS products ever since Windows 3.11 and Visual Basic 1.0. I have used MS Access and SQL Server since they were first released. But, I have never worked with Windows CE or any of the mobile OSs or with SQL CE or SQL Mobile.
We now have a project that requires us to develop a database application on handheld devices using the CE 4.2 and CE 5.0 OSs.We will be using CF 2.0, VS 2005 and SQL 2005 for our development environment.
My questions are: 1. Which version of mobile SQL will allow us to best develop for both the CE 4.2 and 5.0 OSs usinf VS 2005? I have done a lot of reading online and it's pretty confusing. There are quite a number of different versions out there. It seems some work with 4.2 and some with 5.0. Is there a version that will work for both?
2. Is SQL CE 3.0 (Sql 2005 Mobile) available for other than Laptop and Tablet use? When i finaly got to the download page for this version there was verbiage there that suggested it was not available for smart devices.
3. What is a good source of info to resolve these questions? I am using the MSDN areas for CE and SQL Mobile but haven't really found what I need to get started. Any suggestions on forums, books, articles, blogs, etc... would be greatly appreciated.
I know these are very broad questions but I want to get some advice from the experienced before going to far here.
I just want to ask for any good advice here ragarding my project. I'm planning to create a program that will use an sql server as a database. I have 3 branches and I need the branches to be updated always when they create any transaction or changes. The sql server will be installed in a PC with Windows Server 2003 and with a public IP. the question is can I access the sql server from other branches by using the public IP of the ServerPC thru the Internet? If yes, how? if no, is there a best way to do this updating method?
I am totally new to SQL and need to ask some basic questions.
Can anybody tell me if SQL 2005 will run ok on Windows Server 2008 32bit and/or 64bit.
Also, we are currently planning to implement OCS 2007 and we are considering running it on Windows Server 2008 32bit with SQL 2005. Does anyone know if this combination is possibleand if not what implementation should we be looking at.
here is my situation, i have a central database server that contains all the data, running in an intranet, the client application, which is a thick client containing most of the biz logics is installed on a laptop,
i m required to develop a solution so that when the client application is disconnect to the database server (e.g. the laptop is taken away somewhere) the application can still work and when the client application gets connect to the database server again the application will be able to synchronize changes with the database and update all the changes. and i m not allow to develop a web-base application
my initial thinking is to use merge replication, however, i m conerning with the performance on the laptop if i have to install sql server on the laptop.
is there any other approach that can still achieve the requirement and have a better performance?
I'm creating a DB to track clients, programs, and client participation in the programs. They are service programs. A client can be in more than one program and a program can have more than one client. Can someone give me an example of how they would layout the tables? My guess is: tblClient, ClientID tblClientProgramLog, ProLogID, ClientID tblProgramDetails, ProDetailID, ProLogID tblPrograms, ProgramID, ProDetailID I appreciate any suggestions,
Hello,Consider I have a String:Dim MyString As String = "Hello"or an Integer:Dim MyInteger As Integer = 100or a class which its properties:Dim MyClass As New MyCustomClass MyClass.Property1 = "Hello" MyClass.Property2 = Unit.Pixel(100) MyClass.Property3 = 100Or even a control:Dim MyLabel As Label MyLabel.Id = "MyLabel" MyLabel.CssClass = "MyLabelCssClass" Is there a way to save a String, an Integer, a Boolean, a Class, a Control in an SQL database?Something like: Define something (Integer, String, Class, Control, etc) Save in SQL 2005 Database Later in code Retrive from database given its IDIs this possible?How should I do this?What type of SQL 2005 table field should be used to store the information?Thanks,Miguel
Hello,I am creating a simple blog system using SQL 2005.I have a Blog table:[BlogId] > PostId (PK), BlogTitle, ...And a Posts table[Posts] > PostId (PK), BlogId (FK), PostContent, PostLabels, ...PostLabels would have the following format:Label1,Label2,Label3, etc ...I will need to perform 3 actions:1. Get all posts in blog2. Get all labels in a post3. Get all unique existing labels in all posts in a blog and make a list.I am not sure if my approach of using a simple labels column in my Posts table is a good idea.So my other idea would be to add two more tables:[BlogLabels] > BlogLabelId (PK), BlogId (FK), LabelName ...[LabelsInPosts] > BlogLabelId (PK), PostId (PK)So my idea is:1. When creating a post one of the parameters would be a comma separating string with all labels for the post. Inside SQL Procedure I will need to loop through each label and check if it exists in BlogLabels. If not then I added it. For each label I add a records in LabelsInPosts. How to create this loop? Am I thinking this right?2. To get a list of all labels in a blog I would need to go to BlogLabels and get all labels which are related with posts in LabelsInPosts. Those posts must be only the ones that are related with my given BlogId. Grrr, this is getting really confusing for me. Is this possible to to? How?Please, give me some advice about all this.Thanks,Miguel
Hi everyone, My hoster hosts asp.net but does not yet support sql 2005 only sql 2000 and access I want to use either of the starter kits. I am confused on what is needed to make the changes to make either sql 2000 or access work. I know enough that the connection strings will need to change my concern is code. Is there strings that I will need to find and change in the application. I sure would appreciate any and all advice. Thank you for your help. DKB
Hi,I recently contacted my hosting company's customer support about my databases not working - saying that I use sql express (which they support).The guy recommended: "I would suggest you to upgrade the db's to use mssql 2005." "This is because, sql express is built for development environment. When you are in development environment, you are accessing everything with administrator permission. However, in live hosting environment (when there are differnet kind of permission restrictions), sql express often failed on attaching database." Does anyone have any opinion on that? Would it be best to change db's to use mssql 2005? How complicated/time consuming will it be to upgrade?Thanks!Jon
Hi, I just started learning ASP.NET this week and have watched a mountain of videos from this website which has helped me alot However I have been stuck on a problem for 2 days now. I have created an SQL database with the Following 2 tables: USERS COMPUTERSUserid Computerid firstname Useridsecondname Manufacturer Model I have made a relationship between the 2 tables. I then created a dataset with the following query:SELECT COMPUTERS.Computer_ID, COMPUTERS.Manufacturer, COMPUTERS.Model, USERS.First_Name, USERS.Last_Name FROM COMPUTERS INNER JOIN USERS ON COMPUTERS.User_ID = USERS.User_ID I however do not get the option in my grid view when i output this data, to UPDATE. The best i have found from google is that i need to use subqueries and not innerjoins but i just cant seem to get my head around them, please help as i feel my head may just explode if i think about this or try any more ways to get this to work :D
If i'm building an application would it be better to use the asp.net membership api or since i have a local instance of the database and admin rights should i use the logins and roles from sql server? If i do use direct sql server what will be the best way to do it? I ask this because it just might not be a web application
I'm trying to design a database that handles Clients, Cases, Individual and Group Sessions. My problem is that a client can have individual sessions and belong to more than one group at the same time, so I have a many-to-many relationship to deal with. Also I'm trying to design it so that I can have a form that when a group is selected from a drop down it shows all clients assigned to that group and will let me enter new session data for them.
Just looking for some advice on how to handle the relationships. Maybe someone could show me how they see the relationships working.
My take is that the session is linked to the case not the client, I could be thinking incorrectly?
I'm about to embark on a new project and I'm having a few troubles deciding what .NET technologies I need to learn and employ.
The system will be built around SQL Server.
It will involve a client application that will be used to gather data offline using MSDE, then sync back up with the online DB. This will then have a web front-end for users not employing the client app.
I'm pretty fine with the whole front-end and data manipulation thing, but I don't know where to start with the synchronisation.
Could someone please tell me how I should go about getting the systems to communicate? Do I need to use Web Services and do it by XML? Message Queueing? Or are there simpler ways?
I have a web app which is used to do normal insert/update of employee info. Connected to each employee that is entered is some data that is imported from an outside source for each employee. The question I have is currently my database is very normalized and importing data from this outside source will be quite a pain because of this. Is it bad practice to denormalize a specific table if no user will every insert/update it beside DTS?
Hi guys,we has accidently restored our Database with a back up file created months ago. we have no back up for the huge data and Stored procedures ( I know it is very stupid :( and i m sad abt it all ), guys is there any way by which i can get restore my db back to what it was before restoration.the flks in team have also deleted(shift + delete) the transaction log file ( I know it is very stupid again :( and i m very sad abt it all )..please adviceThanx in advance ashish
I'm looking for some advice on the best way to execute a complicated search on my eCommerce web site. My example here is much simpler than the real thing, but it should be enough to get the point across. Imagine that I have two tables: Products Table id as Integer code as VarChar(10) name as VarChar(50) mfr as Integer description as Text Manufacturers Table id as Integer name as VarChar(25) I need to write a keyword search that will return matches on the product code, product name, product description or manufacturer name. I could do a simple join and use a lot of LIKE clauses, but it's not very efficient. SELECT p.id FROM Products p INNER JOIN Manufacturers m ON p.mfr = m.id WHERE p.code LIKE '%Keyword%' OR p.name LIKE '%Keyword%' OR p.description LIKE '%Keyword%' OR m.name LIKE '%Keyword%' Having those wildcards at the start of the like kill performance. So next I thought, maybe I join all of the fields together, then do the LIKE. For example: SELECT p.id FROM Products p INNER JOIN Manufacturers m ON p.mfr = m.id WHERE p.code + p.name + p.description + m.name LIKE '%Keyword%' This is certainly better than above, but it still doesn't seem ideal. Can anyone suggest something I might not be thinking of? The product codes tend to have dashes, periods and numbers that preclude the Full-Text search features of SQL Server, so I'm grasping at straws for a better approach...
