Best Way To Insert Large Amounts Of Data From A Webform To SQL Server 2005
Oct 21, 2007
Hi
I have a VB.net web page which generates a datatable of values (3 columns and on average about 1000-3000 rows).
What is the best way to get this data table into an SQL Server? I can create a table on SQL Server no problem but I've found simply looping through the datatable and doing 1000-3000 insert statements is slow (a few seconds). I'd like to make this as streamlined as possible so was wondering is there is a native way to insert all records in a batch via ADO.net or something.
I'm trying to move my current use of an sql 2000 db to sql 2005.
I need to update a table definition (to change a field to an Identity)
I'm getting a dialog box (in SQL server management studio) on save saying :
'xxxx' table
- Saving Definition Changes to tables with large amounts of data could take a considerable amount of time. While changes are being saved, table data will not be accessible.
I press 'Yes' to the dialog box.
After 35 seconds, I get another dialog box saying:
'xxxx' table
- Unable to modify table.
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Well, the server is responding and I can query that talbe and other, I can add/delete rows to other columns. I can modify other (smaller) tables.
I'm in the process of migrating a lot of data (millions of rows, 4GB+of data) from an older SQL Server 7.0 database to a new SQL Server2000 machine.Time is not of the essence; my main concern during the migration isthat when I copy in the new data, the new database isn't paralyzed bythe amount of bulk copying being one. For this reason, I'm splittingthe data into one-month chunks (the data's all timestamped and goesback about 3 years), exporting as CSV, compressing the files, and thenimporting them on the target server. The reason I'm using CSV isbecause we may want to also copy this data to other non-SQL Serversystems later, and CSV is pretty universal. I'm also copying in thisformat because the target server is remotely hosted and is notaccessible by any method except FTP and Remote Desktop -- nodatabase-to-database copying allowed for security reasons.My questions:1) Given all of this, what would be the least intrusive way to copyover all this data? The target server has to remain running and berelatively uninterrupted. One of the issues that goes hand-in-handwith this is indexes: should I copy over all the data first and thencreate indexes, or allow SQL Server to rebuild indexes as I go?2) Another option is to make a SQL Server backup of the database fromthe old server, upload it, mount it, and then copy over the data. I'mworried that this would slow operations down to a crawl, though, whichis why I'm taking the piecemeal approach.Comments, suggestions, raw fish?
Hello,Currently we have a database, and it is our desire for it to be ableto store millions of records. The data in the table can be divided upby client, and it stores nothing but about 7 integers.| table || id | clientId | int1 | int2 | int 3 | ... |Right now, our benchmarks indicate a drastic increase in performanceif we divide the data into different tables. For example,table_clientA, table_clientB, table_clientC, despite the fact thetables contain the exact same columns. This however does not seem veryclean or elegant to me, and rather illogical since a database existsas a single file on the harddrive.| table_clientA || id | clientId | int1 | int2 | int 3 | ...| table_clientB || id | clientId | int1 | int2 | int 3 | ...| table_clientC || id | clientId | int1 | int2 | int 3 | ...Is there anyway to duplicate this increase in database performancegained by splitting the table, perhaps by using a certain type ofindex?Thanks,Jeff BrubakerSoftware Developer
I was wondering if any one could help me, I need to store large amounts of data in my database, at present I have it set to nvchar (8000), I've looked around and noticed you can use text which stores up to 2 million, but is slow in displaying the information.
Any ideas or points in the right directions would be great.
Does anyone have ideas on the best way to move large amounts of databetween tables? I am doing several simple insert/select statementsfrom a staging table to several holding tables, but because of thevolume it is taking an extraordinary amount of time. I consideredusing cursors but have read that may not be the best thing for thissituation. Any thoughts?--Posted using the http://www.dbforumz.com interface, at author's requestArticles individually checked for conformance to usenet standardsTopic URL: http://www.dbforumz.com/General-Dis...pict254055.htmlVisit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbforumz.com/eform.php?p=877392
We are looking to store a large amount of user data that will bechanged and accessed daily by a large number of people. We expectaround 6-8 million subscribers to our service with each record beingapproximately 2000-2500 bytes. The system needs to be running 24/7and therefore cannot be shut down. What is the best way to implementthis? We were thinking of setting up a cluster of servers to hold theinformation and another cluster to backup the information. Is thispractical?Also, what software is available out there that can distribute querycalls across different servers and to manage large amounts of queryrequests?Thank you in advance.Ben
I have a dataset with 300,000 records and I'm getting the following error with MS Reporting Services. "An error has occurred during report processing. Exception of type System.OutOfMemoryException was thrown. any help with this would be highly appreciated.
I need to be able to graph roughly about 150 employees/ supervisor and their monthly cell phone usage in minutes. I understand that I will need to group this on say one graph for every ten employees so it doesn't look messy and cluttered. I have read some threads here but they dont seem to work for me.
So again each supervisor has 100+ subordinates and I need to graph theie phone usage by month
p.s. my email was incorrect in the last mail. Hi all, is there a sql 2k thread. Am interseted in finding out what the largest database size of a sqlserver database people have worked with. We have a 1.2 Terabyte db with about 150-200 million new rows being processed everyday. Would like to share some thoughts on this with other people who are working with this much data and what they are doing with it.
bhala ---------------------------------------- Please check us out at: http://www.bivision.org/bivision
Hello all, I am having problems running a stored proc from an aspx.cs file. Basically I want to extract data from webForm controls and create a new record in a DB table. I have watched the "Getting Started" videos on thi site, and the only difference between my code and the code of the demonstrator is the connection string. My conn string:- dataSrc.ConnectionString = ConfigurationManager.ConnectionStrings["ProjectTblConnString"].ConnectionString;Demonstrator conn string:- dataSrc.ConnectionString = ConfigurationManager.ConnectionStrings("ProjectTblConnString");When I debug with the string as shown by the demonstrator the error list reports that ConnectionStrings is a property and I am trying to use it as a mothod, which is fair enough, but I am just confused as how it worked for the demonstrator and not myself. When I debug with the code I used above I get no error, but nothing is inserted in the DB.Also, it looks as if the only class in my aspx.cs file that is actually being executed is PageLoad() - the remainder of the code seems to be ignored. Below is the entired aspx.cs file:- 1 using System; 2 using System.Data; 3 using System.Configuration; 4 using System.Collections; 5 using System.Web; 6 using System.Web.Security; 7 using System.Web.UI; 8 using System.Web.UI.WebControls; 9 using System.Web.UI.WebControls.WebParts; 10 using System.Web.UI.HtmlControls; 11 12 public partial class CreateProject : System.Web.UI.Page 13 { 14 protected void Page_Load(object sender, EventArgs e) 15 { 16 17 if (User.Identity.IsAuthenticated == false) 18 { 19 Server.Transfer("Default.aspx"); 20 } 21 22 if (chkbox_startDate.Checked == true) 23 { 24 txtbox_startDate.ReadOnly = true; 25 txtbox_startDate.Text = (System.DateTime.Now.ToString()); 26 } 27 28 } 29 30 protected void btn_create_Click(object sender, EventArgs e) 31 { 32 //connection string for connecting to SQL Server DB 33 SqlDataSource dataSrc = new SqlDataSource(); 34 dataSrc.ConnectionString = ConfigurationManager.ConnectionStrings["ProjectTblConnString"].ConnectionString; 35 36 37 //insert data in table using the CreateNewProject stored proc in the DB 38 dataSrc.InsertCommandType = SqlDataSourceCommandType.StoredProcedure; 39 dataSrc.InsertCommand = "CreateNewProject"; 40 dataSrc.InsertParameters.Add("Project_Title", txtbox_title.ToString() ); 41 dataSrc.InsertParameters.Add("Description", txtbox_desc.ToString() ); 42 dataSrc.InsertParameters.Add("Start_Date", txtbox_startDate.ToString() ); 43 dataSrc.InsertParameters.Add("Primary_Dev_Lang", list_devLang.ToString() ); 44 dataSrc.InsertParameters.Add("Dev_Environment", list_devEnv.ToString() ); 45 dataSrc.InsertParameters.Add("No_Junior_Devs", txtbox_juniorDevs.ToString() ); 46 dataSrc.InsertParameters.Add("No_Senior_Devs", txtbox_seniorDevs.ToString() ); 47 48 int rowsAffected = 0; 49 try 50 { 51 rowsAffected = dataSrc.Insert(); 52 } 53 catch (Exception ex) 54 { 55 Server.Transfer(Error.aspx); 56 } 57 finally 58 { 59 dataSrc = null; 60 } 61 62 if (rowsAffected != 1) 63 { 64 label_confirm.Text = "Error, Contact system admin"; 65 } 66 else 67 { 68 label_confirm.Text = "Success"; 69 } 70 71 }//end btn_create 72 73 //if cancel is clicked set all values back to default 74 protected void btn_cancel_Click(object sender, EventArgs e) 75 { 76 txtbox_title.Text = ""; 77 txtbox_desc.Text = ""; 78 txtbox_startDate.Text = ""; 79 txtbox_seniorDevs.Text = ""; 80 txtbox_juniorDevs.Text = ""; 81 list_devEnv.SelectedIndex = 0; 82 list_devLang.SelectedIndex = 0; 83 chkbox_startDate.Checked = true; 84 label_confirm.Text = ""; 85 } 86 87 //if checked the current dateTime is inserted into txtbox 88 protected void chkbox_startDate_CheckedChanged(object sender, EventArgs e) 89 { 90 if (chkbox_startDate.Checked == true) 91 { 92 txtbox_startDate.ReadOnly = true; 93 txtbox_startDate.Text = (System.DateTime.Now.ToString()); 94 } 95 else 96 { 97 txtbox_startDate.ReadOnly = false; 98 txtbox_startDate.Text = ""; 99 } 100 }//end chkox 101 102 103 }//end class
Any help anyome can give is greatly appreciated. This is part of my Final Year College Dissertation which is due in 3wks !!!! Sláinte á chaire, Seán
this may seem like a simple question, but I have a report/lease agreement I need to put together and wanted to know the simpliest way to add large amounts of text. Basically its all the legal stuff most leases include in the amount of some 14 pages.
Should this be just one long string-- or does ssrs have another way to format this
I was wondering what is the fastest way to UPDATE lots of recods. I heard the fastest way to perform lots of inserts in to use SqlCeResultSet. Would this also be the fastest way to update already existing records? If so, is this the fastest way to do that:
1. Create a SqlCeCommand object. 2. Set the CommandText to select the datat I want to update 3. Call the command object's ExecuteResultSet method to create a SqlCeResultSet object 4. Call the result set object's Read method to advance to the next record 5. Use the result set object to update the values using the SqlCeResultSet.SetValue method and the Update method. 6. repeat steps 4 and 5
Also I was wondering do call the SqlCeResultSet.Update method once per row, or just once? Also would it be possible and faster to wrap all that in a transaction?
Would parameterized updates be faster? Any help will be appreciated.
I have been looking into mirroring a large amount of small databases approx 150 databases.
As I understand this won't be feasible because of the way mirroring threading works, http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=441900&SiteID=1
As I understand it for every database being mirrored sql will ping the mirror second, causing a network bottleneck?.
Also that the amount of threads generated for each mirrored database will cause also cause a bottleneck?
At the moment our database servers are under very little pressure and as an estimate use about 10% of the resources allocated to them such as CPU utilization, memory, disk IO and network. Our server hardware is Dual Quad core Xeons with 4 - 8 gig of memory and variety of 10k SCSCI raid configurations from raid 5 or 1,0 and sql 2005 32bit.
Ive done some calculations on the log file generation rate compared to network bandwidth there is more than enough network bandwidth.
Has anybody had any luck in mirroring many small databases?
My concerns is how much traffic is caused by the pinging of the mirror for each database?,
How many threads will the mirroring cause and what is the max amount of threads sql can handle?
How much memory will be consumed by each one of these mirroring threads?
I am running into a problem inserting large amounts of text into my table. Everything works well when I test with a few simple words but when I try to do a test with larger amounts of text (ie 35,000 characters) the appropriate field is left blank. The Insert still performs (all the other fields recieve their data, but the "Description" field is blank. I have tried this with both "text" and "ntext" datatypes. I am using a stored procedure with input parameters. As I mentioned, the query goes off flawlessly with small amounts of data (eg "Hi there!") but not with the larger amount.I check and the ntext field claims to be able to accept 1073741823 bytes of data. Is there some other thing I should consider with large amounts of text?
I know the standard Microsoft recommendation is to make the pagefile at least 1.5 to 3 times larger then the amount of physical memory. However, if you're talking about a server with lots of memory such as 16GB or 32GB, would following this rule be unnecessary. With SQL 2000 running on Windows 2000 Server or Windows Server 2003 I typically see pagefile usage no more then 12% for a 2GB pagefile. Anything over 15% means I need to look at other indicators to see if a memory bottleneck has developed. If I have 32GB of physical memory and make the pagefile only 1.5 x 32GB I have a 48GB pagefile. 10% of this is 4.8GB, which I would hope I never see consumed.
I need the value stored in tblTwo.GuideMult (ie. One, BaseRate) to be translated into the numerical value shown on a webform (ie. @One, @BaseRate) and then insert the numerical values into tblOne.GuideMult
Clear as mud? Does somebody have a better way to do this?
Hi I've followed a tutorial on how to write and read varbinary(max) data to and from a database. But when i try to read the data i get the error that the data would be truncated, but only when the varbinary(max) is greater then 8kB. I've used a system stored procedure (sp_tableoption) to set the table that holds the data to store data outside rows. To select the data i'm using a stored procedure: SELECT imageData , MIMEType FROM Pictures WHERE (imageTitle = @imageTitle) And then using an .aspx page to Response.Write the data:Using conn As New sql.SqlConnection conn.ConnectionString = ConfigurationManager.ConnectionStrings("myConnectionString").ToString Dim getLogoCommand As New sql.SqlCommand getLogoCommand.CommandType = Data.CommandType.StoredProcedure getLogoCommand.CommandText = "GetPicture" getLogoCommand.Connection = conn Dim imageTitleParameter As New sql.SqlParameter("@imageTitle", Data.SqlDbType.NVarChar, 200) imageTitleParameter.Value = Request("imageTitle") imageTitleParameter.Direction = Data.ParameterDirection.Input getLogoCommand.Parameters.Add(imageTitleParameter) conn.Open() Using logoReader As sql.SqlDataReader = getLogoCommand.ExecuteReader logoReader.Read() If logoReader.HasRows = True Then Response.Clear() Response.ContentType = logoReader("MIMEtype").ToString() Response.BinaryWrite(logoReader("imageData")) End If End Using conn.Close() End Using Can anyone please help me with this?!
Hi,I m using vwd2005 and sql express,c#.I have a question here.In my database i have a table named Table1(id,name,age,country,email,phone) The values for these fields are inserted from the webform.Id is auto generated.name and age is passed from the textbox within the webform.country is passed from dropdownlist within the webform.email and phone is passed from listbox(multiple email and phone) within the webform.Now my problem is i want to retrieve all these row values back into the textboxes,dropdownlist and listbox at the same timeon the click of the retrieve button within the web form . Further more i want to make few changes on it and update the database on the click of another button update within the form.I hope u are getting it.How would the query would be in this case.?can u demonstrate the concept in one simple running example along with code?thanks.jack.
Hello Everyone,I 've trying to retreive data back into webform from the database from past couple of weeks.But i m not able to retrieve it completely.Here is my scenario.I've tried to explain in detail but in simple language.I hope u will get my point.Ok here it is.plz run my code:-i have three tables in sql.they are:-grup(id,grp) (1,abc) (2,xyz) organization(id,oid,gid,organisation,orgphone) (1,1,1,ibm,1234567) (2,1,2,microsoft,6543210) (3,2,1,oracle,7654323) (4,2,2,apple,9876543) userform(id,title,name,address,email,organizationid) (1,mr,jack,usa,jack@jack.com,1) userform.aspx page<%@ Page Language="C#" AutoEventWireup="true" CodeFile="userform.aspx.cs" Inherits="userform" %><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml" ><head runat="server"> <title>Untitled Page</title></head><body> <form id="form1" runat="server"> <div> <div style="text-align: center"> <table> <tr> <td style="width: 100px"> Title</td> <td style="width: 100px"> <asp:DropDownList ID="DropDownList1" runat="server"> <asp:ListItem>mr</asp:ListItem> <asp:ListItem>miss</asp:ListItem> <asp:ListItem>ms</asp:ListItem> </asp:DropDownList></td> </tr> <tr> <td style="width: 100px"> Name</td> <td style="width: 100px"> <asp:TextBox ID="TextBox1" runat="server"></asp:TextBox></td> </tr> <tr> <td style="width: 100px"> Add</td> <td style="width: 100px"> <asp:TextBox ID="TextBox2" runat="server"></asp:TextBox></td> </tr> <tr> <td style="width: 100px; height: 26px;"> Email</td> <td style="width: 100px; height: 26px;"> <asp:TextBox ID="TextBox3" runat="server"></asp:TextBox></td> </tr> </table> </div> </div> <div style="text-align: center"> <div> <div> <asp:ScriptManager ID="ScripManager1" runat="server"> </asp:ScriptManager> <div> <table> <tr> <td style="width: 100px"> Group</td> <td style="width: 100px"> <asp:DropDownList ID="ddl1" runat="server" AutoPostBack="true" OnSelectedIndexChanged="ddl1_SelectedIndexChanged"> </asp:DropDownList></td> </tr> </table> </div> <div> <asp:UpdatePanel ID="UpdatePanel1" runat="server"> <ContentTemplate> <table> <tr> <td> Organisation </td> <td> <asp:DropDownList ID="ddl2" runat="server" AutoPostBack="true" OnSelectedIndexChanged="ddl2_SelectedIndexChanged"> </asp:DropDownList> </td> </tr> <tr> <td> org-phone </td> <td> <asp:TextBox ID="TextBox4" runat="server"></asp:TextBox> </td> </tr> <tr> <td> </td> <td> <asp:TextBox ID="TextBox5" runat="server" Width="45px"></asp:TextBox></td> </tr> </table> </ContentTemplate> <Triggers> <asp:AsyncPostBackTrigger ControlID="ddl1" EventName="SelectedIndexChanged" /> </Triggers> </asp:UpdatePanel> <br /> <table> <tr> <td style="width: 100px"> <asp:Button ID="Button1" runat="server" Text="save" OnClick="Button1_Click" /> </td> <td style="width: 100px"> <asp:Button ID="Button3" runat="server" Text="update" /></td> <td style="width: 100px"> <asp:Button ID="Button2" runat="server" Text="cancel" /></td> </tr> </table> </div> </div> </div> </div> </form></body></html>userform.cs page public partial class userform : System.Web.UI.Page{ protected void Page_Load(object sender, EventArgs e) { if (!Page.IsPostBack) { AssignDDLDataSource(); ddl1_SelectedIndexChanged(null, null); } } private void AssignDDLDataSource() { // SqlConnection string sqlQry = "SELECT GRP DISPMEM, ID VALMEM FROM GRUP"; SqlCommand cmd = new SqlCommand(sqlQry, con); ddl1.DataSource = cmd.ExecuteReader(); ddl1.DataTextField = "DISPMEM"; ddl1.DataValueField = "VALMEM"; ddl1.DataBind(); } protected void ddl1_SelectedIndexChanged(object sender, EventArgs e) { AssignSubjectDDL(ddl1.SelectedValue); } private void AssignSubjectDDL(string val) { // SqlConnection string sqlQry = "SELECT ORGANISATION DISPMEM, GID VALMEM FROM ORGANIZATION where OID = '" + val + "'"; SqlCommand cmd1 = new SqlCommand(sqlQry, con); ddl2.DataSource = cmd1.ExecuteReader(); ddl2.DataTextField = "DISPMEM"; ddl2.DataValueField = "VALMEM"; ddl2.DataBind(); ddl2_SelectedIndexChanged(null, null); } protected void ddl2_SelectedIndexChanged(object sender, EventArgs e) { AssignTextBoxValues(ddl1.SelectedValue, ddl2.SelectedValue); } private void AssignTextBoxValues(string teamval, string memval) { // SqlConnection string sqlQry = "Select ID,ORGANISATION,ORGPHONE from ORGANIZATION where OID = '" + teamval + "' AND GID = '" + memval + "'"; SqlCommand cmd = new SqlCommand(sqlQry, con); SqlDataReader rdr = cmd.ExecuteReader(); while (rdr.Read()) { TextBox5.Text = rdr["ID"].ToString().Trim(); TextBox4.Text = rdr["ORGPHONE"].ToString().Trim(); } } protected void Button1_Click(object sender, EventArgs e) { // SqlConnection string sqlQry = "INSERT into USERFORM (title, name, address, email, organizationid) VALUES(@title,@name,@address,@email,@organizationid)"; SqlCommand cmd = new SqlCommand(sqlQry, con); cmd.CommandType = CommandType.Text; cmd.Parameters.AddWithValue("title", DropDownList1.SelectedValue.Trim()); cmd.Parameters.AddWithValue("name", TextBox1.Text); cmd.Parameters.AddWithValue("address", TextBox2.Text); cmd.Parameters.AddWithValue("email", TextBox3.Text); cmd.Parameters.AddWithValue("organizationid", TextBox5.Text); cmd.ExecuteNonQuery(); Response.Redirect("detailsview.aspx"); }} detailsview.aspx <%@ Page Language="C#" AutoEventWireup="true" CodeFile="detailsview.aspx.cs" Inherits="detailsview" %><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml" ><head runat="server"> <title>Untitled Page</title></head><body> <form id="form1" runat="server"> <div> <asp:DetailsView ID="DetailsView1" runat="server" AllowPaging="True" EnablePagingCallbacks="True" Height="50px" HorizontalAlign="Center" Width="125px"> </asp:DetailsView> </div> </form></body></html> detailsview.cspublic partial class detailsview : System.Web.UI.Page{ protected void Page_Load(object sender, EventArgs e) { //SqlConnection string str = "select title,name,address,email,grp,organisation,orgphone from userform as a inner join organization as b on a.organizationid=b.id inner join grup as c on b.oid=c.id"; SqlDataAdapter oda = new SqlDataAdapter(str, conn); conn.Open(); DataSet ods = new DataSet(); oda.Fill(ods, "info"); DetailsView1.DataSource = ods.Tables["info"].DefaultView; DetailsView1.DataBind(); }}My problem :-Now when user fills up the userform and clicks the save button the details are shown in the details view.Now i want to add a Edit hyperlink next to the records in the detailsview so that when i click on it i would be redirected to userform.aspx form with all the textboxes,dropdownlist etc be populated from the related data from the database.(here table userform).Now i want to edit these data in the userform.aspx page and finally when i click the update button at end i want all the data be updated in the database aswell as in details view too.at the same time i want this form to redirect to detailsview.aspx .i hope ur getting it?i m able to do partial part only.Not completely. So can u now help me modify the code accordingly?Thanks in advance.Jack.
Hi, I have 2 tables. MASTER table as M and TRAN table as T. I need to insert 1 row in table M and with the foreign key need to insert multiple rows in the table T. SQL SERVER used is 2005. Application is in .NET 1.x
I am able to insert one record in the table in M but how do i insert the multiple records in table T?
i have a large table in sql server 2005 (it has about 6 columns and 10 million records).
i need to work in a linear way on all the records (i know it sounds dumb but i need to work on all records).
now, obviously when trying to work on this table sql server get stuck for timeout or something like that...
i've noticed that a simple function like "select top 100 * from ExportTable" still works.
is there any way to have sql send me the data when it access it so that i'll still be able to proccess it on the same time, i basically work using dataset so that fixing the timeout wont be helpfull since windows probably wont allow me to load this amount of data into memory.
I'm running a resource-intensive stored procedure, which reads a filewith about 50,000 lines with a BULK INSERT into a temp table, thengoes through it and inserts a record for each line into another table.While this procedure is running, SQL server stops accepting any otherrequests coming from the website.Question:Is there a way to make SQL server "listen", or emulate an "interrupt"to other requests while in the middle of a long intensive process?I really appreciate your replies.Thank you,Oleg.
Our system runs a SQL Server 2012 DB, it has a table (table_a) which has over 10M records. Our system have to receive data file from previous system daily which contains approximate 3M updated or new records for table_a. My job is to update table_a with the new data.
The initial solution is:
1 Create a table (table_b) which structur is as the same as table_a
2 Use BCP to import updated records into table_b
3 Remove outdated data in table_a: delete from table_a inner join table_b on table_a.key_fileds = table_b.key_fields
4 Append updated or new data into table_a: insert into table_a select * from table_b
As the test result, this solution is very inefficient. Step 3 costs several hours, e.g. How can I improve it?
I have a web form with a text field that needs to take in as much as the user decides to type and insert it into an nvarchar(max) field in the database behind. I've tried using the new .write() method in my update statement, but it cuts off the text after a while. Is there a way to insert/update in SQL 2005 this without resorting to Bulk Insert? It bloats the transaction log and turning the logging off requires a call to sp_dboptions (or a straight-up ALTER DATABASE), which I'd like to avoid if I can.
I’m wondering if anyone can shed light on a problem I’ve noticed that's really made for a major thorn in my side. I recently had a Microsoft patch installed on my server, and now for some reason, trying to run INSERT or UPDATE queries against the SQL 2000 database are severely limited. I constantly get the error:
“Error: A severe error occurred on the current command. The results, if any, should be discarded.�
My Event Logs also return the following:
"Invalid buffer received from client."
I think I’ve isolated the problem to be that I can’t add new or modify existing records that try using a field which is of type TEXT, but now can’t be longer than 4,000 characters, else the error fires. This is really weird, as I’ve used the same ASP.NET script to call a stored procedure to INSERT/UPDATE records thousands of times before with 100% success.
I have a feeling this might have something to do with the patch, but has anyone come across this problem specifically, or know for sure which patch(es) cause it? Why all of a sudden would a TEXT field be so limited in capacity?
I am writing a query to return some production data. Basically i need to insert either 1 or 2 rows into a Table variable based on a decision as to does the production part make 1 or 2 items ( The Raw data does not allow for this it comes from a look up in my database)
I can retrieve all the source data i need easily but when i come to insert it into the table variable i need to insert 1 record if its a single part or 2 records if its a twin part. I know could use a cursor but im sure there has to be an easier way !
Below is the code i have at the moment
declare @startdate as datetime declare @enddate as datetime declare @Line as Integer DECLARE @count INT
set @startdate = '2015-01-01' set @enddate = '2015-01-31'
I have a table that I'm inserting a file into and using the Image data type to store the binary object. Now the code below works fine for files around 1.5 MB, but anything larger and it's like the code won't even execute and I get a Page Not found error. I'm in the process of running some traces to find out what's going on in the backend, but I'm assuming there's something amiss with my code. The Image data type should handle files that size with no problem but for some reason it isn't. Does anyone see anything wrong? Thanks Dim iLength As Integer = CType(File1.PostedFile.InputStream.Length, Integer) If iLength = 0 Then Exit Sub 'not a valid file Dim sContentType As String = File1.PostedFile.ContentType Dim sFileName As String, i As Integer Dim bytContent As Byte() ReDim bytContent(iLength) 'byte array, set to file size
'strip the path off the filename i = InStrRev(File1.PostedFile.FileName.Trim, "") If i = 0 Then sFileName = File1.PostedFile.FileName.Trim Else sFileName = Right(File1.PostedFile.FileName.Trim, Len(File1.PostedFile.FileName.Trim) - i) End If conn = New SqlConnection(eco) conn.Open() cmd = New SqlCommand("INSERT INTO ECO_Attachments (ECOID, FromType, DocName,OldRev,NewRev,NtLogin,DisplayName, FileName, FileSize, FileData, ContentType) VALUES (@ECOID, @FromType,@DocName,@OldRev,@NewRev,@NtLogin,@DisplayName, @FileName, @FileSize, @FileData, @ContentType) ") cmd.Connection = conn Try File1.PostedFile.InputStream.Read(bytContent, 0, iLength) With cmd .Parameters.Add("@ECOID", SqlDbType.Int) .Parameters.Add("@FromType", SqlDbType.NVarChar, 50) .Parameters.Add("@DocName", SqlDbType.NVarChar, 250) .Parameters.Add("@OldRev", SqlDbType.NVarChar, 50) .Parameters.Add("@NewRev", SqlDbType.NVarChar, 50) .Parameters.Add("@NTLogin", SqlDbType.NVarChar, 100) .Parameters.Add("@DisplayName", SqlDbType.NVarChar, 200) .Parameters.Add("@FileName", SqlDbType.NVarChar, 255) .Parameters.Add("@FileSize", SqlDbType.Real) .Parameters.Add("@FileData", SqlDbType.Image) .Parameters.Add("@ContentType", SqlDbType.NVarChar, 50) .Parameters("@ECOID").Value = ECOID .Parameters("@FromType").Value = From .Parameters("@DocName").Value = DocName .Parameters("@OldRev").Value = OldRev .Parameters("@NewRev").Value = NewRev .Parameters("@NTLogin").Value = NTLogon .Parameters("@DisplayName").Value = DisplayName .Parameters("@FileName").Value = sFileName .Parameters("@FileSize").Value = iLength .Parameters("@FileData").Value = bytContent .Parameters("@ContentType").Value = sContentType .ExecuteNonQuery() '.ExecuteScalar() End With Catch ex As Exception Response.Write(ex) 'Handle your database error here conn.Close() End Try
SQL 2005 Express - Database Restore size problem Was this post helpful ?
Hi,
I'm trying to restore a SQL Server DB Backup from a SQL Server DB Server on to my Laptop (SQL 2005 Express)
When I execute a restore filelistonly command on the backup file, It seems that the Database included is 1GB, but the Log file is 91 GB in size, which exceeds my diskspace.
I can restore the Data on its own without the log file, but the the Database stays in "restoring" mode. I've tried to switch the restore flag off (update sys.databases set state = 0 where name = 'G001'), but I can`t seem to be able to do it, even if I try to allow updates via:
sp_configure 'allow updates', 1
GO
RECONFIGURE WITH OVERRIDE
GO
Any ideas how I can restore the database without restoring the enormous logfile?
A table in one of my databases is running very slowly. The IO is very high and below is a printout from the SET STATISTICS IO ON command run on a common query used on the table:
I have a clustered unique index and a nonclustered index on the table. I have ran SQL Profiler and opened the trace in Database Tuning Advisor, DTA displays 0% improvement suggestions. I have a number of statistics on the table and index which are all up to date and fragmentation is less than 1%. I've tried a number of variations on indexes to improve performance but to no avail. There is only one query which runs on the table, and the nonclustered index created on the table did significantly improve performance, however the query still runs at around 23 seconds. The query does bring back a large amount of data however i'm sure there is a way to bring down the IO and logical reads to improve performance.
-- =================== Nonclustered Index ===========================
CREATE NONCLUSTERED INDEX [dta_ix_WebProxyLog_Kaction_clientusername_logtime_uri_mimetype_webproxylogid] ON [dbo].[WebProxyLog] ( [Action] ASC ) INCLUDE ( [ClientUserName], [logTime], [uri], [mimetype], [WebProxyLogid]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
-- =================== Query which is called regularly on the table ===========================
SELECT [User] = CASE WHEN LEFT(clientusername,3) = domain' THEN RIGHT(clientusername,LEN(clientusername) - 3) ELSE clientusername END, logtime AS [Date], desthost AS [Site], uri AS [Actual Site] FROM webproxylog WHERE CONVERT(Datetime,CONVERT(VarChar(25),logtime,106),106) BETWEEN '20 apr 2008' AND '14 may 2008' AND(RIGHT(uri,4) NOT IN('.css','.jpg','.gif','.png','.bmp','.vbs')) AND (RIGHT(uri,3) NOT IN('.js')) AND LEFT(mimetype,6) = 'text/h' AND (uri NOT LIKE '%sometext.local%') AND (uri NOT LIKE '%sometext.co.uk%') AND [action] = 9 AND (clientusername IN ('USERNAME')) ORDER BY logtime ASC;
We are running SQL Server 2005 Ent Edition with SP2 on a Windows 2003 Ent. Server SP2 with Intel E6600 Dual core CPU and 4GB of RAM. We have an C# application which perform a large number of calculation that run in a loop. The application first load transactions that needs to be updated and then goes to each one of the rows, query another table get some values and update the transaction.
I have set a limit of 2GB of RAM for SQL server and when I run the application, it performs 5 records update (the process described above) per second. After roughly 10,000 records, the application slows down to about 1 record per second. I have tried to examine the activity monitor however I can't find anything that might indicate what's causing this.
I have read that there are some known issues with Hyper-Threaded CPUs however since my CPU is Dual-core, I do not know if the issue applies to those CPUs too and I have no one to disable one core in the bios.
The only thing that I have noticed is that if I change the Max Degree of Parallelism when the server slows down (I.e. From 0 to 1 and then back to 0), the server speeds up for another 10,000 records update and then slows down. Does anyone has an idea of what's causing it? What does the property change do that make the server speed up again?
If there is no solution for this problem, does anyone know if there is a stored procedure or anything else than can be used programmatically to speed up the server when it slows down? (This is not the optimal solution however I will use it as a workaround)
