Batch Inserts From DataTable
Jun 24, 2005
Please help guys,I have a DataTable filled from the parsing of a csv file by the OleDb text driver.This DataTable could on occassion contain in excess of 2000 rows.I want to be able to batch the inserts to my backend sql table and be able to recorver on errors during the insert.i.e, maybe send the first 500 rows to insert via an insert dynamic text.... really don't know the optimal insert technic to use.but, if I get an error on say the third batch, I want to be able to recorver, and not have to start all over again and continue the inserts from the batch that failed.....Please help... what is the best way to perform the inserts and how can I track these inserts and recorver on errors like power failures or sql server unavailable etc.Please help....
View 3 Replies
ADVERTISEMENT
Jun 16, 2015
I have a table with 370 million rows and 50+ columns. I need to change the data type of a column from character to numeric. Here's what it contains:
40 million with numbers I want to keep, the rest I just want to set to null:
4k with alpha characters
55 million other numbers
275 million empty strings
An alter column statement fails not just on the alpha characters but on the empty strings. So I tried a couple things on a test database to get an idea of the time it would take:
An update statement to clear out the non-numeric data is too slow (~1.5 days, batched 10000 at a time). I think I probably should create a new column anyway though, so I'm going to copy the data to a new table since it would be faster than adding a new column to the original table.
An insert ... select ... takes about 12 hours; adding WITH (TABLOCK) didn't seem to have any effect, and I'm not sure how to batch it. Recovery model is simple.
A select ... into ... only takes about 1 hour, but can't be batched.
Using a 3rd party ETL tool takes about 5 hours, batched.
I wanted to batch it to minimize impact on other queries but primarily the logs. Is there any way to do a fast batched bulk transfer within SSMS?
View 9 Replies
View Related
Nov 14, 2006
I have transactional replication set up between 2 SQL Server 2005 databases on 2 different boxes. Both the log reader and distribution agent run in continuous mode. The distributor is residing on a third SQL Server box. We are having performance issues with replication when there are large batch deletes/inserts happening on the publisher. There is a batch job that runs for about 8-10 hours everyday on the publisher and deletes/inserts thousands of records as part of transactions. The amount of time to replicate all this data on the subscriber is around 13-15 hours which is not acceptable to our user community. While monitoring I found that the distribution database (MSRepl_commands table) at times has millions of records in it which would explain why the latency is so high. Add to it the fact that there are large transactions occuring on the publisher.
I was wondering if anyone has faced similar problem before. Are there any conifguration changes I can make to the replication infrastructure to reduce latency?
Would appreciate your help.
Thanks.
View 4 Replies
View Related
Sep 3, 2007
Filling a DataTable from SqlQuery : If SqlQuery returns some null values problem ocurrs with DataTable.
Is it possible using DataTable with some null values in it?
Thanks
View 2 Replies
View Related
Aug 7, 2007
HELP,
I need to take a variable from a tabel in SQL Server pass to a Batch file and execute the batch file. Right now I can exec the batch file with XP_CMDSHELL but how can I pass the variable to the batch file and loop through all the variables.
Please help
Phil
View 4 Replies
View Related
Dec 5, 2006
I am using the following batch file to execute a script that creates a db and all its objects in the local sql express:
sqlcmd -S (local)SQLExpress -i C:CreateDB.sql
This works fine, but I'm wondering if there's an easy way to put the script in the batch file, so users don't have to worry about putting the script in the C drive. I tried getting rid of the i parameter and pasting the script from the sql file into the batch file, but it didn't work.
Thanks,
Dave
View 1 Replies
View Related
Dec 11, 2006
Hi I am trying to use the results of a datatable as an input to a ne SQL statement.
The datatable is created and populted by the source below after previously creating MyLookupCommand
MyLookupReader = MyLookupCommand.ExecuteReader(CommandBehavior.CloseConnection)
Dim Mydatatable = New DataTable()
Mydatatable.Load(MyLookupReader)
I now create a new SQL string allong the lines of
SELECT tableA.field1 FROM TableA INNERJOIN Mydatatable ON tableA.field2 = Mydatatable.Filed9
However I get the following error
Invalid object name 'Mydatatable'.
Any suggestions on how to resolve this.
possibley of interest is that I would have like to have done this in 1 SQL but the results in Mydatatable are from a SELECT DISTINCT and Field1 is a text filed that can't be used in a DISTINCT statement, hence using two statements.
Many thanks in advance
View 6 Replies
View Related
Jun 28, 2006
Hi,
I'm wondering if this is possible. I want to use a datatable as a parameter in my SQL query. The error I am getting is "No mapping exists from object type System.Data.DataColumn to a known managed provider native type." Any help is appreciated.
Here is the code.
Private Function GetAvailableTutors(ByVal qualifiedTutorsDataSet As DataTable) As DataTable
Dim TutorAvailabilityDataTable As DataTable = New DataTable()
Using myConnection As New SqlConnection(ConfigurationManager.ConnectionStrings("ASC_Schedule").ConnectionString)
Dim myCommand As SqlCommand = _
New SqlCommand("SELECT TutorID, TimeIndex FROM T_Hours WHERE SessionID > 0 AND TutorID IN (@TutorList) AND DateDropped > GetDATE() ORDER By T_QualifiedTutors.TutorID", myConnection)
myCommand.Parameters.AddWithValue("@TutorList", qualifiedTutorsDataSet.Columns.Item("TutorID"))
Dim myDataAdapter3 As SqlDataAdapter = New SqlDataAdapter()
myDataAdapter3.SelectCommand = myCommand
myDataAdapter3.Fill(TutorAvailabilityDataTable)
Return TutorAvailabilityDataTable
End Using
End Function
View 2 Replies
View Related
Aug 2, 2006
hi all
the usual way to bid a gridview is to data soursce
is there a way to do the folowing , creat a data table from the gridview shown valus " currunt page "
thanks
View 1 Replies
View Related
Dec 27, 2006
Hi,
I am experimenting to make a datatable in C# code in a page. This table should be a disconected table with Only valid for the present session.
I try the following code:
public partial class Default2 : System.Web.UI.Page
{
DataTable DT = new DataTable("TEST");
protected void Page_Load(object sender, EventArgs e)
{
if (!IsPostBack)
{
DataColumn Col1 = new DataColumn("Col1");
Col1.DataType = typeof(Int32);
Col1.AllowDBNull = false;
DT.Columns.Add(Col1);
DataColumn Col2 = new DataColumn("Col2");
Col2.DataType = typeof(string);
Col2.AllowDBNull = true;
DT.Columns.Add(Col2);
DataColumn Col3 = new DataColumn("Col3");
Col3.DataType = typeof(DateTime);
Col3.AllowDBNull = true;
DT.Columns.Add(Col3);
GridView1.DataSource = DT;
}
}
protected void Button1_Click(object sender, EventArgs e)
{
for (int i = 1; i < 20; i++)
{
DataRow MyRow = DT.NewRow();
MyRow["Col1"] = i;
DT.Rows.Add(MyRow);
}
}
}
For one reason or the other. if I click the button I get the message that Col1 dus not make part of the table TEST. It turns out that there are no columns added to the table. altroug the code in the page load part has been run. I suppose I have to do something with the session state to make my DataTable persistent, but I have no idea what. Can somebody help me out?
Thanks!
Rob
View 6 Replies
View Related
Jun 1, 2007
Is the merge method, what will work in this case ? I have two datatables with the exact same structure. How can I append the rows from table 2 onto the bottom of table 1 ? Is looping through the rows collection the only way ?
View 2 Replies
View Related
Sep 11, 2007
Hi,Is it possible to store data that is retrieved by SqlDataSource in a... let's say... DataTable? I mean, If I dragged and dropped a GridView and an SqlDataSource and then I set up those controls in page designer in Visual Studio, is it possible at some time later to retrieve the data retrieved by the SqlDataSource and then store the data in a DataTable?Best regards,Haris
View 7 Replies
View Related
Dec 6, 2007
Is there a built in way of converting a LINQ qurey result into a DataTable or DataView ?
View 4 Replies
View Related
Dec 26, 2007
hai This is rameshi had a small doubt in this public DataTable GetStatelist() { DataTable returnValue = null; SqlCommand selcommand = new SqlCommand(); selcommand.CommandType = CommandType.Text; selcommand.CommandText = "select * from USER_MASTER"; selcommand.Connection = _connectionobject; try { _connectionobject.Open(); DataSet ds = new DataSet(); SqlDataAdapter ada = new SqlDataAdapter(selcommand); ada.Fill(ds); _connectionobject.Close(); return returnValue; ; } catch (Exception exception) { if (_connectionobject.State == ConnectionState.Open) { _connectionobject.Close(); } throw new Exception(exception.Message); } if (_connectionobject.State == ConnectionState.Open) { _connectionobject.Close(); } return returnValue; ; } after compiling its show the return value is nullwhat is error in my codinghow i can return the value to the corresponding function if (!IsPostBack) { GridView1.DataSource=database.GetStatelist(); GridView1.DataBind(); } i am waiting for the result
View 4 Replies
View Related
Feb 12, 2008
I am fetching records from a table and putting in a datatable using the "sqlda.fill(datatable)"
when i see the datable in "data visualizer" i find no rows.
It is confused eventhough it does not throw any error
View 3 Replies
View Related
May 2, 2008
Do I have to save the datatable into sqldatasource's SELECT results? if yes , how?
View 4 Replies
View Related
Mar 20, 2006
I have a table containing prices. This table will be queried very often to provide quotes for clients.So to ease the burden on the server I want to cache the table and then just query the cached version.However it seems that I can only cache the table as a datatable. This means I have to query the datatable to get the prices for each quote.I'm not sure how to query a datatable. Is ther syntax similar to querying a SQL table?In fact is this best way to go about things?Any help would be appreciated.G
View 4 Replies
View Related
Aug 7, 2007
Hi...
In my appllication, i am having a DataTable... For that DataTable i have to write a query for a expression... This epression is more of mathematical expression type. Here is the sample expression-
(column1 < 1 && (column2 = 2 || column3 > 5) || column4 != 3)
How i can write a query for such a equation...? Is it possible to use logical & mathematical operator in the equation & to write a query for this type of equation... Is there is any good article on this...
Thanks in adavnce...
IamHuM
View 3 Replies
View Related
Mar 4, 2007
I have the following: Dim ResourceAdapter As New ResourcesTableAdapters.ResourcesTableAdapter Dim dF As Data.DataTable = ResourceAdapter.GetDataByDistinct Dim numRows = dF.Rows.Count Dim dS As Data.DataTable = ResourceAdapter.CountQuery Dim sumRows = dS.Rows.Count DataList1.DataSource = dF DataList1.DataBind() LblCount.Text = numRows LblSum.Text = sumRows numRows is the number of records in a particular category. CountQuery is a method that returns the total number of records in all the categories. I don't know how to get the result of that query, the code line in bold isn't right. How do I get this number?Diane
View 5 Replies
View Related
Jul 23, 2007
I'm trying this code but nothing is being displayedSqlConnection conn = new SqlConnection(ConfigurationManager.AppSettings["STRING_CON"]);
SqlDataAdapter da = new SqlDataAdapter("my_sproc", conn);DataTable dt = new DataTable();
DataRow dr;
//Adding the columns to the datatabledt.Columns.Add("CategoryIdIn");
dt.Columns.Add("CategoryNameVc");foreach (DataGridItem item in gd_freq.Items)
{
dr = dt.NewRow();
dr[0] = item.Cells[0].Text;
dr[1] = item.Cells[1].Text;
dt.Rows.Add(dr);
}//ForEach
da.Fill(dt);
gd_freq.DataSource = dt;
gd_freq.DataBind();
View 2 Replies
View Related
Jul 24, 2007
Can you use a DataTable to update an SQL Table and can it be done in a batch UPDATE as opposed to incrementing through every row using a stored procedure similar to UPDATE SQLTable SET Position = @Position WHERE ID = @ID?
View 3 Replies
View Related
Nov 11, 2007
I'm trying to use the SQL Bulk copy class to bulk import from a text file.I'm getting the following error: Line 24: bulkCopy.WriteToServer(CreateDataTableFromFile()); System.Data.SqlClient.SqlException: An error has occurred while establishing a
connection to the server. When connecting to SQL Server 2005, this failure may
be caused by the fact that under the default settings SQL Server does not allow
remote connections. (provider: Named Pipes Provider, error: 40 - Could not open
a connection to SQL Server) I've even tried to allow remote connections thro pipes and restarted the database engine but to no avail. Any inputs/suggestions?
View 3 Replies
View Related
Jan 8, 2008
I am quite new to ASP.net 2.0. I have had plenty of experience using ADO.net in standard windows applications.
In my app I am opening a connection to an SQL database and I am creating a DataTable without a DataSet:
Shared m_cnADONetConnection As New System.Data.SqlClient.SqlConnection
Shared m_daDataAdapter As System.Data.SqlClient.SqlDataAdapter
Shared m_rowPosition As Integer
Shared m_dtContacts As New System.Data.DataTable
I am initializing everything and filling my DataTable when the Page first Loads if it isnt a postback.Protected Sub Form1_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Form1.Load
If Not Page.IsPostBack Then
m_rowPosition = 0
m_cnADONetConnection.ConnectionString = "Data Source=.SQLEXPRESS;AttachDbFilename=C:First ASP DatabaseApp_DataMyFirstDatabase.mdf;Integrated Security=True;Connect Timeout=30;User Instance=True"
m_cnADONetConnection.Open()m_daDataAdapter = New System.Data.SqlClient.SqlDataAdapter("Select * From Books", m_cnADONetConnection)m_cbRefillCommand = New System.Data.SqlClient.SqlCommand
m_daDataAdapter.Fill(m_dtContacts)
Me.ShowCurrentRecord()
End IfEnd Sub
The Me.ShowCurrentRecord Sub assigns the values of the current record(row) in the DataTable via (m_rowPosition) to TextBox controls on the form:
I also have record navigation buttons on my form: << < > >> Moving me from record to record (row to row) by incrementing or decrementing m_rowPosition
All is good! I am able navigate the DataTable and the textboxes change their text properties accordingly from record to record.
The only other control on my form is a button which I'm coding its click event to save changes that I made to the current row (record) by changing the values in the textboxes then clicking save.
Here is my code:Protected Sub ButtonSave_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles ButtonSave.Click
If m_dtContacts.Rows.Count <> 0 Then
m_dtContacts.Rows(m_rowPosition).BeginEdit()m_dtContacts.Rows(m_rowPosition)("Title") = TBTitle.Text
m_dtContacts.Rows(m_rowPosition)("Author") = TBAuthor.Textm_dtContacts.Rows(m_rowPosition)("YearPublished") = TBYearPublished.Text
m_dtContacts.Rows(m_rowPosition)("Price") = TBPrice.Textm_dtContacts.Rows(m_rowPosition)("LastReadOn") = TBLastReadOn.Textm_dtContacts.Rows(m_rowPosition)("PageCount") = TBPageCount.Text
m_dtContacts.Rows(m_rowPosition).AcceptChanges()
m_dtContacts.Rows(m_rowPosition).EndEdit()
m_dtContacts.AcceptChanges()
m_daDataAdapter.Update(m_dtContacts)
End Sub
After I click save I can navigate through my records and back to the one I just changed and updated and all is well. The changes were made in the table.
However, when I close the page and exit out of Visual Web Developer and reopen the database: THE CHANGES WERENT UPDATED!!!
This worked all the time in VB2005.net when developing a standard windows app.
Can I use the same approach I was using in my code above or am I missing something.
I have read and searched all over and what I'm thinking is that my UpdateCommand, InsertCommand, DeleteCommand, SelectCommand are empty.
Do I have to do it this way?
View 1 Replies
View Related
Jan 10, 2008
Hi, I have the code below. I need to skip the first row in the datatable as it has the headers. This works now, but my gridview gets the header row inserted as a record.Private Shared Sub InsertData(ByVal sourceTable As System.Data.DataTable, ByVal destConnection As SqlConnection) ' old method: Lots of INSERT statements ' first, create the insert command that we will call over and over: destConnection.Open() Using ins As New SqlCommand("INSERT INTO [tblAppointmentDisposition] ([contactdate], [dnbnumber], [prospectname], [businessofficer], [phonemeeting], [followupcalldate2], [phonemeetingappt], [followupcalldate3], [appointmentdate], [appointmentlocation], [appointmentkept], [applicationgenerated], [applicationgenerated2], [applicationgenerated3], [comments], [newaccount], [futureopportunity]) VALUES (@contactdate, @dnbnumber, @prospectname, @businessofficer, @phonemeeting, @followupcalldate2, @phonemeetingappt, @followupcalldate3, @appointmentdate, @appointmentlocation, @appointmentkept, @applicationgenerated, @applicationgenerated2, @applicationgenerated3, @comments, @newaccount, @futureopportunity)", destConnection) ins.CommandType = CommandType.Text ins.Parameters.Add("@contactdate", SqlDbType.Text) ins.Parameters.Add("@dnbnumber", SqlDbType.Text) ins.Parameters.Add("@prospectname", SqlDbType.Text) ins.Parameters.Add("@businessofficer", SqlDbType.NVarChar) ins.Parameters.Add("@phonemeeting", SqlDbType.Text) ins.Parameters.Add("@followupcalldate2", SqlDbType.Text) ins.Parameters.Add("@phonemeetingappt", SqlDbType.Text) ins.Parameters.Add("@followupcalldate3", SqlDbType.Text) ins.Parameters.Add("@appointmentdate", SqlDbType.Text) ins.Parameters.Add("@appointmentlocation", SqlDbType.Text) ins.Parameters.Add("@appointmentkept", SqlDbType.Text) ins.Parameters.Add("@applicationgenerated", SqlDbType.Text) ins.Parameters.Add("@applicationgenerated2", SqlDbType.Text) ins.Parameters.Add("@applicationgenerated3", SqlDbType.Text) ins.Parameters.Add("@comments", SqlDbType.Text) ins.Parameters.Add("@newaccount", SqlDbType.Text) ins.Parameters.Add("@futureopportunity", SqlDbType.Text) ' and now, do the work: For Each r As DataRow In sourceTable.Rows For i As Integer = 0 To 16 ins.Parameters(i).Value = r(i) Next ins.ExecuteNonQuery() 'If System.Threading.Interlocked.Increment(rowscopied) Mod 10000 = 0 Then 'Console.WriteLine("-- copied {0} rows.", rowscopied) 'End If Next End Using destConnection.Close() End Sub
View 3 Replies
View Related
Jan 31, 2008
I'm reading XML data into a DataTable that is populated into a datagrid. From here I need to update database and wanted to know how to Fill a DATA ADAPTER with a DATA TABLE?
I'm familiar with updating using a SQL command, but in this case, i have the dataTable already created from XML.
Please help.
Thanks,
View 3 Replies
View Related
Feb 8, 2008
With C#, ADO.net, and SQL Server, is there an easy way to run some sort of query that will return only the structure of a table but not have any data in it? Put another way, I want to create a DataTable that is empty but has exactly the same columns as its associated Table in SQL Server.Robert W.
View 3 Replies
View Related
May 10, 2008
Hello.
I have 2 diffrent tables in the database, is it possible to select both of those back to c# on the same time?
I DONT want to combine the tables, just select them into 2 diffrent datatables in one select statement?
At the moment, i have something like:
function XXOpenDBCon();mCom.Parameters.Clear();mCom.CommandText = "Stuff_GetMovieCategories";myAdap = new SqlDataAdapter();myTable = new DataTable();myAdap.SelectCommand = mCom;myAdap.Fill(myTable);CloseDBCon();return myTable;
Can i ask the database to perhaos retrieve Stuff_GetMovieCategories and on the same time retrieve Stuff_MostViewedMovies?
So the SP does 2 select statements and returns them, i get in the function above 2 datatables with the information, one datatable with Categories and one table with MostViewedMovies.
Is this possible?
View 7 Replies
View Related
Apr 21, 2006
hello, i'm using asp.net 2 with VB, i have a table in my db named 'exams' and i have 6 columns in this table one for question number and one for the question and 3 columns for 3 answers and a last column for the right answer, now in my page i want to show 7 questions and each question has 3 answers that i can choose between them i want the question appear in datatable and the answers in radiolist i have a little code for that but it shows an error it says "no row at position 1" so hope u guys can help : Dim con As New SqlConnection("Data Source=.SQLEXPRESS;AttachDbFilename=|DataDirectory|aspnetdb.mdf;Integrated Security=True;User Instance=True") Dim a As New ArrayList(8) Dim i As New Integer i = 0 Dim d As New DataSet() Dim cmd As New SqlDataAdapter("select top(7) question from exams", con) cmd.Fill(d, "exams") a.Add(d.Tables.Item(0, 1)) i = i + 1 Dim ta As New DataTable ta.Rows(i).Item(0).text = d.Tables(0).Rows(0)(i)and by the way this code is for the questions i still dont have the code for the answers that will appear in a radiolist hope u can help .thanks
View 3 Replies
View Related
Jun 16, 2008
I'm populating a gridview with the following code and all works fine until i try to use an input parameter(see bold below)
If I remove the bold text to get all news all works well.
Any help much appreciated.
Cheers.
private DataTable getTable()
{
SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["blabla"].ToString());
SqlDataAdapter adap = new SqlDataAdapter("SELECT newsItemID, LEFT(newsItemTitle, 25) + '...' FROM newsItems WHERE active='True' AND newsCat=@newsCat ORDER BY dateTimeStamp DESC", conn);
SqlParameter newsCat = new SqlParameter("@newsCat", SqlDbType.VarChar, 50);
newsCat.Direction = ParameterDirection.Input;
newsCat.Value = "Staff";
DataTable dt = new DataTable();
adap.Fill(dt);
return dt;
}
View 6 Replies
View Related
Oct 12, 2006
helloo
Can I use "like" in datatable.select method??
meaning:
Dim exp As String = "c.cDesc like " & txtSearch.Text & " + N'%' "
Dim rows() As DataRow = dtCenters.Select(exp)
knowing that txtSearch.text has unicode characters
View 1 Replies
View Related
Apr 18, 2007
Hi!
I'm having a lot of difficulty in creating a user defined function in vs2005.
why I'm doing it in c# instead of vb or tsql is that I know C# better than the others (that and tsql won't let you make temp tables in functions
But as far as this goes, I can't seem to get this working.
I' trying to execute a simple select statement and get a datatable variable before I run it through the creative algorithm I have planned.
But I can't seem to get the data into a table.
I get this exception when I execute the function on the server:
System.InvalidOperationException: Data access is not allowed in this context. Either the context is a function or method not marked with DataAccessKind.Read or SystemDataAccessKind.Read, is a callback to obtain data from FillRow method of a Table Valued Function, or is a UDT validation method.
This is the code I use to get the data:
using(SqlConnection sqlcnct = new SqlConnection("Context Connection=true"))
{ SqlCommand sqlcmd = new SqlCommand(sqlstrng, sqlcnct);
sqlcnct.Open();
SqlDataReader sqldr = sqlcmd.ExecuteReader();
while (sqldr.Read())
{
//inserts into datatable here (I don't yet have code here because i've been trying to figure out how to get it to read first. )
}
}
But unfortunately SQL server 2005 doesn't like that.
Can anyone point me in the right direction or shoot down my hopes right here?
I'm trying to use a function to avoid having to run a sproc on every employee on the employee table and storing it in an actual field /table every time I need to run this function.
View 4 Replies
View Related
Jan 8, 2007
hi,
I have my database stored in the sqlserver 2005. Using the table name i am retrieving the table and it is displayed to the user in the form of datagridview.I am allowing the user to modify the contents of the table, including the headers. Is it possible for me to update the table straightway rather than giving a sql update command for each and every row of the table .
Pls reply asap....
-Sweety
View 3 Replies
View Related
Mar 14, 2008
hi,
can anyone tell me how to retreive a row from a table using rowid (For example i want to retreive fifth row from a table)
View 3 Replies
View Related
