Bulk Inserting Into Sql Server

Apr 26, 2004

Hello Everyone,





I am currently struggling with a problem bulk inserting data into sql server.





The application I am writing is multi-threaded and downloads about 2000 records every 2 seconds on x amount of threads (depending on bandwidth).





What I would like to do is find a 'friendly' way to insert this data into sql server without hammering the cpu.. I have tried the following with little success.





1, Using a single insert and thread.sleep(x * 20) to allow for massive data input, altough this made the application more stable and lowered cpu usage to very little the data takes about 60 times longer to download and process into the database.





2, Using a SqlDataAdapter and DataSet and updating the database via the .Update method of the data adapter.. Simply this was awful and took forever to process the data into the database.. (Took about 30 seconds to process 2000 records and Command Timeout was high).





3, Using OpenXML in SqlServer and parsing the data as an XML string (nText), although this method is fast its still very CPU intensive. I have to set the command timeouts very high to allow for this approach (because of the multi-threaded nature of the app).





Does anyone have any idea's on a cpu friendly approach to this problem ??





Thanks in advance..





Gary.

View 4 Replies


ADVERTISEMENT

SQL Server 2008 :: BULK INSERT Inserting No Rows

Aug 7, 2015

I am trying to BULK INSERT csv files using a stored procedure in SQL SERVER 2008R2 SP3. Although the files contain several thousand lines and BULK INSERT returns no errors, no data is actually imported into the table. Every field in the table is a NVARCHAR(50) datatype.

Here is the code for the operation (only the parameters for the insert itself):

set @open = 'bulk insert [DWHStaging].[dbo].[Abverkaufsquote] from '''
set @path = 'G:DataStagingDWHStagingSourceAbverkaufsquote'
set @params = ''' with (firstrow = 2
, datafiletype = ''widechar''
, fieldterminator = '';''
, rowterminator = ''
''
, codepage = ''1252''
, keepnulls);'

The csv file originates from a DB2 database. Using exactly the same code base I can import several other types of CSV files without problem.

The files are stored on the local server with as UCS2 Little Endian and one difference is that the files that do not import do not include a BOM. The other difference is that the failed files are non-UNICODE files.

View 4 Replies View Related

Triggers When Bulk Inserting

May 21, 2008

Hi,

hope this is the right forum for this...

I'm using asp.net and the SqlBulkCopy command to insert a few lines of data into a table. I have the SqlBulkCopyOption.FireTriggers set, and a simple trigger on the table to take the inserted data and update a couple of other tables. Works fine if I only insert one record, but as soon as there are multiple records it fails.

So I presume that the trigger isn't firing on every line, but only once when the bulk insert is done?

Am I missing a trick? I can post some code if it will help - I'll have to get a screengrab of the error too.

cheers

View 2 Replies View Related

Bulk Inserting A .dat File

Aug 19, 2007

The following produces no errors, but does not place any rows in the table:

USE MoneyManager
GO
BULK INSERT DailyPrices
FROM 'C:DownloadsSecuritiess20070105.dat'
WITH
(
firstrow = 2,
formatfile = 'C:DownloadsDailyPrices.fmt',
errorfile = 'c:downloadsprice_error_file.txt'
)

select *
from DailyPrices

View 5 Replies View Related

Bulk Inserting Uniqueidentifier Column

Oct 10, 2007

Hi at all,I'm trying to bulk insert a uniqueidentifier column from unicode file.In my file I have guid generated from c# application and they areformatted in this way (separated by "|") :guid | field1 | field2fc0c0c42-438e-4897-96db-8b0489e873ef|field1|field2In my destination table I have three column:id (uniqueidentifier)field1 (nvarchar)field2 (nvarchar)I use in bulk insert a format file like this :9.031SQLNCHAR00"|"1IDLatin1_General_CI_AS2SQLNCHAR00"|"2Field1Latin1_General_CI_AS3SQLNCHAR00"|"3Field2Latin1_General_CI_ASand I use this scriptBULK INSERT [dbo].[KWTA2] FROM 'd:WTA2.txt'WITH (FORMATFILE = 'd:wta2Format.FMT')It doesn't work, it prints outMsg 8152, Level 16, State 13, Line 2String or binary data would be truncated.I've also tried to specify in FMT file SQLUNIQUEID instead of SQLNCHARand it works perfectly but it imports another data. For example theguid fc0c0c42-438e-4897-96db-8b0489e873ef became00350031-0039-0033-3100-300030003000Please can you help me?Why sql converts alphanumerical GUID into only numbers ID?How can I bulk insert GUID? (I didn't find anything googling around : )Thanks!Bob

View 2 Replies View Related

Inserting Manually Vs. Bulk Insert

Sep 12, 2006

Hi Andrea,
I have made a table which contain data inserted manually and also data that was inserted by using bulk insertion. I have no problems using the table with Grid View. But when I try to use a query like the following:

SELECT *
FROM dbo.last
WHERE VMake = 'Honda'
AND VType = 'sedan'
AND VColor = 'Red';

I would only get the data that were inserted manually.
When I use the same query to filter data from a table that the data was inserted by using bulk insert, I get column names but no data.

Any help please.

Juvan

View 1 Replies View Related

Bulk Inserting The Datetime Failing

Oct 25, 2007

Hi,
I am using SQL 2k5

I have a table with the follwing schema

Table BulkTest
(

ID nvarchar(100),
SubDate Datetime
)

And i have data in a file as follows ( column delimeter is | and row deleimeter is new line character ( windows ) )

4000000|10/25/2007 6:07:32 AM
4000001|10/25/2007 6:07:32 AM
4000002|10/25/2007 6:07:32 AM
4000003|10/25/2007 6:07:32 AM
4000004|10/25/2007 6:07:32 AM


BULK INSERT dbo.BulkTest FROM 'c:\insert.dat' WITH (FIELDTERMINATOR = '|',ROWTERMINATOR='
')
When i execute this statement its failing with the follwing message.

Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 2 (SubDate ).


Can any one please let me know whats wrong in this and what should be done to make it work
Quick response will be appreciated
Thanks much
~Mohan




View 5 Replies View Related

Bulk Inserting Into Table With Computed Columns

Jul 20, 2005

Using SS2K, I'm getting the following error while bulk inserting:Column 'warranty_expiration_date' cannot be modified because it is acomputed column.Here is my bulk insert statement:BULK INSERT dbo.TestDataFROM 'TestData.dat'WITH (CHECK_CONSTRAINTS,FIELDTERMINATOR='|',MAXERRORS = 1,FORMATFILE='TestData.fmt')The computed column is not referenced in the format file and the data filedoes not contain the computed data.Thanks

View 2 Replies View Related

Bulk Insert - Inserting Only One Column From Data File

Sep 29, 2007



Hi,

I have a data file in the folloing format

SubjectId1|class1
SubjectId2|class2
SubjectId3|class3


I just wanted to insert only SubjectIds into my table 'Subjects' which has the follwing schama ignoring the classes
The row delimeter is "
" and the column delimeter is ' | '

Table Subjects
{

ID (Autoincrement)
SubjectId varchar(20)
}

Can any one provide the format file for doing this or suggest anyway to do this?
Please do note that the file may contain millions of records

Thank u
~mohan

View 5 Replies View Related

Skip Field Terminator While Inserting Data To A Table-Bulk Insert

Oct 10, 2007

Hi,
I have a data file which consists of data as below,
4
PPU_FFA7485E0D||
T_GLR_DET_11||

While iam inserting into table using bulk insert, this pipe(||) is also getting inserted into the table,
here is my query iam using to insert the data using bulk insert.

BULK INSERT TABLE_NAME FROM FILE_PATH
WITH (FIELDTERMINATOR = ''||'''+',KEEPNULLS,FIRSTROW=2,ROWTERMINATOR = '''')




Can any one help on this.

Thanks,
-Badri

View 7 Replies View Related

Bulk Insert - Bulk Load Data Conversion Error

Jan 17, 2008

Im having some issues with bulk insert.

This is the table:

CREATE TABLE [dbo].[tmp_GA_status](

[GA_recno] [int] NOT NULL,

[GA_desc] [varchar](40) NULL

)


This is the file (unicode):
1|"test1"
2|"test2"
3|"test3"
4|"test4"
5|"test5"
6|"test6"
7|"test7"
8|"test8"


and this is the sql:

bulk insert tmp_GA_status from 'C: empTextDumpGA_status.dta'

with (CODEPAGE='RAW', FIELDTERMINATOR='|', ROWTERMINATOR='
', DATAFILETYPE='widechar')



so yeah, pretty simple. But whatever I do I get this;

Msg 4864, Level 16, State 1, Line 1

Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 2 (GA_desc).



So what am I doing wrong ?

View 13 Replies View Related

I Don't Suppose BULK UPDATE Exists?... Like BULK INSERT?

Sep 27, 2007

I have to update a field within a table of 60 records or so. Each record has a different field value. it's type varchar. i was given an excel file with the field values and was thinking of a bulk update like bulk insert, but i don't recall that it's possible that way.

Is the only way to create a table, bulk insert, then merge the two tables together with UPDATE?

Just wanted to see if there was an easier way to do it, otherwise i'll take the latter route. Thanks!

View 1 Replies View Related

Cannot Fetch A Row From OLE DB Provider BULK With Bulk Insert Task

Nov 23, 2005

Hi, folks:

View 18 Replies View Related

Pros: How To Bulk Delete And Bulk Insert?

Oct 11, 2000

I have a table containing 8 million records.
I need to replace 2 million of these records with
a scaled down query that goes something like:
SELECT 1, ShareholderID, Assets1
FROM MyTable (Yields appx. 200,000 recods)
SELECT 2, ShareholderID, Assets2
FROM MyTable (Yields appx. 200,000 recods)
.
.
.
SELECT 10, ShareholderID, Assets1 + Assest2 + Assets3 + ... + Assets9
FROM MyTable (Yields appx. 200,000 recods)

Updates and cursors just seem to be too slow.

So far I have done the following, but was wondering if anyone could think of a better way.
SELECT 6 million records that don't need to be deleted into a #TempTable
Use statements above to select into same #TempTable
DROP and recreate Original Table
SELECT 6 + 2 million records INTO original table.

This seems rather convoluted. Is there a better approach? Would it be worth while to dump data to a file and use bcp / Bulk Insert


Any comments are appreciated,

-Marc

View 3 Replies View Related

How Read CSV File In Other Server Using Bulk Insert Command In Sql Server In Different Server...

Mar 24, 2008

Hi All,
 Please some one help me...
I have to insert a csv into one table in sql server. But the problem is the file is in one server and SQL SERVER 2005 is in other server..
 how do i insert the file....
please help me.....
 

View 1 Replies View Related

BULK INSERT ERROR Using Format File - Bulk Load Data Conversion Error

Jun 29, 2015

I'm trying to use Bulk insert for the first time and getting the following error. I think it might have something to do with my Format File and from the error msg there's a conversion error for the first column. In my database the Field is nvarchar(6) so my best guess is to use SQLNChar for the first column. I've checked the end of each line is CR LF therefore the is correct for line 7 right?

Msg 4863, Level 16, State 1, Line 1
Bulk load data conversion error (truncation) for row 1, column 1 (ASXCode).
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".

BULK
INSERTtbl_ASX_Data_temp
FROM
'M:DataASXImportTest.txt'
WITH
(FORMATFILE='M:DataASXSQLFormatImport.Fmt')

[code]...

View 5 Replies View Related

Questions About Bulk Copy Insert Using 'Memory Based Bulk Copy Operations'

Feb 1, 2007

Hi~,

Before implementing memory based bulk copy insert with IRowsetFastLoad interface of SQL Server 2005 OLE DB provider, I want to know some considerations.

- performance : compared with T-SQL's "BULK INSERT ..." and bcp utility

- SQL Server's resource usage : when running memory based bulk copy, server resource's influence

- server side action(behavior) : when server is busy, delayed-update means IRowsetFastLoad::Commit(true) method can insert right after?

- row-count : The rowcount limitation can be inserted by IRowsetFastLoad::InsertRow() method before IRowsetFastLoad::Commit

- any other guide lines

View 1 Replies View Related

How Do You Use An Identity Column When Doing A Bulk Insert Using The Bulk Insert Task Editor

Apr 18, 2008



Hello,

I'm just learning SSIS and I've hit my first bump. I am doing a bulk import from a tab delimited text file to an empty sql table that has a Idendity column defined. How do I tell the bulk insert task to skip that column when inserting from the text file. If I remove the identity column it imports the data fine, but I want to create the indentity column in the table too.

Thanks.

View 8 Replies View Related

Bulk Insert Into SQL Server

Mar 7, 2008

Hello,
I need help regarding bulk insert into sql server. basically i m making a customized webapplication to save all the file and directories saving it in an arraylist and then want to save the contents of that array list into sql server table. now my questions are:
1) should i use arraylist for this purpose(i mean is it fast as my data is too big)
2) how to save all the data from arraylist into sql server table with bulkinsert or smthing like that. i dont want to use insert query with looping through the arraylist, as it will take lot of time and resources.
hope you get my query.

View 4 Replies View Related

SQL Server Bulk Uploads

Sep 13, 2007

I have tried to upload about 1 million records from a foxpro table to SQL Server. It only loaded about 700,000 records.
I found out that the drive where the SQL Server transaction log file were located had increased to 15 gigs and I run out of disk space.
I guess that is the reason it could not load the rest of the records.



What should be the option setting in my SQL Server database when doing "bulk uploads"?.
Should I delete table indexes when uploading?.
Should I set the Recovery Model to simple?.
Thanks

View 1 Replies View Related

Inserting Image Data Type From SQL Server CE To SQL Server 2000.

Oct 23, 2006

Hi All,

I am manually replicating parts of a SQL Server CE database (running windows mobile 5.0) to a centralized SQL Server 2000 database.

My program is throwing an exception whenever I try to insert an image data type into the 2000 server from the PDA. I am using parameterized queries.

Error is as follows:
[error]
System.Data.SqlClient.SqlConnection.OnError()
at
System.Data.SqlClient.SqlInternalConnection.OnError()
at
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at
System.Data.SqlClient.TdsParser.Run()
at
System.Data.SqlClient.ExecuteReader()
at
System.Data.SqlClient.ExecuteNonQuery()
at
PDASync.Database.ExecuteIDRemote()
[/error]

The code for my ExecuteIDRemote method works fine for other queries. It also works if I remove the image column from the offending query.

Does anyone have any suggestions? Thanks.

View 1 Replies View Related

SQL Server Express Bulk Insert

Oct 25, 2006

does anyone know how to do a bulk insert in SQL Server Express thanks Marcel

View 2 Replies View Related

How To Insert Data Into Sql Server In Bulk Using ADO.net

Dec 27, 2007

Hi!
 I'm building a web application. I need to read data from a text or excel file and process the data and then store the result records into database. The record number is big. I can store the data record into database (SQL Server 2005) one at a time. I think it's slow. Is there any way to insert the data in bulk.
 
Thanks!
ccy

View 4 Replies View Related

Sql Server Bulk Copy Program

Aug 1, 2000

Hi,

I am working on sql server bulk copy program. I am getting data files from
our vendors for shares and stocks. The data files are pipe separated values.
for ex the Ascii file format is

8388182|"ACC consultanats"|"rating for the current financial year"|23

My doubt is i have four columns in my sql server table named stocks.

table structure
---------------
serialno numeric
caption nvarchar(255)
memo1 ntext 16
sno int

In this the third column named memo1 in the data file would be a large
volume data. That is it may be upto one full A4 size page.
One important thing is, data in the third column is not formatted.

Since it is very urgent, let me know what would be the format file for
this type of data file and the bulk copy program utility.

Kindly let me know as early as possible.

Regards
Sivaramakrishnan

View 1 Replies View Related

BULK INSERT FROM REMOTE SERVER

Nov 7, 2001

I am running the following:
BULK INSERT DB.dbo.[stblCLIENT]
FROM 'SERVER1downloadClient.txt'
WITH
(
FIELDTERMINATOR = 'Ř',
ROWTERMINATOR = ''
)
DB.dbo.[stblCLIENT] is on SERVER2. I receive the following error:
"Could not bulk insert because file 'SERVER1downloadClient.txt' could not be opened. Operating system error code 53(The network path was not found.)."

I am able to run a DTS package that imports the same text file from SERVER1 with no error.

Is BULK INSERT limited to importing text files from the server on which SQL Server is running or should I be able to BULK INSERT from another server on my LAN?

View 1 Replies View Related

SQL Server 2005 - BULK INSERT

Sep 4, 2006

I'm trying to import data from flat file in table and have fewproblems.1.Field Delimiter is ',' (comma). If ',' occurs in quotedstring it is still treated as field delimiter. This is BUG or ?2.In table I have datetime field that can be null, but bulkinsert reports error if in flat file is null or ''. It's OK only whenreal date is specified.Table:create table AttachmentList (Code integer not null,ClassID integer null,Description varchar(200) null,ValidUntil datetime null,constraint PK_ATTACHMENTLIST primary key (Code))flat file.1,13,'Naputak, CU 261098', ''Thanks in advanceDavor

View 3 Replies View Related

Using Bulk Insert With SQL Server 2005

Aug 29, 2006

I have a web page that prompts a user to select a csv file. Using a Bulk Insert the data is loaded into a SQL Server 2005 table.

I have been using the Bulk Insert with SQL Server 200 with no problems, but with 2005 I am getting the error "You do not have permission to use the bulk load statement".

My web.config file has the following connection string:
[code]
<add key="connectionString" value="Server=(local);Database=BroadCastOne;trusted_connection=true" />
[/code]

I've given bulkAdmin role to the ASPNET user. It's still not working. What am I doing wrong?

Any help is greatly appreciated,
Ninel

View 3 Replies View Related

Bulk Insert Into SQL Server Table With XML

Sep 9, 2007

Hey There,

Here, is the example of Bulk Insert into SQL Server Table.
From Application you have to pass a XML string to a Stored Procedure and it will insert all data into table using that XML.
Example SP.


CREATE PROCEDURE StoredProcName
(
@strXML varchar(8000)
)
AS
Declare @intPointer int
exec sp_xml_preparedocument @intPointer output, @strXML

INSERT into tbl_plnd_insertion
SELECT Column1, Column2, Column3, Column4, Column5
FROM OpenXml(@intPointer,'/root/tbl_plnd_insertion',2)
WITH (Column1 varchar(20) '@Column1' , Column2 varchar(20) '@Column2', Column3 varchar(20) '@Column3' , Column4 varchar(50) '@Column4', Column5 varchar(50) '@Column5')
exec sp_xml_removedocument @intPointer


Thanks !!!!!

View 10 Replies View Related

SQLXML Bulk Load Of SQL Server Database

May 2, 2007

I need to update a number of sql server tables, the data sources for these coming from a number of stored procedures.  I want a generic way of getting the data and then passing this data to the tables.I am thinking of doing this for each table:Populating a datasetWriting this dataset to XMLUsing  SQLXML Bulk Load to pass this XML to the database to updateI can create the xml data file by:
dataset.WriteXml("C:data.xml")The problem I have is that the example (http://support.microsoft.com/default.aspx/kb/316005/en-us) I looked at relies on the schema being defined:<?xml version="1.0" ?><Schema xmlns="urn:schemas-microsoft-com:xml-data" xmlns:dt="urn:schemas-microsoft-com:xml:datatypes" xmlns:sql="urn:schemas-microsoft-com:xml-sql" > <ElementType name="CustomerId" dt:type="int" /> <ElementType name="CompanyName" dt:type="string" /> <ElementType name="City" dt:type="string" /> <ElementType name="ROOT" sql:is-constant="1"> <element type="Customers" /> </ElementType> <ElementType name="Customers" sql:relation="Customer"> <element type="CustomerId" sql:field="CustomerId" /> <element type="CompanyName" sql:field="CompanyName" /> <element type="City" sql:field="City" /> </ElementType></Schema>Is there any way I can create the schema 'on the fly' similar to how I did for the data source file.As I could then pass these files to the database:objBL.Execute ("schema.xml","data.xml"); 

View 1 Replies View Related

SQL Server 2014 :: Bulk Import XML To Table

Jun 19, 2014

I found loads of things but nothing seems to work...

I'm trying to get a link with XML data inside the page into a table but I can't find anything

View 9 Replies View Related

How Read CSV File In Other Server Using Bulk Inser

Mar 24, 2008

Hi All,

Please some one help me...

I have to insert a csv into one table in sql server. But the problem is the file is in one server and SQL SERVER 2005 is in other server..

how do i insert the file....

please help me.....

View 1 Replies View Related

Cannot Fetch A Row From OLE DB Provider BULK For Linked Server

Nov 15, 2006

I have an SSIS job that is pumping to a SQL Server Destination, hundreds of gigabytes of raw text files. Today I received this strange error - does anyone have insight? Also, how would I make the data tasks more stable and robust so that this doesn't cause package failure (retries, or something?)

[SQL Server Destination [4076]] Error: An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Reading from DTS buffer timed out.".

View 20 Replies View Related

Transact SQL :: Create Server Login In Bulk

Sep 29, 2015

I need to create a T-sql script to create logins in bulk so users can access views on the database. Usernames as used in the local domain are all stored in a table.

Users already exist in the domain (let’s say: MYDOMAIN)ADSI.dbo.Ad.username  is the column wich contains  the usernames as used in the Active Directory domain  (+/- 350)Script needs to check if login already exists to make sure login is not added twice.If possible give all these users db_datareader role, in let’s say: MYDATABASE

View 7 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved