Load Data With BulkInsert In DTS, How To Skip The First Line?
Dec 13, 2007I can skip the first line with Transform Data Task, it look like can not skip the first line in BulkInsert.
But bulkinsert is faster, anybody can help?
Thanks
ADVERTISEMENT
Skip The First Line Of The Data File - Bulk Insert
Oct 1, 2007Hi,
I have a data file and the contents of it are as follows
2 -- This is the header indicating the no of records in my files
1001|s1
1006|s2
The content of format file is as follows. This is to skip first column of the all the rows and get only Subs (i.e s1 and s2 )
9.0
2
1 SQLCHAR 0 100 "|" 0 ID ""
2 SQLCHAR 0 100 "
" 1 Subs ""
Here is my query to get all the Subs from my data file
SELECT * FROM OPENROWSET( BULK 'datafile.txt',
FORMATFILE = 'FormatFile.fmt',
FIRSTROW = 2 ) AS a
But this query retuns only s2 where i was expeting s1 and s2. The reason being is that the firts row i.e header doesn't follow the format
Can any one please let me know how to skip the first line in the data file and get the result as required
~Mohan
Not Able To Load The Application In Case Web Farm Garden When We Load Data Through Background Thread.
Dec 14, 2007Hi,
Here I will describe my problem.
1. We are loading large amount of data from database on background thread which is starting on Application_start event in global.aspx.cs file.The data is later cached for subsquent request to improve the performance.
2. Now when we put the application on web farm garden, it is not able to load the application.
3. We are sending the request the servers through Router kind of application.
4 This application is working fine on single server enviornment.
Please help us.
Ajay Kumar Dwivedi
Load All Data Without Knowing Old One Was Load In The Previous Time???
Apr 27, 2007I just have done the SSIS example in the tutorial document included when install SQL 2005 ENT. I have a problem that whenever I test to run, the service load all data from source with out noticing about the data (I mean it load all the data to the destination), I do it several time and it continue to load all without checking. That mean the data is dublicated when the schedule run???
I think there should be a paramete or something like that to help the engine just load the new data to the destination. Could you help please?
Thank
BULKINSERT
Feb 26, 2008hi all,
can any one give me syntax example of how to use BULKINSERT
Thanx for any help
sql9
Skip Field Terminator While Inserting Data To A Table-Bulk Insert
Oct 10, 2007Hi,
I have a data file which consists of data as below,
4
PPU_FFA7485E0D||
T_GLR_DET_11||
While iam inserting into table using bulk insert, this pipe(||) is also getting inserted into the table,
here is my query iam using to insert the data using bulk insert.
BULK INSERT TABLE_NAME FROM FILE_PATH
WITH (FIELDTERMINATOR = ''||'''+',KEEPNULLS,FIRSTROW=2,ROWTERMINATOR = '''')
Can any one help on this.
Thanks,
-Badri
BULKINSERT Problem
Jan 24, 2005Dear All,
I am importing all the files from a particular folder to a table on my database KB. It is working perfectly if i use it on the same system where the DB exists and not working from
the network. Also this scritp is working perfectly on sqlserver7 and not working on sqlserver2000.
SQL2000 returns error near comma
set @query ='BULK INSERT '+ @Tablename + ' FROM "'+ @Filepath+@Filename+'"
WITH ( FIELDTERMINATOR = ","(returns error here) ,ROWTERMINATOR = "")'
USE TESTDB
--Table Creation Starts here
Create table Account([ID] int IDENTITY PRIMARY KEY, Name Varchar(100),
AccountNo varchar(100), Balance money)
Create table logtable (id int identity(1,1),
Query varchar(1000),
Importeddate datetime default getdate())
--Table Creation ends here
---Stored Procedure Starts here
Create procedure usp_ImportMultipleFiles @filepath varchar(500),
@pattern varchar(100), @TableName varchar(128)
as
set quoted_identifier off
declare @query varchar(1000)
declare @max1 int
declare @count1 int
Declare @filename varchar(100)
set @count1 =0
create table #x (name varchar(200))
set @query ='master.dbo.xp_cmdshell "dir '+@filepath+@pattern +' /b"'
insert #x exec (@query)
delete from #x where name is NULL
select identity(int,1,1) as ID, name into #y from #x
drop table #x
set @max1 = (select max(ID) from #y)
--print @max1
--print @count1
While @count1 <= @max1
begin
set @count1=@count1+1
set @filename = (select name from #y where [id] = @count1)
set @query ='BULK INSERT '+ @Tablename + ' FROM "'+ @Filepath+@Filename+'"
WITH ( FIELDTERMINATOR = ",",ROWTERMINATOR = "")'
--print @query
exec (@query)
insert into logtable (query) select @query
end
drop table #y
--sp ends here
Exec usp_ImportMultipleFiles 'c:myimport', '*.csv', 'Account'
If i use the above Exec like
Exec usp_ImportMultipleFiles '\kb-02C$MyImport', '*.csv', 'Account'
I am getting the following error:
Could not bulk insert because file '\kb-02C$MyImportAccess is denied.' could not be
opened.
Operating system error code 5(Access is denied.).
C Drive and MyImport folder is shared on system kb-02
Also If if execute this script it should insert the following line to logtable for all the available files on myimport folder.
How do i modify the following line in the above script to get the syntax given below:
set @query ='BULK INSERT '+ @Tablename + ' FROM "'+ @Filepath+@Filename+'"
WITH ( FIELDTERMINATOR = ",",ROWTERMINATOR = "")'
Required Syntax
BULK INSERT kbAccount FROM 'c:myimporta.csv' WITH ( FIELDTERMINATOR = ',' ,ROWTERMINATOR = '')
Would appreciate your valuable HELP.
thanking your valuable help in advance.
K006B
BULKINSERT - Any GURU ?
Jan 2, 2008
Hi,
I am trying to import comma delimitted text files which contain data for any particular country.
I have created SSIS package to import data into sql server table. table structure is exactly same as text file except one additional column which is to identify the CountryID. Text files we recieved does not contain CountryID column.
I am using BULK INSERT Task in SSIS to import the data as text files are quite big (up to 1 GB). during processing I do determine the country id for that text file as file name contain countrycode which can be use for lookup and get the countryid.
cyn_bills_20071208_032242.txt
CYN is the country code and country table has got country id for it.
I am using Format file with BULK INSERT task as table has additional column which does not present in source file.
currently after inserting all rows into table i have to run an UPDATE statement to update the value of CountryID column which is very expensive some time taking more then 30 min for (30 million rows)
Can anyone tell me how to insert an expression when using BULK INSERT command / Task in SSIS. there must be a way to insert litterals using Bulk Insert.
Thanks in Advance.
Furrukh Baig
Data Warehousing :: How To Skip PIPE (comes As Data) In PIPE Delimited Flat File
May 13, 2015I have a problem with a PIPE "|" delimited flat file. I have a column "Description" in which we get a string in which we have PIPE "|" as data. How we can skip this and load it as a data into the column Description.
View 7 Replies View RelatedImport A XML File In SQL2005 DB Using BulkInsert?
Apr 15, 2008Hi All,
I am dealing with a lot of XML file data that I load in a SQL 2005 database using Integration Services. Let me give you a high level idea of I how I do it.
The way I do it is that a) I create a blank Integration Services Project, then b) In the "Control Flow" tab I put the "Data Flow Task" by simply dragging "Data Flow Task" on to the "Control Flow" page. c) Then in the "Data Flow" tab I drag the "XML Source" as my data flow source, and drag "OLEB Destination" as my data flow destination; then I create a connection between the "XML Source" and "OLEDB Destination" and point the "XML Source" to my XML file, and I point the "OLEDB Destination" to the Database where I want to load the XML file. I configure the mappings etc if needed and run the package. This works fine for me. But if the XML file is large it takes hours to load the data. I am dealing with huge size XML files and I want an alternative in Inegration Services probably "Bulk Insert Task" but dont know how to use it.
Can someone please explain how I can use bulk insert as a fast alternative to load the data. Please list the steps.
Thans a milliom in advance.
Zee
Problem Showing Data From Select Line With Another Select Line In It
Apr 7, 2007Hi
If i use this code i cant get the data showed, it show nothing."SELECT COUNT(DogImageDate) AS Amount, DogImageDate, DogImageID FROM EnggaardImages WHERE DogImageDate NOT LIKE (SELECT TOP 1 DogImageDate FROM EnggaardImages ORDER BY DogImageDate DESC;) GROUP BY DogImageDate ORDER BY DogImageDate DESC;"
But if i use this code i get data showed"SELECT COUNT(DogImageDate) AS Amount, DogImageDate, DogImageID FROM EnggaardImages GROUP BY DogImageDate ORDER BY DogImageDate DESC;"
Then i get Images(6)Images(1)Images(1)
But i dont want the first to be showed, therefor i use the Select TOP 1, so i get a look like this
Images(1)Images(1)
But i cant get it to work.
Specifyin The Row Delimeter As Newline (Unix Style) In Bulkinsert Format File
Sep 28, 2007Hi,
I am bulk inserting the data from a file into a table using the following format.
9.0
1
1 SQLCHAR 0 10 "
" 1 MSISDN SQL_Latin1_General_CP1_CI_AS
This works perfectly for me.
But my problem is that the firmat may not be generated in Windows and i can't user "
" as row delimeter.
It could be generated in Unix and want to use LF as the row delimeter.
Questions :
1) How do i specify this kind of new line (LF ) in formatfile?
2) Is there anyway that i can specify both the delimeters and let it use which ever is appropriate ?
Please provide some inputs on this??
Thank u so much
~Mohan
Data Access :: How To Load Data From CSV File In Temp Table At Run Time
May 28, 2015how I can load the CSV file data into the sql server table. I know there are ways like bulk insert and other to load the csv file data into the table. But in my case the table doesn't exist and has to be created at the run time. With simple insert in temp table we do like select * into #temp from tablename and that creates the temp table. So. I need something like that which create the temp table and load the data into it. because the CSV file would have different number of columns and names so I can not create the table structure in advance. I have to create the table at run time.
View 3 Replies View RelatedT-SQL (SS2K8) :: Load Data From Flat File Source Into OleDB Destination By Changing Data Types In SSIS
Apr 16, 2014I have an source file and i have to load it into the data base by changing datatype of the columns in ssis
View 1 Replies View RelatedHow Can I Remove The Line Feed/carriage Return In The Last Line Of The Exported Text File ?
Feb 27, 2007Hi,
for some AP issue, the file I upload must be without the line feed/carriage return in the last line.
for example:
original fixed-length file (exported from SSIS)
line NO DATA
1 AA123456 50 60
2 BB123456 30 40
3 CC123456 80 90
4 <-- with line feed/carriage return in the last line
The file format that AP request. The file only has 3 records, so it should end in the third line.
line NO DATA
1 AA123456 50 60
2 BB123456 30 40
3 CC123456 80 90
Should I use script component to do it ? I am new for VB . Anyone would help me ?
Thank you all.
Reporting Services :: Draw Trend Line For SSRS Line Chart 2005
May 4, 2012I need the Trend line for the following data in Line chart they are the following data. The following are the graph are my output and i need the trend line for these Key_gap value.
This is the link [URL] ....
I need the same trend line for the Bar-Chart in SSRS 2005.
Storing And Retrieving Line Breaks/newlines From Multi-line Textbox (C#)
Aug 31, 2007I hope I'm posting this in the correct forum (forgive me if I'm not) since I'm not sure if this is an issue with inserting an item into a db or the processing of what I get out of it. I wrote a basic commenting system in which someone my post a comment about something written on the site. I wanted to keep it very simple, but I at least want the ability for a user to have newlines in their comment without having to hardcode a <br /> or something like that. Is there a way for me to detect a newline if someone, for example, is going to their next paragraph?
Let me know if you need a better explanation.
Thanks in advance!
ISQL: Msg 170, Level 15, State 1, Line 1 Line 1: Incorrect Syntax Near ' '
Nov 8, 2006G'day everyoneThat's a space between the ticks.It's all part of a longer script but seeing as the failure occurs online 1if exists (select * from dbo.sysobjects where id =object_id(N'[dbo].[config]') and OBJECTPROPERTY(id, N'IsUserTable') =1)drop table [dbo].[config]GOThat's three lines only. Does it matter that they're in Unicode?Any ideas?Kind regards,Bruce M. AxtensSoftware EngineerStrapper Technologies
View 3 Replies View RelatedNeed New Line Character In Data
Jan 11, 2008I have a need to have new line character in some data. I am trying to insert and retrieve the data with the new lines char in it, but it does work. I need the new line char in the data as I need to do some further processing whenever the new line starts.
Any ideas on how I can achieve this?
Thanks for the help.
This are the ways I have already tried, but does not work
create table mytest (col1 varchar(max))
insert into mytest values (' line1 ' + CHAR(13) + ' line2')
update mytest set col1 = (' line1 ' + CHAR(13) + CHAR(10) + ' line2')
update mytest set col1 = 'line1
line2
aa
line3 '
update user03.amitatest set col1 = 'line1 ' + CHAR(13) + CHAR(10) +
' line2 ' + CHAR(13) + CHAR(10) + 'aa ' + CHAR(13) + CHAR(10) + 'line3'
select col1 from mytest
The result set from the above select is -
line1 line2 aa line3
Displaying A Trend Line (in Line Chart) In SSRS
Feb 7, 2007We have a line graph which plots the actual data points (x,y), everything is working fine with this graph. Now we need to add a trend line to this existing graph after going thro. the articles we came to know that there is no direct option in SSRS to draw a trend line. So we need to calculate the trend values ourselves which we need to plot as atrend line. This trend line is similar to the trend line which comes in Excel chart, do anyone know how to calculate the trend values from the actual data points. We got through several formulas, but were not clear, have anyone tried out exactly the same, if so please help us out by providing an example to calculate the trend values.
View 1 Replies View RelatedDynamically Change The Color Of The Line On A Line Graph
Oct 26, 2007I have a line graph which shows positive and negative values. Is it possible to have the line one color when its negative and another when its positive?
kam
Data Load
Jun 20, 2001Hi!
We make complete data load once a week. Now we need to make additional steps to append data on daily basis. We have primary key on the table and it doesn't allow appending duplicate rows.
What steps should we create to append data?
Thank you,
Elena.
LOAD Data
Oct 23, 1999Hi everobody!
When I run :
LOAD DATABASE db1
FROM DISK = 'c:mssqldatadb1_backup.dat'
go
I got error message:
Msg 3201, Level 16, State 1
Can't open dump device 'c:mssqldatadb1_backup.dat', device error or device off line. Please consult the SQL Server error log for more details.
How can I fix the problem?
Thank you.
Alona
Pulling Data Into One Line (Aggregation)
Aug 19, 2015eventually I started using DERIVED Tables and Sub queries within them if needed. I like choking down all the queries in the selects joining them and having your result set select there to choose from all the aliases, it also resolves while doing this. So much easier IMO than using CTE's or TEMP Tables. I was big on temp tables for a while...
I'm curious though if you want to count a type of criteria in a column do you use a standard case in your inner query choking it down just for those particular counts? Then do another case on the other criteria and END AS with the alias name?
E.G.
CASE WHEN COLUMN1 = PIZZA THEN COUNT(COLUMN1) END AS PIZZACOUNT
CASE WHEN COLUMN1 = ROOTBEER THEN COUNT(COLUMN!) END AS ROOTBEERCOUNT
ETC...........
Need To Load Data From A Client To SQL.
Nov 13, 2005I'm needing to load data on the client side to load into a DataGrid. I decided to use excel to load the data, but it doesn't need to be. My problem is that it only loads from the server not the client. I browse to find the file and get the path with a control named ctlFindFile. A button labeled ctlLoadData will when pressed display the pathway in label1 and also place the pathway in the function GetDataFromExcel which returns a dataset from the spreadsheet and displays the data in a dataset. Data is returned fine if I’m on the server, but when I'm on a remote machine I receive an error. That is unless I've place a spreadsheet with the same name and pathway on the server as on the client machine, then I’m able to load the file. Now how do I get it to load from on the client machine?
Code below:
Private Sub ctlLoadData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ctlLoadData.Click
Label1.Text = ctlFindFile.Value
DataGrid1.DataSource = GetDataFromExcel(ctlFindFile.Value, "SampleNamedRange").Tables(0)
DataGrid1.DataBind()
End Sub
Public Function GetDataFromExcel(ByVal FileName As String, ByVal RangeName As String) As System.Data.DataSet
'Returns a DataSet containing information from a named range in the passed Excel worksheet
Try
Dim strConn As String = "Provider=Microsoft.Jet.OLEDB.4.0;" &_ Data Source=" & FileName & ";Extended Properties=Excel 8.0;"
Dim objConn As New System.Data.OleDb.OleDbConnection(strConn)
objConn.Open()
' Create objects ready to grab data
Dim objCmd As New System.Data.OleDb.OleDbCommand("SELECT * FROM " &_ RangeName, objConn)
Dim objDA As New System.Data.OleDb.OleDbDataAdapter
objDA.SelectCommand = objCmd
' Fill DataSet
Dim objDS As New System.Data.DataSet
objDA.Fill(objDS)
' Cleanup and return DataSet
objConn.Close()
Return objDS
Catch ex As Exception
' Possible errors include Excel file already open and locked, et al.
Return Nothing
End Try
End Function
Error on Client:Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.Source Error:
Line 31: Private Sub ctlLoadData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ctlLoadData.Click
Line 32: Label1.Text = ctlFindFile.Value
Line 33: DataGrid1.DataSource = GetDataFromExcel(ctlFindFile.Value, "SampleNamedRange").Tables(0)
Line 34: DataGrid1.DataBind()
Line 35: End SubSource File: C:InetpubwwwrootSAI_LoadWebForm1.aspx.vb Line: 33 Stack Trace:
[NullReferenceException: Object reference not set to an instance of an object.]
SAI_Load.WebForm1.ctlLoadData_Click(Object sender, EventArgs e) in C:InetpubwwwrootSAI_LoadWebForm1.aspx.vb:33
System.Web.UI.WebControls.Button.OnClick(EventArgs e) +108
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +57
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +18
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +33
System.Web.UI.Page.ProcessRequestMain() +1292
Version Information: Microsoft .NET Framework Version:1.1.4322.2032; ASP.NET Version:1.1.4322.2032
Load Data Command
Dec 14, 2007Hi foks!
In MYSQL I can insert a whole text/excel/csv file with the Load Data command. Can I do such a thing for MSSQL?
Thank you!
Monitoring Data Load
Jan 16, 2007I have a user that is loading data via an Access load procedure to a table that actually is a SQL Server 2005 table but is linked to the Access database. He is saying the load is extremly slow. How can I monitor what it is doing on the SQL Server side?
Thanks!
duckman2007
Have a great day!
Load Data Command
Dec 14, 2007Hi foks!
In MYSQL I can insert a whole text/excel/csv file with the Load Data command. Can I do such a thing for MSSQL?
Thank you!
Data Load Query
Sep 21, 2005Hi,I'm extracting data from a mainframe application with a view to loadingit into a MS SQL database. I'm trying to determine the most efficientway to format the mainframe extract file to make loading into thedatabase easier.The problem I have is that the existing record structure includes anarray that can vary between 1 to 50. If I include this array in asingle record the table I use to import the data would need 50 columnsthough not all these would be populated. There is a field in the recordto identify how many occurances of the array there are.Current Record Structure :Account NumberAccount NameOther Account DetailsTotalNumberOfArrayFieldsPopulatedArray :Value1Value2Value3....up to Value50 (if required)i.e.12344,Mr Agent,$29.95,2,BX123,BX12412345,Mr Jones,$14.95,3,XX123,XX124,XX12512345,Mr Jones,$14.00,1,XY12312345,Mr Jones,$15.95,2,XZ124,XZ12512346,Mr Smith,$19.95,3,AX123,AX124,AX12512346,Mr Smith,$19.00,1,BY12312347,Mr Acant,$99.95,7,CX123,CX124,CX125,CX126,CX127,CX128 ,CX129There may be up to 3 records created for each Account Number withdifferent values in the array fields.Am I better to break this file into two files .. one with the corecustomer information and a second file with a row for each array valuewhich has a link to the customer information file.OrIs there a way to efficiently process the original file once it isloaded into the staging tables in the database ?i.e.File 1 - Core Customer Information====================================Current Record Structure :Record NumberAccount NumberAccount NameOther Account DetailsTotalNumberOfArrayFieldsPopulatedFile 2 - Array Information====================================Record NumberArray :Value1Value2Value3....up to Value50 (if required)File 1========================12344,Mr Agent,$29.95,212345,Mr Jones,$14.95,312345,Mr Jones,$14.00,112345,Mr Jones,$15.95,212346,Mr Smith,$19.95,312346,Mr Smith,$19.00,112347,Mr Acant,$99.95,7File 2========================12344,BX12312344,BX12412345,XX12312345,XX12412345,XX12512345,XY12312345,XZ12412345,XZ12512346,AX12312346,AX12412346,AX12512346,BY12312347,CX12312347,CX12412347,CX12512347,CX12612347,CX12712347,CX12812347,CX129At times the individual array values will be used for look ups thoughessentially the Customer Information record will be the primary lookupdata.I'm leaning toward changing my COBOL code and creating the 2nd outputunless someone can suggest a simple way to process the information onceloaded into the table.Any help that could be suggested would be greatly appreciated.
View 1 Replies View RelatedUse SSIS To Load Data Into CRM 3.0
Jan 18, 2007Hi,
I have a little experience with SSIS (but a lot with DTS) and none with CRM 3.0.
Is it possible to use SSIS to import flat files into CRM 3.0 ?
And how ?
I read something that CRM 3.0 uses SQL server 2000. Should I upgrade it SQL server 2005 ?
Any suggestion where to read about this ?
Thanks in advance
Constantijn Enders
Load Data From .DAT File
Jul 14, 2006Hi All,
I am using Bulk Insert task to laod data from .dat file to SQL table but getting an error below.
[Bulk Insert Task] Error: An error occurred with the following error message: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.".
Any help will be appreciated.
Thanks.
Data Load From The Different Server
Aug 21, 2007I tried to load data from the table in different server.
If I just want to limit one year data ( using date_key in that table ) then what task do I need to do that?
Please let me know.
Thanks.
Load Data Using ADO.NET In SSIS
Nov 8, 2006Hi,
How to extract and Load Data using ADO.NET in SSIS.i hope to extract data we have DataReader source .but how to load (Insert) data with ADO.NET ?.and is ADO.Net quicker than OLEDB ?
Thanks
Jegan.T