Dynamically Importing Csv File Into MS SQL
Feb 14, 2007
I have an app written in delphi v5 with a MS SQL Database and I'm struggling with an import feature i need
The user needs a function to import Contact data from a txt/csv file into the Contact Table
Details of feature:
the user enters the different parameters into the Delphi app e.g.
File Location
Files Includes Column Headings or not
Whether the file Comma or Tab Seperated
Mapping the fields
Then i need to use those parameters and run some sort of import routine putting the data into the specific table.
The tables consists of 3 fields - First Name, Surname, Mob Number - but these fields can be in any order in the file. for example Col 1 (in file) needs to go into Field 3 in DB. this is sorted in the Mapping Fields above
How do I do this using MS SQL??
I've been looking at using the BULK INSERT command e.g
BULK INSERT Contact
FROM 'C:Import_data.csv'
WITH
{
FIELDTERMINATOR = ',',
ROWTERMINATOR = '
'
}
but at the minute i just get error -
[Microsoft][ODBC SQL Server Driver]Syntax error or access violation.
Is this the correct command to use??
Do you know any websites that can point me in the right direction??
I have an app written in delphi v5 with a MS SQL Database and I'm struggling with an import feature i need
The user needs a function to import Contact data from a txt/csv file into the Contact Table
Details of feature:
the user enters the different parameters into the Delphi app e.g.
File Location
Files Includes Column Headings or not
Whether the file Comma or Tab Seperated
Mapping the fields
Then i need to use those parameters and run some sort of import routine putting the data into the specific table.
The tables consists of 3 fields - First Name, Surname, Mob Number - but these fields can be in any order in the file. for example Col 1 (in file) needs to go into Field 3 in DB. this is sorted in the Mapping Fields above
How do I do this using MS SQL??
I've been looking at using the BULK INSERT command e.g
BULK INSERT Contact
FROM 'C:Import_data.csv'
WITH
{
FIELDTERMINATOR = ',',
ROWTERMINATOR = '
'
}
but at the minute i just get error -
[Microsoft][ODBC SQL Server Driver]Syntax error or access violation.
Is this the correct command to use??
Do you know any websites that can point me in the right direction??
Hope all this makes sense
Thanks
View 1 Replies
ADVERTISEMENT
Jul 22, 2004
Hi,
I have a dts package that imports a number of text files into a SQL Server 2000 database table. The package has been set up to accept a text file with a row delimiter of carriage return and line feed ({CR}{LF}). Some of the text files I receive only have a line feed ({LF}) as the row delimiter and the dts package fails the file. Is there an activex script i can use that will scan the file and change the row delimiter as required?
i was going to use the filesystemobject which allows me to read a line at a time, however the Readline method doesn't read the new line character. The text files are too big to read into one variable and then do a replace.
Any help would be appreciated
I am using SQL Server 2000 SP3, Windows Server 2000 and Windows XP Professional. All systems are fully patched
Regards Justin
View 3 Replies
View Related
Sep 2, 2015
Currently have a single hard coded file path to the SSRS config file which parses the file and provides the reporting services web service url. Â My question is how would i run this same query against 100s of servers that may or may not share the same file path as the one hard coded ?
Is there a way to query the registry to find the location of the config file of any server ? which could be on D, E, F, H, etc.Â
I know I can string together the address followed by "reports" and named instance if needed, but some instances may not have used the default virtual directory name (Reports).
Am I going about this the hard way ? Is there a location where the web service url exists in a table ? I could not locate anything in the Reporting service database. Basically need to inventory all of my reporting services url's.
View 2 Replies
View Related
May 15, 2006
good morning,
I want to load data that i receive everydays from my customers in .xls file format (excel) or cvs file format, to the database that i have created on this purpose. but when trying to do that whith SSIS; i got an error message .... that i can't import redudant data in my database column.
Best regards.
View 1 Replies
View Related
Apr 11, 2008
I have CSV file as source for SSIS package every time the filename will be changing like trd_1990M1_1990M12.csv
trd_1991M1_1991M12.csv , trd_1992M1_1992M12.csv so it will vary as per user selection . i need to run the same SSIS package to execute the package whichever the file name.
Please let me know the solution for that how to pass the file name dynamically to SSIS package.
View 3 Replies
View Related
Oct 7, 2007
Hi,I have a requirement to load data from a flat file.The flat file has a naming convention and thus a timestamp is appended to it. eg : RMRS_07102007_001.delThis makes the file name different each day, Is there any method where by I can provide a dynamic name in the file connection manager.P.S. I tried a variable in For each loop with file enumatror and stored that in a variable , but I was not able to use that variable in File connection manager. pls guide if i was missing anything.Thanks in Advance, Ashish Prasad
View 1 Replies
View Related
Feb 13, 2006
Hi
Some one Please guide me... How to create a Excel File Dynamically in DTS.
Once i run my DTS Package the result should be moved to a new Excel sheet( This should be created dynamically).
I tries in this way but it says....Syntax error at the DTS GlobalVariables.....
Function Main()
Dim appExcel
Dim newBook
Dim oSheet
dim oPackage
dim oConn
Set appExcel = CreateObject("Excel.Application")
Set newBook = appExcel.Workbooks.Add
Set oSheet = newBook.Worksheets(1)
oSheet.Range("A1").Value = "au_lname"
oSheet.Range("B1").Value = "au_fname"
oSheet.Range("C1").Value = "phone"
oSheet.Range("D1").Value = "address"
oSheet.Range("E1").Value = "city"
DTSGlobalVariables("fileName").Value = "C:\" & Month(Now) & "-" &
Day(Now) & "-" & Year(Now) & "-" & Hour(Time) & "-" &Minute(Time) & "-" &
Second(Time) & ".xls"
With newBook
.SaveAs DTSGlobalVariables("fileName").Value
.save
End With
appExcel.quit
set oPackage = DTSGlobalVariables.parent
set oConn = oPackage.connections(2)
oConn.datasource = DTSGlobalVariables("fileName").Value
set oPackage = nothing
set oConn = nothing
Main = DTSTaskExecResult_Success
End Function
View 2 Replies
View Related
Apr 10, 2008
hi,
Can you guys please give me the steps of how to create a destination file dynamically. What i mean is for example i want to get everything from a table and ssis should create a file distination according to today's date and save it in a specific folder.
thanks
View 7 Replies
View Related
Dec 17, 2007
Hi,
I am trying to import a SQL dataset run through SQL Execute task to an excel file.
I would like the excel file to be generated each night.
My question is
1. How would I dynamically name the excel file to be report_daterun?
2. Also I would like to automatically zip the file up can ssis do this as well?
thanks
View 1 Replies
View Related
Dec 1, 2007
hi all
I have a table which has the paths where the source files need to be collected.
So I have a set of files which I need to collect. So I need to iterate through the result set which retrieves the paths
where the files can be found and set this dynamically to connection string of flat file connection manager.
Can anyone please let me know how to do this.
Can you be a little elaborate as to how we can do this because I am really new to SSIS.
Please help me as I am new to SSIS.
Thanks
Sai
View 1 Replies
View Related
Jun 15, 2008
Hi
My SSIS package automatically creates Excel files everyday.
i want the excel files to get todays date when they are created .
Eg:06152008 for today and 06162008 for tomorrow
please help
View 3 Replies
View Related
Dec 6, 2007
I have one share folder ,every month end-user will copy & paste excel file into particular share folder. Ok .
Now i have to create new SSIS package as schedule should run every month to find the file and then load automatically into Sql server tables and then move those excel file to another share folder if file successfully loaded only.
The excel file name will be changing every month. but the format wont change. If any body knows this process or steps.
Please share with me .
Thanks in Advance.
View 5 Replies
View Related
May 21, 2007
I am a relative newbie to SSIS. I have been tasked with writing packages to import data from our clients. We have about 100 clients. Each client has a few different file formats. None of the clients have the same format as each other. We load files from each client each day. Each day the file name changes. I have done all of my current development work with a constant file name in a text file connection manager.
Ultimately we will write a VB application for the computer operator to select the flat file to load and the SSIS package to load it with. I had been planning on accomplishing this thru the SSIS command line interface. Can I specify the flat file to load via a variable that is passed through the command line? Do I need to use a Script Component to take the variable and assign it to the connection manager?
Is there a better way to do this? I have seen glimpses of a VB interface to SSIS. Maybe that is a better way to kick off the packages from a VB app?
Thanks,
Chris
View 9 Replies
View Related
Feb 16, 2007
I am trying to create a text file from an SQL query on a SQL table. I would like the SSIS package to prompt for the file name and path. The text file is tab delimited and the text qualifier is a double quote.
Thanks,
Fred
View 9 Replies
View Related
Jun 19, 2007
I am looking a way to export SQL Server 2005 DB tables to Access file dynamically. Like if i have added or removed any tables from SQL DB then when i run SSIS package it should export that table with data to Access file. is there any easy way to do this. if it is not them please someone please tell me which controls should i look at and which technique i should use to do this.
View 4 Replies
View Related
Apr 30, 2008
Before I launch into a long description of the issue, I was wondering whether its possible to dynamically amend the file path held in a configuration string? (I'm looking at the Package Configurations Organizer)
Basically recreating the /CONF switch in the dtexec utility
So a parent package could somehow tell a child package to pull its configuration from a selection of different xml config files at run time... I couldn't see a variable in the child package to set from the former
My only idea at the moment is to replace the Execute Package Task with an Execute Process Task, and directly invoke the child using dtexec & an expression to dynamically set the /CONF switch..
thanks
James
View 3 Replies
View Related
Jul 14, 2015
I have a query to pass .csv file data into sql table using bulkinsert . I used :
CREATE TABLE #CSVTest
(CouponCode VARCHAR(200))
/* Insertion From csv Source To Table */
BULK
INSERT #CSVTest
FROM 'C:Apple.csv'
[Code] ....
This worked very fine to me. Now, the thing is i should pass the filename as parameter and so that what ever file i specified in execute command , query should take that file and push that file data from .csv file to sql table.
View 2 Replies
View Related
Jun 5, 2007
Hi All,
I am struck at one point. I am trying to this operation and not able to go further.
1. I have got the dataset to a variable in the control flow.
2. I am looping through the dataset based on a coloumn.
3. Now inside my For each loop i have a dataflow task.
4. In the data flow task i am trying to build a dynamic query using the OLEDB Source and i have selected SQL Command from variable. And the variable build the Query as select * from @othervariable.
Now my question is
Can i send the data of each of the Query resultset to an out put text file using Flat File Connection? If yes pls guide me how? I have tried to create a flat file connection but i am failing how to map the data comming from step 4 dynamically for every query, since every query gives you a different resultset with different coloumns.
Thanks in advance..
Regards,
Dev.
View 15 Replies
View Related
Jun 22, 2007
Hi i'v e installed the file: SQLServer2005_DTS.msiBecause i've heard that i need that to import a csv file into a ms sql database. now i have no idea how to work it, like to even make it open.Where would i open it from? I reckon i'v eread enough literature about using it, to be able to have a bash... thanks in advance...
View 3 Replies
View Related
Oct 5, 1999
Hi, this is my first time and was wondering what's the best way to do the following:
i've sql server 6.5. also have a sql database(.DATfile) in a folder.
i need to import this database to my server. should i just LOAD/import it? or do i need to first create a database device on my server(how much size should i allocate etc) and then load it??
thanks a lot
rohit
View 3 Replies
View Related
Jun 24, 2002
Hi guys,
I have to write a Store procedure which will pickup a txt file from a destination, read it and update some table.
How to pick and read a txt file in a Stored procedure.
Thanks in advance
Vineet
View 1 Replies
View Related
Jun 11, 2007
Hi,
i am having problems while importing a .csv file.
when i import a .csv file in sql server I get the folowing error message.
"Cannot create an OLE DB accessor. Verify that the column metadata is valid".
Any help would be appreciated
View 3 Replies
View Related
Jul 23, 2005
I have a .txt file that I need to add to an existing table, which ismade up of Varchar, Char, numeric and int fields.What is the best way to do it.The first thing I tried was importing the .txt file, and then goinginto the design and changing the field type, but I hit problems on theNumeric fields.Then I tried changing the field types from varchar to different typesat the transfom stage of importing, but that failed too.Regards,Ciarán
View 2 Replies
View Related
Nov 22, 2007
I have a CSV file that i need to import into a SQL table. The problemis the values in the first column are not brackited in "". There areover 700K rows. Is there an easy way to fix the data or have SQLcorrectly import the the data?The data looks like this1, "xxx","zzz"2, "aaa","bbb"an so on...Thanks
View 1 Replies
View Related
Sep 19, 2006
Hi all-
I am in need of some help importing a .CSV file into a SQL Server 2005 Express database. I can't use SSIS because it's SQL Server Express. The operation needs to run as a parameterized Stored Proc which I will call from ASP.NET.
The problem is that I need to get the only 2 columns from the text file, then "tack on" 3 more columns that will have data that changes, which will come in as parameters to the T-SQL stored proc. This prevents me from using a straight BULK INSERT.
The data in each of these columns will be the same for each record in that column - that is, every record in every field in column 3, for example, will be the same.
Some of the files will have 4 million records and up, so speed is of the essence here. I tried using BULK INSERT to dump the data into a #temporary table - which took 38 seconds and was acceptably fast - but then, my next step was to dump the additional data into the other columns using UPDATE... SET. I gave up on this after the query ran for THIRTY MINUTES! My next step was going to be to move the data from the temporary table into the target permanent table somehow, but I never got that far when I saw how long the previous step took...
It's a little odd, because I can do the same thing in MS Access in under 5 minutes by using a SQL statement like this in my ASP.NET code:
"INSERT INTO [Target_Table] (field1, field2, field3, field4)" & _
"SELECT F1, F2, '" & strSource & "', Now() AS DateTimeStamp " & _
"FROM [Text;HDR=NO;DATABASE=" & strPath & ";].[" & strFilename & "]"
strSource is a string that the User enters (properly vetted for security); strPath and strFilename are strings holding the path and filename to the .CSV file. I create a unique filename from the file the user uploads. This works in under 5 minutes for several millions of records in MS Access, as I said above. I've had no luck getting anything similar to work in SQL Server, though.
Anyway, does anyone have any ideas? This is somewhat urgent, as the project was about to go "live" when it was discovered that the actual, live data had grown to the point where Access couldn't hold it, and a move to SQL Server Express was necessary.
Thanks in advance,
-Andrew
View 9 Replies
View Related
Sep 19, 2005
I need to be able to create a DTS package that imports a CSV file which is loated at URL. I.E. HTTP://www.url.com/csv/thefile.xls I tried copying the URL an pasting it in the file location when in SQL wazird but I got an error message.
View 1 Replies
View Related
Jun 4, 2007
I'm a newbie to SSIS, so there is probably an easy solution:
I'm importing from a csv file into a db table, but would like to remove the quotation marks from around the data before inserting into the table.
Are there any suggestions of how to remove the "data" marks? Would a Foreach Loop container work for this?
View 1 Replies
View Related
Dec 5, 2014
Is it possible to dynamically grow a data file for TempDB?
I have some TSQL that I want to execute (this is raw tsql, not something I'd typically share) when I get an alert that TempDB is growing.
SET NOCOUNT ON;
DECLARE @TempDB TABLE (FileID tinyint, name varchar(50), size bigint)
DECLARE @TotalSize decimal(19,4), @TotalFiles int, @CurrentTempDBSize bigint, @SQL varchar(1000), @DBName varchar(50)
SET @SQL = ''
INSERT INTO @TempDB
[code]....
ENDThe code runs fine, but doesn't actually increase the size of the file.
View 7 Replies
View Related
Feb 23, 2007
I have a Parent package and a child package.
I have hard coded the value for the User::FilePath variable in the parent package.
I am mapping this variable to the value of the same variable in the Child Package.
I created a directory and sql file: "C: empsqlb.sql". I have verified that the path variable value is passed to the child package by using a Script task with a messgbox call.
How do I define an execute sql task to execute sql file: @[User::FilePath] & "sqlb.sql". I'm using this expression for the SqlStatementSource property. I have entered the OLEDB server information and specified the SQLSourceType = fileconnection.
However I get the error:
[Execute SQL Task] Error: Connection manager "D:
lewisdevZsqlb.sql" does not exist.
What am I doing wrong?
View 3 Replies
View Related
Mar 12, 2008
Hi,
I tried to follow the widely talked about method to dynamically populate the connection string property of my flatfileconnection manager from a variable. I keep getting the following non-fatal error.
TITLE: Microsoft Visual Studio
------------------------------
Nonfatal errors occurred while saving the package:
Error at Package [Connection manager "FFCM"]: The file name ""C:ProjectsSSISHLoadTOutputOut.csv"" specified in the connection was not valid.
Error at Package: The result of the expression "@[User::CsvFullFileName]" on property "ConnectionString" cannot be written to the property. The expression was evaluated, but cannot be set on the property.
Here is what I am trying to do.
I have a foreach loop that iterates through a list of xml config files and reads the config information including the destination csv file name and does a data transformation.
So I created a flatfile connection to a csv file did my data mappings.
Created a package level variable to hold the destination file path
In the Flat file conn. manager's properties -> expression -> set the @[User::CsvFullFileName] (which even evaluates fine)
When I try to run the package I keep getting the above mentioned non-fatal error..I checked the path and it is valid. I even tried
the c:\projects\...notation and the UNC path notation...all seem to give the same error
Anyone experience this before ? any thoughts would be appreciated.
Thanks
View 5 Replies
View Related
Apr 18, 2007
Trying to figure out the best method of reading in a number of flat files, all with different number of columns and data types and outputting them to a database.
Here's the problem: They are EBCDIC encoded and some of the columns are packed decimal. I've set up one package that takes the flat file, unpacks the decimal (Using UnpackDecimal component) and then sending the rest through a second component to go from EBCDIC -> ASCII.
What I need is a way to do this for every flat file based on the schema for that flat file. One current solution is to write a script/app to create the .dtsx XML file and then execute that for each flat file. It appears like this may be possible, but I haven't gotten far enough to know for sure. So my questions are this:
1) Is there an easier way to do this (ie somehow feed the schema to the package and use it to dynamically set up the column makers and determine which columns get fed to the unpack decimal component.
2) If there isn't a better way, will dynamically creating the .dtsx XML file based on the necessary input/output columns for each flat file work? If so, what is a good source of information on this (information about how the .dtsx XML file is set up, what needs to be changed/what doesn't, etc).
Thanks,
Travis
View 1 Replies
View Related
Sep 25, 2006
Hi ,
I have a table (source file) where all sourse file(its all text file) name and location stored. i have a dts packeage, through this package all the sourse file will upload. So I need to dynamically change the source file name in dts packages.
How i can do it??????
Can any body plz help me out!
Thnaks
SAndipan
View 1 Replies
View Related
Feb 7, 2008
Good Afternoon,
So this one has been bugging me for a while and I am ready to punt...
Is it possible to dynamically create a text file destination in SSIS and then pump the results of a query stored in a variable to this text file?
So my package looks like this
1) SQL task that pulls back a list of tables to be exported
2) For Each Loop ADO enum that passes the table name to a SQL Task that builds the select...ie select * from <DTS.Variables()>
3) Data flow task that sets the command from variable from step 2
4) Text File destinaiton that is built using a varable as the connectionstring
I am delaying validation in steps3 and 4 above without any luck...basically I am curious if i can even do what I am thinking I should be able to do here...I get as far as getting metadata errors because SSIS can't seem to handle dynamically filling the pipeline with the columns from the variable/SQL statement.
Am I missing something? Is this possible?!
Thanks in advance.
Dave
View 6 Replies
View Related