Import Files Based On Whether They Have Already Been Imported Or If They Have Changed
Feb 9, 2008
Hi there,
Today I tried out Integration Services and after a couple of hours of confusion I am impressed by the power of this product. I was wondering if I might get some help on the 'best practice' for the following requirements:
I have a foreach container that scans a directory and calls a data flow task for each file in the directory. The files import to the database fine, but I want to modify the procedure so it only imports files that have not yet been imported. There are a couple of scenarios:
- a file is created in the directory
- a file is modified in the directory
In both cases I want to insert (or reinsert) the file. How can I modify my package to accomodate this behaviour? Storing the filename is an option, but I am not sure how to also bring in the file creation/modified dates.
I have a record in an Excel format (Excel 2010) and I would like to bulk import that into SQL Server 2008 and also while importing, SQL Server will automatically create a new table based on the header fields or row of the source file.
I am not sure if SQL Server 2008 has this capabilities.
hi, I am importing data daily to many tables, I want to keep track of the #of rows for each import process. I already have created a trigger as follow: CREATE TRIGGER tr_bcp_log ON dbo.A FOR INSERT AS
declare @name varchar(30), @row_count int
select @name=name , @row_count= @@rowcount from inserted
insert into bcp_tracks (name,row_count) values(@name,@row_count) GO
The problem is that I am getting a row for each inserted row in table A.for instance if I have 500 rows in table A, I will get 500 rows in the log table like this table_name,#of rows A 1 A 1 A 1 etc up to 500 rows for table A
This is not what I want, I want to capture the num of rows for every bcp process , so in the log table I want to see the following : table_name, #of rows A 500 B 600 C 450 A 250 etc
I am trying to create a stored proc that will update exactly one row. Simple. For insurance purposes, I want to create some logic that will rollback the entire transaction if more than one row is updated.
I know that I could force the primary key into the WHERE clause, but I was looking for some logic that will allow me to bypass that.
I want to import via DTS to big table only the records that changed last day. can i do it without a time field? because this is a key table for DWH and not a fact table. THX Inon
Hi all (newbie @ asp.net)(oldie @ ASP 3)What is the purpose of using an attached MDF database files in the App_Data folder on a web site as to importing it into the SQL server directly or creating it on the SQL server. Does a mdf database attached file purely use the SQL server as a connection interface.Is it something similiar to DSN(ODBC) Connections for ms access databases.
In the For Loop, How to Iterate from Older flat files to Newer flat files based on File's Timestamp. If there are some older files in that folder, it should be processed first and then continue with the newer one.
I tried to open mdf file without any success. Then, all database files icon were changed or grayed out. How can i repair or replace the file to it's original file? ex... the file named "event.mdf" has now a wrong icon not a database icon like before.
Brief overview...Running SQL Server 2003 Server Enterprise 64 bit - All Service Packs and patches current SQL Server 2005 Enterprise Edition 64 bit Build Microsoft SQL Server 2005 - 9.00.3054.00 (X64) Mar 23 2007 18:41:50 Copyright (c) 1988-2005 Microsoft Corporation Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)
I cannot import any SSIS packages nor crete any new folders under stored packages. I hve googled the news groups and looked at BOL to no avail. HELP!!!!
I need reflecting changes of csv file in oracle DB. Suppose, Â I load single csv file in oracle DB which contains 10 rows. After some time, I have loaded another CSV file which has the modified row of the previously loaded csv file. So, how can I capture the CSV changes and how it is going to get reflected in oracle DB?There is no unique column in csv file to identify particular row.
Hi i have a question. How can i import dump from a text file , to an exsiting database, i´m will not useing DTS, i what to use BULK cmd in sql queryanalyser etc.
I have SQL 2000 running under windows 2000 server.
Question: I'm receving CSV files from the field. I would like to built SQL database from daily CSV files. I've been doing this manually :( however, is there anyway I can do this task automactically so that once new daily CSV file ~200KB appear in the folder, progarm automatically insert recent file to SQL database...:) Do I need to write VB progarm for this? Please help...
I have created a stored procedure to handle xml data. One of the inputs to this procedure is xmldata which is basically the contents of the xml in a file. Now I would like to create a SSIS package that takes the xml files from the network and uses this SP. Should I have a file system task inside a for loop container? How do I get started on this please? Thanks
I have created a stored procedure to handle xml data. One of the inputs to this procedure is xmldata which is basically the contents of the xml in a file. Now I would like to create a SSIS package that takes the xml files from the network and uses this SP. Should I have a file system task inside a for loop container? How do I get started on this please? Thanks
Hi, I have about 300-400 XML files I want to load in my SQL database (2005). The following code will load one (1) file. How do i do a mulitple collections? INSERT INTO MEL (DATA) SELECT * FROM OPENROWSET (BULK'C:TempCHAPTER1.xml', SINGLE_BLOB) AS TEMP Thanks,
I'm new to SQL server. I would like to ask how to do the following. Now I have many CSV files in a directory, which are uploaded through ftp from my client computers daily. How can I automatically import those CSV files to the SQL server? Can I do it by a script or a stored procedure or something else?
I have data in excel files I want to import it to existing SQL Server database. I can use SQL Server built in import tool but before Importing, the data needs to be validated. What is the best way to do this?
I have 8GB of text files which are basically log files from the past few years. There is 24 text files per directory which are labeled for every day (so they are not all in 1 folder). It would make reading them much easier if I could import them to SQL but I only seem to be able to import 1 at a time? (with the wizards :eek: )
Surely there is a way to mass import without all the costly applications that google searches give me? cheers :P
I have a job that calls about 5 DTS's. Each DTS imports a text file. The problem is that some of the files may not be there every day. How do I exit the DTS without an error if the file does not exist, but I want my job to keep running...
I have a script which imports the contents of a csv file from our CRM system and updates a table in my database. This works OK but the problems I have are that a) sometimes there is more than one file in the folder, and b) that I wish to move any csv files that have been imported into an archive folder. The csv files arrive with a time/datestamp and I currently rename them manually to FREXPORT before importing (the name is in the format FREXPORT_20141101_1217.csv).How do I:
1) get it to process the file without me having to manually rename the file(s) each time, 2) if there is more than 1 file in the folder process all the files and 3)move the correctly processed files to an archive folder which is: importarchive?
Ultimately, I would like the script to be run as a scheduled job, so it also has to deal with the fact that sometimes there will be no files to import too.
I have several hundred .csv files that have a specific cell that I need to get into a SQL table. These files are named after the date on which they were created...ie 8252005 would be todays date. Im looking for a way to import this cell to SQL... the same cell in each file.... Thanks for any help
I have a task to import *.log files in to the Microsoft SQL Server and need help in this regards.
This import should be the best table structure/format for the web log (considerations for import time, size, indexing, querying, reporting, etc.), and best utilize all available space on the server
I'd like to set up a dts package to import .csv files and on testing one .csv with only one row I am getting this error:
Error Source: Microsoft Data Transformation Services Flat File Rowset Provider
Error Description: Too many columns found in the current row; non-whitespace characters were found after the last defined column's data.
data set from .csv: -31,0,"CENTER","","","64 ROCK DR. #4100","BETH","MD.","20417","0470-5",30109,10598,"52492000 083","1018","406800","6896CE","7007","G00887",1,13983.24,2139.66,42741.9,150101.36,830317.41,1562843.99,30128.45,2,506,0,0,0,0,0,0,37.87,1,0,0,476.39,472.15,0,1191.12,0,90073.46,43038.97,0,309926.59,-14880.49,20060102,20090101,"A","","",""," ","C62454",0,0,0,0,0,0,0,0,8.39,0,0,0,0,"",0,"00000000",""
i'm trying to find out if SSIS is a right toolbox for importing of many SPSS-files into the large market research data warehouse.
A simplified version of SPSS-datasheet looks like this:
- many columns (up to 3000). some of them are interview attributes (InterviewID, Date, etc.) and variables, representing survey questions
- relative few rows with "cases" or "interviews"
- in cells there are answers to the questions (in columns) by respondent (in rows)
As first step i would like to unpivot the dataset in the following way:
- Pass-Through: all interview attributes, that shouldn't be unpivoted
- Input Columns: rest 2990 columns
- Destination Column: Question
- Pivot Key Value Column Name: Answer
It works allright, except that i has to manually define destination column for all 2990 input columns, which takes a lot of time (multiplied by the number of SPSS-files i want to import). Is there a way to automate this (default value for destination column and/or scripting?)
Does anyone have a completed script that will import Perfmon logs from csv format into an existing database that follows the SQL Log File Schema EG; CounterData, CounterDetails, DisplayToID tables accurately. I don't know enough T-SQL to get it right. Thanks for any input you may have.
Thank you for the help and support you have given me. Now i am confronted with a new problem. I have to import some textfiles to SQL Server Tables . I have to create a tool to automate the porting using C# .The columns in textfile is seperated with pipe"|" . I f any body knows this please help me .
Thank you for the help and support you have given me. Now i am confronted with a new problem. I have to import some textfiles to SQL Server Tables .The columns in textfile is seperated with pipe"|" . I f any body knows this please help me .
Hi, I try to import csv files to Sql Server using .net. The code is as following: string strCsvConn = @"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=;Extended Properties='text;HDR=Yes;FMT=Delimited(,)';"; using (OleDbConnection cn = new OleDbConnection(strCsvConn)) { string strSQL = "SELECT * FROM " + strFileName; OleDbCommand cmd = new OleDbCommand(strSQL, cn); cn.Open();using (OleDbDataReader dr = cmd.ExecuteReader()) { // Bulk Copy to SQL Server using (SqlBulkCopy bulkCopy = new SqlBulkCopy(strSqlConn)) { bulkCopy.DestinationTableName = strSqlTable; bulkCopy.WriteToServer(dr); } } } And the data is as following (simplified): Model,Serial AFICIO 3045,K9464900965 AFICIO 3045,K9464900932 Fax 5510L,A3761290041 Fax 2210L,A4978800008 AFICIO 3025,K8565201014 AFICIO 3025,K8565102398 The result of the 2nd column is: 9464900965, 9464900932, null, null, 8565201014, 8565102398 - either the first character is missing or the whole entry is missing. One more weird thing is that some other files work fine, though I am not able to tell any difference between them. Any idea is hoghly appreciated. shz
