T-SQL (SS2K8) :: Importing CSVs With Unpredictable Format And Name
Dec 5, 2014
I'm having a bit of an issue with a project I'm working on presently. Originally, I was supposed to be importing Excel files placed into a folder with unpredictable names, but consistent format. That was easy enough, and that's handled.
However, the project has now been expanded; I need to also import CSV files, still with unpredictable names (easy), but both the Excel and CSV files can now have unpredictable columns.
For example, I was originally told I'd be working with columns A,B,C,D,E, but now I'm working with files that can have that format, or A,B,F,G,C,D,E, or any variation as such.
For the Excel files, I can easily just do a SELECT * INTO... with a staging table and OPENROWSET, so that's no problem. The CSVs, however, I'm not so lucky with; I can't seem to use OPENROWSET for it. I could use BULK INSERT, but I can't use it in the form of a SELECT * INTO...
Compounding this is that there can be typos in the column headers, which I also just ran into
Am I missing something here? Is there means of easily adapting to the varying column header names and positions?
I have a CSV with some columns containing quoted text with the text itself containing quotes.
I'm trying to use a Flat File Source in SSIS. I've set a double quote as the text qualifier (") and comma as the delimiter. The trouble I'm having is getting to text that contains a double quote like in the following example;
123,"xyz","Don't you hate when MS say's it's a "feature" and not a bug","more data"
SSIS is seeing the first quote as a text qualifier and the second quote as the end of the text. Because the next delimiter is not found (in this case a comma), it returns;
[staging_output [4195]] Error: The column delimiter for column "Column 3" was not found.
How I can get SSIS to treat the whole column as a single piece of text that just happens to contain quotes?
Using bcp to load data from a flat file into a database table will load the records in the exact same order as they are in the flat file; whereas, a DTS package may not load the records into the table in the exact same order as they are in the flat file. Why? ... the table has no keys or indexes ... the DTS package was created with MS SQL 2000 and is now being used in MS SQL 2005 ...
We are importing xer formats through the wizard to sqlserver database and It takes upto 35-45 mins for each import (single project), any option to reduce the time.Is they any other import options - which can give us faster results.
I have table1 with col1 varchar,col2 int , col3 xml , col4 bit ...best way to fetch the col3 into file (.txt or .sql) into seprate file for each col3 record . I want to export this in different files for each record if possible with date and time stamp .
i am trying to convert a string like this 'le dd/mm/yyyy' into a datetime.I have removed the 'le ' part and used covert(datetime, 'dd/mm/yyyy',103) to convert into datetime. This works for example for 'le 22/11/1799' but for 'le 09/11/1716' it does not work.
select convert(datetime,RIGHT('le 22/11/1799', LEN('le 22/11/1799') - 3), 103) -> it works select convert(datetime,RIGHT('le 09/11/1716', LEN('le 09/11/1716') - 3), 103) -> it does not work
declare @deadline Datetime = '2014-03-23 15:30:10.000' SELECT CONVERT(VARCHAR(30),@deadline, 100) AS DateConvert ----With this I am able to produce that like ----o/p: Mar 23 2014 3:30PM
i m creating one google map application using asp.net with c# i had done also now that marker ll be shown from database (lat,long)depends on the lat,long i wanna display customer,sales,total sales for each makers in html table format.
when i m importing data from excel to Sql using DTS the column which has text content was not imported as same in excel sheet. whereas a special character is appearing in between the lines. the text field contains multiple lines but the conetent is imported in single line .
I have created SSRS report which has many overlapping objects, the output in PDF format seems to good but in word format it is not giving the required output.
I have date coming to one page as a string in the following format"May 4 2005 12:00AM" I need to query one of my tables using this date in combination of other nondate values. How can I convert this date into valid sql server datetime format before I query a database tables Please help
I have a table which stores date-of-birth in varchar 19861231(yyyymmdd). A view takes this data. I want to store this date as mmddyyyy in the view. How can we achieve this?
I have a set of csv files and a set of Format Specification files for each of the csv files. I need to convert the csv files into another format of csv files as specified in the Format Specification files. All the columns of the input csv files do not have a mapping with the columns of the output csv files. How can I achieve this using SSIS ? This is an urgent requirement. Please reply asap. Thanks.
I am stuck at one place, where I have to convert CSV format file data into SAP IDOC format file. In SSIS we don't have any such SAP adapter (though we have .NET Data Provider for mySAP suite [SSIS SAP Adapter] but this is still not fully supported by Microsoft, plus it doesn't have feature to convert data into IDOC format) that can do this. Can someone here please provide me some pointers on any third party adapters available in market to do this job or if anyone has already developed some custom approach to achieve this task?
Your quick response on this is highly appreciated.
dear all can anybody help me soon.... i am using visual studio 2005 webapplication based on sql server 2005 database. i can get one date from sql using one query. I am selecting my field based on following code CONVERT(varchar, Oman.Positions.Datum, 9) AS LastUpdate this case my output is May 4 2008 3:19:45:000AM..... this output is correct but from this output i want to avoid millisecond part. ie i want the output like May 4 2008 3:19:45 AM.... how i can do this regards
E.g, i have a store procedure. The start date is long date (4/15/2007 3:00pm). i want to select the start date with a particular date (short date format 4/15/2006). Thanks in advance.
I have a table that has a DATE field named. AccountingDate that is in the format YYYY-MM-DD. It's not a VARCHAR field. I simply want to convert this date field into the format MM/DD/YYYY and call it New_Accounting_Date.
I've played with various combinations of CAST & CONVERT but haven't been able to get it to work.
Below is my latest effort which returns the error:
Incorrect syntax near the keyword 'as'
What code would work to return a MM/DD/YYYY value for New_Accounting_Date?
Select GLBATCH.AccountingDate, convert(GLBATCH.AccountingDate as date),101) AS New_Accounting_Date from GLBATCH
I am trying get my VB6 application to insert a record into a table (SQL Express) which has a datetime column but it would not process if the data format is differ to *American Date format*.
The date() function in VB returns 15/11/2006 which is in Australian Date format (DD/MM/YYYY) according to my setting in "Reginal and Lanuage Option-> Locale 0> English (Australia)" setting.
I get the following error:
Msg 242, Level 16, State 3, Server KITSQLEXPRESS, Line 1 The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value. The statement has been terminated.
My computer's locale is set to English (Australia) and I expect the datetime format would follow what is set in system locale
I've read an article somewhere on the net about how SQL 2005 eliminate the confusion of date conversion when read/write datetime records into a table...but it seems to me that it is still as in-flexible as MS Access
Is there a setting in the database that takes care of it?
I have a column date in my database which I should send it to Oracle database. The Date format in Oracle is number. I don’t know how should I convert the date to that format? Example : SQL FormatOracle Format 02/16/05 105046
hi siri have table hh .it has two columnsone is hhno is integer datatype another hhdoc is xml data type likehh tablehhno hhdoc---------------------------------------------------------------------------------------------100<suresh>sfjfjfjfjf</suresh>....................................101<ramesh>hhfhfhf</ramesh>..................................how to convert the xml data format into the general data format plshelp me with examples
Ok, here's the deal. I am 1 of 2 developers at my organization, the other is a COBOL programmer. Long story short, every night he sends me a 22MB XML file containing our company's inventory. This contains about 40,000 items. In C#, I'm basically looping through all the XML nodes and on each node I load up an object with the particular properties (manufacturer, SKU, serial number, etc.) and call a Stored Procedure to to save the item.
In saving the item, I'm checking to see if it was already in my inventory and I either perform an update or an insert on the table as approriate. The query does a couple of other things like retreiving the Purchase Order Number, Cubic Feet, etc. from other tables. All in all, the query takes about 1-2 seconds to complete. I've optimized the query as much as I know how (including the addition of indexes), but this still takes at least 6 hours to complete the 40,000 items. Is this the best approach? Any other suggestions?
So far the only things I've thought about changing is: 1. Break it out into multiple queries to avoid poor compilation of execution plans
2. Multithread the C# app so that I'm running like 10 queries at once.
3. Get the COBOL programmer to give me a delimited file and do a BULK INSERT on it into an empty HEAP, then run a single INSERT and a single UPDATE (using CASE statements)
