T-SQL (SS2K8) :: Importing CSVs With Unpredictable Format And Name
Dec 5, 2014
I'm having a bit of an issue with a project I'm working on presently. Originally, I was supposed to be importing Excel files placed into a folder with unpredictable names, but consistent format. That was easy enough, and that's handled.
However, the project has now been expanded; I need to also import CSV files, still with unpredictable names (easy), but both the Excel and CSV files can now have unpredictable columns.
For example, I was originally told I'd be working with columns A,B,C,D,E, but now I'm working with files that can have that format, or A,B,F,G,C,D,E, or any variation as such.
For the Excel files, I can easily just do a SELECT * INTO... with a staging table and OPENROWSET, so that's no problem. The CSVs, however, I'm not so lucky with; I can't seem to use OPENROWSET for it. I could use BULK INSERT, but I can't use it in the form of a SELECT * INTO...
Compounding this is that there can be typos in the column headers, which I also just ran into
Am I missing something here? Is there means of easily adapting to the varying column header names and positions?
View 1 Replies
ADVERTISEMENT
Jun 11, 2009
I have a CSV with some columns containing quoted text with the text itself containing quotes.
I'm trying to use a Flat File Source in SSIS. I've set a double quote as the text qualifier (") and comma as the delimiter. The trouble I'm having is getting to text that contains a double quote like in the following example;
123,"xyz","Don't you hate when MS say's it's a "feature" and not a bug","more data"
SSIS is seeing the first quote as a text qualifier and the second quote as the end of the text. Because the next delimiter is not found (in this case a comma), it returns;
[staging_output [4195]] Error: The column delimiter for column "Column 3" was not found.
How I can get SSIS to treat the whole column as a single piece of text that just happens to contain quotes?
View 11 Replies
View Related
Oct 9, 2006
Using bcp to load data from a flat file into a database table will load the records in the exact same order as they are in the flat file; whereas, a DTS package may not load the records into the table in the exact same order as they are in the flat file. Why? ... the table has no keys or indexes ... the DTS package was created with MS SQL 2000 and is now being used in MS SQL 2005 ...
View 14 Replies
View Related
Sep 8, 2006
Hi There
I am having alot of trouble with the order in which messages are being delivered, here is my scenario.
I have a transaction action table :
message 1 - xml schema A.
message 2 - xml schema A.
message 3 - xml schema B.
message 4 - xml schema D.
I have the following SP:
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
AT the target i get message 1 in the queue first them message 2. But then i try this.
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 3
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 4
COMMIT
At the target i get message 4 first it has the lower queuing_order on the queue, somehow it got ont he queue first.
I have tried turning validation off on the mssage types as i thought it may have something to do with the xml, but same problem.
I then tried to do a commit after each message like this:
BEGIN TRAN
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
COMMIT
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
COMMIT
But message 4 still gets on the queue first . WTF is going on ?
The only way i can get this to work is like this.
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
WAITFOR DELAY '00:00:01'
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
Why does this work ? and why do i not have this issue with message 1 and 2? If it has soemthing to do with the xml what is it?
Please help.
View 3 Replies
View Related
Aug 9, 2015
We are importing xer formats through the wizard to sqlserver database and It takes upto 35-45 mins for each import (single project), any option to reduce the time.Is they any other import options - which can give us faster results.
View 0 Replies
View Related
Aug 4, 2015
how to import data from word document into sql server table in SSIS.
View 18 Replies
View Related
May 21, 2014
I have table1 with col1 varchar,col2 int , col3 xml , col4 bit ...best way to fetch the col3 into file (.txt or .sql) into seprate file for each col3 record . I want to export this in different files for each record if possible with date and time stamp .
View 5 Replies
View Related
Jun 17, 2014
i am trying to convert a string like this 'le dd/mm/yyyy' into a datetime.I have removed the 'le ' part and used covert(datetime, 'dd/mm/yyyy',103) to convert into datetime. This works for example for 'le 22/11/1799' but for 'le 09/11/1716' it does not work.
select convert(datetime,RIGHT('le 22/11/1799', LEN('le 22/11/1799') - 3), 103) -> it works
select convert(datetime,RIGHT('le 09/11/1716', LEN('le 09/11/1716') - 3), 103) -> it does not work
View 3 Replies
View Related
Mar 12, 2014
How to get this out put.
Details:
declare @deadline Datetime = '2014-03-23 15:30:10.000'
SELECT CONVERT(VARCHAR(30),@deadline, 100) AS DateConvert
----With this I am able to produce that like
----o/p: Mar 23 2014 3:30PM
declare @deadline1 Datetime = '2014-03-03 15:30:10.000'
SELECT CONVERT(VARCHAR(24),@deadline1, 100) AS DateConvert
--o/p: Mar 3 2014 3:30PM
--expected O/p: Mar 03 2014 03:30PM
What is the correct date format to achieve this.
View 2 Replies
View Related
Jun 5, 2014
I need to export the records of a table in xml format.
create table ##prova
( Valuta varchar(2),
Misura float
)
insert into ##prova values ('EU',1000)
insert into ##prova values ('$',2000)
The final result must be something like this:
<root>
<obs id=”0”>
<dim name=”Valuta” value=”EU” />
<dim name=”Misura” value=”1000” />
</obs>
<obs id=”0”>
<dim name=”Valuta” value=”$” />
<dim name=”Misura” value=”2000” />
</obs>
</root>
View 2 Replies
View Related
Feb 16, 2015
I have a date file with no delimiter like bellow
0080970393102312072981103378000004329392643958
0080970393102312072981103378000004329392643958
I just know 5 first number in a line is for example "ID of bank"
or 6th and 7th number in a line is for example "ID of employee"
How can I create a XML format file?
View 2 Replies
View Related
Sep 15, 2015
I am working on a ssis package ,where I have date parameters as format below
"2015-09-01-00.00.00.000000"
How to get this date format in TSQl expression in SSIS package?
View 1 Replies
View Related
Dec 4, 2014
I need to insert and update a table. One column of the table needs to conform to a format as: XXXXXXX.XXXXX
If any data is not complying to the format an error needs to be thrown.
How to implement this constraint!
View 3 Replies
View Related
Dec 21, 2014
I have a standard datetime and I need to convert it to the client specification of:
YYYY-MM-DDThh:mm:ssTZD
eg: 2009-04-16T19:20:30+08:00
I am not sure of the easiest way to do this.
The test code below gets me part of the way but I am unsure on how to get the offset on the end without hardcoding to much.
DECLARE @datetime DATETIME = '2014-12-20 12:30:00'
SELECT CONVERT(VARCHAR(30),@datetime,127)
View 2 Replies
View Related
Apr 28, 2014
I'm sending report based on below DB Mail query from mentioned data format table. But i'm getting invalid format output in csv file.
Table Data:
CREATE TABLE Sales_Data
(
ITEM_CODE varchar(25),
Price float,
Purchased_Date Datetime
[code]...
View 5 Replies
View Related
Mar 16, 2014
i m creating one google map application using asp.net with c# i had done also now that marker ll be shown from database (lat,long)depends on the lat,long i wanna display customer,sales,total sales for each makers in html table format.
View 2 Replies
View Related
Jul 17, 2007
hi
when i m importing data from excel to Sql using DTS the column which has text content was not imported as same in excel sheet. whereas a special character is appearing in between the lines. the text field contains multiple lines but the conetent is imported in single line .
ex:
ARIZONA
ALABAMA
STATE
but i m getting imported
as :
ARIZONA ALABAMA STATE
How to Format a single column while importing?
Regards
Raj
View 1 Replies
View Related
Aug 19, 2015
I have created SSRS report which has many overlapping objects, the output in PDF format seems to good but in word format it is not giving the required output.
View 5 Replies
View Related
Nov 19, 2014
We are migrating data from old DB2 systems to sql server 2012, the DATE FORMAT in those systems is in decimal format with 7 digits. CYYMMDD format.
I need to convert this into DD/MM/YYYY format.
View 9 Replies
View Related
Jun 15, 2005
I have date coming to one page as a string in the following format"May 4 2005 12:00AM"
I need to query one of my tables using this date in combination of other nondate values. How can I convert this date into valid sql server datetime format before I query a database tables
Please help
View 3 Replies
View Related
Jun 4, 2015
I have a table which stores date-of-birth in varchar 19861231(yyyymmdd). A view takes this data. I want to store this date as mmddyyyy in the view. How can we achieve this?
View 18 Replies
View Related
Dec 19, 2007
Hi,
I have a set of csv files and a set of Format Specification files for each of the csv files. I need to convert the csv files into another format of csv files as specified in the Format Specification files. All the columns of the input csv files do not have a mapping with the columns of the output csv files. How can I achieve this using SSIS ? This is an urgent requirement. Please reply asap. Thanks.
View 1 Replies
View Related
Jun 16, 2006
Hello Expert!
I need help to I translate this data...
Table "LeaveDetail"
StaffNo | StartDate | EndDate | LeaveType |
1 | 23/04/2006 | 26/04/2006 | AL |
2 | 24/04/2006 | 25/04/2006 | MC |
3 | 26/04/2006 | 27/04/2006 | EL |
1 | 30/04/2006 | 02/05/2006 | EL |
Into this format...
|Apr|Apr|Apr|Apr|Apr|Apr|Apr|Apr|May|May|May|May|
StaffNo |23 |24 |25 |26 |27 |28 |29 |30 |01 |02 |03 |04..
---------------------------------------------------------
1 |AL |AL |AL |AL | | ... |EL |EL |EL |
2 | |MC |MC | | |
3 | | | |EL |EL |
Parameter:
Date From e.g. 23/04/2006 to 23/05/2006
Using only query statement...
Is this possible??
TIA
Regards.
View 7 Replies
View Related
Nov 16, 2006
Hi All,
I am stuck at one place, where I have to convert CSV format file data into SAP IDOC format file. In SSIS we don't have any such SAP adapter (though we have .NET Data Provider for mySAP suite [SSIS SAP Adapter] but this is still not fully supported by Microsoft, plus it doesn't have feature to convert data into IDOC format) that can do this. Can someone here please provide me some pointers on any third party adapters available in market to do this job or if anyone has already developed some custom approach to achieve this task?
Your quick response on this is highly appreciated.
Regards,
Kuldeep Chauhan
View 2 Replies
View Related
May 4, 2008
dear all can anybody help me soon....
i am using visual studio 2005 webapplication based on sql server 2005 database.
i can get one date from sql using one query.
I am selecting my field based on following code CONVERT(varchar, Oman.Positions.Datum, 9) AS LastUpdate
this case my output is May 4 2008 3:19:45:000AM.....
this output is correct but from this output i want to avoid millisecond part.
ie i want the output like May 4 2008 3:19:45 AM....
how i can do this
regards
View 4 Replies
View Related
Feb 1, 2008
E.g, i have a store procedure. The start date is long date (4/15/2007 3:00pm). i want to select the start date with a particular date (short date format 4/15/2006). Thanks in advance.
View 1 Replies
View Related
Nov 27, 2015
I have a table that has a DATE field named. AccountingDate that is in the format YYYY-MM-DD. It's not a VARCHAR field. I simply want to convert this date field into the format MM/DD/YYYY and call it New_Accounting_Date.
I've played with various combinations of CAST & CONVERT but haven't been able to get it to work.
Below is my latest effort which returns the error:
Incorrect syntax near the keyword 'as'
What code would work to return a MM/DD/YYYY value for New_Accounting_Date?
Select GLBATCH.AccountingDate,
convert(GLBATCH.AccountingDate as date),101) AS New_Accounting_Date
from GLBATCH
View 11 Replies
View Related
Nov 14, 2006
Hi
I am trying get my VB6 application to insert a record into a table (SQL Express) which has a datetime column but it would not process if the data format is differ to *American Date format*.
The date() function in VB returns 15/11/2006 which is in Australian Date format (DD/MM/YYYY) according to my setting in "Reginal and Lanuage Option-> Locale 0> English (Australia)" setting.
I get the following error:
Msg 242, Level 16, State 3, Server KITSQLEXPRESS, Line 1
The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value.
The statement has been terminated.
My computer's locale is set to English (Australia) and I expect the datetime format would follow what is set in system locale
I've read an article somewhere on the net about how SQL 2005 eliminate the confusion of date conversion when read/write datetime records into a table...but it seems to me that it is still as in-flexible as MS Access
Is there a setting in the database that takes care of it?
Thanks
View 6 Replies
View Related
May 28, 2008
Hi,
I have a column date in my database which I should send it to Oracle database. The Date format in Oracle is number. I don’t know how should I convert the date to that format?
Example :
SQL FormatOracle Format
02/16/05 105046
Thanks.
View 6 Replies
View Related
Jun 9, 2006
hi siri have table hh .it has two columnsone is hhno is integer datatype another hhdoc is xml data type likehh tablehhno hhdoc---------------------------------------------------------------------------------------------100<suresh>sfjfjfjfjf</suresh>....................................101<ramesh>hhfhfhf</ramesh>..................................how to convert the xml data format into the general data format plshelp me with examples
View 1 Replies
View Related
Sep 14, 1998
I am trying to import a database into Sql 6.5 from Access 97`.
How shall I go about this ?
Thanks.
mark.
View 3 Replies
View Related
Aug 26, 2004
Hello everyone and thanks in advance.
Ok, here's the deal. I am 1 of 2 developers at my organization, the other is a COBOL programmer. Long story short, every night he sends me a 22MB XML file containing our company's inventory. This contains about 40,000 items. In C#, I'm basically looping through all the XML nodes and on each node I load up an object with the particular properties (manufacturer, SKU, serial number, etc.) and call a Stored Procedure to to save the item.
In saving the item, I'm checking to see if it was already in my inventory and I either perform an update or an insert on the table as approriate. The query does a couple of other things like retreiving the Purchase Order Number, Cubic Feet, etc. from other tables. All in all, the query takes about 1-2 seconds to complete. I've optimized the query as much as I know how (including the addition of indexes), but this still takes at least 6 hours to complete the 40,000 items. Is this the best approach? Any other suggestions?
So far the only things I've thought about changing is:
1. Break it out into multiple queries to avoid poor compilation of execution plans
2. Multithread the C# app so that I'm running like 10 queries at once.
3. Get the COBOL programmer to give me a delimited file and do a BULK INSERT on it into an empty HEAP, then run a single INSERT and a single UPDATE (using CASE statements)
Again, Thanks in advance!
View 4 Replies
View Related
Mar 17, 2004
Hi everybody:
I want to know how to import to SQL Server an XML File
Thanks for your help and quick answer.
Cristopher Serrato
View 1 Replies
View Related