There's Just No Escaping It... Data Containing Quotes Ruining CSV Files
Jul 11, 2006
Much to my surprise and dismay, after burning hours and hours developing various SSIS packages to extract data from my SQL 2005 DB to CSV flat files, I found out the hard way that SSIS doesn't properly escape quotes (") in the data (my output file is required to have quotes as text qualifiers).
It seems like I'm pretty much forced to create derived columns to test for the existence of quotes in my data, then either escape them or replace them with another character.
This is really NOT COOL. Not only will this significantly hinder performance, it makes my packages a heck of a lot more complicated (especially when I'm exporting lots of text columns).
Am I doing something wrong, or does SSIS really not escape the text qualifier in a delimited flat file? If so, someone PLEASE tell me this will be fixed in the very near future.
I have a parameterized query. The parameters contain data from my tables. Some of the parameters could include single quotes. The single quotes are wreaking havoc in my parameterized query. How can I replace single quotes with double quotes inside of my SQL stored procedure?
I know that it's something similar to REPLACE(@variablename, '''''', ''''''''), but I can't get the number of quotes right.
All of the examples that I am seeing are converting the quotes inside of an application. This is not an option for me, as I am calling this stored procedure from a SQL job that will run daily.
I have some code (C#) that runs an SQL update query that sets thevalue of a column to what the user passes. So, this causes an errorwhen anything the user passes in has a ' character in it. I'm surethere's other characters that'll break it too. So, I was wondering,how do I get around this? Is there some commonly accepted regexpattern that will make the value safe to run in an SQL query? How canI take care of any values that need to be escaped?I'm not using any fancy ado.net objects:string sql= [whatever the user passes in]SqlConnection connection = newSqlConnection(ConfigurationManager.ConnectionStrin gs[Utils.GetConnectionString].ToString());connection.Open();SqlCommand command = connection.CreateCommand();command.CommandType = CommandType.Text;command.CommandText = sql;try{int result = command.ExecuteNonQuery();if (result != 1){Response.StatusCode = 500;Response.Write("The file has been uploaded, but wecould not update the DB");Response.End();}}catch (InvalidOperationException){Response.Clear();Response.Write("error");Response.StatusCode = 500;Response.End();}connection.Close();
I am running into an issue with the SSIS when I try to load a CSV file that contains double quotes wrapped around a field (CSV files have double quotes when field contains a comma; example: "Streams, Inc")
i need to add the double quotes in all the records from start and end.
source data col1 col2 col3 col4 1 abdul this is email it was very good ,and very relative posts. Target data col1 col2 col3 col4 "1" "abdul" "this is email" "it was very good, and very relative posts"
I've got a flat file data source, that is to large to edit with most Windows apps on my server that contains both single and double quote characters that I need to load in a varchar column.
So I attempted to do it with a Replace in data transformation, but I can't get SSIS to allow me to use a variable or pair of single or double quotes within the replace.
If I don't replace the single quote characters with a pair then the records containing these characters all end up in my failed records output file.
Here are 5 example property legal descriptions from my FLAT FILE data source:
COM 441'6" N OF SW/C OF NW4 OF SEC 22-29-20 ELY1340' N200' CROSSING THE CNTR OF TR AT 100 WLY1240' S200' TO POB CONTAINING 6 3/10 ACRE MOL
N 50' OF S 330' OF W 122' OF E 735' OF SW4 OF NE4 OF SEC 28/28/18 A/K/A LOT
Running a SQL2K-server on a Win2K(adv) I have sort of an enigma (to me atleast!):
One of my SPs sometimes seems to hang. Arrgh. This is of course not the total extent of my problem - naah, I've had/created my share of faulty SPs which hang on a regular basis - no, the enigma is that the code inside my SP *never* hangs when executed from the Analyser? The test is done using ctrl-a/copy&paste - remove 'Create...' and putting a 'Declare in front of the only parm, then inserting a select parm='value_of_parm'. As far as my knowledge goes this 'running code from SP in Query Analyser' should introduce no extra room for slack (which in turn could help me explain why the code works in Query Analyser)?
Btw, when I say 'hang' it's not the whole truth - using the debugger I can see the code (mostly a bunch of updates on one table) being executed - but extremely slow. The code takes approx. 1 min in Query Analyser but hasn't finished after 60 mins when executing the SP.
A little info on the SP: The user runs another SP which gathers some data from an Oracle DB and inserts these data into a SQL2000 table. Then the user activates the above flamed SP in vain hope of marking the newly fetched data with a statusid. The problem allways arises when the user tries to let the SP engage newly fetched data - and rarely when engaging old (already marked, hence 'remarking') data. I suspect this could be a clue for anyone with some insight?
I'm looking for 1) a solution, 2) a finger pointing out the bug(s) 3) any suggestions on how to do further research into this evil SP!
I have text data files from a third party and they use comma as field delimiters and enclose the text for each column in double-quotes. Not a problem for most of the data files until they start sending files where there is " within the column values. SSIS package fails with the error:
The column delimiter for column "Column 1" was not found.
Any ideas on how to resolve this issue will be greatly appreciated.Thankspcp
I had a procdure in SQL 7.0 in which I am using both single quote and double quotes for string values. This proceudreused to work fine in SQL 7.0 but when I upgraded SQL 7.0 to SQL 2000, this proceudre stopped working. When I changed the double quotes to single quotes, it worked fine.
I have data in a trace file, and I need to extract some info such as phone number.The problem is the phone number could be varying lengths, and various positions in the row.
For example:
@City='New York', @Phone='2035551212' (10 characters, no dashes) or @City='San Francisco', @Phone='918-555-1212' (12 characters, with dashes) or @City+'Berlin', @Phone='55-123456-7890' (14 characters, with dashes)
I can use CHARINDEX to search & find @Phone=' so I know where the phone number starts, but stuck on a programatic way to find the data between the quotes since it can vary.
I have to import a flat file with commas and double quotes as the text qualifier in a SSIS package. However, when I try to import the data into a table, the data moves the information to right, therefore, the last field will capture mulitple fields worth of data. When I create the Connection Manager for the flat file, I have the format as [Delimited] and Text qualifier as ["]. I do not check the unicode button, but under Advanced, make each field a Unicode string [DT-WSTR]. I have included a sample of my data below.
I have a CSV file that encloses all the data fields with quotation marks. Here is a sample:
"08/01/2007","3","021200012","123","0.03"
Is there any way in SSIS that I can tell the Flat File wizard to ignore the quotation marks? I don't want to import the quotes in the database since that will really mess up other applications that need to use the data.
Hi, i'm having problems executing the followingSET @SQLAH = 'SELECT sub_id WHERE 'SET @SQLAH = @SQLAH + 'VENUE_TYPE = Hotel'EXEC(@SQLAH)Its getting stuck at Hotel. I realise that it should include an apostrophe either side like so:..SET @SQLAH = @SQLAH + 'VENUE_TYPE = 'Hotel' '..But this escapes the string, how would i escape an apostrophe in a string?I thought maybe:SET @SQLAH = @SQLAH + 'VENUE_TYPE = 'Hotel' 'But no joy :confused: Thanks
myCommand2.CommandText = "INSERT INTO testimonials(name,email,testimonial,approved,time) VALUES ('" + exp.escString(txtName.Text) + "','" + exp.escString(txtEmail.Text) + "','" + txtTestimonial.Text + "','" + false + "','" + DateTime.Now + "')"; ok the form has a field txtTestimonial.Text these strings work when i submit them thoughthats all folksthat"s (double quotes) that\"sthat\"s these failthat's that's that\'s that\'s tried up to 7 just to make sure Exception Details: System.Data.SqlClient.SqlException: Incorrect syntax near 's'.Unclosed quotation mark after the character string ')'. I figure the " works because it ends up being ' " ' in the sql statement and it doesnt mind thatI tried researching this. I could not find any info stating that perhaps sql escapes things differently. I was on the assumption that in the sql itself the database would reconise the before the '.I read i can use html decode but it seems that then i will have to undecode everytime I read from the database and it could be a major pain if I am using a datagrid or something and a big mess. Any help would be greatly appreciated. Jim
Y'all:I am needing some way, in the SQL Server dialect of SQL, to escape unicodecode points that are embedded within an nvarchar string in a SQL script,e.g. in Java I can do:String str = "This is au1245 test.";in Oracle's SQL dialect, it appears that I can accomplish the same thing:INSERT INTO TEST_TABLE (TEST_COLUMN) VALUES ('This is a1245 test.");I've googled and researched through the MSDN, and haven't discovered asimilar construct in SQL Server. I am already aware of the UNISTR()function, and the NCHAR() function, but those aren't going to work well ifthere are more than a few international characters embedded within astring.Does anyone have a better suggestion?Thanks muchly!GRB-----------------------------------------------------------------------Greg R. Broderick Join Bytes!A. Top posters.Q. What is the most annoying thing on Usenet?---------------------------------------------------------------------
I proposed on a new server that we separate Data Files, Log Files, tempDB, Backups, etc. onto separate LUNS on a SAN with High Speed Solid State Drives.I was told that with the new technology with solid state SAN's that it would decrease performance and that it did not work the same way as it did when you had RAID 5's etc.I thought that if things were cared out correctly by a SAN Administrator they would know how to configure for optimal performance.
I'm working on a script to produce XML. There are two parts to the XML: an envelope (called "message") and a payload ("payload). Here's an example of what the result should look like:
(Note that I had to add a space between the '&' and 'lt' and 'gt' to get it to post in the forum without the forum converting them back to < and >!)You can see that SQL escaped the < and >. Also the CDATA escape sequence is missing.I want to see < instead of & lt and > instead of & gt. Also, I need the CDATA escape sequence.
We are facing a problem while passing a string containing the "&" character into Full Text search CONTAINS predicate. The records that do have this character are not being returned by the search.
I'd like to raise two questions:
1) Is there any special way to escape this character? 2) Does FTS index it at all?
We have tried all known (to us) ways of escaping like doubling the character, using char(38), using ESCAPE etc..Nothing seem to work. Any help would be appreciated.
We have a .NET drop down, which gets populated as the user types in letters(last name). If the user types in the single quote we get the error about not escaping the single quote. Question is, which way would it be easier to fix, in the .NET code or in the SQL procedure? I am not to sure if we have full access to the source code since that is a 3rd party control, so if that is not feasible how would I fix that in the stored procedure? This is the current proc that we are using:
Code Snippet select @str = 'SELECT DISTINCT TOP ' + @Top + ' e.DisplayName as DbComboText, e.EmployeeID as DbComboValue FROM DepartmentDirectory.dbo.Employees ee INNER JOIN DataMart.dbo.Employees e ON ee.UIN = e.UIN WHERE e.LastName like ''' + @LastName + ''' AND e.FirstName like ''' + @FirstName + ''' ORDER BY e.DisplayName'
Hi! I'm using replication with two database on SQL 2000,when begin, the log files size is 50mb and the data files size is 150mb. But now the log files size is 2Gb and the data files size is 4Gb. I would like to decrease the log files and the data files ??? How do i do this??? (I using Truncate and shrink doesn't change ) Thanks!!!
Hello all. Before my arrival at my current employer, our consultantsphysically set up our MSSQL 7 server as follows:drive c: contains the mssql enginedrive d: contains the transaction logdrive e: contains the data filesNo filegroups were set up and the data files consist of only 1 largephysical file. Currently, our data file is >10GB. When I was trained onthe physical aspects of sqlserver, I was told to never create physical files[color=blue]> 2048MB each. If I did, I could expect inefficient physical storage of[/color]data and slower performance (due to the OS).Our server has 2 RAID-5 arrays. Drive c: and e: are located on the firstarray and drive d: on the second. We're running Windows 4.0 NT Server SP6with NTFS.Can someone comment on the use of 1 single large data file vs. more smallerdata files?
I have an MVC asp.net application that stores many records in a table on sql server, in its own system. used the system for 2 months, worked fine accessing, changing data.
Now that other users are logging in? there is cross coupling going on. one user gets the data from another users sql search.
In the mvc app it had used the get async method to read the ID record from the db, i set that to synchronous. no effect; the user makes their own login id but that does nt matter either.
i am really in need of help. i have a text file consiting of some data.i want to update my database from that text file periodically say 12 hours.the text file is being updated by another server program in every 12 hours can any one help me in this case? i am lost for this scenario?? help me please.....
Hi,I have(had) an old Win2k Server server with about 30 web site databases(SQL 2000) that just went under due to hardware problems. Thankfully, Ihave backups of all the databases plus the MDF and LDF files from thehard drive.I want to move all of these sites and their data to a newer server(Win2003) running SQL2000.What's the best way to copy the database from the old server hard drive(now mounted as an extrnal drive to a local machine; I'm currentlyFTPing all of the web site directories from it to the new server)?Just upload the original data to the new server and then mount the MDFand LDF files within the new SQL server? Or do I restore the backupfiles in the new SQL2000?All of my previous data migrations have been DTS operations from onelive server to another, so no experience with either of the abovescenarios. I'll certainly have a lot more experience at one of them bythe time this weekend is through.Thanks for any help you can offer.
below is the code of my sp it's giving my wrong result: 1 instead 0 and vice versa could you please check what's wrong with it: i think there is something related to @var2 in the code colored in orange.
Thanks!
Code:
ALTER PROCEDURE sp_name ( @param1 int = NULL, @param2 int = NULL, @Result int OUTPUT )
AS
DECLARE @var1 AS varchar(255) DECLARE @var2 AS varchar(255) DECLARE @x AS varchar(1000)
IF @param1 IS NOT NULL BEGIN SET @var1 = ( SELECT t1col1 FROM tabl1 WHERE t1col2 = @param1)
SET @var2 = ''
DECLARE crsr CURSOR FOR
SELECT t2col1 FROM tabl2 WHERE t2col2 = @var1
OPEN crsr FETCH NEXT FROM crsr INTO @x
WHILE @@FETCH_STATUS = 0 BEGIN SET @var2 = @var2 + '''' + @x + '''' + ', '
FETCH NEXT FROM crsr INTO @x END
CLOSE crsr DEALLOCATE crsr
SET @var2 = SUBSTRING(@var2, 1, LEN(@var2) - 1)
END ELSE BEGIN SET @var2 = (SELECT t2col1 FROM t2 WHERE tb2col3 = @param2) SET @var2 = '''' + @var2 + '''' END
PRINT @var2
IF EXISTS (SELECT * FROM tabl3 WHERE t3col1 IN (@var2)) OR EXISTS (SELECT * FROM tabl4 WHERE t4col1 IN (@var2)) BEGIN SET @Result = 1 END ELSE BEGIN SET @Result = 0 END
Your help would be appreciated if you could give me some idea about why SQL 2000 takes single quote(') as a default for string searching instead of double quotes("). I have couple of stored procedures which were developed with string comparison with double quotes in SQL 6.5. If any one knows how I can change default as a double quotes(") instead of single quote(') in SQL 2000, would be great. Thanks
select * from tblCalls where TNum = [NUMBER] and Street like '[TEXT]'
The problem is that the Street field contains street names with quotes (ex. DE L'ACADIE). When I run this query using this street name, it doesn't work because of the ' between the L and the A. How can I tell SQL to "ignore" the ' in the text?
Hi, I am creating a flat file connection to a .csv file In the columns section of the flatt file connection manager editor, I am not sure why the texts in the .csv file are shown with double quotes arouond them. They do not have "" in the .csv file. Thanks
