Truncate Value (of Datatype Dt_ntext) Being Imported
May 14, 2008
Hi All,
I am importing data from an oracle db and one of the columns is a nclob. I would like to truncate the value if it is more than a certain length. I want to use a derived column transformation. I cannot use the len and substring functions because it is not a string. I could convert it to a unicode string and use those functions, but I am afraid there is a size limitation (4000 characters) on it. Is there smth I am missing? Can anybody please suggest any ideas?
Thanks a lot.
I have another problem, this time with a flatfile source component. The file is pretty simple organized and in fact everything works as expected.
The file itself comes from an FTP server, so it is not available at designtime. For setting up the flatfile source i downloaded it, but afterwards i delete it, set 'dalayvalidation' to true and try to execute.
FTP download works fine, the file is there but then i get the errormessage:
Copier ErrorCodes [586]: The data type for "output column "Flat File Source Error Output Column" (610)" is DT_NTEXT, which is not supported with ANSI files. Use DT_TEXT instead and convert the data to DT_NTEXT using the data conversion component.
When i try to set the datatype of the Error Output Column (610) to DT_NEXT i only get an errormessage telling me that this would not be a valid value for this property.
The only chance i have to get it working again is to completely delete the flatfile-source, readd and setup it again, then everything works... but only until i remove the files and try to download them via FTP. It's strange :-( and annoying.
I import data from multiple excel files into SQL DB. I have trouble with fields that could contain >255 chars.
If I have the col type = DT_Ntext in my Data Flow, the package fails for files that do not have any values >255 chars.
If I have the external coltype=dt_wstr and the output coltype=dt_wstr(4000) the package fails if the file contains any value >255 chars.(Implicit conversion does not occur, as expected).
I worked around by adding a dummy first row with >255 chars.
Is there a way to use a cast function to solve this prob? I tried using Select dt_ntext(fieldname) from Sheet1$, but that does not work.
Is there some clean way to get around this problem?
The IC column referenced in the script below is DT_NTEXT, so when I run it the output value column gets "Microsoft.SqlServer.DTS.Pipeline.BlobColumn" instead of the individual codes (separated by "*"). Also, there is only 1 output row per input row, instead of 1 output row per code.
I found some references to the GetBobData() method, but replacing Row.IC.ToString() with Row.IC.GetBlobData(0, CInt(Row.IC.Length)).ToString() puts "System.Byte[]" in the value output column. There is still only output row per input row.
So how do I convert the IC column to a String that can be Split()?Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer) Dim seq As Integer = 0 For Each code As String In Row.IC.ToString().Split("*"c) 'Add a row to the output buffer: Output0Buffer.AddRow()
'Preserve columns from the input buffer: Output0Buffer.itemid = Row.itemid Output0Buffer.attributeid = Row.attributeid 'Output0Buffer.IC = Row.IC
'Add output columns: Output0Buffer.seq = seq Output0Buffer.value = code seq += 1 Next End Sub
I have a script component that I have written and works as long as the Output columns on the script are string types. When I change the output column type to text (since the size could be essentially unlimited) it gives an error in the script component that the property is read only.
Here is the code line that fails with Property Payments is read only.
Output0Buffer.Payments = fieldValues(i)
If I change the column payments to DT_Wstr it works without issue, but I want to use text incase the value is large.
Here is the error if you try to run the actual script even though I know it has an error.
Error at Data Flow Task [Script Component [85]]: Error 30526: Property 'Payments' is 'ReadOnly'. Line 86 Column 13 through 69 Error 30526: Property 'Ops' is 'ReadOnly'. Line 155 Column 13 through 65
Error at Data Flow Task [Script Component [85]]: Error 30526: Property 'Payments' is 'ReadOnly'. Line 86 Column 13 through 69 Error 30526: Property 'Ops' is 'ReadOnly'. Line 155 Column 13 through 65
Error at Data Flow Task [DTS.Pipeline]: "component "Script Component" (85)" failed validation and returned validation status "VS_ISBROKEN".
Error at Data Flow Task [DTS.Pipeline]: One or more component failed validation.
Error at Data Flow Task: There were errors during task validation.
The documentation on the fuzzy lookup transform mentions that only columns of type DT_WSTR and DT_STR can be used in fuzzy matching. I interpreted this as meaning that you could not create a mapping between an input column of type DT_NTEXT and a column from the reference table. I assumed that you could still have a DT_NTEXT column as part of the input and mark this as a pass through column so that it's value could be inserted in the destination, together with the result of the lookup operation. Apparently this is not the case. Validation fails with the following message: 'The data type of column 'fieldname' is not supported.' First, I'd like to confirm that this is really the case and that I have not misinterpreted this limitation.
Finally, given the following situation
- A data source with input columns
Field_A DT_STR Field_B DT_NTEXT
- A fuzzy lookup is used to match Field_A to a row in the reference table and obtain Field_C.
- Finally, Field_B and Field_C must be inserted into the destination.
With a database size of almost 2 GB, I run the 'truncate table eventlog command' which completes successfully, but the database size only decreases by about 10 MB so stays too large - indeed the number of rows in the eventlog table is minimal, but the otehr tables in this database don't show such an amount of tables large enough to cause the size issue either. What could be the reason and how can I reduce it (possibly truncating another table but then which one, how could I determine which is too large and needs truncating?).
insert into scn_transaction (sourceSystemName) values(@sourceSystem);
SELECT @txOut = @@identity
Whose purpose is to perform an insert into a table and return me the identity value of the inserted record, which I'll then use throughout the rest of my package. The identity column in the inserted table is numeric(18,0).
I execute the stored proc with the following sql with an OLE DB connection manager:
exec sp_newTransaction ?, ?
The first parameter is a string variable from earlier in the package, and the second is the output parameter. I have the following parameter mappings to the execute sql task:
The proc is correctly called, and the row insesrted, however I get a type conversion error when SSIS attempts to map the return parameter to my package variable... I've tried all sorts of combonations, and can't seem to get it to execute.
At one point I wasn't returning a numeric, but rather an int from the stored proc, and all was well until I went to use the variable in a derived column later in the package, and the type was converted quite incorrectly (a 1 was 77799789080 or some such), indicating a type conversion error likely related to the encoding of the number.
I'd like to keep the datatypes as numeric and make ssis use those - any pointers are greatly appreciated as to what type my package variable should be to allow proper assignment of a sql server numeric type to it.
Please forgive the elementary nature of my question, but could someone please explain the differences between these two database backup types:
1. Log backup 2. Log backup no truncate
From what I understand and have read, the "no truncate" backup method keeps the entire transaction log indefinitely. Using the truncation method, the transaction log is either 1) compressed or 2) cleaned up so that any completed transactions are removed from the log. Which one of these is true?
And, for the big question: is it better to run a backup of the transaction log with truncation or not? Our current backup scheme is similar to the following:
Full backup every 24 hours transaction log backup every hour with no truncation
Should we insert a truncation backup somewhere in here? What is the danger of removing (or compressing) parts of the transaction log? Will this affect the restore process?
Hi, I have imported 3 DTS from SQL 2000 to the SQL 2005 server. The wizard went fine, everything is ok. When I close the wizard window, I cannot see any of them. If I reimport it, it asks confirmation to override it. Where are the 3 DTS in the management Studio if they are not under Legacy DTS? Thanks
I have a field in a table that stores date of birth. The field's datatype is char(6) and looks like this: 091703 (mmddyy). I want to convert this value to a datetime datatype.
What is the syntax to convert char(6) to datetime?
I am using the following SQl statement in a DTS vb file. The data is coming from an Excel spreadsheet and being put into a SQl server table.
oCustomTask1.SourceSQLStatement = "select `Order No`,`Country`,`Desc`,`Amount` from `Sheet1$` WHERE `Order No` = 1 "
I want to validate the data being put into the table to ensure no duplicates records are entered UNLESS the record has changed in any way. E.g. If record 1 had a column called "NumberOne" which changed to "Number1" change this information and move the original value to another table.
Hi. I am working on a dynamic website and backed up the data from our database (SQL Server) into a .bak file from our server. I want to develop the site locally on another machine (doesn't have SQL Server, but MSDE) using the database so I restored the database backup using RESTORE DATABASE FROM... (in the DOS prompt). I found a tutorial how to do this at
It did the restore successfully (or so it said) but all that it restored was the database structure (table structure, user information, etc.) but not the actual data in the database. I am using Specify (www.specifysoftware.org) as a database manager.
Could anyone help me out? Can I not go from SQL Server to MSDE? This doesn't seem to be the case because the structure restored okay. What should I do to correct this problem?
I am developing a DB with others in my group. When I import tables created on other servers to my server, the primary key and other properties do not import with the tables. Can anyone explain why this is happening? Is there a setting I have over looked?
I am in the process of importing data that is in a text format to thesql server, I want to add couple of fields and insert general data inthe fields added, this data is going to be similar to all the recordsimported. Your help in this regard will be greatly appreciated.
I am making web application using Asp.net C#(Visual Studio2005). And Sql server 2000 as a back End.
Actually I am fetching data from other database(named as retailexcel but in same server). I have wriiten a view (in retailexcel database) to get required data from four table of other database. My problem is when I fetch those data(about 40000 to 50000 record) It taking to much time. I tried catch feature of asp.net2.0. Time reduced but not significantly. So I am trying to import those tables to my database (named Inventory) from that retailexcel.
Now problem is every day or two, record of those tables changes. How can I keep update my tables (which are importing from retailexcel)
We have an OLTP application with multiple one to many tables relationships that are edited client side. For Example the primary table has descriptive components. One of the multiple table have things like parts and quantity ordered linked by a record number back to the primary table.
When our client transmits their transaction, they actally running a batch job which sends all the client side tables to the SQL Server and then we are trying to issue a stored procedure server side to push this data into our 'dbo' owned server side master tables.
Our problem is generating the fully qualified table name for the from clause.
We have tried declaring a variable but the stored procedure won't compile. Tried user, user_name().
HiI have a script that uses bcp to import data from an ascii text fileinto SQL tables. The french characters are not copied properly. Theyare converted to letters of the alphabet. I tried to change all thefields to nvarchar instead of varchar and nchar instead of char, but Igot Greek characters instead.How can I fix this?Here is some code:--------------------------CREATE TABLE [dbo].[1_HLGT_HLT_COMP_f9.0] ([hlgt_code] [int] NOT NULL ,[hlt_code] [int] NOT NULL) ON [PRIMARY]GOPRINT 'HLGT_HLT'DECLARE @s as nvarchar(300)SET @s='bcp MedDRA..[1_hlgt_hlt_comp_f9.0] in ' + char(34) +'F:MedDRA9.0FrenchMedAsciihlgt_hlt.asc' + char(34) + ' -c -t' +char(34) + '$' + char(34) + ' -r$ -e' + char(34) +'F:MedDRA9.0Frenchlogshlgt_hlt.err.txt' + char(34) + ' -b250 -m50-SDEV -Usa -Ppassword -h' + char(34) + 'TABLOCK' + char(34)EXEC master..xp_cmdshell @s
I have a file I'm pulling from another type of database into an Excel spreadsheet and then using my dtsx package to import the spreadsheet into my SQL database. The problem I'm having is that one of the fields coming out of the database to the spreadsheet has the thousands seperator in the field that I want to use as a numeric field without the ",". Right now I have a macro that I run on the spreadsheet to reset the field to straight numbers without commas before importing it, but would like to configure my Integration package to do it automatically.
I have used the wizard to import a DTS package from a SQL 2000 server to our new SQL 2005 server and need to make edits to reference the new server and database. I am able to see the package within the SQL Management studio under Integration Services, but I cannot find it in the Development studio?
I used a DTS package to import data from one database to a new database (different servers) and it seemed to work fine with the exception that it lost all the passwords. I got one error telling me that for security reasons, it left all passwords NULL. Sure enough - they are now all blank.
Ideas - any good articles with quick fixes ???
Thanks a bunch - I am just getting into a lot of unusual problems this week. Nancy
By the way - what happened to the SQL*Pass message boards ??? Nobody visits there now....
hi, I am importing data daily to many tables, I want to keep track of the #of rows for each import process. I already have created a trigger as follow: CREATE TRIGGER tr_bcp_log ON dbo.A FOR INSERT AS
declare @name varchar(30), @row_count int
select @name=name , @row_count= @@rowcount from inserted
insert into bcp_tracks (name,row_count) values(@name,@row_count) GO
The problem is that I am getting a row for each inserted row in table A.for instance if I have 500 rows in table A, I will get 500 rows in the log table like this table_name,#of rows A 1 A 1 A 1 etc up to 500 rows for table A
This is not what I want, I want to capture the num of rows for every bcp process , so in the log table I want to see the following : table_name, #of rows A 500 B 600 C 450 A 250 etc
I have a stored procedure which will run automatically. I've got try...catch code in the procedure, but I found a bug with the code where if there are any import errors, it doesn't recognize that that there was an error and it runs through the try code as through there was no problems. (I reported the bug). I added some code using @@rowcount to check if there were rows imported, and if not, it moves the data file to a error folder so I know there was a problem with the import. But this only checks if at least one row was imported, not if all the rows in the datafile have been imported. (i.e. if the first row imported correctly, and the second did not, it still sees it as successful). The problem is some of the data files have only one row to import and some have multiple rows. Is there a way to count the number of rows in the datafile, then count the number of rows imported, to verify they are the same number imported? Thanks, Laura
Today I tried out Integration Services and after a couple of hours of confusion I am impressed by the power of this product. I was wondering if I might get some help on the 'best practice' for the following requirements:
I have a foreach container that scans a directory and calls a data flow task for each file in the directory. The files import to the database fine, but I want to modify the procedure so it only imports files that have not yet been imported. There are a couple of scenarios:
- a file is created in the directory - a file is modified in the directory
In both cases I want to insert (or reinsert) the file. How can I modify my package to accomodate this behaviour? Storing the filename is an option, but I am not sure how to also bring in the file creation/modified dates.
I have a stored procedure which will run automatically. I've got try...catch code in the procedure, but I found a bug with the code where if there are any import errors, it doesn't recognize that that there was an error and it runs through the try code as through there was no problems. (I reported the bug). I added some code using @@rowcount to check if there were rows imported, and if not, it moves the data file to a error folder so I know there was a problem with the import. But this only checks if at least one row was imported, not if all the rows in the datafile have been imported. (i.e. if the first row imported correctly, and the second did not, it still sees it as successful). The problem is some of the data files have only one row to import and some have multiple rows. Is there a way to count the number of rows in the datafile, then count the number of rows imported, to verify they are the same number imported? Thanks, Laura
i have few fields that contain foreign characters with diacritic marks which are not getting imported correctly.
below is the import format:
- File type: ASCII - Row delimiter: carriage return and line feed {CR/LF} - Column delimiter: Tab - Text qualifier: None
Please advice.
Here is the errors i'm getting:
- Executing (Error)
Messages
Error 0xc02020a1: Data Flow Task: Data conversion failed. The data conversion for column "Country_str_local_long_name" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.". (SQL Server Import and Export Wizard)
Error 0xc020902a: Data Flow Task: The "output column "Country_str_local_long_name" (37)" failed because truncation occurred, and the truncation row disposition on "output column "Country_str_local_long_name" (37)" specifies failure on truncation. A truncation error occurred on the specified object of the specified component. (SQL Server Import and Export Wizard)
Error 0xc0202092: Data Flow Task: An error occurred while processing file "L:Country.txt" on data row 6. (SQL Server Import and Export Wizard)
Error 0xc0047038: Data Flow Task: The PrimeOutput method on component "Source - Country_txt" (1) returned error code 0xC0202092. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. (SQL Server Import and Export Wizard)
Error 0xc0047021: Data Flow Task: Thread "SourceThread0" has exited with error code 0xC0047038. (SQL Server Import and Export Wizard)
Error 0xc0047039: Data Flow Task: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. (SQL Server Import and Export Wizard)
Error 0xc0047021: Data Flow Task: Thread "WorkThread0" has exited with error code 0xC0047039. (SQL Server Import and Export Wizard)
I have imported some tables from msaccess but as the datetime field from access is different from SQL Server I had to change all columns type from datetime to varchar...
Now in SQL Server i'm trying to convert this data into datetime, and I'm using cast(field as datetime) and sql give me this message..."The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value."
how can I select these rows that are giving me errors ??? OR is there a way to convert this data ??
I don't know if in this data it will give the error... but as the table is too big...I can't post everything here...
------------------------------- EDIT
I got the problem...
I have data like 4/11/2006 9:23:19 AM and others like 11/21/2005 6:02:13 PM
So the first one 4/11/2006 9:23:19 AM it converts without problem, but wrongly...cause the month is "4" and not "11" the other one 11/21/2005 6:02:13 PM it tries to convert the month as 21 and not as 11....
