I am using SSIS to load a lot of Excel, csv files. Some of the files will fail for various formating/validation reason. Is it a good way to capture the error and generate a nice error report so the provider can read it easily and correct the data files?
The error log of the package is difficult to read.
I don't want show the DBMS error to my clients. I want show a simple error instead of SQL error.
View 3 Replies View RelatedHi
I have Lookup task to determine if source data should be updated to or insert to the customer table. After Lookup task, the Error Output pipeline will redirect to insert new data to the table and the Output pipeline will update customer table. But these two tasks will be processing at the same time which causes stall on the process. Never end.....
The job is similiart to what Slow Changing Dimention does but it won't update the table at the same time.
What can I do to avoid such situation?
Thanks in advance,
JD
I have various SSIS packages running on my SQL Server 2005, they are mostly schadule to perform data updations from different locations over remote network.
In every task there is an email object added in so that it can notify about sucessful completion of SSIS package.
Here I want my email should contain some information about how many records updated in system ... or how long it take to complete the job.
Can i include sort of information into email object? if yes ... then how to do it?
Thanks
MIZ
Hi Guys,
I have a bulk process. During the processing incase of any error I need to log error to the SQL Server Error Log.
I checked the help and found the external SP(Srv_Message_Handler) for this purpose.
I am not clear on the usage of this SP. Pls Help.
Vineet
Hello All,
I am trying to do a complex ordering of my query results. Hopefully I can explain what I am trying to do. To simplify the explanation I have a table with two fields in it.
Table name: SPACE
Column names: SPACE_TYPE_ID, OWNER_ID
What I would like to to is order the results in this exact way:
SPACE_TYPE_ID = 2
SPACE_TYPE_ID = 5
SPACE_TYPE_ID = 6
OWNER_ID Alphabetically (if SPACE_TYPE_ID = 3,4,9)
SPACE_TYPE_ID = 1
SPACE_TYPE_ID = 7
SPACE_TYPE_ID = 8
Basically it will be ordered by the id unless the id is 3,4 or 9. I thought I could possibly use a UNION with three select statements but I believe I can't have three order by statements with a UNION. I also was told about temp tables in the database but have no idea how to use them. Is this a possible solution.
Any help on how to achieve this customized order would be greatly appreciated.
Thanks
jlmoshier
I need a sales report that would display weekly amounts either sold or forecasted in matrix (pivot) form.
The data used for the report is like following (INV is sold and FC is forecasted):
rtype
region
week
wgt
INV
EU
1
150
INV
US
2
200
FC
US
2
400
FC
US
3
1000
FC
EU
2
100
I want the report to show data like this:
WEEK 1
WEEK 2
WEEK 3
TOTAL
INV
INV
FC
FC
INV
FC
EU
150
100
150
100
US
200
400
1000
200
1400
So I put region as rows, week and rtype as columns, and wgt as data field. Everything works fine except that there will be no grand totals for INV/FC. What I get is:
WEEK 1
WEEK 2
WEEK 3
TOTAL
INV
INV
FC
FC
EU
150
100
250
US
200
400
1000
1600
How can I get my totals? I know I could tailor my data to get INV/FC values into different columns to show as data fields in matrix report. But in this case, every week would always display 2 columns, which is certainly not what I want.
I also know I could add another matrix report to create a simulated total columns, but I wasn't able to "join" these 2 reports properly, there is always some space between them and the report looks unprofessionally. So I am looking for a way to do that with a single matrix.
Second question: how can I paint entire columns into different colors, so that INV is always green and FC is always blue? I tried customizing background color, but it only paints cells with values inside, leaving empty cells white. Is it possible to have the INV columns green, entirely?
Thank you in advance,
Wapper
I am creating a key-wording module where I want to search data using the comma separated words.And the search is categorized into comma ',' and minus '-'. Take a look on the example what I exactly want to do is
I have a main table name `tbl_main` in SQL
AS_ID KWD
1 Man,Businessman,Business,Office,confidence,arms crossed
2 Man,Businessman,Business,Office,laptop,corridor,waiting
3 man,business,mobile phone,mobile,phone
4 Welcome,Greeting,beautiful,bride,celebration,wedding,woman,happiness
5 beautiful,bride,wedding,woman,happiness,mobile phone,talking
6 woman,girl,Digital Tablet,working,sitting,online
7 woman,girl,Digital Tablet,working,smiling,happiness,hand on chin
If serch text is = Man,Businessman then result AS_ID is =1,2
If serch text is = Man,-Businessman then result AS_ID is =3
If serch text is = woman,girl,-Working then result AS_ID is =4,5
What is the best why to do this?
Has anyone customized the parameters area to pretty it up and make it more configurable like control number of columns etc?
Or can anyone direct me to a third party component?
Thanks.
Hi...
I am creating an application where I need to export the reportviewer data into formatted excel sheet.
Ex excel sheet should have formatting-
1-Column width set to 8.3
2-Wrap text turned OFF
3-Left Justify all fields
4-Landscape default for orientation.
If I apply these formatting into the .rdl files then it is going to be applicable for all other formats
like csv,xml etc but I don't want to apply these fomatting except the excel report.
I think if I can trap any event when export button is clicked when excel is selected from dropdown menu
then it is going to solve the problem a little bit.
I need this solution as soon as possible.
Regards
Chinmay Chandra Dey
(Software Engineer)
I have a dataset which is like:
Month, Day, Location, TotalSales
Jan 1 A 100
Jan 1 B 200
Jan 14 A 120
Feb 2 A 130
Mar 5 B 150
I want to transform the dataset using sql query into the following format:
Month, Day, LocationATotalSales, LocationBTotalSales, TotalSales
Jan 1 100 200 300
Jan 14 120 0 120
Feb 2 130 0 130
Mar 5 0 150 150
We have Microsoft SQL Server Management Studio Version 10.50.2500.0.I change to "Results To Text", and "Display Results in a Separate Tab". Then I restart SQL Server Management Studio; and my settings are gone.Is there a way to save my settings?
View 3 Replies View RelatedHi!
I have created a setup in visual studio and set the sql server as a pre-requsites.
now when the setup is run from
setup.exe
it checks whether sql server is installed... if installed then continues otherwise install sql server express edition, now this installation is silent,
I need to set the sa password during installation of sql server . i have no option to set the password during installation, can any one help
Hi all,
I am developing a program to connect to SQL 2005 Express. I don't know if I need any CAL license in order to make the connection. If it is not necessary for the Express Edition case, how about if my client upgrate to SQL 2005 in the future? Do I need CAL license for SQL 2005?
Thanks for any advice given.
i have a weird situation here, i tried to load a unicode file with a flat file source component, one of file lines has data like any other line but also contains the character "ÿ" which i can't see or find it and replace it with empty string, the source component parses the line correctly but if there is a data type error in this line, the error output for that line gives me this character "ÿ" instead of the original line.
simply, the error output of flat file source component fail to get the original line when the line contains hidden "ÿ".
i hope you can help me with issue.
Thanks in advance.
Hi guys i'm a newbie to this forums so hopefully i've posted in the right place, so here goes .....
My company has been given a CMS to look at, thing is everything seems to be working except for this beasty stored procedure.. the purpose of this stored procedure is to gather data and output it as xml, but for some reason and i havent a clue why - i am getting an error stated below - i have also attahced the stored procedured in hopes of some kind gurus can help me
Appreciate any and all help guys
andy
ERRROR MESSAGE:
Microsoft OLE DB Provider for SQL Server error '80040e21'
Parent tag ID 1 is not among the open tags. FOR XML EXPLICIT requires parent tags to be opened first. Check the ordering of the result set.
I would like to get the actual name of the column that has the error. Using the ErrorColumn (int value) I thought there would be some type of lookup collection based on the input (like column names)- if there is, can someone tell me how to get to it?
I have my error output writing to a stored proc, but instead of "32226" as the column name, I need to have the actual name of the column. I am going from Flat File to OLE DB Destination. I have a Script Component getting the output to write to my sproc, and I just need to get the column name.
Suggestions?? Thanks
I used the oTable.Script method to output the DDL for a Sql Server 2000 user defined table. The result is DDL with an error. I don't think the problem is with DMO itself so I posted this here. Note the 'TEXTIMAGE_ON ' clause. The table does not have a text column, and the DDL will not execute.
CREATE TABLE [MyTable] (
[FilterID] [int] IDENTITY (1, 1) NOT NULL ,
[LoanAgentID] [int] NOT NULL ,
[Key] [varchar] (16) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[IsActive] [int] NOT NULL ,
[tStamp] [timestamp] NOT NULL ,
[Formula] [varchar] (4000) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[_AuditAddDt] [datetime] NOT NULL ,
[_AuditUpdateDt] [datetime] NOT NULL ,
CONSTRAINT [PK_MyTable] PRIMARY KEY CLUSTERED
(
[FilterID]
) WITH FILLFACTOR = 90 ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
Server: Msg 1709, Level 16, State 1, Line 1
Cannot use TEXTIMAGE_ON when a table has no text, ntext, or image columns.
Any ideas?
Thanks!
What is the purpose of the error output for an OLE DB Source component. Any sql that would cause an error such as converting a character to a number or division by zero causes the OLE DB Source component to fail regardless of the settings for the error output. Works perfect for OLE DB Destination but I cannot come up with any scenario where it would work for the OLE DB Source component.
Thanks in advance,
Brad
In the Input and Output Properties tab under Advance Editor for OLE DB Source, I cannot remove columns. I copied this Source from a standard template and have made the normal changes to make it work. However I keep getting this error...Error: 0xC020837B at Load Server Security, OLE DB Source [1]: The output column "DBName" (1632) on the error output has no corresponding output column on the non-error output.Error: 0xC004706B at Load Server Security, DTS.Pipeline: "component "OLE DB Source" (1)" failed validation and returned validation status "VS_ISBROKEN".DBName of course is one of the columns that no longer exist, but I can't remove. Whenever I try to remove one of the columns, I get this error...Error at Load Server Security [OLE DB Source[1]]: The column cannot be deleted. The component does not allow columns to be deleted from this input or output. Is there anything that I can do to remove the columns? Is there just a simple setting that I can change to make this work?
View 6 Replies View RelatedHello,
I realize I have a question about what constitutes an "error" for an error output.
For example, a flat file source has an error output for "bad rows", that is, when it encounters "unexpected data". What specifically is "unexpected data"? Is this documented somewhere?
Another example would be an OLE DB source that uses a query to retrieve row. This too, has an error output, but I realize I have no clue what would constitute bad data from a table. I mean, data in a table is just data, so what would constitute an error from an OLE DB source? I can't think of one thing. Where are these "rules" documented, if anywhere?
Thanks
Hello,
I have a simple SSIS package running under SQL 2005 SP2a.
In the data flow, I have several lookup transforms with error outputs. Each error output links to its own audit transform and then writes data to its own flat file destination.
After the data flow is complete, my control flow will then use a ForEach file container to mail the flat files to the source system data owners with a message about incomplete/inconsistent source data. I am effectively providing feedback to the source owners so that we may improve the quality of data that gets sent to us.
My problem is that the flat file (which contains the offending rows) seems to get created everytime even when there were no errors in the lookup in question. Thus my ForEach container will always send a mail to the data source teams even if there were no errors as the error flat file will always exist albeit empty.
How can I stop this happening? How can I only create flat files when there really were errors? How can I prevent the source teams from receiving feedback emails when there is no reason to?
Thanks in advance
MGale1
I want to automate the dbcc checkdb process. I create a temp table called #CheckDbTbl and run the following command:
INSERT INTO #CheckDBTbl
dbcc checkdb(MyDbName) with tableresults
I plan to send myself an email if any problems are found.
Does anyone know what Error numbers or Levels or anything else I should look for in the #CheckDBTbl that will tell me a problem exists? Right now I'm only checking for: Level >= 16.
Many thanks for your responses!
-bh
I have a c# app. This is a piece of code out of a stored proc. it is erroring: Procedure or function getTopParentDealerFromChildDealer has too many arguments
OR
@dealerID is not a parameter for procedure getTopParentDealerFromChildDealer.(if I put ",@dealerID=@parentID)
I have tried all combinations "@dealerID",@dealerID=@parentID" etc.
BEGIN
--get the top parent dealerID
DECLARE @parentID INT
SET @parentID = 0
EXEC getTopParentDealerFromChildDealer @dealerID, @parentID OUTPUT
IF (@parentID>0)
BEGIN
------------------------------------------------------
here is the getTopParentDealerFromChildDealer as called
------------------------------------------------------
ALTER PROCEDURE getTopParentDealerFromChildDealer @childDealerID INT
AS
SET NOCOUNT ON
DECLARE @dealerID INT
DECLARE @parentID INT
SET @dealerID = 0
SELECT @dealerID = dealerParentID from dealerRelations where dealerChildID = @childDealerID
WHILE @dealerID <> 0
BEGIN
declare @temp INT
set @temp = @dealerID
IF (SELECT count(dealerParentID) FROM dealerRelations WHERE dealerChildID = @temp)>=1
BEGIN
SELECT @dealerID = dealerParentID
FROM dealerRelations where dealerChildID = @temp
END
ELSE
BEGIN
SET @dealerID=0
set @parentID = @temp
END
END
if (@parentID IS NULL)
BEGIN
set @parentID = 0
--set @parentID = @dealerID
END
return @parentID
I don't usually use stored procedures but the job I have taken over previously used them. Any help would be much appreciated.
Thanks
I have the following stored procedure working with an Access 2000 front end. The output parameters returned to Access are both Null when the record is successfully updated (ie when @@Rowcount = 1), but the correct parameters are returned when the update fails. I'm a bit new to using output parameters, but I have them working perfectly with an insert sproc, and they look basically the same. What bonehead error have I made here? The fact that the record is updated indicates to me that the Commit Trans line is being executed, so why aren't the 2 output parameters set?
TIA
EDIT: Solved, sort of. I found that dropping the "@ResNum +" from "@ResNum + ' Updated'" resolved the problem (@ResNum is an input parameter). This implies that the variable lost its value between the SQL statement and the If/Then, since the SQL correctly updates only the appropriate record from the WHERE clause. Is this supposed to happen? I looked in BOL, and if it's addressed there I missed it.
CREATE PROCEDURE [procResUpdate]
Various input parameters here,
@RetCode as int Output, @RetResNum as nvarchar(15) Output
AS
Declare @RowCounter int
Begin Tran
UPDATE tblReservations
SET Various set statements here, LastModified = @LastModified + 1
WHERE ResNum = @ResNum AND LastModified = @LastModified
SELECT @RowCounter = @@ROWCOUNT
If @RowCounter = 1
Begin
Commit Tran
Select @RetCode = 1
Select @RetResNum = @ResNum + ' Updated'
End
Else
Begin
Rollback Tran
Select @RetCode = 0
Select @RetResNum = 'Update Failed'
End
GO
Am new to SSIS and developing a component which pulls data from a staging table and drops them into another table in the same database.
Am using a
1) OLE DB Source to get the data from the staging table.
2) OLE DB Destination to insert or push the data into another table of the same database.
3) Script component to get the error rows and to update the staging table column with a flg value.
The rows that throw an error like primary key violation, or any other error should be redirected to the script component and the process should get completed.
The Error Output of the OLE DB Destination doesnt show any columns to be selected for Redirect Row option
The script executes without any error and the records are shown in error path but the records are not updated in the DB.
This is what i have in the script
Public Class ScriptMain
Inherits UserComponent
Dim sqlConn As SqlConnection
Dim sqlCmd As SqlCommand
Dim connMgr As IDTSConnectionManager90
Dim txnIdParam As SqlParameter
Dim errorDescParam As SqlParameter
Public Overrides Sub AcquireConnections(ByVal Transaction As Object)
connMgr = Me.Connections.ErrorConnection
sqlConn = CType(connMgr.AcquireConnection(Nothing), SqlConnection)
End Sub
Public Overrides Sub PreExecute()
sqlCmd = New SqlCommand("UPDATE STG_TRANSACTION SET ERROR_FLG='Y' AND ERROR_DESC=@errorDescParam WHERE TXN_ID=@txnIdParam ")
fueltxnIdParam = New SqlParameter("@txnIdParam", SqlDbType.BigInt)
errorDescParam = New SqlParameter("@errorDescParam", SqlDbType.VarChar)
sqlCmd.Parameters.Add(errorDescParam)
sqlCmd.Parameters.Add(txnIdParam)
End Sub
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
sqlCmd.Parameters("@txnIdParam").Value = Row.TXNID
sqlCmd.Parameters("@errorDescParam").Value = Row.ErrorCode
End Sub
End Class
Pleas let me know the solution and any help will be appreciated.
Hello,
Is there a way that I can set up an error handler on a DataFlow so that errant rows within the DataFlow are delivered to a user via email?
I'm looking for Primary Key Violations. In this case, there are two columns of data involved in the DataFlow (AcctCode, RebateRate).
Thank you for your help!
cdun2
When configuring error output, I want everything that is good in the row to make it to the destination, and then the offending column that is causing an error to be set to NULL, and then sent to the destination as well. In addition, I want to take the offending column's data, and route it over to an error holding table. I know about the ability to redirect the whole row, but I just sort of want to redirect just that column. For example....
Have a table with 5 columns
col1 int null,
col2 int null,
col3 char(3) null,
col4 bit null,
col5 int null
My data flow loads data from a flat file and has a record that looks like this
1 5 ABC R 3
I want the row to make it to the destination as follows....
1 5 ABC NULL 3
Then the offending data needs to go over to my error table
err# errcolumn errdata errdesc
1 col4 R Could not convert "R" to bit data type
Any way to do this?
I am developing a custom destination component and I have encountered a few areas where there seems to be a lack of helpful documentation and examples.
1. I have not been able to find any information on or examples of creating custom destinations with an error output. The OLE DB Destination has an error output so I investigated the input and error output properties in the advanced editor and found that the OLE DB Destination error output is synchronous with the input (its SynchronousInputID matches the input's ID) and has its ExclusionGroup value set to 1. Using this information, I modeled my error output after the OLE DB Destination.
ProvideComponentProperties:
AddErrorOutput(ERROR_OUTPUT_NAME, input.ID, 1);
ProcessInput:
int errorOutputID = -1;
int errorOutputIndex = -1;
GetErrorOutputInfo(ref errorOutputID, ref errorOutputIndex);
...
buffer.DirectErrorRow(errorOutputID, 0, errorOutputIndex);
Checking the input and error output properties in the advanced editor for my custom destination component I find the following:
Input
-----
ID: 3515
Error Output
------------
ExclusionGroup: 1
ID: 3516
IsErrorOut: True
SynchronousInputID: 3515
Shortly after I start my SSIS package and it encounters an error row, I get the following exception:
[My Destination Adapter 1 [3512]] Error: System.ArgumentException: Value does not fall within the expected range. at Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSBuffer90.DirectErrorRow(Int32 hRow, Int32 lOutputID, Int32 lErrorCode, Int32 lErrorColumn) at Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.DirectErrorRow(Int32 outputID, Int32 errorCode, Int32 errorColumn) at MyDestination.ProcessInput(Int32 inputID, PipelineBuffer buffer) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostProcessInput(IDTSManagedComponentWrapper90 wrapper, Int32 inputID, IDTSBuffer90 pDTSBuffer, IntPtr bufferWirePacket)
2. My custom destination component is used for writing a file with a fixed schema. I followed the means by which source component examples add their output columns, but applied this to my external metadata columns. In my Validate() I check if the ExternalMetadataColumnCollection.Count == 0 and return DTSValidationStatus.VS_NEEDSNEWMETADATA; to force a call to ReinitializeMetaData(). In ReinitializeMetaData() I call a method that creates the input's external metadata columns that reflect my external data source.
This works fine except every time I add my custom destination component to a SSIS Package and go to edit the component I am greeted with a dialog box that states: "The component is not in a valid state. ... Do you want the component to fix these errors automatically?" Pressing the Yes button, I assume, makes the call to ReinitializeMetaData() and I have my external metadata columns. Where is the correct place to add the external metadata columns so the user does not have to take this extra step every time they add my component to their package?
When setting an output's "IsErrorOut" property to true, is it also possible to add additional columns to that error output?
I'd like to add a message beyond the standard errorCode and errorColumn columns, a column which is the "specific error message", not just a lookup on the errorCode.
IDTSOutput90 outError = ComponentMetaData.OutputCollection.New();
outError.Name = "Error Output";
outError.IsErrorOut = true;
// Add extra column here, e.g. ErrorMessage
I have a data flow that takes an OLE DB Source, transforms it and then uses an OLE DB Command as a destination. The OLE DB Command executes a call to a stored procedure and I have the proper wild cards indicated. The entire process runs great and does exactly what is intended to do.
However, I need to know when a SQL insert fails what record failed and I need to log this in a file somewhere. I added a Flat File Destination object and configured appropriately. I created 3 column names for the headers in the flat file and matched them with column names existing for output. When I run this package the flat file log is created ok, but no data is ever pumped into the file when a failure of the OLE DB Command occurs.
I checked the Advanced Editor for the OLE DB Command object and under the OLE DB Command Error Output node on the Input and Output Properties tab I notice that the ErrorCode and ErrorColumn output columns both have ErrorRowDisposition set to RD_NotUsed. I would guess this is the problem and why no data is written to my log file, but I cannot figure out how to get this changed (fields are greyed out so no access).
Any help would be greatly appreciated.
Hi,
Iam redirecting the error output of a OLEDB destination component to a script component. My aim is to create a HTML report having the information about the bad records, the error occuring in the rows and the column name that fails. The error output provided two new columns i.e the errorcode and errorcolumn , the errorcolumn value for a bad record gives the linage id for the column, is there a way to derieve the name of the column by using the lineage id?
Regard,
pritesh
I'm in the process of running some tests to determine which method is faster...
I created a data flow task OleDB Source -> Data Conversion -> OleDb Destination. Error outputs from the OleDB destination is sent to a flat file destination. This works great.
I'm importing millons of rows and found that using SQL Server Destination (local) is much faster than the OleDB Destination. However, I have not figured out how to output errors to a flat file destination like I did when using the OleDB destination.
Is there any way to trap errors in a flat file when using a SQL Server Destination?
Thanks!
NLC