Complex Transformations (SSIS Components Vs TSQL)
Jun 13, 2006
Greetings SSIS friends,
I have been attempting to implement one of our numerous ETL processes in SSIS but hit a brick wall when I tried replacing a complex stored procedure with a series of Merge Join components.
In the end, I had to settle with using a SQL task which merely calls the stored procedure and this proved to be the better option as the other version where I used SSIS components only took forever to run.
How do people cope with complex transformations?! Do you guys opt for pure TSQL to perform complex transformations and use SSIS components for control flow+simple(ish) data flow tasks?
I am confused.
View 23 Replies
ADVERTISEMENT
Sep 7, 2006
Everything I've read says that custom data flow components are built by inheriting from the Microsoft.SqlServer.Dts.Pipeline.PipelineComponent class.
But the stock components such as the Derived Column data flow transformation must each be implemented by their own class. So how do I base my custom components on those classes? The documentation for the PipelineComponent class doesn't list any such subclasses.
View 1 Replies
View Related
Sep 25, 2007
I am not sure if I am posting this is the right place, please let me know if a better place exists.I am trying to figure out the distance from a point on the earth to a route between two other points. All points that I have are in latitude and longitude secondsIE: 51-52-40.6700N => 186760.6700NI am able to find the distance between two points on the globe using a custom function FindDist(latitude1, longitude1, latitude2, longitude2)I did the legwork for the math and was able to come up with this psudo code: p1 = start of routep2 = end of routep3 = point of referencep4 = INTERSECT(LINE(p1,p2), PERPENDICULAR_LINE(p1,p2,p3)) The direct distance between p4 and p3 is somewhere in the range of 0 - 70 and encompasses all points in the US (We only deal in the US) which is incorrect. If I do the conversion of p4 to latitude and longitude I get numbers that start out at 1000 which is also incorrect.I am doing the conversion with the following: DECLARE @lat2 VARCHAR(25)SET @lat2 = (@p4x * 3600)IF @p4x < 0BEGINSET @lat2 = @lat2 + 'S'
ENDIF @p4x > 0BEGINSET @lat2 = @lat2 + 'N'
ENDDECLARE @lng2 VARCHAR(25)SET @lng2 = @p4y * 3600IF @p4y < 0BEGINSET @lng2 = @lng2 + 'W'
ENDIF @p4y > 0BEGINSET @lng2 = @lng2 + 'E'
END I am not sure, however, how to either:a) find the latitude / longitude of p4b) find the distance in miles between p3 and p4Either of these will solve my problemI know this is a rather odd question, but if anyone can help me that would be great. If you need more information, let me know and I will try and explain things further.
View 1 Replies
View Related
Feb 21, 2007
SQL writes "Hi,
i am getting to trip with SQL Server and i find it is quite interesting.
please send me complex TSQL queries which would help me to test my SQL server expertise.
do let me know what are the URLs where i can find complex TSQL Queries
i appreciate pointers in right direction
Thanks in advance"
View 1 Replies
View Related
Jan 30, 2006
I was hoping to get a little input on a problem I'm having.
In the DataAccessLayer of my application, I have a "search" function written that takes a bunch of parameters (in the form of a class object) and depending on what each of the paramaters are set to (to include search type parameters) it builds an appropriate select statement. The issue is that my company has recently decided to require all DataAccessLayer functions to use TableAdapters. TableAdapters can use StoredProceedures, and StoredProceedures can make external calls (it all seems a bit backward to me, but there does seem to be a bit of logic in that TableAdapters contain a connection string which is set in the app.Config file)... Anyway, here's an example of how I am doing it currently, and I was hoping someone could suggest a way I could do it with either TSQL, or otherwise. If CLR is the way to go, how does that work?
internal static dsStrongTypeDataSet GetAll(clsMyClass inData)
{
bool first = true;
dsStrongTypeDataSet data = new dsStrongTypeDataSet ();
string selectStatement = "Select * from tblMyTable where ";
//There is one of these if statements for each parameter
if ((inData.Paramater1 != null))
{
if (!first)//not as important in this section of code,
//but there are areas where there are up to 30 parameters
selectStatement += " and ";
if (inData.SrchParameter1 == SearchType.Fuzzy)
selectStatement += " Column1 LIKE ('%" + inData.Parameter1 + "%') ";
else if (inData.SrchParameter1 == SearchType.Literal)
selectStatement += "Column1 = '" + inData.Parameter1 + "'";
first = false;
}
//More if statements like above for EVERY parameter possible
SqlCommand selectCommand =
new SqlCommand(selectStatement, CorrectSqlConnection());
SqlDataAdapter dataAdapter = new SqlDataAdapter(selectCommand);
dataAdapter.Fill(data, "tblMyTable");
return data;
}
View 3 Replies
View Related
May 29, 2007
In good old fashioned DTS there was the ability to perform custom transformations using activeX / vbscripty type language - does this still exist or are we stuck with the derived column editor?
View 3 Replies
View Related
May 10, 2006
Hi,
I saw some thing called custom properties for the "Derived transformation" in the msdn site. I tried to use them in a simple package, but I am getting an error as "can't write to derivedoutputcolumnname.friendlyexpression". Friendly expression is one of the custom properties available for the derived transformation output columns.
The steps I followed to get to this error are as follows:
1) Get data from a table using OLEDB Source. Suppose I am getting firstName, LastName etc.
2) Derived column input is values from the above OLEDB Source.
3) I have added a new column called "Concatenated name" which is concatenated value of first and last names.
4) Then in the properties editor of this data flow task in expressions option I clicked on ellipse available. I got an editor for property expression, which contained two columns called "Property" and "Expression". Property column contains dropdown with friendly expressions propety for the derived columns and expression column is a text box, where in we can enter expression to be evaluated for the corresponding friendly expression property.
5) Now when I click on OK and try to debug it gives an error as "Can't write to concatenatedname.friendlyexpresiion".
If anybody has already faced this problem and solved it please let me know, because I am struck here a long time.
Thanks&Regards,
Sreekanth Ammisetty
View 1 Replies
View Related
May 25, 2006
I've read that SSIS tries to do all transformations in memory as a way of enhancing processing speed. What happens though if the amount of data processed exceeds the available RAM? Are raw files then used (similar to staging tables) or is an error generated?
Barkingdog
View 1 Replies
View Related
Apr 20, 2007
I'm trying to find if there is a combination of dataflow transformations that will produce the following result
SELECT
period,
project,
task,
employee = CASE
when empid in (SELECT DISTINCT empid FROM EmpTable) then empid
else 'Deleted Employee'
end
FROM ProjectTable
I know I can create a dataflow task with this query as a data source and then send it to a destination, but I was wondering if that is the best way to do it or if there was a better way to do this using the data transformations available in SSIS.
Any insight would be most appreciated.
Regards,
Bill Webster
View 4 Replies
View Related
Nov 1, 2006
Hi
I am new to SSIS and have the following problem. I used the following script to clear data in columns of any CR/LF/Commas and char(0)'s. Can I just transfer this to SSIS and how exactly do I do that? Any help or advice would help.
Function Main()
Dim x
For x=1 to DTSSource.count
If Isnull(DTSSource(x)) = False Then
DTSDestination(x) = replace(replace(replace(Replace(DTSSource(x) , chr(13),""),chr(10),""),chr(0),""),","," ")
Else
DTSDestination(x) = DTSSource(x)
End If
Next
Main = DTSTransformStat_OK
End Function
Andre
View 8 Replies
View Related
Jun 2, 2006
Hi,
I'm finding that the standard components often just don't quite meet my
needs, but would only need some fairly minor changes to save me and my team a lot of
work (and produce more elegant solutions). So I was just wondering whether the source code was available for the standard components that come with SSIS, or if there is anyway to extend their functionality?
Or do you just have to start form scratch?
Thanks,
Lawrie
View 4 Replies
View Related
Jan 2, 2008
Hi
I am looking for a function or so to format a number to two digits plus one decimal point.
Some examples:
- if input=4 then output= 04.0
- if input=40 then output= 40.0
- if input=4.1 then output= 04.1
- if input=4.32 then output= 04.3
Any idea?
View 1 Replies
View Related
Oct 10, 2007
hello all -- I'm an SSIS newbie who is exploring (custom) programming functionality through third party components, to be plugged into SSIS, for SMTP, FTP, PGP, TAR, WinZIP, basically for a scheduling & automation utility.
Is there such a concept of third party components (and marketplace for this?) or is VB a much better choice for this vis-a-vis components variety, $$ and robustness ?
thanks much,
Cos
View 1 Replies
View Related
Oct 10, 2007
Hi,
As a hobby, I am creating custom SQL Server Integration Services components, which are installed as an add-on to out of the box components. They are mainly used to reduce the development time and minimize the maintance overhead. I would like to know, how I should market them? What are the mediums to use? Since this is the first time, I am creating a software for mass markets, I would really appreciate your help while marketing this baby.
Just FYI, the first commercially available package that I develop is for creating text based files. My package connects to any OLE DB datasource. Executes the query and stores the output in a text file. It is very useful while developing database driven custom feeds. Instead of working with multiple dataflow tasks, data flow sources, you just need to configure one single component and the rest is taken care for you. Since, the package implements the standard Microsoft Task Editor, it is very easy to learn.
Thanks for your help.
Cem
View 3 Replies
View Related
Feb 10, 2006
Especially interested in the CodePageConvert.
Installing the .msi just creates a project folder in my Visual Studio directory. I'm unclear how to get from this point to being able to choose this component from my Toolbox items in SSIS. There was a readme file that talked about gacutil.exe and .snk files that was a above my head.
Can anyone dumb it down for me?
View 6 Replies
View Related
Mar 3, 2006
How can we install the SSIS runtime components alone.
We are calling SSIS packages from .NET code. We would like to know what are the minimum components we need to install in the machine in which the .NET code is running.
Thanks,
Loonysan
View 4 Replies
View Related
Jul 17, 2006
Is it possible to use versioning at all with SSIS components?
My experience so far is that I have not figured out how version them at all, because if I change the version number, then apparently the <property name="UserComponentTypeName" ...> has to be changed in every single reference to the component, in every dataflow, in every single dtsx file that uses the component at all -- this will be a tremendous headache as some of this stuff is in source control. It seemsl clearly infeasible to check out and substitute and change all these references, and check them all back in, except very infrequently -- surely not feasible to do it for each small change.
I am in the habit of versioning dlls and assemblies, and actually changing at least a minor build number for each actual change, so I am a bit unhappy to find that apparently I cannot feasibly do this for a custom SSIS component :(
(I just find a dislike at apparently being forced to leave the version number invariant when I make code changes.)
What I would be very happy with, is if there were a way to adjust all the references in all the dtsx files, to refer to only the major and minor build numbers say, specifying that I am not picky about the revision numbers below those, so that it will find the latest component in the GAC matching the major and minor numbers.
That way I could still update the component version with each code change, which would make me happy, and not have to edit a zillion dtsx occurrences (not having to do that would also make me happy).
Anyone know if that is possible?
I'm hoping it is, and that it is only my ignorance at component references preventing me from doing so.
View 4 Replies
View Related
Aug 4, 2006
What SSIS components must be loaded onto the server?
To run an SSIS ETL program, exactly what executable, components, programs, etc. must reside on the server that will be running the program. Our developers write an test their ETL programs locally, on their desktops or laptops, but must upload the scripts to the production server for execution. Obviously, they are not going load Visual Studio and SQL 2005, etc. up to the server just to run the scripts (tho SQL 2005 may be on the same box as one of the DB engines used for ETL).
Any info would be appreciated.
View 7 Replies
View Related
Jan 15, 2015
MSDN states the following on: Readable Secondary Replicas (AlwaysOn Availability Groups) for SQL Server 2014:
Limitations and Restrictions:
Change tracking and change data capture are not supported on secondary databases that belong to a readable secondary replica:
Change tracking is explicitly disabled on secondary databases.
Change data capture can be enabled on a secondary database, but this is not supported.
This confuses me: You can not track the changes. However you can enable CDC?
The scenario I am trying to achieve is to use SSIS CDC components on an asynchronous secondary replica. Is this possible? If not what would be other viable approaches?
View 0 Replies
View Related
Nov 15, 2007
I have a SQL Server 2005 Package (32 bit, 4 processors/16 GB of RAM, 3 GB switch for SSIS) with 1 DataFlow and 5 Custom Components:
> An OLEDB Source that reads a stoed proc from Sybase
> 3 Transform Components that perform: 6-required fields check, 4-lookups (3 cached, 1 non-cached), and a calculation that is performed(called) from a .NET Library.
> A SQL Destination that uses the .NET BulkCopy Class to insert the data.
It takes about 3 hours to load 1 million records that contain 46 columns (the largest being a varchar(100)). I have tried a good number of tuning options as well as indexing/SQL techniques to improve perfromance.
Some pieces of info:
1) We dynamically create the buffer during runtime based on an SSIS variable.
2) The actions of the components are determined at runtime through config files.
We do these two things so we can have multiple data flows run through the same package for better scalability and flexibility. Since we are dynamically creating the buffer I can NOT use any of the out of the box components because of lineage and metadata inconsistency issues.
3) I noticed on PerfMon that I am spooling buffers to disk after 100K rows.
I realize this is bad, but what are my alternatives?
Should I go to 64 bit? Is there anything else I can do?
Thanks in advance,
Anthony
View 10 Replies
View Related
Sep 29, 2007
Hi,
I have a SSIS Package which I would like to modify using SSIS API. I need to put new component between some two existing data flow's components. During this process I need to disconnect two data flow's components using SSIS API. How can I do that?
Thanks,
Rafal
View 1 Replies
View Related
Feb 6, 2008
How do i access global variables in SSIS scripts ?
the Dts.variables("VarName").Value
should work but doesn't ?
do i need to reference the variable another way ?
View 3 Replies
View Related
Jun 30, 2006
Hi,
In another thread Jamie Thomson very informatively said "The components in SSIS
are deliberately atomic (i.e. they do something very specific) so that
its easy to put them together to build something greater than the sum
of the parts". Which does make a lot of sense. However, I've been finding that I end up having to create exactly the same "pattern" of combined transform components again and again in order to solve the same problem but in different dataflows (or even within the same dataflow). Cut-and-paste-tastic! In order to obtain real re-use, it seems to me like SSIS is crying out for an easy way to create new components by using composition - i.e. the ability to take commonly-used combinations of existing components and create new "super" components (without having to write Custom Transform Components in C#/VB.Net and handle everything in code).
Does anyone know if this sort of functionality is likely to make it into SSIS in the forseeable future?
Regards,
Lawrie
View 6 Replies
View Related
May 10, 2007
The following is a list of questions that I have not been able to obtain concrete answers. I am probably missing something:
1) ReadWriteVariables -- can the updated value for a ReadWriteVariable be accessed within the same data flow? It appears not as I think the PostExecute() fires at the completion of the data flow not the end of the Script Component. Secondarily, the Script Component is a non-blocking transformation so the component does not "see" the end of the pipeline prior to sending data down stream.
2) Record Count -- Because of #1 above, How could you calculate a record count for a data stream? It does not appear that one can calculate the number of records for a data stream within a data flow and then access the count from within the same data flow.
3) FinishOutputs() -- Is the concept of FinishOutputs() applicable to Script Component Destinations? Asked another way, is FinishOutputs() executed at the end of the data stream regardless of whether there are "real" outputs for the component? I can create a "Dummy" output to create FinishOutputs() but is this ok?
4) Script Component -- It appears that the Script Component Source, Transformation or Destination are really defined based on the columns defined in "Inputs and Outputs". Can you convert an Source script component to a transformation script component by simply adding an Output?
Sorry for these basic questions but I am not getting it completely. As you can tell...
View 12 Replies
View Related
Jun 30, 2006
Hi,
Can anyone tell me where I might find the Class and Sequence Diagrams for the SSIS framework (for Custom Components)?
I've just started trying to create some Custom Transform Components and
I'm really struggling to get my head around the component lifecycle
(i.e what methods are called when, with what arguments, and why) with
just the BOL documentation to guide me.
Thanks in advance,
Lawrie
View 1 Replies
View Related
Nov 28, 2006
Now that SSIS has been out for a year, how is SSIS helping the data warehouse developers in the real world. I have seen etl projects done in terms of TSQL where data is ELTL in stead of ETL. The entire data is written into some staging database, transformed and then loaded in to the datawarehouse. Are there any real success stories out there where SSIS really made a huge difference over DTS or TSQL. I really like the product but just wanted to hear from the experts out in the field.
Also, do you see IT industry being eager to start using SSIS in their production applications?
View 2 Replies
View Related
May 21, 2007
Hi,
I am trying to make a call to a third-party web service in my SSIS package. The request has custom complex data type as the parameter. As has been pointed out in this forum before, the Web Service Task only lets you assign the outside parameter from a variable, not the internal parameters needed to create the complex data type.
To be more specific, the web service input wants a 'ContactSearchRequest' parameter. I can assign this from a variable. If I click on the 'value' field under the 'Input' section for the web service task, it shows me that the 'ContactSearchRequest' data type is made up of the following:
contactId - long
numResults - int
offset - int
passKey - string
searchParam - string
sortType - int
Unfortunately, I can't assign these internal parameters from a variable, at least not through the web service task interface.
My next thought was to create a variable of type 'object' and then set it in a script task prior to calling the web service task. However, I'm not sure exactly how to do this. How will my script know about the class definition of 'ContactSearchRequest'? Do I just create a class called 'ContactSearchRequest'?
I've used this same web service in a .NET C# project and after I imported the web service, visual studio knew all about the custom data types. How do I do something similar in SSIS?
Of course, the easiest solution would for Integration Service to allow me to set those internal parameters via variables, but we're apparently not there yet.
Any suggestions?
Thanks,
Trey
View 8 Replies
View Related
Jun 23, 2008
I'm copying almost all contents of one table from one server/db to another. In relation to speed, what is the difference of using SSIS vs a linked server and sp? I'm going to do the benchmarks, but I'm curious about behind the scenes kinda stuff, theoretically which one if any should be faster and why?
View 5 Replies
View Related
Mar 29, 2007
I created a SSIS package and several data flow componenets for this package.
What does strategy exist to deploy SSIS package and data flow components into a enterparise server?
Thanks in advance.
View 2 Replies
View Related
Nov 2, 2006
We're experiencing a problem where intermittently our SSIS packages will hang. There are no log errors or events in the event viewer. It will happen whether the package is executed from the SQL Job Agent or run from BIDs. When running from BIDs it appears to hang inside one of the data flows (several parallel pipes with sorts, merge joins etc...). It appears to hang in multiple pipes within the data flow component. The problem is reproducable, we just kill it and re-run, and it appears to hang in the same places.
Now here's the odd thing: as we simply open and close some of the components in the pipe line after the place it hangs, a subsequent run will go further in the pipeline before hanging. If we open and close all the components after the point it initially hung, the data flow will run fine, from there on out. When I say "open and close" I mean no changes are made, we simply double-click the component, like a merge join, then click 'close.'
To me this does not seem like a memory problem but likely something is wrong with the metadata, where opening a component and closing it somehow alters the metadata to "right it".
This seems to occur intermittently after we make modifications to the package. It's like if you make any mod, even unrelated to the data flow, you then have to go through and open and close every component in your package to ensure it will work. Again, no errors or warnings are fired.
Has anyone seen this type of problem?
View 10 Replies
View Related
Feb 7, 2006
I had this (what seems to be a) simple question asked today and I'm afraid I didn't like my answer. Does anyone know the proper answer to this one:
Any ideas on how I can constrain a lookup or merge join based on the dimension row's effective and expired dates so three criteria are needed as follows:
1. DataStagingSource.ModifyDate < DataWarehouseDimension.RowExpiredDate AND
2. DataStagingSource.ModifyDate >= DataWarehouseDimension.RowEffectiveDate AND
3. DataStagingSource.NaturalKey = DataWarehouseDimension.NaturalKey
-- Brian
View 3 Replies
View Related
Oct 9, 2007
Hi,
Need a guidance on consuming the WCF service with complex type in the SSIS package.
I have a WCF service with complex type inside the complex type (Nested complex type) in the web method as an argument. When I try to use this WCF service in the SSIS web service task, I get an error "The web method has unsupported arguments".
I am able to consume the WCF service with the web method having Complex type and simple/prmitive type in side that as argument.
For example:
The web method in WCF service which accepts the argument as comlex type say "Employee" whose structure is:
Employee
{
FName String;
LName String;
Age int
}
It is possible to consume this WCF service and pass the arguments.
But when the Employee complex type is changed to have one more complex type in side it it give the above mentioned error. The Employee type is modified as:
Employee
{
FName String;
LName String;
Age String
Type EmployeeType;
}
EmployeeType
{
type string;
}
Now I get the error Web Method has unsupported arguments.
Can this be done in SSIS?
Virendra
View 3 Replies
View Related
Oct 30, 2007
Hi,
I'm trying to replicate a SQL join across two seperate data sources in SSIS. If I were to write SQL to do this, it would be as follows:
SELECT Costs.CostRateEntryId,
Costs.UserId,
Costs.HourlyRate * 8 AS DailyCostRate,
Dates.DateKey,
Dates.ActualDate,
FROM Costs
INNER JOIN Dates ON Dates.ActualDate >= Costs.EffectiveDate AND Dates.ActualDate <= Costs.EndDate
Unfortunately, as the tables 'Dates' and 'Costs' are in two seperate SQL2005 systems, I can't really do this. I was hoping that it could be achieved in SSIS, but I cant seem to find any way that I can do a join that's <= or >=.
Can anyone help?
Thanks
Jeremy
View 7 Replies
View Related
