What Is The Best Most Efficient Performant Way To Do A Control Break In SSIS?
Feb 13, 2007
I have an EDI file with Different Transaction types. I would like to read for a Header, capture some specific info on the header, and read for another specific tran type that comes after and capture additional info. I have a couple of ideas like tagging the records with a key and loading into two temp tables for matching later but that means I would have to do double processing. Maybe a conditional split for the two transaction types and then unioning them downstream, but not sure if the right records would be associated. Possibly tagging sequential key and writng to raw files and matching on the raw file keys downstream.
I have a lot of ideas but I am looking for the best proven practice here so i don't spin my wheels or have to go back and re-engineer later.
Thanks,
Larry
View 15 Replies
ADVERTISEMENT
Nov 15, 2007
Hello everyone,
I would like to develop an SSIS solution which loads data from large-sized csv files into the database either in REPLACEMENT or UPDATE mode.
In REPLACEMENT mode I delete all the exisitng data (if any) in the corresponding tables and then I do insertions with a few transfomations beforehand. I have accomplished this mission successfully with SSIS as follows:
- Control Flow: Execte SQL Task for performing pre-execution tasks --> Data Flow Task --> Execte SQL Task for performing post-execution tasks
- Data Flow: Flat File Source --> Some transformation components --> OLE DB Destination with FastLoad options.
In UPDATE mode I would like to update existing records in the database. I have found a great deal of articles talking about updating existing records. Based on my readings, here are the statements, I have concluded:
1. If the expected number of updates is high, use a staging table and perform the update as a batch operation from the Control Flow.
2. If the expected number of updates is low, use an OLE DB Command transformation.
Question:
Well, in my particular case all the rows in the source file represent record updates. And I am expecting a large number of update records. Thus, according to the previous conclusions, I should use the staging table alternative.
What I do not like about this alternative is that I will have to insert all the source rows in a staging table first and then I will have to update them all. This seems like too many extra work.
Is there a way to avoid this extra insertion work? I am thinking of something like "FastLoad Update" similar to the "FastLoad Insert", where I directly map input columns to output columns after defining a certain WHERE-CLAUSE?
Thank you in advance.
Samar
View 7 Replies
View Related
Jan 2, 2008
Hello,
I have a package that has over 80 sequence containers and as I added more sequence containers to the package I started encountered System.OutOfMemoryException error. I am planning to break up this package to six small packages. Is there a quick and safe way to do this? I have tried to use the copy and paste method by creating a new project and move a list of sequence containers over, that seemed to create even more work. Also, if the main package is splitting up to six smaller packages (the main package has the global variables that need to pass the values to the other child packages) how do the variables values pass onto the child packages?
Thank you so much in advance,
Andrew
View 4 Replies
View Related
Nov 27, 2005
Hi there,I'm looking for advices from people with more experience using SQL Server. I have a situation where from my perspective i have 3 ways to accomplish my task, but i dont know which way would be more efficient and performant.Here's the deal, i have a table which holds millions of records. This table will eventually receives between 30-40 insertion a second. So it's pretty busy. To display the data correctly in reports over the web, i need to manipulate the data in such a way that certain transaction are modified and others are completly eliminated depending on a set of preferences choosen at the moment of the request for the report. Obviously i dont see any ways to do this with just one SQL query, so my choices are the following:1. Create a second table, which would hold the manipulated data. Create a trigger and on each insertion manipulate the data and modify the second table.2. Create a stored procedure, from the stored procedure i query my records, create a temp table, manipulate my data and return it to the user3. Simply fetch the data, return it to the IIS server and process it there before returning the result to the client.The manipulation process consists of querying about 1000 to 2000 records, eliminating duplicate results (or similar results within a chosen range) by comparing them to the last one, and to indicate for each record how many other records were similar. In other words, it's a basic loop and i compare each record to the last one to see if it changed.Personally i think choice #2 would be better, but i could be wrong. I'd like to hear your comments and suggestions.Thanks,
View 9 Replies
View Related
Jun 27, 2007
We have what I think is a pretty common setup for records with a dynamic set of descriptive fields. Something like:
People
PersonID PersonType
1 Consultant
2 Partner
PeopleFieldDefs
FieldDefID FieldName
1 FirstName
2 LastName
PeopleFields
PersonID FieldDefID FieldValue
1 1 John
1 2 Smith
2 1 Alice
2 2 Johnson
Of course, we need to be able to search and display this data in a tabular format like:
PersonID PersonType FirstName LastName
1 Consultant John Smith
2 Partner Alice Johnson
We have been building dynamic queries based on which fields are needed (users can select the fields), e.g.:
SELECT p.PersonID, p.PersonType, pf1.FieldValue AS 'First Name', pf2.FieldValue AS 'Last Name'
FROM People p
LEFT JOIN PeopleFields pf1 ON (p.PersonID=pf1.PersonID AND pf1.FieldDefID=1)
LEFT JOIN PeopleFields pf2 ON (p.PersonID=pf2.PersonID AND pf2.FieldDefID=2)
This is very flexible but slow. We've done lots of optimization but can't get this below 5 seconds for common scenarios.
So I'm back to the drawing board now trying to figure out a better way to approach this.
I'm wondering if it would be better (if even possible) to break this out into some sort of view or table UDF that would contain a full representation of all person data, pre-joined. Problem is, this is almost certainly going to have to involve dynamic SQL since we can't know anything about what fields are defined. I think that rules out any sort of view or table UDF, no?
Does anyone have a suggestion for a good approach? Thanks.
View 4 Replies
View Related
Dec 6, 2007
I have a report with several matrix objects. The data contained in each matrix is simple. One matrix has one column of labels and a column of data. The other has 3 columns of data. Both matrix's only show about 25 rows of data. A variable amount of data is displayed above the matrix's so that sometime the render across a page boundary. Whenever this happens the matrix doesn't render rows down to the bottom of the page. A considerable amount of empty space is left on the page, the completed matrix is displayed on the next page. I'm wondering how to get the matrix to render in this empty space.
This problem only occurs when I view the report on screen, in the Visual Studio "Preview" window, or from the Report Server web site. Tiff, pdf and printed output doesn't contain the extra space.
I've tried adding my matrix to a List and a Rectangle to see of this would fix the problem but it didn't help.
I've check the dimensions and margins or the page and I don't think I have any sizing problems, everything should fit on the page.
--Mark
View 1 Replies
View Related
Jun 7, 2006
Hi All,
I'm new on SSIS, but have worked for some time with DTS and a long time with other ETL tools like Informatica or OWB.
I would like to know in which way can i, easily, control my project/package versions. At the same time i need to implement a concurrency management system, which will control what developer is using which package, and when finished update the central repository (As it does Informatica or even OWB).
I have heard that i could implement versioning with source safe, but can i implement this in the way that i've referenced before. Can i use CVS?
Thanks,
Vítor Ferreira
View 3 Replies
View Related
Aug 22, 2007
I am having a hard time with what appears to be something simple. I want to import an excel spreadsheet into a table on a daily basis from a command line. I created a package from the Import Wizzard in the SQL Management Studio and saved it. Since I want a clean table each day, my process needs to be create a temp table, import from the Excel file into the temp table. If that is successful, delete the original table and rename the temp table the original name. The point of this process is to provide for a fail-safe if there is some unforseen problem downloading the data on a particular day.
When I run the package, the first thing it does is delete the original table. I know this because the process shows the time that it finished is before anything else has started or finished. The time shown for the completion of the data flow task is about 2 minutes after that time.
This is maddening!!! The one thing I do not want to happen I can not seem to prevent. I have my control flow set on success. Why does it do this?
View 3 Replies
View Related
Oct 9, 2007
hi all,
is there any ways to control SSIS packages once it is deployed in MSDB.
now if a person deletes any package from MSDB,where to check the log or history that who has deleted the package.
particularly if the package is deployed on a server.
thanks in advance.
srikanth katte
View 1 Replies
View Related
Aug 26, 2007
Our team has the following problem:
I create a SSIS project in VS and add it to Team Source Control. When the project and all it's files exist on a machine and I want to open the project from Source Control, everithing works fine, but when another developer, who doesn't have the files on his machine, wants to open the project from Source Control he is required to check out the project file.
I've investigated the problem and found out that each project has a database file (with .database extension) and project file contains a reference to it. Once the user opens from Team Source Control a project that isn't presented on a target machine, the VS creates a new database file on a local machine and insists to change the reference in the project file.
Does anyone have a workaround for the problem?
Thanks
View 5 Replies
View Related
Aug 11, 2006
have a job that loads data from a data file into a table.
Is there a "task" that can be used to check to see if a file exists
and put error handling around it without programming ?
View 1 Replies
View Related
Apr 22, 2007
I have all the config files for my various packages in a folder call configs. It seems that I cannot add folders to the SSIS solution. I can add config files, but they get copied into the root folder, which breaks the directory structure that I have.
Am I missing something? Is there a recommended to manage config files, and other external files used by the SSIS package, in SourceSafe and in the SSIS solution? Thank you.
View 1 Replies
View Related
Apr 18, 2007
Hello all,
Is there documentation somewhere about multiple execution paths in SSIS control flow? I didn't find documentation anywhere. I have a situation where I have two tasks that take considerable time, but could be executed in parallel (to speed up things) and I was wondering whether SSIS supports parallelism.
To illustrate the issues in simultaneous execution, I created a test SSIS package. In the package, I have five tasks, let's call them T1, T2, T3, T4 and T5. The taks are connected with "green arrows" like this:
T1->T2
T1->T3
T2->T4
T3->T4
T5 is not connected. The tasks can be e.g. Send Mail tasks, that's not relevant to this issue. I put a breakpoint in each task and execute the package.
When I execute package, T1 and T5 become active, i.e. the arrow that displays where the package execution currently is, is in two tasks simultaneously. Now F10 (step over) doesn't seem to work "Unable to step. Not implemented". If I press F5 nothing happens. After I press F5 for a second time tasks T1 and T5 and executed. Why don't they execute with the first pressing of F5? I would additionally like to know whether these two tasks are executed in parallel or sequentially, i.e. in the same thread or in two threads? Is there documentation of this?
The execution stops at T2&T3. Again, pressing F5 doesn't do anything, but the second time I press F5 T2 and T3 are executed.
View 11 Replies
View Related
Apr 29, 2008
Hello All Experts,
I have created one custom task with PropertyGrid Control and two button on it. I have everything under one class library project.
Problem I am facing is when i load task and clik on Edit I can not see those properties into that GUI and even functionlity of those two buttons (OK and Cancel) not working but I am able to see those properties in default property window.
If I create this GUI as a seperate window application then I am able to see those properties in GUI and buttons also working but in SSIS I am not able to load the task.
After reading on internet about SSIS they suggest to create everything under one project which I did.
Basically I am trying to populate connection managers like Source Connection and Destination Connection when I load this task and there are much more backend functionlity but at first step i m stuck and not able to see those properties in GUI.
Please help and give your input on it. I was following "Increment Task" example given by MSDN.
If you need more info let me know.
Thanks
View 6 Replies
View Related
Aug 10, 2007
How do you go about setting up configuration files to control connection strings when SSIS packages are migrated from a test environment to a production environment? Specifically, changing the server you are connecting to and possibly different login and password.
View 1 Replies
View Related
Nov 12, 2007
Hi Guys,
This is a question to the SSIS development team. I would like to know what are the requirements to implement custom SSIS Control Flow task in C++ . There is a documentation describing the process when implementing a managed task, but no such documentation exists for implementing a task in C++.
Thank you,
Ivan
View 10 Replies
View Related
Aug 25, 2006
HI i need to write the Condition for Insert and Update Reccord depending upon the Prod_ID. How to write the Follwing condtion in the Condition Split? pls Anyone give me the Solution?
" if Prod_ID Exist then UPDATE Records
if Prod_ID Not EXIST then INSERT Records "
how to write the above conditon in the Condional Split?
Thanks & Regards,
Jeyakumar.M
chennai
View 10 Replies
View Related
Apr 16, 2007
I want to get the start time of data load and end time after data load and store it in a table which has mapping_id , mapping_name,start_time,end_time.
i use ActiveXScript task to get the start time before data load and store the mapping_id in a global variable,then data flow transformation occurs.
i want to use a global variable to store the mapping id ,so that i can update the end time after data load with that variable.how to do this?
is there any other way, i can get the start and end time of data load (other than the logging information)?
View 3 Replies
View Related
May 2, 2006
I am working on an SSIS project where I create two flat files for submission to a data contractor. This contractor requires a control record be the first line in the file. I create the control record based on the table information being exported.
What I would like to know is, is it possible to utilize the Header Section of the Flat File Destination Editor to insert the control record? And, as it is dynamic, what kind of coding must I do in order to utlise this functionality?
Thanks.
View 4 Replies
View Related
Oct 25, 2015
I installed the Feature Pack Balanced Data Distributor control on my PC to use with SQL Server 2014 64 bit. I have used the control with SQL Server 2014 and SSDT so I was familiar with the process. Unfortunately, I cannot get the control to appear in the toolbox. No error messages appear, BDD just doesn't appear in the toolbox. I have tried un-installing, reinstalling, installing SQL Server 2014 SP1, installing again, rebooting a number and nothing works. The control just does not appear in the toolbox. It doesn't not appear when I go to choose items either. What does it take to get BBD to appear in the SSIS Toolbox for VS?
View 2 Replies
View Related
Dec 26, 2007
Why isn't there some documentation on how to do this. This should be really simple and it has taken me 2 weeks and I still haven't gotten an answer. Please Help Does anyone know the answner or some place where there is some documentation!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I get the following error when I try to substitute the strings in the databasedetails collection with variables:
Error: Object reference not set to an instance of an object. StackTrace: at Microsoft.SqlServer.Dts.Tasks.TransferObjectsTask.TransferObjectsTask.CheckLocalandDestinationStatus(Database srcDatabase, DatabaseInfo dbDetail) at Microsoft.SqlServer.Dts.Tasks.TransferObjectsTask.TransferObjectsTask.TransferDatabasesUsingSpAttachDetach()
I created the following variables:
strDestinationDB = AirCL2Exp_new3
strDestinationDBPath = C:Program FilesMicrosoft SQL ServerMSSQL.2MSSQLDATAAirCL2Exp_new3_Data.mdf
strDestinationLGPath = C:Program FilesMicrosoft SQL ServerMSSQL.2MSSQLDATAAirCL2Exp_new3_Data.ldf
strSourceDB = AirCL2Exp
strSourceDBPath = C:Program FilesMicrosoft SQL ServerMSSQL.2MSSQLDataDataNewAirCL2Exp_Data.mdf
strSourceLGPath = C:Program FilesMicrosoft SQL ServerMSSQL.2MSSQLDataDataNewAirCL2Exp_Log.ldf
I then assigned those variable to DatabaseDetails Collection:
DatabaseName = @strSourceDB
DestinationDatabaseName = @strDestinationDB
Inaddtion I also assigned the following to the two DatabaseFiles Collection:
for 0:
DatabaseFileSize = 0
DestinationFilePath = @strDestinationDBPath
FileType = DatabaseFile
SourceFilePath = @strSourceDBPath
SourceSharePath = @strSourceDBPath
for 1:
DatabaseFileSize = 0
DestinationFilePath = @strDestinationLGPath
FileType = LogFile
SourceFilePath = @strSourceLGPath
SourceSharePath = @strSourceLGPath
View 13 Replies
View Related
Feb 3, 2014
I recently upgraded to on 2012 SP1 CU5 and have found the SSDT gui for SSIS to be almost unusable. I can't drag or resize items. Any time i try they either automagically shrink to the tiniest possible size, shoot off to some extreme or just shake uncontrollably I didn't have these problems on previous versions (dont remember what It was).
Is there a fix for this?
View 9 Replies
View Related
Jun 26, 2006
Hi ,
I am Using Derived column between Source and Destination Control. the Source input column PriceTime is String Data type. but in the Destination is should be a DATE TIME column. How to Convert this string to DateTime in the Derivied Column Control.
I already tried to in the Derived column control
PRICEDATETIME <add as new column> ((DT_DBTIMESTAMP)priceDateTime) database timestamp [DT_DBTIMESTAMP]
But still throwing Error showing type case probelm
Pls help me on this
Thanks & Regards
Jeyakumar.M
View 23 Replies
View Related
Jan 27, 2008
Hi All,
I am placing a Matrix inside the table control for grouping requirements,but when we export the report to the Excel, the contents inside the table cell are ignored. Is there any way to get the full report exported, as per the Requirement.Please help me with this issue.
With Thanks
M.Mahendra
View 5 Replies
View Related
Oct 25, 2007
does any one have and example of how to embedd a flash swf file onto a report.??? Is it possable? any examples would be helpful.
View 1 Replies
View Related
Apr 27, 2006
SSIS doesn`t start after applying SQL Server 2005 SP1
I get an error Message in event log
event id 7000 source: Service Control Manager Type: Error
Message:
The SQL Server Integration Services service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
regards
Lothar Belle
View 4 Replies
View Related
May 17, 2007
Hi everyone,
Primary platform is 64 bit cluster.
How to move information allocated in SSIS variables from Data Flow to Control Flow layers??
We've got a SSIS package which load a value into a variable inside a Data Flow. Going back to Control Flow how could we retrive that value again????
Thanks in advance and regards,
View 4 Replies
View Related
Aug 3, 2007
Hi there,
I'm using a Repeater at the moment which is bound to a SQLDataSource. I expect much load on that Website, should I choose another DataSource? Which other DataSource is better if it's about Performance?
I read some stuff about the SQLAdapter and a DataSet.. is that better in performance? Why is it better?
What about LinQ?
Thanks a lot for any clarification.
View 3 Replies
View Related
Dec 13, 2007
Hi all, I have the code listed below and feel that it could be run much more efficiently. I run this same code for attrib2, 3, description, etc for a total of 21, so on each postback I am running a total of 21 different connections, i have listed only 3 of them here for the general idea. I run this same code for update and for insert, so 21 times for each of them as well. In fact if someone is adding a customer, after they hit the new customer button, it first runs 21 inserts of blanks for each field, then runs 21 updates for anything they put in fields, on the same records. This is running too slow... any ideas on how I can combine these?? We have 21 different entries for EVERY customer. The Pf_property does not change, it is 21 different set entries, the only one that changes is the Pf_Value.
Try Dim queryString As String = "select Pf_Value from CustomerPOFlexField where [Pf_property] = 'Attrib1' and [Pf_CustomerNo] = @CustomerNo" Dim connection As New SqlClient.SqlConnection("connectionstring") Dim command As SqlClient.SqlCommand = New SqlClient.SqlCommand(queryString, connection) command.Parameters.AddWithValue("@CustomerNo", DropDownlist1.SelectedValue) Dim reader As SqlClient.SqlDataReader command.Connection.Open() reader = command.ExecuteReader reader.Read() TextBox2.Text = Convert.ToString(reader("Pf_Value")) command.Connection.Close() Catch ex As SystemException Response.Write(ex.ToString) End Try
Try Dim queryString As String = "select Pf_Value from CustomerPOFlexField where [Pf_property] = 'Attrib1Regex' and [Pf_CustomerNo] = @CustomerNo" Dim connection As New SqlClient.SqlConnection("connectionstring") Dim command As SqlClient.SqlCommand = New SqlClient.SqlCommand(queryString, connection) command.Parameters.AddWithValue("@CustomerNo", DropDownlist1.SelectedValue) Dim reader As SqlClient.SqlDataReader command.Connection.Open() reader = command.ExecuteReader reader.Read() TextBox5.Text = Convert.ToString(reader("Pf_Value")) command.Connection.Close() Catch ex As SystemException Response.Write(ex.ToString) End Try
Try Dim queryString As String = "select Pf_Value from CustomerPOFlexField where [Pf_property] = 'Attrib1ValMessage' and [Pf_CustomerNo] = @CustomerNo" Dim connection As New SqlClient.SqlConnection("connectionstring") Dim command As SqlClient.SqlCommand = New SqlClient.SqlCommand(queryString, connection) command.Parameters.AddWithValue("@CustomerNo", DropDownlist1.SelectedValue) Dim reader As SqlClient.SqlDataReader command.Connection.Open() reader = command.ExecuteReader reader.Read() TextBox6.Text = Convert.ToString(reader("Pf_Value")) command.Connection.Close() Catch ex As SystemException Response.Write(ex.ToString) End Try
Thanks,
Randy
View 2 Replies
View Related
Apr 1, 2008
what's the difference, if I use SQLDataReader at code level, making a query of that retrieves 500 rows and 2 columns, and making a query that retrieves 2 rows and 500 columns?
View 6 Replies
View Related
Feb 12, 2002
I have a table that has the following...
ID Status Type Check_Num Issued IssueTime Paid PaidTime
-----------------------------------------------------------------
1 I <null> 10 10.00 2/1/02
2 E IDA 10 <null> <null> 10.01 2/3/02
3 E CAP 10 <null> <null> 10.00 2/4/02
4 E PNI 11 <null> <null> 15.00 2/6/02
I want to return the Check_Num,Type, Paid, and Max(PaidTime) from this...
Example:
Check_Num Type Paid Time
---------------------------
10 CAP 10.00 2/4/02
11 PNI 15.00 2/6/02
Any assistance will be greatly appreciated.
Thanks,
Brian
View 1 Replies
View Related
Sep 7, 2000
Hey people
I'd be really grateful if someone can help me with this. Could someone explain the following:
If the following code is executed, it runs instantly:
declare @SellItemID numeric (8,0)
select @SellItemID = 5296979
SELECT distinct s.sell_itm_id
FROM stor_sell_itm s
WHERE (s.sell_itm_id = @SellItemID )
However, if I use this WHERE clause instead -
WHERE (@SellItemID = 0 OR s.sell_itm_id = @SellItemID)
- it takes 70 micro seconds. When I join a few more tables into the statement, the difference is 4 seconds!
This is an example of a technique I'm using in loads of places - I only want the statement to return all records if the filter is zero, otherwise the matching record only. I think that by using checking the value of the variable in the WHERE clause, a table scan is used instead of an index. This seems nonsensical since the variable is effectively a constant. Wrapping the entire select statement with an IF or CASE works, but when I've got 10 filters I'd have to 100 select statements.
I DON'T GET IT!! There must be a simple answer, HELP!!
Jo
PS this problem seems to occur both in 6.5 and 7.0
View 4 Replies
View Related
Jun 12, 2008
This query is giving me very slow search .What could be the efficient way
SELECT (
SELECT COUNT(applicationID) FROM Vw_rptBranchOffice
WHERE ( statusDate between '2008-03-13 16:12:11.513' AND '2008-05-30 00:00:00.000'
AND SearchString like '%del%')) AS
totalNO,ApplicationID,SearchString,StudentName,IntakeID,CounslrStatusDate
FROM Vw_rptBranchOffice
WHERE statusDate between '2008-03-13 16:12:11.513' AND '2008-05-30 00:00:00.000' AND
SearchString like '%del%'
View 5 Replies
View Related
