SSIS Design Question - Best Pratices
Apr 9, 2007
I have an application with 4 enviornments, test, dev, qa, prod - each with dedicated databases - some on seperate servers - some on same server seperate instance, some in same instance.
Plan to develop one SSIS Project with multiple packages and use DTSConfig files to store the connection strings.
Question one:
Where to store deployed packages - in SQL (msdb) or File System
Question two:
Where to store DTSConfig files - default directory or dedicated SQL Configuration directory.
Question three:
Retain one DTSConfig file per package or merge the connection strings into one global DTSConfig file per enviornment.
I know there is no right or wrong answer to any of these questions - just looking for ideas.
My plan (as of now)
- Deploy the SSIS packages to SQL Server (stored in msdb) for each Server
- Create dedicated folder (F:SQLConfigs) to store DTSConfig files for each server
- Manually combine the each packages DTSConfig into one global DTSConfig for each enviornment. Use this DTSConfig when configuring job steps in SQL Server Agent.
Thanks for your ideas.
View 2 Replies
ADVERTISEMENT
Feb 20, 2008
Hello everyone!!
I am completely new to SSIS and have been given a large project (of course with a tight deadline) that has the absolute requirement of using SSIS. I am/was very, very good with DTS and could easily accomplish what I need to do with an ActiveX script task in DTS in no time, but as this is new development, we are not to use ActiveX script tasks within SSIS since it will not be supported in the next SQL Server release. I'm thinking script task, but please give some comments on how you would accomplish the following in SSIS (please remember I'm new to SSIS, so don't assume I know anything. )
I must accomplish this: in a nutshell, I need to create separate tab delimited text files of customer informaion. One for each region. Each region consists of X amount of states and we have X amount of regions. (Pseudo code followed by standard explanation)
Select a max value from region lookup table in SQL (this is the # of regions)
for N=1 to MyMaxValue
select states from region lookup table where region code = N (the current region we are on)
'this returns a list of states in a region, need these in array or recordset object or something
Open an output file which will be a tab delimited text file we will write results below in loop to (in DTS I would programatically kick off a transformation task in the package)
'loop thru states returned, so if in a rs object...
do while not rs.eof
execute customer stored procedure, passing as a variable the current state we are on
'this will return all customers within a state, this whole result set (approx 1 million) needs to go to the tab delimited file
'I have to execute this stored procedure for each state & then write results to the SAME file, until we are onto a different region
rs.movenext
close file
loop
next
OK, so basically, as you can see, Its sort of simple in a way what I need to do, i just have no idea how to go about doing this in SSIS. I can not hard code any state or region values. I MUST read them in from the lookup tables as region codes are constanatly changing and we are constantly adding in new states and new regions, so with above coding idea, it would always dynamically pick up any new states, new regions or changes.
So in a nutshell, I need to create separate tab delimited text files of customer informaion. One for each region. Each region consists of X amount of states and there are X amount of regions. Pretty strait forward, huh? The requirements are strait forward, but SSIS is throwing me for a loop... it does not seem flexible enough to be as dynamic as I need it to be to do this. I'm sure it is, just my understanding of it is very basic so far.
Please provide your suggestions! I think a lot of newbies would benefit from some SSIS design info... how to do common things in SSIS, but beyond just retrieving a recordset and writing it to a file... what do you do when you need to add just a few layers of decision processing, and retriving recordsets and writing files based on that decision processing?????
THANK YOU!!!!!!
Kathy
View 7 Replies
View Related
Jul 12, 2007
Hi,
I would like to merge data from several excel sheets and load into a SQL Server 2005 database. The excel sheet are uploaded by users from several locations and I'm storing them on a folder on the web-server. I've written the SSIS packages for the required transformations, and for loading the data onto the database. Now, my question is:
1. What is the most efficient way to trigger execution of the SSIS package each time a file is uploaded onto the server (and stored in that specific folder)?
2. Would it be possible to not statically set the "excel source" for the SSIS package data flow transformations, but instead use the folder as the source and start the SSIS package each time a file is newly transfered onto the folder? (Should I use the Script task to get a list of excel files in the folder ordered by the system time?)
I tried using the Service Agent and scheduling it to execute the package on the server. But, my current SSIS design has a static excel source filename and thus when the package is executed more than once, it fails due to DB key constraints.
Thanks!
-Ganesh
gv1@cise.ufl.edu
View 1 Replies
View Related
Jul 11, 2007
Hi,
I need some help in designing the SSIS Package. The requirement is as follows:
I need to import data from an excel sheet to a table in the database. I have 2 lookup tables which has the master data. Before importing, I need to check whether the values of 2 columns in Excel sheet exist in those Master data tables or not. If not exist i need to log that error along with the row number and proceed to next row. It has to loop through all the rows and log the errors. The Import happens when there is no error occurred ie i need to roll back (but loop through all rows to log the errors) if there is any error occurs.
It would be great if you could help me out in desining this scenario in SSIS Package.
Thanks in advance.
Gnan
View 1 Replies
View Related
Feb 21, 2007
Hello,
I have designed a quite small SSIS package and everything worked just fine .
Untill I moved the SSIS project folder to another solution folder and added whole solution to Visual source safe.
Since then opening the package for design has become unbelievably slow, debugging it even worse.
I think the cause is in moving the SSIS project folder. But I have no idea how to fix this.
Any suggestions ?
Janez
View 10 Replies
View Related
May 15, 2008
Hi
I am busy designing a ETL solution and have a question about how to design the packages.
We have over 30 source systems for different customers. We are building a WH that will combine all this data for analysis. The main issue is that these systems are always at different versions of the software. When a patch is released, it usually goes to one or two for a Beta process before it moves to the rest. These patches can affect the DB design, and we would want to be able to extract any new data as it comes available from the systems that have it.
Solution 1 - Package Versions
The idea was to create the SSIS packages with a version number in their name. For each change, you would create a new version. The Batch control application that is being developed will store which system needs to use which version of the package.
Solution 2 - Multiple paths within a package
This was to create a single package for each table, with a conditional split as the first task. The batch system will still provide which path the package needs to take with different Data Flow tasks containing the different column mappings.
Both have pros and cons, but was wondering if anyone has experience with this type of setup and which way worked best, or if there are any other options.
Many Thanks
Michael
View 3 Replies
View Related
Nov 28, 2006
Greetings SSIS friends,
I am in the process of designing a package for populating a Dimension table for my new data warehouse. I would like to discuss with you my proposed solution in the hope that some of you will be able to advise me on any mistakes that I have made so far.
I have 3 source tables which are used to populate my Dimension table (I will restrict each source table to a few columns to keep this simple).
The source tables are :
Meetings (MeetID, NameID, Meeting_Start_Date, Meeting_End_Date, Meeting_Time_Stamp)
Events (EventID, MeetID, Event_Start_Date, Event_End_Date, Event_Time_Stamp)
Names (NameID, Long_Name, Short_Name, Names_Time_Stamp)
My Dimension table structure is very basic as follows :
Product_DIM (EventID, MeetID, Meeting_Long_Name, Event_Long_Name..... Meeting_Time_Stamp, Event_Time_Stamp)
Please note that I haven't included the start and end dates to keep this simple, but in my real solution, the Dimension table has many more columns.
The Meetings and Events tables contains hundreds of thousands of records so when my package is run, I should only retrieve information that has changed since the last time my Dimension table was populated. This is why I store the time stamps in my Dimension table.
So when my package runs, I get max(Meeting_Time_Stamp) and max(Event_Time_Stamp) from my Dimension table then retrieve all records from the source table where their timestamps are GREATER than the max values.
The problem is this :
If in the source system, an event is changed, the time stamp associated with that record will change value but the meeting record associated with that meeting will not have its time stamp changed.
How do I ensure that when my package is run, I retrieve all meeting and events information even if one hasn't changed?
Should I build a table made of IDs?! And if I do need to build a table made up of IDs, should I store this as a permanent table in my database or is there an SSIS data structure I can use to hold this information?
I hope this makes sense!
View 5 Replies
View Related
Oct 20, 2006
We are downloading 4 large (500mb) zip files in the package. Those come from 4 different FTP sites. Sometimes the FTP download on one of those fails.
Zip files contain images which need to be uploaded for existing listings. Each zip file is processed the exact same way. The question is how can I make the SSIS package so that each downloaded zip file can be processed immediately after the download - and not duplicating the processing logic?
View 3 Replies
View Related
Nov 29, 2007
I have 3 'separate' but related processes that I am developing in SSIS. Each process shares a few global variables and connection objects. None of the processes depend on inputs or outputs from the other processes. However, there is a sequence order in which they are exectuted.
Should I develop these three process in the same package or 3 separate packages in the same solution?
From the point of view of sharing the same variables and connection objects, maybe just one package? If so, would it be advise to use the toolkit's 'Sequence Container' to ensure the order in which the 3 processes are run? I noticed that 'global variables' are limited to the scope of individual packages so it would appear that a variable can't be shared between packages in the same solution? I was also thinking that in terms of configuration settings, there might be an advantage of keeping everything in the same package. The only disadvantage I can think of in relation to keeping everything in the same package is that the design layout might look a bit cluttered.
I'm tending to think one package here but would welcome comments and suggestions from those that have experience of similar design scenerios.
Thanks,
Clive
View 4 Replies
View Related
Nov 15, 2006
It appears that some of the methods in a custom source data flow component are called (repeatedly) at both design time and run time?
Is there a way to determine that "I'm in runtime" vs "I'm in design time"?
View 2 Replies
View Related
Apr 2, 2008
Hello,
I am new to SSIS. I need to transfer data from SQL Server 2005 Operational Database to SQL Server 2005 Report Database. The upload needs to work every night. There are few master tables and remaining are transactions tables.
I am planning to create 2 packages one for master tables and other for transactions tables.
Is it the good approach?
Also few of transaction tables are heavy in terms of number of records. Will it better if i further break them in many packages?
I am using book "Microsoft SQL Server 2005 Integration Services Step by Step".
Can you suggest any other good book?
Regards,
Shailendra Shelar
View 7 Replies
View Related
Nov 21, 2015
Win 7 SP1 x64 PC. I installed SQL Server 2014 Dev Edition + Visual Studio 2015.
I'd like to create some basic ETL SSIS packages, and I worked very comfortably in 2008R2.
For 2014, I started with this tutorial:[URL]However, it says to go to Start->All Programs->Microsoft SQL Server->SQL Server Data Tools.
I did explicitly install SSDT when I installed VS2015. I also installed it separately. I see SSDT listed in Programs, and SSIS is running according to SQL Server Config Manager, and Services. Half of Microsoft's docs seem to be 2012 era, which is a shame because 2014 is out and it's nearly 2016...
how do I get to the GUI where I can design ETL packages?
View 7 Replies
View Related
Jun 8, 2007
OK, I was able to successfully migrate all of my DTS packages to SSIS, for SQL 2005. I can log into intergration services and see my packages listed under:
servername --> stored packages --> msdb. Now my question is, how can I open these packages, not run them open them in a design mode like you can in SQL 2000, you can double click on the package name and view the design of the package. how can i do that now that I have them in SQL 2005?
View 6 Replies
View Related
Oct 2, 2007
Hi, all experts,
Is there a way in SSIS dev environment that will let you convert/cast data type at desing time?
Under data flow mappings, there are so many things you can do, but why I did not find any easy way to convert?
I got this validation warning about not supportng converting type DT_STR to DT_I4, or something like that on many occassions..
I am using OLEDB Source as source and SQL Server Destination as destination.
Am I missing something obvious?
Thanks!
View 6 Replies
View Related
Dec 13, 2007
I need to copy all the data from all the tables in a database to a copy of this database on another server.
What feature of SSIS should I take advantage of to accomplish this?
We have an SLA for 8am, most times the data warehousing jobs complete at 8:05am. Adding an additional process/set of tasks to this package would obviously make matters so I'm trying to update/copy/replicate the data in the fastest manner. Typically we're talking 2 marts (10-20GB) with 2 large tables (5-10 mill records) and 20 marts (0.5 - 5 GB) with many more smaller tables (~40 tables with record count ranging from 1 to a million)
Additionally please indicate if the design/feature you suggest can handle (pushing schema changes and additions to the target server) schema changes or new tablesviews added to the source database.
My only idea so far...is using the import wizard (in Management Studio) to create an SSIS package (top copy all the tables from one server to another) and saving it to the server, Then executing this package after the job is complete. However this would not work if the schema of a table changed, or if a a table is added. Moreover I don't think I can edit this package in visual studio.
View 3 Replies
View Related
Aug 17, 2007
Hi i tried designing a SSIS package which loads only those rows which were different from existing rows in the table , i need to timestamp the existing row with an inactive date when a update of that row is inserted (ex: same studentID )
and the newly inserted row with a insert time stamp
so as to indicate the new row as currently active, in short i need to maintain history and current rows in same table , i tried using slowly changing dimension but could not figure out, anyone experience or knowledge regarding the Data loads please respond.
example of Data would be like
exisiting data
StudentID Name AGE Sex ADDRESS INSERTTIME UPDATETIME
12 DDS 14 M XYZ ST 2/4/06 NULL
14 hgS 17 M ABC ST 3/4/07 NULL
New row to insert would be
12 DDS 15 M DFG ST 4/5/07
the data should reflect
StudentID Name AGE Sex ADDRESS INSERTTIME UPDATETIME
12 DDS 14 M XYZ ST 2/4/06 4/5/07
12 DDS 15 M DFG ST 4/5/07 NULL
14 hgS 17 M ABC ST 3/4/07 NULL
Please provide your input as much as you can even though it might not be a 100% solution.
View 4 Replies
View Related
Jun 22, 2006
In visual studio 2005, I create a new Integration Services Project. It tries to create the first package by default "Package.dtsx". The "Package.dtsx[Design]" tab displays
Microsoft Visual Studio is unable to load this document
Object reference not set to an instance of an object
I try to create new SSIS package or edit an existing one (from tutorial), I get the same error in the SSIS graphical user interface tab.
Thanks for your help.
View 3 Replies
View Related
May 13, 2015
I have a scenario like below
Product1
Product2 Product3
Product4 Product5
Product1 1
1 0 0
1
Product2 1
1 0 0
1
Product3 0
0 1 1
0
Product4 0
0 1 1
0
Product5 1
1 0 0
1
How to design tables in SQL Server for the above.
View 2 Replies
View Related
Feb 13, 2002
Ok, I'm doing a football database for fixtures and stuff. The problem I am having is that in a fixture, there is both a home, and an away team. The tables as a result are something like this:
-------
Fixture
-------
fix_id
fix_date
fix_played
----
Team
----
tem_id
tem_name
-----------
TeamFixture
-----------
fix_id
tem_id
homeorawayteam
goals
It's not exactly like that, but you get the point. The question is, can I do a fixture query which results in one record per fixture, showing both teams details. The first in a hometeam field and the second in an away team field.
Fixture contains the details about the fixture like date and fixture id and has it been played
Team contains team info like team id, name, associated graphic
TeamFixture is the table which links the fixture to it's home and away team.
TeamFixture exists to prevent a many to many type relationship.
Make sense? Sorry if this turns out to be really easy, just can't get my head around it at the mo!
View 2 Replies
View Related
Aug 18, 2015
I would like to create a table called product. My objective is to get list of packages available for each product in data grid view column while selecting each product. Each product may have different packages type (eg:- Nos, CTN, OTR etc). Some product may have two packages and some for 3 packages etc. Quantity in each packages also may be differ ( for eg:- for some CTN may contain 12 nos or in other case 8 nos etc). Prices for each packages also will be different that also need to show. How to design the table..
Product name :
Nestle milk |
Rainbow milk
packages :
CTN,OTR, NOs |
CTN, NOs
Price:
50,20,5 |
40,6
(Remarks for your reference):CTN=10nos, OTR=4 nos
| CTN=8 Nos
View 3 Replies
View Related
Nov 7, 2003
Hi all.
I'm going to create a big DB that will hold important info
(the usual stuff - Clients, Products, Orders...)
I wonder where should I use the IDENTITY field,
for example - on Orders I will have Order_ID...
(and where does SQL server the numbers of a deleted records)
My fear is that IDENTITY fields will go wrong somehow so I can
loose connections within the tables
(maybe when restoring my DB to some other locations... with DTS... or other issues)
1. When should I use IDENTITY field ?
2. If I do NOT - how can I lock a record when I add a new one
View 17 Replies
View Related
Jan 13, 2004
Hi,
I have 7 web forms wizard. after the user edit the fields, I need to keep a history log of the changed data only.
I am thinking of the following structure
log_id, user_id, field_id, old_value, new_value, transaction
Where field_id is a number indicates the changed field.
for example 1> First Name, 2> Last Name etc
Where transaction is ADD, DELETE, EDIT
Also i need to keep history of the details
for example an enterprise has 10 branches.
State, employees
NY, 1000
MD, 500
My log table will look like this
log_id, user_id, field_id, old_value, new_value, transaction
1, 1, 1, , Mick ,John , EDIT
2, 1, 3, ,NY , HI , EDIT
3, 1, 3, ,MD , , DELETE
4, 1, 4, ,500 , , DELETE
5, 1, 3, , ,NV , ADD
6, 1, 4, , ,700 , ADD
Is there a better way to log changes?
Thanks
View 1 Replies
View Related
Jan 16, 2004
I'm going to create a hardware/software inventory program to keep track on all the computers at the office. This program should be able to search the database using queries like "all computers with a cpu faster than 500Mhz".
I want some tips on my database design.
One solution is to create columns for each piece of hardware, i.e. cpu, ram, hdd etc etc. Then just run simple SELECT queries against them. The problem is that a computer may have many HDDs/CDs etc, and also other type of equipment may be entered in the database like switches and routers.
Another solution is to specify the valuetype + value in one table having a relation to another table containing the actual machines/routers/switches. The problem here is that I can't do numeric comparisons this way since "11" is less than "2", ("all computers with a cpu faster than 500Mhz").
Any suggestions?
View 3 Replies
View Related
Feb 10, 2004
Table A
[ID] [SalesmanEmail] [SalesmanName]
Table B
[ID] [QuoteNo] [SalesmanID]
Table C
[ID] [Product] [QuoteNo]
Program A creates the records for Table C. In the process it reads from Table A and Table B.
Program B reads record from Table C. Program B now needs [SalesmanEmail] field.
The suggestion is to add [SalesmanEmail] to Table C. Is this good or bad design?
View 5 Replies
View Related
Oct 15, 2004
hello,
i am working on a project with another guy as well. well, we have a table called lets say student and student_unit where student_unit holds the unit names the student is doing. and i was planning to use the same student_unit table to hold the result of that unit for that student as it is the case of adding one field but this guy who happens to be my boss thinks that result should be separated. but is it a better idea to make a separate table just for that result which is totally dependent on unit and student? that doesn't make any sense to me.
if asked why, he says for performance, modularity and object oriented approach and blah blah and honestly i think he knows nothing. so i would like u guys to tell him if there is really a need to create a separate table for attributes which is totally dependant on the same PK? having five more attributes on the same table hurts than creating a new table?
View 1 Replies
View Related
Oct 27, 2007
I think I might have dug myself in a hole here, so could use some advice on database design.
Currently, I have a table that records meeting minutes. The design is:
Code:
minutesid | int | 4 | no nulls
minutescat | varchar | 255 | allow nulls
minutesnotes | text | 16 | allow nulls
What this does is set up a minutes category, such as "Members Present", and the notes for that category, such as "Mrs. Peacock, Mr. Green, Professor Plum, Colonel Mustard, Miss Scarlett". Each individual category has it's own ID.
I need to develop an archive of each of the minutes. So if there are 15 categories and associated notes for the 15 OCT 2007 meeting, how can I design the archive table to include all of those categories and notes into one record for 15 OCT 2007, then for 15 NOV 2007 and so on ...?
Hope that makes sense. Thanks in advance ...
View 5 Replies
View Related
Sep 21, 2007
I have a table to store members. For each member, I have flags like manager, verified, etc. How should I be storing the flags? As individual columns in the same table and using bit datatype or creating a new table called MemberStatus and creating 1 row for each flag?
Thanx in advance.
Premal.
View 14 Replies
View Related
Mar 26, 2008
Hi,
I need to design a database, but i i just don know if i am doing it correctly.
Basically, i have an equipment which is sending me a lot of gauges readings (high limit, low limit, current level), electrical readings, hydraulic readings, alarms levels such as Fuel level, temperature level, etc etc, about 200 different types of readings, continuously. I may have different types of equipments and although most of the data are the same, some data types and its availability may be different too., depending on the type of equipment.
I want to keep this readings in a database, and this database will be updated constantly.
Should i create all of this different gauges readings in one big single table? If i do so, i only have one long row of data then for a particular equipment.If i put in different table, i can't see how i can create relationship among my tables with such kind of data.
Can you help on the possible ways of putting this equipment real time readings into DB?
View 3 Replies
View Related
Mar 12, 2004
I am trying to create a database that is used to create/store estimates for a manufacturing company. There are many things to consider in this estimate but I will isolate this question to the takeoff itself:
There are several categories in which costs are estimated and they are as follows : Product(s), Site Work, Transportation (Shipping), Field Materials, etc...
Should each of the above have their own "Takeoff" table, or would using one table and a gategory table be a better way to go?
Mike B
View 2 Replies
View Related
Jun 18, 2008
The data that I have is the following:
Car Manufacturer Factory CarCode
---------------- ------- -------
Ford Houston F-Hou
Ford Reno R-Hou
Chevy Houston C-Hou
Chevy Las Vegas C-LV
Honda SLC H-SLC
Ford SanFran F-SF
Chevy Miami C-Mia
I have a database design implemented already, but I'm receiving some disagreement about it from a co-worker. The way I see it is you have a car manufacturer and a factory location that are both unique, so I created a lookup table for each. I then created a branch table for the many-to-many relationship that exists between Manufacturer and Factory, and within that branch table I placed a column for CarCode, because it seems to me that Manufacturer and Location both determine the CarCode.
I also have an autonumber field within the branch table, which I then use for foreign key relationships.
Is there anything wrong with my design here?
Thank you in advance for your help!
View 5 Replies
View Related
May 1, 2007
I've fought with this issue for a week now and I can't come up with a simple way of doing it.
I have a database, with the following tables:
Members
MainCategories
Categories
Answers
The idea is that Members will log into the site, select the Categories from the MainCategories, fill out some data, which is stored in the Answers table. The problem is, each of the main categories have different fields. So a Main Category might be "Shoes" and the fields would be "Color,Size,Brand", and another Main Category might be "Cars" and the fields "Make,Model,Year,Color,Type,IsNew".
I thought about creating the Answers table with all the possible answers, linking it to the categories and members table, so we can view which members filled out answers for which categories.
Can anyone help me come up with a solution for this?
View 4 Replies
View Related
May 13, 2007
Please ignore my earlier post...
Please help me in designing the tables...
I used to get flat files which i need to import into a tables..
The flat files data which contains.....
FlatFiledata:
------------
100,1,xx,yy,,20030101
100,2,20030101
100,3,01,20030101
Below are the tables which i design please correct me..
Table1ForRow1
--------------
col Identity,
col1,100
col2,1
col3,xx
col4,yy
col5,,,
col6,20030101
Table1ForRow2
-------------
col Identity,
col1,100
col2,2
col3,20030101
Table1ForRow3
------------
col Identity,
col1,100
col2,3
col3,01
col4,20030101
When i queried then I want the output:
100,1,xx,yy,20030101,2,3,01
View 1 Replies
View Related
Jul 23, 2005
Hello all,I am trying to correctly model the relationship between products andversions within my db. The problem I have is how to store the versiondata. I need to collect the following information:1) Major Build Number (int)2) Minor Build Number (int)3) Build Number (int)4) SP number (int)What is the "correct" schema for this? How many tables and whatcolumns?(On a side note if any one has any links to useful examples of DBdesign could they post them?)Thanks,Jose
View 1 Replies
View Related
