Package Design Architecture
Oct 8, 2007
Can anybody give me design ideas on the following?
I have numerous tasks, any one of which can fail. I want to point them all (via 'Failure' constraint) to a SendMail task for a "failure notification" email. This I have setup, and is working fine. Now, I want to have a changing message for the email's body (MessageSource) to say "Process A has failed", or "Process B has failed", etc.
My initial thought was to add a variable, then add a ScriptTask between each task and the single SendMailTask, have the script update the variable, and have the MessageSource (body of email) mapped to the variable. Is this the proper way to go about this? Seems like if I have 20 processes that could fail, then I'll have to add 20 script files; this becomes a bit unwieldy the more processes that I want to monitor.
Is there a better way to accmoplish this?
Thx,
Mojo
View 4 Replies
ADVERTISEMENT
Jan 24, 2008
Any design suggestions for the best way to architect this report using SQL Reporting Services 2005 are appreciated!
My website features a catalog of roughly 50,000 items, each of which may be appear in a list of search results or in a detailed view. There are counters on the pages that update totals for such appearances and track other item-specific information in several tables in a SQL database. The catalog of items changes frequently, so the list of item IDs is never exactly the same from month to month.
I've been asked to produce a monthly report of this data for each of the items in the catalog, with reports for the current and previous months (for many years) accessible at all times. Some -- but not all -- items are useful for one purpose or another and so can be considered as belonging to a group of items. Although I have not yet been asked to create a report that aggregates the values for all Group members into a single report for that Group, I can clearly see it would be valuable and will be requested soon.
To ensure the report captures the data for an entire month, it must be run at the very end of each month. That means I will need to run the report using a Schedule that kicks off the process at 12:01am every 1st of the month. The report must be processed and stored for later retrieval and rendering on demand.
Considering the number of items and the indefinite length of time the report data must be retained, my question is really what's the best way to set all this up?
Should I create a report for each item separately? That would mean the scheduled task would have to somehow discover the current list of item IDs (which is available via query from the database) and create and process (but not render) a report for each (passing the item ID as a report parameter?), adding it to the report history. Although each report would be small take only a short time to run, overall that seems like it would take a long time to run and create a huge number of reports to store each month.
Or should I create a single 'master' report that contains all the data for every item for the month, and then use the item ID as a filter on the data when it is rendered? While that means only one report is created each month and added to the history, it would be a much larger report and take much longer to run (with more potential for timeouts and errors to scuttle the whole report). It also means all the data for the entire report has to be loaded every time the report is rendered, even though only 1/50,000 of the data (the data for 1 of the 50,000 items) will actually be viewed with any given rendering. But that would seem overly cumbersome, slow, and wastefully band-width intensive.
Any alternatives, suggestions, considerations, etc. -- all welcome!
Thanks
BillB
View 2 Replies
View Related
Jun 5, 2014
We have a Silverlight based application which currently supports only one production version. Idea is to support three concurrent versions of the same application and user will switch to the newer versions based on their interest or they can still continue with the older version.
We still have to use the existing database for all these three versions.
What is the best way to architect this so that we can differentiate the code between the versions and still keep the data in sync and run all the versions in parallel.
View 7 Replies
View Related
Oct 24, 2006
Hi everyone,
Is Microsoft thinking over the possibility to implement Control+Z in order to avoid drawbacks when you're writing a package? I mean, when you drop a container then you aren't able to retrieve again
.
That same behaviour happened with Sql2k-dts.
Thanks in advance and regards,
View 3 Replies
View Related
May 1, 2001
Hi,
Does any one have a workaround this issue. If I designed a package from SQL2000 and save it on SQL 7, I can't open/execute it from SQL 7.0. My workstation is SQL 2000 and the server is SQL 7.0, even if I have the same SQL 7 but different SP.
Thanks
View 2 Replies
View Related
Nov 23, 2005
Hi all,I'm fairly new to this...I't trying to design a DTS package on SQL Server 2000`, which willconnect to the server, export some table date into a .txt file (basedon a select statement) and then delete the data from the table (basedon a delete statement) upon successful completion of the export.So far so good..Now, upon the successful deleteion, I want to export some more data andthen (upon the successful export) to delete the data. I want to repeatthis process approx. 10 times.Though, when I try to link the deletion to the new export on success, Iget the message: "Defining the precedences between the selected itemsis not valid."Any idea on how I can accomplish what I want to do?thanks a lot
View 3 Replies
View Related
Aug 31, 2007
All:
This proably is an unsual request. I have developed a package that runs fine and does what it is supposed to do. I am jsut not sure if I have developed it in the most efficient way. I have several years of ETL experience but only about 6 months with SSIS. I know I can benefit a lot if I had my package reviewed by an expert.
I realize that I am asking for some time committment on your part and so would understand if I do not get any takers. But if you would like to review my package and offer suggestions on its improvement please let me know. We can work on the logistics of getting the package to you.
Thanks!
View 7 Replies
View Related
Feb 21, 2007
Hello,
I have designed a quite small SSIS package and everything worked just fine .
Untill I moved the SSIS project folder to another solution folder and added whole solution to Visual source safe.
Since then opening the package for design has become unbelievably slow, debugging it even worse.
I think the cause is in moving the SSIS project folder. But I have no idea how to fix this.
Any suggestions ?
Janez
View 10 Replies
View Related
May 15, 2008
Hi
I am busy designing a ETL solution and have a question about how to design the packages.
We have over 30 source systems for different customers. We are building a WH that will combine all this data for analysis. The main issue is that these systems are always at different versions of the software. When a patch is released, it usually goes to one or two for a Beta process before it moves to the rest. These patches can affect the DB design, and we would want to be able to extract any new data as it comes available from the systems that have it.
Solution 1 - Package Versions
The idea was to create the SSIS packages with a version number in their name. For each change, you would create a new version. The Batch control application that is being developed will store which system needs to use which version of the package.
Solution 2 - Multiple paths within a package
This was to create a single package for each table, with a conditional split as the first task. The batch system will still provide which path the package needs to take with different Data Flow tasks containing the different column mappings.
Both have pros and cons, but was wondering if anyone has experience with this type of setup and which way worked best, or if there are any other options.
Many Thanks
Michael
View 3 Replies
View Related
Nov 28, 2006
Greetings SSIS friends,
I am in the process of designing a package for populating a Dimension table for my new data warehouse. I would like to discuss with you my proposed solution in the hope that some of you will be able to advise me on any mistakes that I have made so far.
I have 3 source tables which are used to populate my Dimension table (I will restrict each source table to a few columns to keep this simple).
The source tables are :
Meetings (MeetID, NameID, Meeting_Start_Date, Meeting_End_Date, Meeting_Time_Stamp)
Events (EventID, MeetID, Event_Start_Date, Event_End_Date, Event_Time_Stamp)
Names (NameID, Long_Name, Short_Name, Names_Time_Stamp)
My Dimension table structure is very basic as follows :
Product_DIM (EventID, MeetID, Meeting_Long_Name, Event_Long_Name..... Meeting_Time_Stamp, Event_Time_Stamp)
Please note that I haven't included the start and end dates to keep this simple, but in my real solution, the Dimension table has many more columns.
The Meetings and Events tables contains hundreds of thousands of records so when my package is run, I should only retrieve information that has changed since the last time my Dimension table was populated. This is why I store the time stamps in my Dimension table.
So when my package runs, I get max(Meeting_Time_Stamp) and max(Event_Time_Stamp) from my Dimension table then retrieve all records from the source table where their timestamps are GREATER than the max values.
The problem is this :
If in the source system, an event is changed, the time stamp associated with that record will change value but the meeting record associated with that meeting will not have its time stamp changed.
How do I ensure that when my package is run, I retrieve all meeting and events information even if one hasn't changed?
Should I build a table made of IDs?! And if I do need to build a table made up of IDs, should I store this as a permanent table in my database or is there an SSIS data structure I can use to hold this information?
I hope this makes sense!
View 5 Replies
View Related
Oct 20, 2006
We are downloading 4 large (500mb) zip files in the package. Those come from 4 different FTP sites. Sometimes the FTP download on one of those fails.
Zip files contain images which need to be uploaded for existing listings. Each zip file is processed the exact same way. The question is how can I make the SSIS package so that each downloaded zip file can be processed immediately after the download - and not duplicating the processing logic?
View 3 Replies
View Related
Apr 2, 2008
Hello,
I am new to SSIS. I need to transfer data from SQL Server 2005 Operational Database to SQL Server 2005 Report Database. The upload needs to work every night. There are few master tables and remaining are transactions tables.
I am planning to create 2 packages one for master tables and other for transactions tables.
Is it the good approach?
Also few of transaction tables are heavy in terms of number of records. Will it better if i further break them in many packages?
I am using book "Microsoft SQL Server 2005 Integration Services Step by Step".
Can you suggest any other good book?
Regards,
Shailendra Shelar
View 7 Replies
View Related
Nov 21, 2015
Win 7 SP1 x64 PC. I installed SQL Server 2014 Dev Edition + Visual Studio 2015.
I'd like to create some basic ETL SSIS packages, and I worked very comfortably in 2008R2.
For 2014, I started with this tutorial:[URL]However, it says to go to Start->All Programs->Microsoft SQL Server->SQL Server Data Tools.
I did explicitly install SSDT when I installed VS2015. I also installed it separately. I see SSDT listed in Programs, and SSIS is running according to SQL Server Config Manager, and Services. Half of Microsoft's docs seem to be 2012 era, which is a shame because 2014 is out and it's nearly 2016...
how do I get to the GUI where I can design ETL packages?
View 7 Replies
View Related
Jun 8, 2007
OK, I was able to successfully migrate all of my DTS packages to SSIS, for SQL 2005. I can log into intergration services and see my packages listed under:
servername --> stored packages --> msdb. Now my question is, how can I open these packages, not run them open them in a design mode like you can in SQL 2000, you can double click on the package name and view the design of the package. how can i do that now that I have them in SQL 2005?
View 6 Replies
View Related
Sep 6, 2004
Hello,
I want to kow if the following architecture is good :
Disque 1&2 ( Raid 1)
c: OS
d: sqlserver + system tables + log files
Disque 3&4&5 (Raid 5)
e: data
View 1 Replies
View Related
Sep 29, 2004
Hi,
Someone can tell me if the following architecture is good :
f:master db
g:soft db + log
View 2 Replies
View Related
Dec 18, 2007
where can we get to know the architecture of sqlserver 2005
View 4 Replies
View Related
Aug 17, 2007
Hi i tried designing a SSIS package which loads only those rows which were different from existing rows in the table , i need to timestamp the existing row with an inactive date when a update of that row is inserted (ex: same studentID )
and the newly inserted row with a insert time stamp
so as to indicate the new row as currently active, in short i need to maintain history and current rows in same table , i tried using slowly changing dimension but could not figure out, anyone experience or knowledge regarding the Data loads please respond.
example of Data would be like
exisiting data
StudentID Name AGE Sex ADDRESS INSERTTIME UPDATETIME
12 DDS 14 M XYZ ST 2/4/06 NULL
14 hgS 17 M ABC ST 3/4/07 NULL
New row to insert would be
12 DDS 15 M DFG ST 4/5/07
the data should reflect
StudentID Name AGE Sex ADDRESS INSERTTIME UPDATETIME
12 DDS 14 M XYZ ST 2/4/06 4/5/07
12 DDS 15 M DFG ST 4/5/07 NULL
14 hgS 17 M ABC ST 3/4/07 NULL
Please provide your input as much as you can even though it might not be a 100% solution.
View 4 Replies
View Related
Jan 24, 2006
Hi,
I would like to design a SSIS package, which have couple of variables. It loads a xls file specified in a variable [varExcelFileFullPath] .
I will run it by commands: exec xp_cmdshell 'dtexec /SQL ....' (pls see an example below).
It seems it does not get the values passed in for those variables. I deployed the package to a sql server.
are there any grammar errors here? I copied it from dtexecui. It worked inside Dtexecui not in dos command.
exec xp_cmdshell 'dtexec /SQL "LoadExcelDB" /SERVER test /USER *** /PASSWORD ****
/MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW
/LOGGER "{6AA833A1-E4B2-4431-831B-DE695049DC61}";"Test.SuperBowl"
/Set Package.Variables[User::varExcelFileName].Properties[Value];"TestAdHocLayer"
/Set Package.Variables[User::varExcelWorkbookName].Value;"Sheet1$"
/Set Package.Variables[User::varExcelFileFullPath].Value;"D: estshareTestAdHocLayer.xls"
/Set Package.Variables[User::varDestinationTableName].Value;"FeaturesTmp"
/Set Package.Variables[User::varPreSQLAction].Value;"delete from FeaturesTmp"
'
thanks,
Guangming
View 2 Replies
View Related
Jan 22, 2001
We have database thats transaction intensive, so we are trying to sepetrare ldf file from mdf file to a different disk array. what raid should I use for the Transactional log file(.ldf).
Thank You,
John
View 1 Replies
View Related
Apr 2, 1999
I would like to know where I can find a senior database architect. Someone who can develop
and implement the database and its stored procedures. I am looking for an experienced person.
It is a contract position, in San Franicisco. The pay is good. Could anyone help me?
I tried Dice, Monster and it seems all of you are working...
View 2 Replies
View Related
Jun 2, 2004
I am replacing the corporate SQL Server at work. The new server will have 6 striped disks of 160G with about 4G of RAM. The current SQL Server currently has two instances which run web applications and a small database warehouse about 6G. Analysis Services is also installed.
Due to a couple of new apps being added to the server and the SQL Server 2000 enterprise license we acquired, i was thinking of adding 2 more instances so that the applications can be independently managed in terms of restarting the SQL Server. I also would like to permanently fix the memory settings on each instance to give more resources to more important applications. The log and data files would also be spilt onto 2 separate hard disks. i understand there are implications on performance such as CPU etc. Is it normally advisable to have more than 1 or 2 instances ? Most of the applications are not very CPU intensive. What other implications or performance issues would l have ?
View 4 Replies
View Related
Sep 1, 2006
We will be creating a moderately high-volume OLTP database application that needs 24/7 availability. We are planning to offload OLAP processing to a second copy of the system. We will be using SQL Server 2005.
I originally planned to set the second server up with SQL Server 2005 mirroring to cover the 24/7 availability requirement, with the idea that we could also do OLAP reporting off of the mirrored copy of the database. But I've gotten some indications that a mirror database is offline and not available for querying. So I figured I would use transactional replication to keep the OLAP database current. Now I am wondering if I need to use mirroring at all, or if I should just use transactional replication on the entire database and swap to the replicated database if the production server crashes.
What is everyone's opinion?
Replication only, for both OLAP reporting and failover?
Mirroring to one database for failover, with replication to a another database for OLAP reporting?
View 3 Replies
View Related
Feb 21, 2004
Hello, Everyone:
What duties are included within SQL Server Architecture? Thanks a lot.
ZYT
View 1 Replies
View Related
May 13, 2008
Can any one of you please explain the SQL architecture in a short..
Or can you give me the link, so i can refer it..
Its very urgent..
Pls help me..
Thanks In Advance
Ganesh
Solutions are easy. Understanding the problem, now, that's the hard part
View 4 Replies
View Related
Jun 14, 2007
We are designing a SQL Server architecture that will need to handle a lot of inserts and many users.
We are not sure what approach to take if we should build a distributed server architecture with a Single Master SQL and many Slave SQL Servers.
Or if we should get a really powerful machine with many processers to handle all the connections.
We project we will have about 5,000 Inserts per sec coming from 500,000 users.
What do you think or what has your previous experience been with handling many inserts and few reads for many users?
Thank you
View 2 Replies
View Related
Oct 20, 2007
Hi there,
My question is more "architectural" than technical.
One of our standard is to systematically enclose the code of our stored proc within a TRY/CATCH block.
What is your point of view?
Should a TRY/CATCH be included even for simple operations?
Are there any drawbacks?
Thanks!
EDIT: Thanks for your input.
View 4 Replies
View Related
Jun 29, 2007
I've implemented a solution with application, database and report server on seperate machines. The application is a web app and is Internet facing. What is the best method for executing reports on the RS server that are initiated from the web server? Using URL access requires a login or anonymous access neither of which are desired. Web services works but I loose access to the toolbar. Is there some other way to pull this off where I can let the public access reports and give them access to the toolbar?
Thanks.
View 4 Replies
View Related
Jun 5, 2007
In a situation where one may have a single master SQL Server that ultimately needs to communicate information back down to 1000's of downstream servers, what is the recommended architectural approach?
It doesn't feel right to have to add 1K-5K routes to the master SQL Server. Is there a way to have the dowstream servers "broadcast" their existence to the master, so that new servers can be added and updates can happen seamlessly? Does this fall into a pub-sub scenario or is there a better way? And, if so, how to ensure an open conversation (so that one server doesn't miss information that all the other servers received)? Should the master dynamically create routes or better to rely on an open conversation initiated by the downstream server?
View 20 Replies
View Related
Dec 31, 2007
Hi Everybody ,
Could Anybody please explain in detail about the Achitecture of Sql server reporting services?
1) What a Report processor in detail and what it does?
2) What is the exact work of Extensions?
3)What is SOAP and WMI?
Thanks in Advance
Mahasweta
happy new year
View 4 Replies
View Related
May 31, 2007
hi
i am haveing 1.5 years exp in asp.net(c#),i want to improve my knowlege in Database Architecture (datmodeling,uml,normalization,etc..).could anyone suggest me course or any books
View 4 Replies
View Related
May 23, 2007
I have the following stored procedure:
INSERT INTO MyTable ( Value1, Value 2)
VALUES( @Value1, @Value2)
SELECT SCOPE_IDENTITY()
How do I put this sp in the DAL typed dataset, so I can get the Identity value in the Business Layer?
View 2 Replies
View Related
May 22, 2008
When having multiple files for log, do they get filled one after the other (like Oracle) or proportionately fill?
------------------------
I think, therefore I am - Rene Descartes
View 2 Replies
View Related
