Multiple Packages Sharing One Solution XML Config File
Jun 22, 2007
Hey guys and girls,
This seems like a no brainer, but it's driving me nuts. I want one XML file for the entire solution. There are multiple packages in the solution which have different Connections in the conntion Managers. The packages share some connection names, and some are unique to the package.
Example:
LoadData.dtsx would have a source database connection named (SourceDB_OLEDB) and a oledb connection (DataWarehouse_OLEDB).
LoadDataMart.dtsx would use the same name for the (DataWarehouse_OLEDB) connection and have another oledb connection (DataMart_OLEDB)
I want one XML config file that has all the connection strings, but the problem is that the LoadDataMart.dtsx will throw an error:
Error 1 Error loading LoadDataMart.dtsx: The connection "SourceDB_OLEDB" is not found. This error is thrown by Connections collection when the specific connection element is not found. c:ssisLoadDataMart.dtsx 1 1
I would like to use one configuration file for multiple SSIS packages. How I want to do this is to save the connection string of my production server with the initial database field excluded and set that per package file. I am able to read in the configuration file into 2 different packages without issue but I cannot see how to extract the connection string into a ODBC Database Connection object in order to set the initial database. Is there some way I am not thinking of to do this, either through the designer or programatically?
Thanks for any help you all can provide! I am so n00b to SSIS.
I have several packages that all use the same DTSConfig file to map the OLEDB connections. This works fine in the development environment, however when i run a rebuild with Deployment set to true i get the error
Error 75 System.ApplicationException: Could not copy file "C:Dev_TFSTowergateTowergate.BdwTowergate.Bdw.Ssis.TransformPackagePath.dtsConfig" to the deployment utility output directory "C:Dev_TFSTowergateTowergate.BdwTowergate.Bdw.Ssis.TransforminDeployment". ---> System.IO.IOException: The file 'C:Dev_TFSTowergateTowergate.BdwTowergate.Bdw.Ssis.TransforminDeploymentPackagePath.dtsConfig' already exists. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite) at Microsoft.DataTransformationServices.Project.DataTransformationsProjectBuilder.CopyFiles(ICollection fileNames, String outputPath) --- End of inner exception stack trace --- at Microsoft.DataTransformationServices.Project.DataTransformationsProjectBuilder.CopyFiles(ICollection fileNames, String outputPath) at Microsoft.DataTransformationServices.Project.DataTransformationsProjectBuilder.CreateDeploymentUtility(IOutputWindow outputWindow)
The DTSConfig file is included in the project under Misc in the development studio as all developers need to have access to the same config file.
The net result is that the MSI deployment package doesnt get built so we cannot do an automated delivery of the data.
Is there a work round or a setting that i can implement to remove the error?
I've read that a single config file can be used across multiple packages, but am unsure if it applies to what I'm trying to achieve. As an example, I have 2 packages, PackageA and PackageB. Each package has a connection manager defined - ManagerA and ManagerB and I want a single package configuration file to contain the connection string for each manager.
I can define the package configuration of PackageA with no problems. I choose XML file, define the filename and add the connection string to the config.
I then open PackageB Package Configuration and point it to the existing config file, IS tells me the file already exists and allows me to choose the option to "Reuse Existing". I then supply a configuration name and the wizard finishes, without letting me expose any properties. At this point, the configuration file still contains config details from PackageA.
If I then choose to Edit the package configuration in PackageB, I get a warning saying the config values in the config file (relating to PackageA) cannot be found, and the wizard continues. The subsequent PackageB properties which I add overwrite the existing PackageA properties.
So it would appear that what I'm trying to do cannot be done, and I need a separate configuration file for each package?
I am working on a project currently where we have many SSIS packages and we want to minimize the number of config files to be used. What I was thinking was using one master config file which will have Server/Database info which will be used by all the packages. Now the thing is there are multiple packages which are being executed from within a master package and I was wondering if this will be an issue. Is there anything else that I will have to keep in mind? Any help is appreciated.
I'm using an XML file to configure my dataBase connection string. I try to deploy my package on a new server and it works perfectly.
Then I made a second package which also need a configuration for the dataBase connection string. (I made the connection with the connection manager inside packages). The configuration is the same that for my first package, so I thought to use the same configuration file.
I can use the same configuration file but the problem is when I try to generate a deployment for my solution. I got an error which tell me that the xxx.dtsConfig file already exist and can not be copy again.
When I made the configuration in the second package I said that I want to reuse the file ... and I thought that for the deployment SSIS would know that it has to include that file only once ...
I have a rather dumb question, it appears that we can have more than 1 package (.dtsx) in a solution. If I have multiple packages within the same solution, how do I invoke them from the main package in the solution? Thanks in Advance.
I have a parent package that uses Execute Package Task against three child packages. They are all setup to use the same configuration file, and the packages are all located in the same folder. Purpose of the configuation is to use the same connection strings for each child package.
I removed the path from the config. file specification to facilitate deployment. During debugging, the child packages cannot find the config file when they are run via the parent unless the full path is specified. If run standalone, a child package can find the config. file without the path. As I said, all dtsx filaes and the config file is in the same folder.
Also, the password-removal issue is killing me in development. Everytime I touch the config. file, it removes the password from the connection string. I have "protection level" set to "don't save sensitive" - I'd appreciate help with this too.
Currently have a single hard coded file path to the SSRS config file which parses the file and provides the reporting services web service url. My question is how would i run this same query against 100s of servers that may or may not share the same file path as the one hard coded ?
Is there a way to query the registry to find the location of the config file of any server ? which could be on D, E, F, H, etc.
I know I can string together the address followed by "reports" and named instance if needed, but some instances may not have used the default virtual directory name (Reports).
Am I going about this the hard way ? Is there a location where the web service url exists in a table ? I could not locate anything in the Reporting service database. Basically need to inventory all of my reporting services url's.
Ok, I understand it is possible, but I still can't quite get the mechanics of it to work.
I create a new BIDS project, and add a package to it, and add 3 connection managers, and set the server, instance, DB, etc. for SQL connections. I change the name of each connection manager so the server name is not there, but use something more like a generic name of what the database is. Fine. I right click in the Control Flow area and go to configurations. I enable configurations, and save to a common place on the C drive. Save everything, exit, VS, fine, and I export all connection managers properties.
Then I start Visual Studio again. Create another new BIDS project, add a package to it, and add 3 connection managers. I don't actually connect them, but use those same generic names from the first iteration. I enable configurations for the package. I am somewhat expecting to see the connections change from the same-o-default to the data I used in the first go around.
The scenario is an ETL that takes flat file feeds via FTP to move data into varous production SQL server databases nightly.
There are a number of packages involved, and this depends upon the type of data being sent.
There are a set number of servers and databases to receive the transformed data. I would like to be able to define say 3 servers, and maybe a couple of databases in each one time in the configuration. For simplicity lets say 6 databases total. I would like a single point of maintenance for these 6 locations. I would like all connection managers in all packages in all solutions to share these 6 settings in all connection managers. Is this possible? From my initial attempts, it would appear each package gets its own independent list of connection managers and which must be configured separately. I don't see how to share settings, which is really where the power of SSIS configurations would be.
Similarly, I would like to be able to locate flat files at given paths. The package may know the name of the file it is looking for, or the file it will create, and the folder path needs to be computed from a configured folder root, and for the package connection manager to store only the name of the file, which never changes.
I can not see how to set something like this up with connection managers and configurations. Perhaps I still need a highly customized solution to achieve this, as we did for DTS, where we would only execute packages using our own executor, which would load the package, search out all environment specific settings and modify them on the fly prior to executing the package.
Thanks for any direction you can give here. The books I've read seem to imply I might be able to do what I need here, but I can't seem to find the mechanics of making it happen.
It is pretty easy to create the package configuration to the shared, never-changing path to XML file. But to get this to work, I have to add a bunch of package variables which are driven from the configuration. I then use expressions to compute actual paths to source files and packages.
This business of copying a block of variables from one package to another is extremely tedious, and at least to me, something of a maintenance issue. I have found if I get ahead of myself and forget to create the variables first, then Visual Studio is NOT very forgiving, and will complain repeatedly just prior to crashing without the offer to save anything. If I add a new variable, it is like an easter egg hunt. This does NOT feel very single point of maintenance when variables must be used in this way to really effectively apply configuration, and it is so tedious and error prone, and unforgiving. I seem to spend a great percentage of time copying the names and datatypes and values of variables ONE AT A TIME. What is that all about?
I hear people telling the praises of configurations, but I must be missing something.
With DTS I used a system where global scope settings were "injected" into packages as they ran, and I never had to worry about anything, and it was damn hard to goof it.
Please someone, evangelize me!! Are people bypassing VS and using NOTEPAD on the packages or something like that to save time?
Basically it seems that whilst you can indeed share a config file, it doesn't aggregate sets of say different connection managers, into a superset - you choose to reuse, but it actually overwrites. Any ideas, anyone??
I have a question regarding Raw Files. I am breaking a large package into more modular components for better processing and debugging.
The process will start with a preparatory dataflow that will create a Raw File(s). This Raw File will then be used as the source in possibly 6 data flows and/or packages.
My question is whether 1 Raw File can be read concurrently by the multiple jobs and how this would affect processing. I'm assuming that this would slow processing.
My other option is to Multicast the writing of the Raw File to 5 other versions of the file. All would be identical except for filename. Obviously this would use more disk space but this is not a concern as we have lots of disk space. Our concern is for speedy processing.
If you have experience with Raw Files, please let me know how you approached this issue. As always, blogs and specific examples are always great!
My current environment has multiple packages stored in SQL server (MSDB). When working on a set of packages I want to bring them into my local development area Add existing package only allows you to pull one package at a time - anyone have the secret to selecting multiples
I am currently migrating from Oracle to MS SQL Server 2005 using SSIS. Since the new schema being used on the SQL Server is very different, we have created separate packages for each target table.
We have several different sets of data stored as separate Oracle schemas, and I would now like to change to a different schema for the OLE DB Source objects in my packages to use (Table/column names to remain the same). While I can go through all packages and search through the drop-down list to select the table of the same name from other Schema, doing so is an extremely slow process.
Is it possible to use configuration files to set a schema to be used for a number of OLE DB Sources over a number of packages?
At the moment I am doing a find and replace over the .dtsx files, but can't believe I have to resort to this for something that I imagine many people would want to do with SSIS.
Hi guys,I've got two apps that are going to have pretty much the same users and i wanted them to have the same ASPNET.MDF user database. Is it possible to store the ASPNET.MDF in a different location and then have the two apps access it from there or alternatively have it in the App_Data folder of one of the apps and then have the second app access it from there?Thanx
I'm looking for some advice on how to manage reports that use the same query in their datasets. I have multiple reports that use several datasets that are the same. If I need to make a change to one dataset, I need to remember to update the other datasets. Of course I don't always remember to do that!
Is there a way to create a dataset in a single location and then share it? I was thinking of using a View but I don't think it'll accept the parameters.
I've been cutting & pasting the entire query as I make change but I'm afraid it'll mess that up or forget to update a dataset.
Good afternoon, This is a simple question....how can I order packages by name in solution explorer ? I have a solution with more or less 50 packages and it's very ime consuming to find a package ! ehehe
I have a situation where I would like to set up one DB server instance for US and anther one for AUStralia. The Instance should share the same data base/hardsisk. One of the instances will be running in using US time zone and anothe one in Australia time zone. There will be 2 Application instaces one poinint to each SQL Server instance. This applications does write into the database
So the underlying requirment is to have 2 database server running in differnet time zones but accessing the same disk/set of data What is the best way to do this? Is it linked servers?Girish
I have a package that executes 3 other packages in it. From the command line I pass in which config file to use (just contains db/seerver) does this setting get carried over into the child packages as well?
I have created a solution which contains only 2 packages say Package1.dtsx and Pakage2.dtsx. I want to create a deployment utility to deploy onto other developers machince. I changed the project properties "CreateDeploymentUtility" to TRUE. When I do the build it is not creating the files in "Deployment" folder. It is saying Rebuild All Failed but the error is not showing.
For more information the 2 packages have 4 indirect configurations from environment variables which are storing the actual config file path.
I was hoping that someone may be able to help me with an issue that I am having while trying to deploy my projects using MSI. Here is what I am trying to do.
I have two programs that are using MS SQL CE to share information. The first program is a windows service. The second program is a windows application that I am using mainly as a UI for the service. Each program has a seperate MSI setup program. I am using windows vista and VS 2005 on my developement machine.
The issue that I am having is when I deploy the programs. The windows service is working fine. It reads and writes to the database as expected. I know this because when I debug the UI program through VS 2005, I can read and write from the common SQL CE database with no issues. I am also pretty confident in the MSI program for the UI program because if I install the UI program on the developement computer and run the UI program as administrator it runs fine.
The issue arises when I simply open the UI program or when I try to run the UI program on a windows xp sp2 machine. I am no longer able to access the database file. I have tried many diffent things, but so far with no luck. Most of the attempts to fix this so far have been concerning the location of the database file these have had little or no effect. Again if I open the UI program using the right click option of 'Run as Administrator' in windows vista, the problem disappears.
Thanks in advance for any help. Even a push in the right direction would be helpful at this point. I will keep working on this issue, if I find a solution myself I will repost it here.
Hi,I have some database files (.MDF, .LDF,...) on the server. When I tryto delete them, the warning "Cannot delete file: There has been asharing violation. The source or destination file may be in use."appears.Since I am new to the environment I don't know where the files comefrom and where they might be used.Can anybody tell me what to do to delete those files?Thank you.
Hi, I am facing a wierd problem working with SQL Server Mobile edition database. The code works fine for some time and then it suddenly starts throwing "There is a file sharing violation. A different process might be using the file." I am disposing and closing the connection after each operation. Please find the function code below which is being used for any database operation.
I have deployed to production a number of nested packages (parent packages that call child packages) to the SQL msdb via the Save As option rather than building a deployment utility. These packages reference configuration files in a static location off of the c: drive on the production server. In the development environment, when connection changes are made and I run the Reload with Upgrade option the connection manager takes on the new server and user id settings. However, out on the production side I get the following error from the SQL job log:
Cannot load the XML configuration file. The XML configuration file may be malformed or not valid.
As a result the SQL job uses the default connection information which references the development database rather than the production database. I did research the error but found no good solutions. Is there a way to ensure the configuration files are formed correctly and that the packages are correctly referencing the configuration files? We are trying to run the ETL updates via a SQL job.
