Sharing An XML Configuration File Between Packages
Jan 22, 2007
Hello,
I'm using an XML file to configure my dataBase connection string. I try to deploy my package on a new server and it works perfectly.
Then I made a second package which also need a configuration for the dataBase connection string. (I made the connection with the connection manager inside packages). The configuration is the same that for my first package, so I thought to use the same configuration file.
I can use the same configuration file but the problem is when I try to generate a deployment for my solution. I got an error which tell me that the xxx.dtsConfig file already exist and can not be copy again.
When I made the configuration in the second package I said that I want to reuse the file ... and I thought that for the deployment SSIS would know that it has to include that file only once ...
Ok, I understand it is possible, but I still can't quite get the mechanics of it to work.
I create a new BIDS project, and add a package to it, and add 3 connection managers, and set the server, instance, DB, etc. for SQL connections. I change the name of each connection manager so the server name is not there, but use something more like a generic name of what the database is. Fine. I right click in the Control Flow area and go to configurations. I enable configurations, and save to a common place on the C drive. Save everything, exit, VS, fine, and I export all connection managers properties.
Then I start Visual Studio again. Create another new BIDS project, add a package to it, and add 3 connection managers. I don't actually connect them, but use those same generic names from the first iteration. I enable configurations for the package. I am somewhat expecting to see the connections change from the same-o-default to the data I used in the first go around.
The scenario is an ETL that takes flat file feeds via FTP to move data into varous production SQL server databases nightly.
There are a number of packages involved, and this depends upon the type of data being sent.
There are a set number of servers and databases to receive the transformed data. I would like to be able to define say 3 servers, and maybe a couple of databases in each one time in the configuration. For simplicity lets say 6 databases total. I would like a single point of maintenance for these 6 locations. I would like all connection managers in all packages in all solutions to share these 6 settings in all connection managers. Is this possible? From my initial attempts, it would appear each package gets its own independent list of connection managers and which must be configured separately. I don't see how to share settings, which is really where the power of SSIS configurations would be.
Similarly, I would like to be able to locate flat files at given paths. The package may know the name of the file it is looking for, or the file it will create, and the folder path needs to be computed from a configured folder root, and for the package connection manager to store only the name of the file, which never changes.
I can not see how to set something like this up with connection managers and configurations. Perhaps I still need a highly customized solution to achieve this, as we did for DTS, where we would only execute packages using our own executor, which would load the package, search out all environment specific settings and modify them on the fly prior to executing the package.
Thanks for any direction you can give here. The books I've read seem to imply I might be able to do what I need here, but I can't seem to find the mechanics of making it happen.
It is pretty easy to create the package configuration to the shared, never-changing path to XML file. But to get this to work, I have to add a bunch of package variables which are driven from the configuration. I then use expressions to compute actual paths to source files and packages.
This business of copying a block of variables from one package to another is extremely tedious, and at least to me, something of a maintenance issue. I have found if I get ahead of myself and forget to create the variables first, then Visual Studio is NOT very forgiving, and will complain repeatedly just prior to crashing without the offer to save anything. If I add a new variable, it is like an easter egg hunt. This does NOT feel very single point of maintenance when variables must be used in this way to really effectively apply configuration, and it is so tedious and error prone, and unforgiving. I seem to spend a great percentage of time copying the names and datatypes and values of variables ONE AT A TIME. What is that all about?
I hear people telling the praises of configurations, but I must be missing something.
With DTS I used a system where global scope settings were "injected" into packages as they ran, and I never had to worry about anything, and it was damn hard to goof it.
Please someone, evangelize me!! Are people bypassing VS and using NOTEPAD on the packages or something like that to save time?
This seems like a no brainer, but it's driving me nuts. I want one XML file for the entire solution. There are multiple packages in the solution which have different Connections in the conntion Managers. The packages share some connection names, and some are unique to the package.
Example:
LoadData.dtsx would have a source database connection named (SourceDB_OLEDB) and a oledb connection (DataWarehouse_OLEDB).
LoadDataMart.dtsx would use the same name for the (DataWarehouse_OLEDB) connection and have another oledb connection (DataMart_OLEDB)
I want one XML config file that has all the connection strings, but the problem is that the LoadDataMart.dtsx will throw an error:
Error 1 Error loading LoadDataMart.dtsx: The connection "SourceDB_OLEDB" is not found. This error is thrown by Connections collection when the specific connection element is not found. c:ssisLoadDataMart.dtsx 1 1
I have two packages share same configuration file. There are 4 connections in the first package and 2 connections in the second. Configuration file contains properties for 4 connections. First package opens in VS designer with no error. The problem with second package. I get error that connection exist in the configuration but not in the package and "Could not copy file " Master_Configuration.dtsConfig" to the deployment utility output directory. Master_Configuration.dtsConfig' already exists." Any help?
Basically it seems that whilst you can indeed share a config file, it doesn't aggregate sets of say different connection managers, into a superset - you choose to reuse, but it actually overwrites. Any ideas, anyone??
I am currently migrating from Oracle to MS SQL Server 2005 using SSIS. Since the new schema being used on the SQL Server is very different, we have created separate packages for each target table.
We have several different sets of data stored as separate Oracle schemas, and I would now like to change to a different schema for the OLE DB Source objects in my packages to use (Table/column names to remain the same). While I can go through all packages and search through the drop-down list to select the table of the same name from other Schema, doing so is an extremely slow process.
Is it possible to use configuration files to set a schema to be used for a number of OLE DB Sources over a number of packages?
At the moment I am doing a find and replace over the .dtsx files, but can't believe I have to resort to this for something that I imagine many people would want to do with SSIS.
I'm fairly new to the SSIS world, and I've recently ported a bunch of dts packages over to SSIS. I'm an ASP.NET developer so I'm very familiar with the capabilities that configuration files give you, and I attempted to set up my solution as follows:
All of my "Data Sources" are at the project level, and added (with the same name) to each package. I wanted to have a single config file that had all of the project-level settings (i.e. connection strings, data file paths, etc). I then have a config for each package with the package level settings - i.e. variables, etc.
The problem becomes that all packages do not use all data sources. This results in an error when I try to open up a package for editing, it complains that it doesn't have a reference to data source XYZ that it is seeing in the configuration file.
Is there any way that I can get around this? If I have a password to a database change, I don't want to have to look through every config file and change it in multiple places.
Hi, I am using custom dll in script component in SSIS package. This dll is looking for some configuration settings and dsplays the message as "Configuration section could not be found in the configuration source" . Please tell me the configuration source it looks for.
I was hoping that someone may be able to help me with an issue that I am having while trying to deploy my projects using MSI. Here is what I am trying to do.
I have two programs that are using MS SQL CE to share information. The first program is a windows service. The second program is a windows application that I am using mainly as a UI for the service. Each program has a seperate MSI setup program. I am using windows vista and VS 2005 on my developement machine.
The issue that I am having is when I deploy the programs. The windows service is working fine. It reads and writes to the database as expected. I know this because when I debug the UI program through VS 2005, I can read and write from the common SQL CE database with no issues. I am also pretty confident in the MSI program for the UI program because if I install the UI program on the developement computer and run the UI program as administrator it runs fine.
The issue arises when I simply open the UI program or when I try to run the UI program on a windows xp sp2 machine. I am no longer able to access the database file. I have tried many diffent things, but so far with no luck. Most of the attempts to fix this so far have been concerning the location of the database file these have had little or no effect. Again if I open the UI program using the right click option of 'Run as Administrator' in windows vista, the problem disappears.
Thanks in advance for any help. Even a push in the right direction would be helpful at this point. I will keep working on this issue, if I find a solution myself I will repost it here.
Hi,I have some database files (.MDF, .LDF,...) on the server. When I tryto delete them, the warning "Cannot delete file: There has been asharing violation. The source or destination file may be in use."appears.Since I am new to the environment I don't know where the files comefrom and where they might be used.Can anybody tell me what to do to delete those files?Thank you.
Hi, I am facing a wierd problem working with SQL Server Mobile edition database. The code works fine for some time and then it suddenly starts throwing "There is a file sharing violation. A different process might be using the file." I am disposing and closing the connection after each operation. Please find the function code below which is being used for any database operation.
I've run into a problem with SSIS packages wherein tasks that write or copy files, or create or delete directories, quit execution without any hint of an error nor a failure message, when called from an ASP.NET 2.0 application running on any other machine than the one where the package was created from. By all indications it appeared to be an identity/permissions problem.
Our application involves a separate web server and database server. Both have SQL Server 2005 installed, but the application server originally only had Integration services. The packages are file system-deployed on the application server, and are called using Microsoft.SqlServer.Dts.Runtime methods. For all packages that involve file system tasks, the above problem occurs.
When the above packages are run using the command prompt (either DTEXEC or DTEXECUI) the packages execute just fine. This is expected since we are using an administrative account. However when a ShellExecute of the same command is called from ASP.NET, the same problem occurs.
I've tried giving administrative permissions to the ASPNET worker process user to no avail.
I have likewise attempted to use the SQL Server Agent job approach but that approach might not be acceptable for our clients since it means installing SQL Server 2005 Database services on the application server.
I have read the relevant threads in this forum, namely http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1044739&SiteID=1 and http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=927084&SiteID=1 but failed to find any solution appropriate for our set up.
I would just like to give my feedback on the configuration packages implementation in SSIS:
THIS STUFF SHOULD BE A NO BRAINER AS IT IS A COMMON REQUIREMENT ON ETL PROJECTS.
I have trolled through these forums and found that I am not alone on this issue. Note for SSIS Team: Please bring out a HOWTO on this subject that details all the little work arounds because intuitively used it does not work.
Consider this:
I develop in BIDS on my client and everything runs. I create a config package and I choose only the Name and ConnectionString Properties for each Connection Manager. (Thanks to Jamie for that tip)
I build the app and a deployment utility is created. I navigate to the deployment directory, and deploy. I point to a mapped share on the production server, change all the properties for production and continue until finished.
On the production server I run a script containing:
Still: Cannot find the configuration file: source.dtsConfig
I have to MANUALLY edit the dtsx file to put the path in from the the name of the config file and THEN it works.
Thinking that it might not like a mapped share from desktop to deploy to, I copIED the contents of the depLoyMENT directory on my desktop to a folder on the production server and deploy from there. It should pick up the proper path from there. NOT!!!!
So it was back to editing the dtsx file.
So there you have it. I await all comments with great interest.
I hope this is all fixed in SP2, Because I don't want to waste anymore time on what should be a simple task.
We manage some SSIS servers, which has only SSIS and SSIS tools installed on them and not the sql server DB.
SSIS packages and configuration files are deployed on a NAS. We run the SSIS packages through DTEXEC by logging in to the server.
We want to allow developers to run their packages on their own on the server, but at the same time we dont want to give them physical access on the server i.e we do not want to add them into RDP users list on server properties. We want them to allow running their packages remotely on the server.
One way We could think of is by using powershell remoting and we are working on that. But is there any other way or any tool already present for the same.
I need some clarification. I am trying to utilize the XML Direct Configuration in my SSIS packages to utilize database connections in the package. I am wanting to utilize this the same way you could use UDLs in the SQL Server 2000 DTS packages.
Currently, I am creating the dtsConfig file and saving it to my desktop. I then modify it with notepad and add the following configuration where it looks like this:
Once this is created, I am trying to "re-use" this in SSIS packages created moving forward, where they all point to this configuration for the same database connection.
What I don't understand is when "Enabling Package Configuration" and then pointing to this dtsConfig file doesn't create a connection in "Connection Manager" NOR does it provide a way to create a blank connection and point to this configuration.
I feel like I am missing something thats so "great" about XML configuration files. Any help would be appreciated.
We have been conducting some testing regarding package deployment and SQL Server based configuration. It seems there is a problem that was documented in an MS Feedback entry (
I searched for a viable answer, work-around, or hotfix that would address this issue but found none.
The jist is that a package that uses SQL Server based configuration must have an xml-based configuration entry to "re-point" the SQL Server based configuration connection manager to the deployment target server. We cannot use environment variable or registry as this is against internal policy. The problem is that even though the xml based config is specified first in the list of package configurations it does not get applied first at run-time. So, when the package runs from a SQL Server Agent job the package's connection manager for the configuration entries is not updated.
The package runs correctly through BIDS. You can change the connection string in the .dtsConfig file and the SQL Server based package configuration is obtained from the correct source.
Environment is SQL Server Enterprise Edition 64-bit w/ SP1, Windows Server 2003 Enterprise Ed. 64-bit.
Does anyone have any experience with this issue? Know any hotfixes, other work-arounds?
I currently have multiple (parent and child) packages using the same config file. The config file has entries for connections to a number of systems. All of them are not used from the child packages. Hence, my child package throws an error when it tries to configure using the same config file because it can't find the extra connections in my connection collection.
Does anyone have any ideas on the best way to go about resolving this? Is multiple config files (one for each connection) the only way?
I've found this problem that when I change settings in my configuration file it does not automatically apply to all child packages which uses the same configuration file if run from a job in SQL Server Agent. I need to open the package and save it again from BIDS. I use one "load group" package to execute all other packages.
Is there a way from the job configuration to set a setting so the package allways will have the newest configuration?
Hi everybody, I moved my packages to the new machine, and have problem with parent variables. When child package tries to get parent variable value I get an error:
Information: 0x40016042 at LoopPackage: The package is attempting to configure from the parent variable "MAIN_SesId".
Warning: 0x80012028 at LoopPackage: Process configuration failed to set the destination at the package path of "Package.Variables[User::MAIN_SesId].Properties[Value]". This occurs when attempting to set the destination property or variable fails. Check the destination property or variable.
I cannot get the log file path read from the configuration.
If the path in the Connection Manager is invalid, package throws an error "SSIS logging provider has failed to open the log" instead of reading it from the config.file. What am I doing wrong?
Here is the portion of the config file. Everthing else is read from the config file correctly.
I have a simple SSIS package (stored in the file system) that gets a file path from a configuration file. The configuration type is an indirect XML configuration file that uses an environment variable to store the location of the configuration file.
When I run the package using dtexecui, or just dtexec from a command line, the package successfully picks up the file path from the configuration file (for verification I am writing out the variable containing the path to the log file). However when I run the package from a SQL Agent job it appears that the configuration file is not being used (the path is set to the same dummy path that I used during development). I have tried running the job as both a CmdExec and an Integration Services job and both fail on the same thing (invalid file path).
Both the SQL Server Agent service and the Integration Services service use a domain account as their start up account. This domain account has been included in the local administrators group on the server (in case it was having trouble accessing the environment variables).
What is the problem here €“ surely changing the way in which the package runs should not affect the configuration file settings! Any help would be appreciated€¦
We are deploying our SSIS packages into different folders. For example: Test1 est2 production
Test1 points to the Test1 database, etc.
So, I configured my SSIS package to use a database connection called dbMAIN.
I then setup the SSIS File Configuration (XML) so that "dbMain" points to Test1. This xml file, called Global.dtsConfig, sits in C: est1, the same place as my ssis.dtsx file.
I want to copy my dtsx file and my dtsconfig file into est2 and production. The problem is that the the location of dtsconfig is HARDCODED in the ssis package!
How do I dynamically change it?
I tried to hardcode the location to ".Global.dtsconfig", but that did not work! Please advise!
I'm sitting down again to see if I can get configuration files to work. To test, I've set up a simple package that reads a flat file and writes to a table.
I've enabled configurations by clicking the checkbox in the configurations window. I am using an xml file on the local server, and there is an absolute path defined to the file. I also added a password to the xml file so the connection will work. To this point, everything works.
However, when I go into the configuration file and edit it to point to another server, it doesn't pick it up!
That is, when I run my package, it still writes the data to the old server. It's like it's completely ignoring the configuration file and just using the OLE DB connection manager instead.
However, if I delete the OLE DB conn mgr, the package doesn't work.
That's the question. I'm sure I will be encourage to use xml or other configuration types rather than INI files. But INI file has the advantage of not embedding any object property path and can be used for both packages and other applications. Imagine a large implementation with hundreds of packages and dozens of class libraries and applications which are configured from the same INI file. Imagine also that the production environment must support both DTS 2000 and SSIS 2005 packages as it's impossible to migrate all at once.
I've read that SSIS would support INI configuration files:
In Package Configurations Kirk says:
"...The two types of package configurations that haven't been documented yet in books online is INI and SQL configurations. INI package configurations have basically been provided for backward compatibility..."
And in BOL, in the DTSConfigurationType Enumeration description we can find that the INIFile configuration Type is also mentioned.
Anyway, a possible workaround is building a custom task with an UI that allows to assign INI keys to object properties, but I wanted to know if it's possible to avoid this.
I created an SSIS package that is connecting to 2 SQL Databases, one connection is using windows auth and the other database connection is using SQL Auth to connect to the database. I setup a config file to use for my ID, servername, database, and password but when I run my SSIS package it fails stating'
"[Execute SQL Task] Error: Failed to acquire connection "dbConnection". Connection may not be configured correctly or you may not have the right permissions on this connection."
If I open up my connecting dbConnection in Connection Manager the username is filled it but the password is blank, if I enter in the password and click 'test connection' it passes. I have the ProtecionLevel = EncryptSensitiveWithUserKey but the password is still 'blanked out' So my question is
1) how can I use the config file to store my id and pwd for my db connection? 2) how can I get my package to read the config file to get the ID and password for my database. 3) is there another way to do this? Would a varaiable work for this, if so how do they work?
I have a package where I store connection managers in a configuration file (XML file). I updated manually the XML file to add the passwords.
When I launch the command line generated by dtexecui, everything is good: "dtexecui /FILE "E:mssql90DTSPackagesBVOMDWRefreshStage Refresh EUR 1.dtsx" /CONFIGFILE "E:mssql90DTSPackagesBVOMDWRefreshEnv_Prod.dtsconfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW "
When I launch the command line with dtexec: "dtexec /FILE "E:mssql90DTSPackagesBVOMDWRefreshStage Refresh EUR 1.dtsx" /CONFIGFILE "E:mssql90DTSPackagesBVOMDWRefreshEnv_Prod.dtsconfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW ", I have the familiar error: "Failed to decrypt protected XML node "DTSassword""
I have the package level set to "EncryptSensitiveWithUserKey", but I think I don't need to update it to "EncryptSensitiveWithPassword" because it works with dtexecui.
