Performance Issues, Size Of SSIS-packages
Feb 7, 2007
Hello!
I have a general question concerning the impact on the performance of massive parallel data imports in one SSIS-package.
We have a Database on a SQL2005 SP1-Server (2 Xeons 3,8 Ghz, 4GB of RAM) for a report web-app which is updated every day with data of the last year/3 years. The data is extracted from several different DBs on multiple machines at different locations. Right now, there are imports/transformations from 7 companies at 3 locations. The table has ~80 columns and about 2 Mio. Rows. I built a SSIS-Package with one companies import and added the others by c&p-ing all the tasks in the package and changing connection parameters and values. Soon there will be 6 more companies to do imports with, and there will possibly be about 20 some day.
Now, when these 7 imports run parallel, there are 3 simultaneous imports from the same Source Server. Sometimes one of these imports seem to hang up. I cannot reproduce it, when I run the package 2 or 3 times, it´s gone. So I put some of the imports in line to reduce the parallel working tasks to 4. Then the problem dissappears. The "MaxConcurrentExecutables" Value is set to 6. "Retain same connection" is set "TRUE".
My questions, regarding stability and performance, are:
1.) Is it better to do those imports in seperate packages, if yes can I schedule multiple packages to execute parallel at the "SQL Server Agent"?
2.) Or should they be combined in one package, running (partly) parallel?
3.) What is the appropriate value for the "MaxConcurrentExecutables" Value and what options do I have to speed up those imports?
View 7 Replies
ADVERTISEMENT
Feb 13, 2006
Is there any information around what the SSIS packages are doing in the first 5-10 seconds of execution, and ways to speed this process up?
View 3 Replies
View Related
Jan 9, 2007
I've run into a problem with SSIS packages wherein tasks that write or copy files, or create or delete directories, quit execution without any hint of an error nor a failure message, when called from an ASP.NET 2.0 application running on any other machine than the one where the package was created from. By all indications it appeared to be an identity/permissions problem.
Our application involves a separate web server and database server. Both have SQL Server 2005 installed, but the application server originally only had Integration services. The packages are file system-deployed on the application server, and are called using Microsoft.SqlServer.Dts.Runtime methods. For all packages that involve file system tasks, the above problem occurs.
When the above packages are run using the command prompt (either DTEXEC or DTEXECUI) the packages execute just fine. This is expected since we are using an administrative account. However when a ShellExecute of the same command is called from ASP.NET, the same problem occurs.
I've tried giving administrative permissions to the ASPNET worker process user to no avail.
I have likewise attempted to use the SQL Server Agent job approach but that approach might not be acceptable for our clients since it means installing SQL Server 2005 Database services on the application server.
I have read the relevant threads in this forum, namely http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1044739&SiteID=1 and http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=927084&SiteID=1 but failed to find any solution appropriate for our set up.
Anybody got any idea on how to go about this?
View 33 Replies
View Related
Apr 22, 2015
We manage some SSIS servers, which has only SSIS and SSIS tools installed on them and not the sql server DB.
SSIS packages and configuration files are deployed on a NAS. We run the SSIS packages through DTEXEC by logging in to the server.
We want to allow developers to run their packages on their own on the server, but at the same time we dont want to give them physical access on the server i.e we do not want to add them into RDP users list on server properties. We want them to allow running their packages remotely on the server.
One way We could think of is by using powershell remoting and we are working on that. But is there any other way or any tool already present for the same.
View 4 Replies
View Related
Jan 20, 2006
I'm having serious problems with the IDE for SSIS for projects that contain more than 5 packages. Especially if these packages call each other with a run package task thats configured with a file connection. Especially annoying are the 20+ "Document contains one or more extremely long lines of text.." messages that pop up during loading / validating. For my project with around 30 packages it takes me around 10 minutes to click through all these pop ups in addition to the long loading time.
Anyone got any tips on this specifically or how to improve performance in the IDE in general? As it is now, the product is a REAL pain to work with for large projects.
View 2 Replies
View Related
Jul 17, 2006
We're having a performance problem with a package since an error occurred. The original error came from the Job Manager whicj was unable to start a thread. Our understanding is that this is a memory related issue and we'll deal with that.
What is realy odd is what happened after that. The package executes from a SQL Agent job that includes 3 other packages. Each package is stored on the file system. Package execution time for the affected package changed from less than a minute to over 5 minutes. The other packages continued to execute normally.
In checking the logs there is large time gap between the start of the SQL Agent step and the first pre-validation message that accounts for 4 minutes, as if there is some issue in loading the package. The issue is ongoing. Does anyone know what happens between the start of the SQL Agent job and the first prevalidate message? Is this some type of caching issue?
SQL Agent step starts at 4:05:27
First Prevalidate message: 4:09:24
Package Execution start: 4:09:57
End Execution 4:10:59
thanks
Peter
View 1 Replies
View Related
Apr 9, 2004
We have an application that runs across the WAN to multiple locations. Performance is poor and we are looking at ways to improve performance. One suggestion from our Sr. Network Administrator is to change our Network Packet Sizes across all points, SQL Server & NIC to match the outgoing Router. This would be a size of 1440.
Does anyone have any thoughts or recommendations on this?
View 3 Replies
View Related
Apr 29, 2004
Hello, everyone:
I am not sure wether transaction log file size affect the database performance. My SQL 2K suddenly became slow yestoday. The data file is 3GB, and transaction log file is 11GB. Someone suggested I should shrink transaction log file. Can it work?
Thanks a lot.
ZYT
View 7 Replies
View Related
Oct 21, 2007
Hello,
Working with SQL Server 2000, I have a table with the following structure:
ID (INT)
userID (INT, foriegn key)
productID (INT)
productQTY (DECIMAL(5,2))
purchaseDate(smalldatetime)
I have about a 1000 users, entering about 20-30 rows per day each, i.e ~20,000 - 30,000 new rows per day. The table might be queried with a simple "SELECT" for the products a user ordered per day or per time frame (purchaseDate column).
My question (finally) is - when should I expect to see performance degradation? Is there anything I can do to prevent it (i.e splitting this table somehow to several tables)?
Thank you all in advance
View 2 Replies
View Related
Nov 29, 2001
SQL Server 6.5
I am running a Query in my Production Server. It is hardly taking 15 Mins. The same Query is taking more than 3 Hours in my test server. I can see the only difference between these two servers is Tempdb Size. Will tempdb size matters the performance of a Query. Can anyone reply me?
Thanks in Advance
Krishnakumar
View 1 Replies
View Related
Apr 19, 2006
I just upgraded our application from SSCE 2.0 to SQL Mobile. Our app is written in C++, and we use OLE DB for most of our queries, including the routine the downloads and inserts our lookup table data. This application is running on a Dell Axim X51.
Using SSCE 2.0, this routine takes 236 seconds, with most time spent inserting data into various tables (using OLE DB). The resultant database size is 15.1 MB.
Using SQL Mobile, this routine now takes 675 seconds, with a resultant database size of 27.9 MB!! There is a noticable increase in time when the downloaded data is being inserted into the database.
What would be the reason(s) for the slower performance and the increased size of the database? This appears to be a monumental step backwards in performance. Any suggestions regarding improving the perfomance and size?
View 6 Replies
View Related
Jun 21, 2007
Hello All,
When creating my database I have modeled some of the tables after the Adventureworks sample database.
There are some fields or entire tables in Adventureworks that I do not see an imediate use for, however; I would hate to ommit them to find out later they would have been benificial. (.eg territory table).
In general terms what would the impact be on size and performance of a database which contains tables or fields that do not contain data.
Thanks for your help!
View 1 Replies
View Related
Aug 25, 2004
Hi,
We currently have a fairly new SQL server 2000 db (currently about 18mb is size) as a backend to an application (Navision). Performance seems to be below what it should be.
The db is increasing quite rapidly in size, with a lot of data scheduled to be loaded onto the db and also more and more shops and users coming onto the system with alot more transactions going onto the db.
The initial setup of the db has the database File properties set to "Automatically grow file" by "30%" and has an unrestricted file growth.
The server that the db sits on is high spec and very large disk space.
Because the database will be expanding alot and thus reaching its maximum space allocation and then performing a 30% increase in size (which I guess affects performance quite a bit??) quite regularly.
Is it best to set the intitial size of the db to a alot bigger size in the first place as we have large disk space availiable and also set the % increase bigger also.
any advice on best performance would be much appreicated.
Regards,
David
View 1 Replies
View Related
Apr 24, 2006
Hi,
Is there a way to reuse existing Configuration Files in SSIS Packages?
I have two packages with the same connection managers, properties and variables.
I want to reuse the Config file of the 1st package in the second one. While building the packages it gives error like
“Config file already exists cannot recreate”.
View 1 Replies
View Related
Jun 8, 2007
Hi All,
I have created SSIS (.dtsx) files and have stored in different servers.
Now my query is I want to move all dtsx files from filesystem to Sqlserver2005 database how should i do it.
Please help
Regards
Hassan
View 2 Replies
View Related
Oct 16, 2007
I need to create the ssis package in business intelligence developement studio i am need to sqlserver 2005.When i opened the BID studio i am not able to see the integration services packages type..
Please help the steps to design the package.
I have experience of using the 2000 in dts designer mode.
Thnks for your help in advance.
View 1 Replies
View Related
Mar 14, 2008
Hi,
iam New to Packages. So Please give me the guidelines to Learn Packages.
Thanks for Advance
View 3 Replies
View Related
Mar 12, 2007
Hi,
I upgraded to Microsoft SQL Server 2005 Service Pack 2 and now when I run the master SSIS package( that has several packages in it), all the packages run twice.
After removing SP2, they work fine. Any ideas how to make this work with SP2?
View 2 Replies
View Related
Aug 24, 2006
I am writing a vb application that is supposed to let the users set the connection string for the datasources in the package. After new connection strings are entered the application is supposed to run 8 packages in a certain order, but I haven't been able to set a new connection string successfully. Is there a way to programmatically modify the connection string of a package's datasource? (the packages are moving data from a D3 database to sql server 2005)
Here is what I have tried so far:
A.
Dim pkgLocation As String
Dim app As Application = New Application()
pkgLocation = "c:Package1.dtsx"
Dim pkg As Package = app.LoadPackage(pkgLocation, Nothing)
Dim myConns As Connections = pkg.Connections
MessageBox.Show(myConns(0).ID.ToString)
Dim myConnMgr As ConnectionManager = myConns(0)
Dim connProperties As DtsProperties = myConnMgr.Properties
connProperties(0).SetValue(myConnMgr, "notsupposed=towork;")
MessageBox.Show(myConnMgr.ConnectionString)
B.
ConMgr = p.Connections.Add("OLEDB")
ConMgr.ConnectionString = "Provider=SQLNCLI.1;Data Source=192.168.0.233; User ID=sa; Initial Catalog=Rmt_New"
ConMgr.Name = "SqlDatabase"
ConMgr.Description = "Sql Connection"
Any help would appreciated.
View 1 Replies
View Related
Feb 28, 2006
I am connecting to a DB2 mainframe to pull data into SQL 2005. Very simple import. SSIS package works fine on 32 bit. However, once deployed to the 64 bit machine, I get "invalid product license" on the Acquire Connection method.
I've worked with IBM support. I have the correct version of the DB2 Connect client installed. The license is there and in the right place. I can connect to the mainframe from the 64 bit server using the DB2 client tools. I just can't seem to execute the package from Integration Services or run a job in SQL Server that executes the package.
According to BOL, the package should automatically detect the 64 client I installed. It and the 32 bit client I developed with share the same name/id.
Am I missing something? Any hints?
View 3 Replies
View Related
Feb 7, 2007
Where are SSIS packages stored and how can I see them in SQL Server Management studio?
View 6 Replies
View Related
Sep 15, 2006
I read in Kirk Haselden's book "Microsoft SQL Server 2005 Integration Services" that if SQL Serfver 2005 and 2000 are installed on the same machine as seperate instances then you can view the SQL Server 2000 DTS packages in 2005 Management Studio under the Management tree, Legacy, Data Transformation Services node.
But in my case, I am not able to see DTS packages in Management Studio. Is there a property or a setting that we need to configure for that?
Thanks.
View 2 Replies
View Related
Jan 17, 2007
I'm still new to SSIS packages and I'm NOT a developer. I am in the process of doing preliminary/prepatory work for migrating our SQL 2000 platforms to SQL 2005.
I am having a REAL headache with migrating/moving DTS packages from SQL 2000 to SQL 2005. Here are things that I know :
1. I know that some packages cannot be migrated due to ActiveX issues and other issues. Fine.
2. I know that I can install DTS backwards compatibility components on the server in order to be able to edit the DTS packages using a SQL 2000 DTS GUI. Fine.
3. I know that I can use the Migration wizard to migrate packages (and that some of them can't be migrated this way). Fine.
Here's what I don't know/or am conjecturing:
1. In a clustered environment, I have to edit the <%Install Path%>/90/DTS/Bin/MsDtsSrvr.ini.xml file to set the <ServerName> property to the Virtual Server name. Correct? Why can't M$ do this for me?
2. Do I HAVE to export the SSIS package to a .DTSX file in order to be able to edit it with Visual Studio? Is there ANY way around this?
3. If I am running in a clustered environment and I use the File System for storing packages, then the pacakges must be stored on a shared volume, right?
4. I did not find SQL Server Integration services on the B- (Passive) node. Do I have to install it separately onto the B server (much like having to install the Client Tools)?
If anyone has some guidance or tips on running SSIS in this brave, new, wonderful world, I would sure appreciate it.
And yes, I am going to go out right now and order a new book on SSIS.
Regards,
hmscott
View 3 Replies
View Related
May 11, 2007
Hello,
When I try to save modifies in packages with many components the system show me a information dialog telling me that there was a System.OutOfMemoryException
Anyone knows how to solve this problem without divide the package in 2 or more packages?
View 6 Replies
View Related
Apr 17, 2008
HI Gurus,
What permissions one sholud have to cretae and execute SSIS packages.
Thanks,
ServerTeam
View 1 Replies
View Related
Jun 3, 2008
Hi All,
I've been assigned a task by one of the programmer in my team to create packages from some of the databases(One from SQL2005&Other one from SQLEXPRESS)
I've created and saved the packages using the export wizard.I saved the Packages as Integration Services Packages(On file System).
Now he wants me to execute the packages using SSIS But in different time,like maybe after 5min.Other package runs.
I really have no clue how to do that,I've added Both packages in SSIS,But i really dont know how to run them in different time.
If anyone could help please do so!
View 3 Replies
View Related
Jun 23, 2008
Hi Gurus,
How can i know SSIS packages history as this package is executed with issues or without issues
Thanks,
ServerTeam
View 2 Replies
View Related
Jun 24, 2008
Hi Gurus,
How can i know SSIS packages history as this package is executed with issues or without issues
Thanks,
ServerTeam
View 1 Replies
View Related
Feb 16, 2006
I am in a situation where we are redesigning our Datawarehouse. Currently we have our Datawarehouse in SQL 2000 and we are rebuilding from scratch in SQL 2005. This means that even though we will get the same tables but we are planning to rename each & every attribute to make it more meaningful.
The question is like this:
We can migrate the current DTS packages to SSIS but since all the ODBC connections , field names(attributes) will change is it worth it to leverages the DTS packages ?
Also we convert the julian date to gregorian date in our DTS packages but since SSIS has a feature to convert julian date it would be redundant to migrate the packages and my feeling is to create new packages in SSIS and start on a clean slate.
Is there anybody who was in such situation ?
Please advise.
Adarsh Mathur
View 2 Replies
View Related
May 30, 2006
I thought so but I'm looking for a couple of packages and I am not be able to find any there at all.
View 3 Replies
View Related
Nov 7, 2007
Hi,
I am asking something very basic and theoretical, here.
Are there any design tools available in the market for designing SSIS packages?
By "design tools" I mean tools which enable us to "plan" or "design" the architecture of a SSIS Solution that will be implemented later, using SSIS, of course.
We have several design tools available for designing a Web Application solution, for example. Similarly do we have something for SSIS?
Are there any design approaches, best practices and/or design techniques published for designing a SSIS solution?
Please note that I am not talking about the SSIS Designer or the BIDS. I am talking about a tool/approach for designing the SSIS solution which can be delivered as a project artifact before the actual coding phase starts?
Well, I have tried to express myself as best as I could.
If someone can help me with this, it will be really great!
Thanks in advance.
Regards,
B@ns
View 10 Replies
View Related
Aug 22, 2007
On my local desktop, I can create and run ssis packages, when I try to do the same thing on server I get he following error right clicking on running packages or stored packages.
failed to retrieve data for this request. Library not Registered. (exception from HRESULT: 0x8002801D
there are 2 instances of SQL on this server. Both are named with one being use by SQLExpress and the other by SQL2005 Std
View 16 Replies
View Related
Jun 13, 2006
hi everyone,
Up to moment I've been using Dtsbackup2000 for do backup for our DTS but now, which let's allow do the same with dtsx???
TIA
View 4 Replies
View Related
