I have created a SSIS package that reads 500 text files splits them into 4 raw files then reads them again and writes then to 4 database tables different Tables.
The reason form this is that my raw files have multiple types of records in them and it is only 1 Coolum. I split this out into the different types of records and load whole rows into the database.
End up begin rows in the T6 Table
1000178 18148821-00 40204043 1 EX201036259NZ 0000304862
1511158 18084863-00 40617044 1 EX201033969NZ 0000302981
T5 Table gets a new record
1511313 18126485-00 2006032510230300 EX201033399NZ
and T1 Table get a record
18148821-00
Anyway all this works find but I find that the DTExec process work fine until it has used up all the memory in the laptop in general it take 400megs to run this SSIS. I'm wondering am I missing something like don't run in a transaction. I know in the old DTS you could commit on each package and how do I turn all logging off eg what you see in the DOS box (can I do this?) would love some help on this and if anyone want a copy of this ssis package ie your trying to do the same then I'm more than happy to email it.
I have a custom .net application that uses SQL 2000 server. All users are compaining performance issues and white-outs while they are using the application. I am almost certain that it's the SQL server that is the curprit. All the other components involved in the application hardly has any CPU or memory usage when I check the performance in the task manager.
On SQL server, I see that the process sqlserver.exe is taking like 2.8GB of memory. Is there a way to find out which exact SQL query or process is taking so much of memory? I belive there may be a bad SQL process that is stuck and taking all the memory? Is there a way to find out?
Hi, Situation: I have a web application that is running on a server. IT has told me that the application I developed is taking up enormous amount of ram. It seems that the amount of ram that is taken up, is slowly increasing. About 10 megs per hour (memory leak). Furthermore, he has informed me that the ram is taken up by the sql. My senior developer told me that I probably have sqlconnections open that are not closed. I have checked through the application and it seems they are all closed properly. But I may have missed some, as the application is pretty big. Questions: 1. Is there an application or piece of code that can monitor how many sql connections I have open during runtime? 2. Could there be any other common causes to such a kind of memory leak. Any help with this matter would be very much appreciated. Thank you for your time. Sincerely, Jeff
I am running SQL Server 2000 on a desktop PC. Just recently upgraded my PC to 2gig of ram from 1 gig, in part, to try and fix the problem below. Didn't work. Have SP3 installed but not SP4 at this point.
When I open up query analyzer and edit some code, regardless of whether I actually execute the code, SQL server eventually sucks up available and cached memory to the point that my system comes to an effective halt --takes forever to do anything either in SQL server or other applications. In the task manager PF Usage climbs to just over the 2gig memory limit.
I conceptually understand the dynamic memory operation of SQL server ... but why is it sucking up most available memory when nothing is executing?
Is there a way I can release/clear the memory? Ideally, code that coould be run under a stored procedure would best meet my needs. Right now, I am "fixing" by shutting down and then restarting SQL server.
Hi, We have 32 individual publications setup on SQL 2000 and have 32 distribution agent running against it. I think each distribution agent opens a process called distrib.exe in Windows task Mgr.
Problem I'm having is that these 32 distribution agents are taking up fair amount of memory 1/2 gig and each process seem to be ever increasing in memory usage so I suspect a memory leak?
Is there something I can do to fix this problem? I don't want to upgrade memory as we already have over 1 1/2 gigs on this machine and it would not be feasible if we have a memory leak.
We have run into an issue on a dedicated SSAS 2012 SP1 server where the allocated memory is not being utilized, causing some slowness in use, connections, and queries.
Total Memory on the server is 512, and after startup, the utilized memory gets up to a max of 60GB and stops there. Checking the Resource Monitor, msmdsrv.exe is only taking around 39GB overall. With the current properties, that should be at 330GB. Am I missing something in the settings or in configuration that should be changed?
Version: SQL Server 2012 SP1 Enterprise (11.0.3000) OS: Windows Server 2012 Datacenter - Fully patched and up to date Databases: 2 Tabular models Server: 512GB RAM
I have a serious problem with my SSIS Package while executing using 32-bit DTExec and 64-bit DTExec.
Here are the details:
Environment:
Windows Server 2003 64-bit (Build 3790: Service Pack 2) SSIS 32-bit & 64-bit installed SQL Server 2005 (Microsoft SQL Server 2005 - 9.00.1399.06 (X64) - RTM)
SSIS Package details (compiled in 64 bit)
Script tasks only Microsoft Visual Basic .NET (using TRY...CATCH block) PreCompileScriptIntoBinaryCode = TRUE Run64BitRunTime = TRUE
Execution
Batch file that uses DTExec to execute the Package.
SCENARIO I am trying to exeucte the above SSIS package using both 32-bit and 64-bit DTExec to make it failure by providing invalid connection string. Here are the details,
Wrong connection String using 32-bit Execution
While establishing the connection the error message has been nicely captured in to my Exception block and writes into the log file.
Wrong connection String using 64-bit Execution
While establishing the connection the error has not been catpured anywhere (although I have TRY CATCH block) and it haults there itself with the message "Process is terminated due to StackOverflowException". Later I found that the error is due to the connection string along with the unhandled exception.
Please suggest any one of the following my findings, also if you have any other advice would be very much appreciated.
1. Shall I go ahead and fix the issue by handling those unhandled errors? (e.g Appdomain, application). I tried several but still not working using 64-bit DTExec.
2. Shall I go ahead and use 32-bit DTExec to execute the package? If so, is there any other major issue...like performance or anyother bug?
P.S: We cannot apply any service pack for SQL Server 2005 at the moment. Sorry abt it. If you have any specific hotfix for DTExec (without affecting SQL Server) then we can decide.
Sorry for the lengthy one and Thanks very much for you help in advance .
Hello. I have received the follwoing error upon an attempt to Browse the Cube. All other tabs are functional, including the Calculations tab. We are running Windows Server 2003 SP2 and SQL Server 2005 SP2. Any suggestions would be greatly appreciated!
**EDIT** - Have confirmed SP1 for VS2005 is installed both locally and on server, also.
Attempted to read or write protected memory. This is often an indication that other memory is corrupt. (Microsoft Visual Studio)
------------------------------ Program Location:
at Microsoft.Office.Interop.Owc11.PivotView.get_FieldSets() at Microsoft.AnalysisServices.Controls.PivotTableFontAdjustor.TransformFonts(Font font) at Microsoft.AnalysisServices.Browse.CubeBrowser.UpdatePivotTable(Boolean translate) at Microsoft.AnalysisServices.Browse.CubeBrowser.UpdateAll(Boolean translate) at Microsoft.AnalysisServices.Browse.CubeBrowser.InitialUpdate() at Microsoft.AnalysisServices.Browse.CubeBrowser.SupportFunctionWhichCanFail(FunctionWhichCanFail function)
I've been researching AWE to determine if we should enable this for our environment.
Currently we have a quad core box with 4 gb of RAM (VMware). OS: Windows 2003 std, SQL Server 2005 std. 3GB is not set but will be as soon as we can perform maintenance on the server.
I have read mixed feedback on AWE, either it works great or grinds you to a hault. I would assume that the grinding to a hault is due to not setting the min/max values correctly or not enabling the lock page in memory setting.
We only have one instance of SQL on the server and this box won't be used for anything else aside from hosting SQL services. We do plan on running SSRS off of this server as well.
1. Will running SSRS and enabling AWE cause me problems? Will I have to reduce the max setting by the SSRS memory usage or will it share and play nice?
2. How do I go about setting the Max value? Should it be less than the physical RAM in the box? Right now its set to the default of 214748364, even if I don't enable AWE should this default value be changed?
3. It seems that even at idle the SQL server holds a lot of memory and the page file grows. If I restart the process in the morning, memory usage in taskmon is at 600mb or so. By the end of the day, its up around 2gb. How can I track down whats causing this, should this even concern me?
4. The lock Page in memory setting worries me. Everything I've read on this seems to give a warning about serious OS and other program support degradation. In some cases to the point where they have to restore the settings on the server before they can bring it back up. What are your thoughts on this.
I have a Windows sever 2012 with sql server 2012 enterprise. Ram size is 22GB. Sometimes SQL sever takes 95% memory.My question, How to reduce memory size without killing any process because it's production server.So there are many background process is running. And,Is there any guides to learn why Memory is raise d so high and how to reduce it.
Hello, I understand that we should use SSMS -> Server Properties -> Memory to put a cap on the SQL server memory usage, therefore it gives some space memory for OS, this is based on the fact if the max memory is not specified, SQL will use whatever available memory and eventually crash the system.
My question is that when a server has SSIS and SSAS services installed along with the SQL service. Would the max memory setting covers the SSIS and SSAS memory usage, or the SSIS and SSAS has to shared the memory with OS?
I am running Visual Studio 2005. I have an SSIS Package which is consuming a huge amount of memory. During the execution of the package the memory keeps increasing. Until finally i get an Out of Memory exception. I have run this package using dtexec, and in the BIDS. No difference. I do have some script components and have added some code to get the assemblies in the current appdomain. I do see that one particular assembly is increasing on every loop. VBAssembly every time it hits the script component is increasing by 6, and along with it the memory is climbing. What is this VBAssembly being used for is there an update to SQL Server Integration Services that I need?
sql server 2000 is running on windows server 2003 ... 4gb of memory on server .... 2003 was allocated 2.3gb nd sql server was allocated (and using all of it) 1.6gb for total of approx 4gb based on idera monitor software ... all memory allocated betweeen the OS and sql server .... then 4 more gb of memory added for total now of 8g ... now idera monitor shows 1.7gb for OS and 1.0 gb for sql server ..... 'system' info shows 8gb memory with PAE ... so I assume that the full 8gb can now be addressed .... why are less resources being used now with more total memory .... especially sql server ..... i thought about specifying a minimum memmry for sql server but i amnot convinced that would even work since it seems that this 1gb limit is artificial .... it it used 1.6 gb before why would it not use at least that much now ??
I've a database with a memory optimized filegroup on it. How can I remove it?I have removed the memory optimized table I had on it, but when I try to remove the filegroup I receive an error.
I have a command line as following, with DTEXEC to launch the execution of a package and to set a value (13335) of an user variable called CIB (string type):
But I have got an error message saying that the object is not known in the package. My variable does exist in the variable window of the dataflow part. Thank you for telling me what to set so that the variable can be set by the command line.
I understand to schedule an SSIS package to run, I need to use the dtexec utility. I want to schedule the job right within SQL. I walked through how to set up the job, but I really could not decipher the proper syntax to use as a Step.
I have the following:
dtexec /File "C:...file.dtsx"
Is the the proper way to schedule and execute an SSIS package or is there some other way I should be doing this. The pacakge will run unattended nightly. I am using SQL Server 2005.
I have lots of Stored procedures in which we use 'dtsrun' to run DTS Packages. Now, after I upgrade the server to SQL 2005 will I have to change all the Stored procedure's to reflect 'dtexec' instead of 'dtsrun'?
Hi I have a SSIS package which pulls files from a network share and loads data into SQLServer Database. When I execute the application using DTExecUI , It runs fine without any issues , but where as when I run it using the command line arguments, It seems to go in sleep mode and nothing happens. I need to kill the package from Task manager. Following is the command I use to run my application
(Optional). Specifies a configuration file to extract values from. Using this option, you can set a run-time configuration that differs from the configuration that was specified at design time for the package. You can store different configuration settings in an XML configuration file and then load the settings before package execution by using the /ConfigFile option.
Does this mean that I can specify which configuration file to use during runtime? Or is just because I'm too desperate for that, I understood that way
The DTEXEC Utility has the capability of returning ReturnCodes which have specific meanings e.g.
ReturnCode=3 means: The package was canceled by the user.
Is it possible to set my own ReturnCode values?
For example, my Package contains a Script Task which contains code to search a folder for a file. If the file cannot be found then I make the Script Task fail and hence the Package fails. But, when the Package is invoked by the DTEXEC utility, then the ReturnCode is always set to 1. Is it posssible to set the ReturnCode from within the Package (in, say, a Script Task in the Event Handler) to a different value?
It is necessary to be able to set custom application-specific return codes because these return codes are passed to the Batch Scheduler which alerts Operations (Support) staff in the event of failure. Meaningful (custom) ReturnCodes expedite problem solving and are (usually) mandatory in large production environments.
If it is not possible to make a Package return my own returncodes via DTEXEC then can you suggest alternative solutions, please?
I'm trying to execute DTExec from a workstation and I got some help from a different group without luck maybe someone here can help me.
This is what I try so far.
1. My package run in command line from my sql box using dtexec with parameters. 2. Set the package with DontSaveSensitive and import into IS under MSDB with the same option setup. 3. Set package role to public. 4. Share DTS folder with everyone permission just for testing. 5. Execute the package from a workstation using //sqlServerbox/DTS/BINN/dtexec /dts "msdb/mypackage" /SER "MySQLServer" /set package.variables[myvariable].Value;"myvalue1,myvalue2" (myvariable is string and I can pass multiple values separate by commas) 6. Still getting error: Error: 2007-02-09 10:31:34.31 Code: 0xC0010018 Source: Execute DTS 2000 Package Task Description: Error loading a task. The contact information for the task is "E xecute DTS 2000 Package Task;Microsoft Corporation; Microsoft SQL Server v9; ? 2 004 Microsoft Corporation; All Rights Reserved;http://www.microsoft.com/sql/supp ort/default.asp;1". This happens when loading a task fails. End Error
I can't crack this problem and my Google-Fu is failing me. I am using DTEXEC from a stored procedure to set a global variable in an SSIS package and execute the SSIS package, but I am getting an error. I will include the SQL code and the error that I am getting:
Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved. NULL Started: 3:17:02 PM Warning: 2008-03-13 15:17:02.76 Code: 0x80012017 Source: ProductionToDevelopment Description: The package path referenced an object that cannot be found: "Package.Variables[User:electMail].Properties[Value]". This occurs when an attempt is made to resolve a package path to an object that cannot be found. End Warning DTExec: Could not set Package.Variables[User:electMail].Properties[Value] value to 1. Started: 3:17:02 PM Finished: 3:17:02 PM Elapsed: 0.375 seconds NULL
All the examples I find over DTEXEC seem to follow what I am doing in the code and making variables in an SSIS package isn't rocket science, so I don't understand where I am going wrong.
I have seen people suggest indirect configuration as a solution, but when I set it up (using the environmental variable TEMP from the dropdown) it doesn't seem to help anything. I have used the exact path from the configuration wizard, though, so I'm pretty sure I have the path right.
HI, with a dataflow that has delay validation property = true, DTExec will not try to validate it when I call it with /validate option. Is there a way to see if the dataflow validate even though the delay validation property is set to true?
It was quite strange with dtexec.... The problem started with my package being scheduled under the SQL Agent. I have two packages... one with simple SQL tasks and another with complex loop,sql tasks and activex scripts. Both using the XML configuration files for those DB Connection strings, user names and passwords. First I tried to schedule the two packages under Sql agent using SSIS steps. Both failed and I found out that I should set the ProtectionLevel to non-default. So I set the protection level to 'DontSaveSensitive' and it works fine with the first package but the second package failed. After I tried many way (changing configuration files, setting and reinstalling the packages inside SQL DB etc..) I found a blog that I should try with CmdExe step in Sql agent scheduler rather than SSIS step. So I tried but still failing. the message said dtexec could not find the configuration xml file. Even I tried in the command prompt with the same command, it couldn't find the configuration. If I went into the directory where the configuration files are stored and run the command, it run fine. Seem like dtexec could not read the configuration from other directory. but still the sqlagent is failing... Somebody have a clue?
In my previous post here: https://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1044739&SiteID=1 Michael Entin provides a number of responses to my questions regarding programatic execution of remote SSIS packages.
Having experienced some significant reliability problems with the Microsoft.SqlServer.Dts.Runtime components from an ASP.NET process (the page either times out, or inevitably just stops responding), I have been prototyping the DTExec command option which Michael suggests as being a better approach to remote programability.
So, off I've been prototyping this all day today...
I have a stored procedure that wraps a call to xp_cmdshell which takes the DTS (DTEXEC) params as a big long argument. This scenario would hopefully allow me to call the sproc from an ASP.NET application.
The proc is deployed to a SQL 2005 machine running SSIS (which I now understand is a REQUIREMENT for targetting SSIS "remotely"). The package targets a seperate SQL 2000 machine and includes two connection managers which are set to use SSPI. I use configuration option files to allow for configurable connection manager target/sources.
In this scenario, it does not seem that the DTEXEC command runs in the same context as the caller. and as a result, a peculiar account called MACHINENAME$ is used (where MachineName is literally the name of the SQL 2005 machine). The account authentication fails (obviously) when the package tries to establish a connection to any of the connection managers because MACHINENAME$ does not exist on the connection manager servers.
Based on the following excerpt from the MSDN doc on xp_cmdshell, it would seem that MACHINENAME$ is probably the LOCAL SYSTEM, which is the process tha the SQL Server Service is running under:
When xp_cmdshell is invoked by a user who is a member of the sysadmin fixed server role, xp_cmdshell will be executed under the security context in which the SQL Server service is running. When the user is not a member of the sysadmin group, xp_cmdshell will impersonate the SQL Server Agent proxy account, which is specified using xp_sqlagent_proxy_account. If the proxy account is not available, xp_cmdshell will fail. This is true only for Microsoft® Windows NT® 4.0 and Windows 2000. On Windows 9.x, there is no impersonation and xp_cmdshell is always executed under the security context of the Windows 9.x user who started SQL Server.
Obviously, settting the ENTIRE SQL Server service to run as a fixed account or even a domain account is probably not appropriate for client sites. Any opinion to the contrary is welcome.
In reading Kirk Haselden's walkthrough for setting up a SQL Agent Proxy, this seems incredibly involved. Before I go through this exercise, can anyone validate that this is the way to go for doing SSPI?
A work-around is to use SQL Server Auth for the connection managers and use configuration files to try to obfuscate these details, but my preference would be SSPI/Windows Integrated.
Hi, I have a package which has 2 file system tasks and 2 data flow tasks all in a for each loop container. if i execute the package from the development studio its executes fine but when i try to run it from the command prompt using the dtexec utility..it just runs upto the first data flow task and then it hangs unexpectedly. Any help on this would be really appreciated.
So I started a new job recently and have noticed a few strange configurations. Typically I would never mess with min memory per query option and index create memory option configuration because i just haven't seen any need to. My typical thought is that if it isn't broke... They have been modified on every single server in my environment.
From Books Online: • This option is an advanced option and should be changed only by an experienced database administrator or certified SQL Server technician. • The index create memory option is self-configuring and usually works without requiring adjustment. However, if you experience difficulties creating indexes, consider increasing the value of this option from its run value.
