One question I haven't found to date is concerning context. First of all my setup is importing 250 fox tables into sql 2005. I have one main package with 4 child packages. These child packages have in turn on average 3 packages which in turn contain several tables each. This setup is due to resource limitation. Currently it takes about 12 hours, mainly due to my transformations, table size and resource limitations.
Anyway, my question is if i simply changed the defaultbuffermaxrows from default 10,000 in one of my tables to say 100,000, then executing this table import ad-hoc to analyse import time would not be a true reflection of package performance?
What I mean is if I analyse this table stand alone and due to change above time is reduced by say 1 minute, i might think great. But overall, I reckon increasing defaultbuffermaxrows to 100,000 will probably reduce resource allocation to other tables in package and thus could have a detriment to package, so the only way to accurately compare package performance is to make changes at table level and the run main package? - which of course taking 12 hours is not very practical......
I am executing a package programmatically from ASP.NET using C#. The package is being loaded from SQL Server at runtime and then it is executed from a custom class written in C#.
Problem:
When a task fails, I'd like information on why it failed. I read about implemeting the IDTSEvents interface as a way to have my package call back to code. This works fine (I have code in the OnTaskFailed event handler), but I'm not sure how to find out why a task failed. Since my task is loading a flat file to SQL Server, failure will likely come in one of two flavors: either the file format will be wrong or the data will violate a database constraint. In etiher case, I'd like that feedback. I am looking that the TaskHost object that gets passed to my event handler and I haven't figured out how to access this information. Am I going about this the right way? If yes, how do I access the error information I am looking for?
I am trying to understand how to predict what user context a package will execute under, more specifically when it comes to operating system access.
For example we have a package that creates a flat file, during developement it was simply created on the C: drive of the users pc.
Now it must create the files on a remote server.
How do permissions to folders work is this case?
When you execute it in BIDS debug does it use the window account you logged onto the server as to connect to the remote folder ?
If so what happens when deployed to sql server and scheduled in a job? Does it use the domain account that the sql server agent account runs under ? Or will it use the credentials of the proxy account?
I am unclear in this scenario ? Obviously we need to grant the correct permissions on the remote server shared folder to the correct user, but i am not sure under what user account the package will be trying to create files on the remote server as ?
In a nutshell i need clarity on what user a package is executed as when running in a sql job ?
Anyone have a good link or the time to clarify this for me please.
Lately, I have been experimenting with SSIS and I created a generic custom error logging component that saves all offending data on data flow component failure. However...
Instead of re-directing rows at the data flow level and handling/logging the data at that level, is it possible to catch all of this information at the package level and handle/process it there?
Hi, I created a package that loads more than 9 crore records in to the destination table. It takes hours to complete a single dataflow.Is there any way to increase the performance of the package?
Dear Friends, I always use this forum to find support and to try help others. But this time I need to receive your feedback about my package that will be in prodution in few weeks. So.. could you give me your opinions? I prefer i write the comments in the blog, but you can write here to... http://pedrocgd.blogspot.com/2007/10/bicasestudy-package-v2.html
I created procedure which completes execution in 20 mins in sql server 2005 but if i kept the same procedure in Execute SQL Task in SSIS and executing means, it is taking 3 hrs.
Is there any way to increase the performance for the above same.
I have multiple data flow tasks defined in my package. The task of the package is to extract data from Oracle/InfoLease tables and put them on to a SQL Server 2005 database.
Listed below are few queries that I had:
1. In SSIS package, I need to add "Data Conversion" component to convert from Unicode string datatype to String datatype. This was not required in SQL Server 2000 DTS package.
2. By default, Individual transformation is created for each column. Is there a way, to create one transformation for all columns.
3. This SSIS package is being executed as part of a job. The execution time takes around 33 mins. The same functionality of the SSIS package was replicated in form of SQL Server 2000- DTS package and was executed in form of a job. This execution got completed in 9 mins. So there has been a drastic increase in job execution time. Are there any ways to increase performance.
I know the SSIS memory problem has probably been covered quite a bit, but being a newbie in using the SSIS, I'm not quite sure how to improved the performance of my SSIS package.
Basically, I've a package that loops through all the subdirectories within a specified directory, and it then loops through each file in the subdirectory and with the use of the Data Flow, process each file (according to their filenames) with a Script Component to insert data into a SQL DB.
Each subdirectory has up to 15 different csv files, but each is less than 5kB. I probably have about 100 subdirectories.
When I run the package, it functioned properly, but the package stalled (no error but just stuck in one Data Flow) after a while, and when I checked my CPU memory, it was running at 100%.
I'm not sure how I could fix it or improved the memory allocation. I was not expecting to have any memory problems as the file size is small and the number of rows of data going into and out of the Script Component is no more than 20.
I would like to know if any one out there has the same performance issue with ssis packages once you move from a 2GB of RAM, 4 processors to a 16 GB of RAM with 8 processors. My SSIS packages performance is extreemly slow especially for one of my loading packages. The execution time for that package is almost double as compare to the execution time on the smaller box (2GB, 4 processor). There is no transformation for this package. It is just one lookup for existing or not and then do the insertion. This package is handling insertion of millions of rows.
The 2GB of RAM, 4 processor box is 32 bit with SP1 and the 16GB of RAM, 8 processors is 64 bit with SP2. However, I did try to force the execution of the package in 32 bit on the 64 bit server. The execution time is still the same as it is executed in 64 bit mode.
If any one out there has experienced the same thing and has known the cause to the problem, please let me know I would greatly appreciated.
I'm working on a fairly straight forward data transfer package and have found that the package runs dramatically faster when I run the package inside BIDS than with DTExec. When I run the package on the server using debug in BIDS, the job completes 1 million rows in around 6 minutes. When I run DTExec with the same package on the same server it is much slower and the package takes roughly 25 minutes to complete.
I know this sounds crazy and that it's supposed to be the other way around with DTExec running much faster, but I'm stumped as to what could be causing the issue. The machine this is running on is a two processor, dual core CPU with GB of RAM and I'm using terminal server to login and create the package with BIDS on SQL Server 2005 SP2.
The main feature of this package is a Foreach container that uses an ADO record set to loop over a set of values from a control table. There are a large number of iterations so the package loops frequently, but the data flow task is fairly simple and uses an OLEDB source and OLEDB destination to transfer data between two SQL Server 2005 databases.
The package works in either BIDS and DTExec, but I'm really puzzled why it would run so much faster inside BIDS?
I have a performance related question about the DTS package in sqlserver 2000 which i have developed We have developed a DTS package which will migrate a view 'ATTRITION' from Sqlserver 2000 to an Oracle database.The design of the package is as follows First step: It checks for the existance of the table 'ATTRITION' in oracle database, if table 'ATTRITION' is not there it will create a table called 'ATTRITION' in the oracle db.If the table 'ATTRITION' is already present in the oracle db,then the table is truncated.
Second step: The view 'ATTRITION' is migrated to Oracle table 'ATTRITION'. For the migration, i have used a connection object which connects to sqlserver 2000 and for oracle connection i have used another connection object 'Microsoft ODBC driver for oracle' and i have joined both the connection objects with 'Transform data task' task which maps one to one from sqlserver 2000 where view 'ATTRITION' exists with oracle database where Table 'ATTRITION' exists. Roughly i have around 65000 rows in 'ATTRITION' view of sqlserver 2000 which needs to be migrated.When im running the package on my system it takes around 4 minutes to migrate all the rows but when im running it on the server it takes a lot lot of time more than 1 hour.
The view definition im using has more than 10 tables joined together.But if its a problem of query used in the view,and if i run the view seperately it quickly displays the data hardly takes 1 minute. and even if i run the package on my local pc it doenst take much time.Now my confusion is why its taking soo much time on server.If i create a indexed view then will it solve my problem.Please suggest... Thanks in advance
When I installed SP1 on my SQL Server 2005, the performance of the package designer dropped enormously. I have to wait like 30 seconds after I drag-and-drop a datasource from the toolbox to the design area. Also it takes a very long time to run the very simple package. Not to mention opening some complicated package that was done before applying SP1. The package scheduled in SQL Agent went from 3-4min to 5-6min.
Can anyone address this issue? Maybe there's something I did wrong?
While installing SP1 I ecountered strange behaviour (some messages about file locking, rebooting and stuff), but in the end it said that it was successful
My SSIS package is running very slow taking so much time to execute, One task is taking 2hr for inserting 100k records, i have disabled unused index still it is taking time.I am rebuilding/Refreshing indexes and stats once in month if i try to execute on daily basis will it improve my SSIS Package performance?Â
I have a performance related question about the DTS package in sqlserver 2000 which i have developed We have developed a DTS package which will migrate a view 'ATTRITION' from Sqlserver 2000 to an Oracle database.The design of the package is as follows First step: It checks for the existance of the table 'ATTRITION' in oracle database, if table 'ATTRITION' is not there it will create a table called 'ATTRITION' in the oracle db.If the table 'ATTRITION' is already present in the oracle db,then the table is truncated.
Second step: The view 'ATTRITION' is migrated to Oracle table 'ATTRITION'. For the migration, i have used a connection object which connects to sqlserver 2000 and for oracle connection i have used another connection object 'Microsoft ODBC driver for oracle' and i have joined both the connection objects with 'Transform data task' task which maps one to one from sqlserver 2000 where view 'ATTRITION' exists with oracle database where Table 'ATTRITION' exists. Roughly i have around 65000 rows in 'ATTRITION' view of sqlserver 2000 which needs to be migrated.When im running the package on my system it takes around 4 minutes to migrate all the rows but when im running it on the server it takes a lot lot of time more than 1 hour.
The view definition im using has more than 10 tables joined together.But if its a problem of query used in the view,and if i run the view seperately it quickly displays the data hardly takes 1 minute. and even if i run the package on my local pc it doenst take much time.Now my confusion is why its taking soo much time on server.If i create a indexed view then will it solve my problem.Please suggest... Thanks in advance
Hi, I need to build up dynamically cursor and fetch variables in a script. I use exec(@sql) to declare the cursor and my hope was I could use the same method for doing the fetch. Unfortunately I run into a
SET @sql ='DECLARE C1 CURSOR FOR SELECT * FROM OPENROWSET(' + @apo+@int_provider_name+@apo+ ',' + @apo+@int_provider_string+@apo + ',' + @apo+@int_query_string+@apo + ') FOR READ ONLY' PRINT @sql EXEC(@SQL)
OPEN c1
set @fetch = 'FETCH c1 INTO '+@int_fetch print @fetch EXEC (@fetch)
... snippet
Output: DECLARE C1 CURSOR FOR SELECT * FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0','Excel 8.0;Database=C:work2007schenker.xls','SELECT * FROM [Datei$]') FOR READ ONLY
FETCH c1 INTO @c1,@c2,@c3,@c4,@c5,@c6,@c7,@c8,@c9,@c10,@c11,@c12,@c13,@c14,@c15,@c16,@c17,@c18,@c19,@c20,@c21,@c22,@c23,@c24
Msg 137, Level 15, State 2, Line 1 Must declare the scalar variable "@c1".
How can I declare the dynamically built 'fetch into' variables (@c1 etc.) so they are in the context of dynamic sql ?
Hi,I am having a SQL Server 2005 problem with my Insert statement. I amsending a command via my website to my database. It comes up with anerror I'll put below. The code is here:"INSERT INTO tblUsers (userName) VALUES ( userNameTest)"This is the error I get:The name "userNameTest" is not permitted in this context. Validexpressions are constants, constant expressions, and (in some contexts)variables. Column names are not permitted.Now, userName is a varchar field in the database. What is wrong?Kivak
I have a web application that I need to search based on what the user entered in the input box. e.g when the user enters in the box something like "Brain Boom"
I need to search the column in the DB table where there is anything word like Brain or has Boom or all the above. How will I accomplish this?
There are some concept about context switch block me .
---executor: dbo create user u1 without login
go
create table t
(
col int
)
go
create proc proc1
with execute as 'u1' as
insert into t values(1)
go
exec proc1
I think proc1 can't be run successfully beacuse the executor of proc1 is u1 and I didn't grant any permission to user u1. So it will return a error message like The INSERT permission was denied on the object 't', database 'tempdb', schema 'dbo'
To my surprise, the "exec p1" command could run successfully..
I'm currently creating a database that will only allow data operations through stored procedures, ie users will not be able to directly modify tables. I'd like to use an Insert trigger which will run in response to a stored procedure that inserts records. The trigger will check business logic and additionally modify records in a couple of other tables. Given that I've disallowed direct access to the tables, will it run or will the security set-up prohibit that?Thanks in advance for any answers.
I was trying out the Building an End-to-End Application (VB.Net) exercises on Web Matrix Guided Tour and encountered the mentioned errors. Please help. Thanks.
' TODO: Update the ConnectionString and CommandText values for your application Dim ConnectionString As String = "server='WRPBI'; user id='sa'; password='sa';database=MatrixOrders;Integrated Security=SSPI"
Dim CommandText As String = "select OrderID, OrderDate, CustomerName from Orders"
Dim myConnection As New SqlConnection(ConnectionString) Dim myCommand As New SqlDataAdapter(CommandText, myConnection)
Dim ds As New DataSet() myCommand.Fill(ds)
MasterGrid.DataSource = ds MasterGrid.DataBind()
End Sub
Sub BindDetailGrid()
' get the filter value from the master Grid's DataKeys collection If MasterGrid.SelectedIndex <> -1 Then
' TODO: update the ConnectionString value for your application Dim ConnectionString As String = "server='WRPBI'; user id='sa'; password='sa';database=MatrixOrders;Integrated Security=SSPI"
' TODO: update the CommandText value for your application Dim filterValue As String = CStr(MasterGrid.DataKeys(MasterGrid.SelectedIndex)).Replace("'", "''") Dim CommandText As String = "select OrderDetailID, ProductName, Quantity, UnitPrice from OrderDetails where OrderID = '" & filterValue & "'"
Dim myConnection As New SqlConnection(ConnectionString) Dim myCommand As New SqlCommand(CommandText, myConnection)
Cannot generate SSPI context. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
I have an application that executes the USE command as it's first order of businees with the database. According to the SQL Books On-line, all USE commands will result in the following message being generated:
However, my application has never reported this message, execept for at one customer site. I cannot figure out why at this one particular site, the customer is seeing this message each time the application starts. Once they ok the message, everything else with the application is fine, but it is a nuisance for them (gotta love customers!).
I am writing a trigger to audit changes to certain columns. In the trigger I would like to record to the "Audit" table as much information about the current execution context as possible (current user, login, nt user, inputbuffer, etc.). I couldn`t find the way to find out what is the currently executed* stored procedure. @@PROCID returns object id for the trigger itself. Any help would be highly appreciated as this is an urgent production environment issue.
We have a sqlserver 7 on windows nt 4. We'd just change a startup account for sqlserver agent to a not domain admin for security reason. The startup account for sqlserver service is still domain admin. We are now getting the following error every 5 second in our application event log.
"SQLServeragent security context does not have server-autorestart privileges"
Is there a way to get rid of this error without putting a domain admin as a startup account for sqlserver agent?
All of a sudden I cannot connect to sql server from Visual Studio. Nor I can't conenct to Sql server through IIS and it gives me the same error saying "Login failed: Cannot Generate SSPI Context". Lots of my development projects got stuck because of the SSPI Issue. I know there was a network upgrade from Windows NT (users) to Active directory. I can only access the sql server from the host machine.if someone can guide or give me a quick run down on things to look for that would be wonderful.
Inserting info into a table, where one of the columns meets a certain criteria.
insert into employee (/*emp_id,*/ fname, lname, start_date, dept_id, title, assigned_branch_id) values (/*null,*/ 'Michael', 'Smith', '2001-06-22', (select dept_id from department where name = 'Administration'), 'President', (select branch_id from branch where name = 'Headquarters'));
But I'm getting this error:
Msg 1046, Level 15, State 1, Line 5 Subqueries are not allowed in this context. Only scalar expressions are allowed.
I have a SSIS package developed by a different user which does a lot of DML. This package sits on the server.
The package needs to be executed on a regular basis.
I have given RO access for a regular user on production DB, he is executing the package from his client desktop.
I was expecting this execution fail, since the package is doing lot of INSERTS , where the user has ONLY RO access.
I understand from the above experience that there is an “execution context� for SSIS execution. Can someone tell me how can I define the execution context for SSIS?
------------------------ I think, therefore I am - Rene Descartes
HiMy company uses a Windows 2000 server with MS SQL Server 2000.We have many clients with administrative software that use the MS SQLserver, the problem is that we experimented randomly the error"Cannot Generate SSPI Context" and the only way to work with DB is logout the client.I'm looking the MS Technical sheet but there's no resolving hints thatcan help.Anyone has a similar problem ?Thanks.
