Does anyone know why SSIS sometimes just sits in the Pre-Execute phase of a data flow and does nothing? It doesn't matter how elaborate the data flow is or the volume of data. It can sometimes take 80% of the task's run time.
View 44 RepliesWhat happens during the pre-execute phase?
Is there anything I can do to optimize it's execution?
I have a package that takes data from 2 SQL Server sources, unions them, and writes to a sql table.
The pre-execute takes 20 minutes, and the execution takes 45 seconds.
Thanks
BobP
Hearing complaints from users about speed on db server (I have almost no control on design) it just has to work. Ran profiler looking for all sql statements over 4000 millsec and in one hour returned over 715 tsql statements. Over 300 of these were over 10000 milliseconds. THis is on an 8 way Dell with 8 gig of RAM. Looking for opinions, how bad does this look compared to other servers you are taking care of? Cache hit ratio is at 99 % and system queue length still under 1, but this does not look good.
View 2 Replies View RelatedHi,
View 9 Replies View RelatedHi,
I have a SSIS package which pumps data from one server to other without any additional steps. There are 11 tables for which data is transferred. And this packages runs fine on two different environments but fails in one environment i.e. on SIT.
It doesn't throw any error and every time stops at the below step
[DTS.Pipeline] Information: Pre-Execute phase is beginning.
Progress: Pre-Execute - 0 percent complete
Progress: Pre-Execute - 1 percent complete
Progress: Pre-Execute - 2 percent complete
Progress: Pre-Execute - 3 percent complete
Progress: Pre-Execute - 4 percent complete
Progress: Pre-Execute - 5 percent complete
Progress: Pre-Execute - 6 percent complete
Progress: Pre-Execute - 7 percent complete
It doesn't complete neither throws an error. Any pointers on what the problem could be
Thanks
Hi all,
I have a monster SQL query (SQL Server connection) that I'm trying to export to flat file. My package appears to just stick on the "Pre-Execute phase is beginning" stage. (I left it running overnight, and it's still going... no error message, I think it's just hung. If I include a "Top 10" in the my source query it completes without a problem. I'm wondering if it's an out-of-memory issue, but when I've run into this in the past there have been error messages generated. I've tried using both a Data Reader and an OLEDB Datasource. Does anyone have any ideas of what's going on and how I might fix it?
Thanks!!!Jess
Hi All
I have a SSIS package running and tested fine on my desktop. However when I deploy the package to my server I get the error given below.
SSIS package "CR Sec Watch List 2.dtsx" starting.
Information: 0x4001100A at Transform Master Files: Starting distributed transaction for this container.
Information: 0x4004300A at Transform Master Files, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Transform Master Files, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Transform Master Files, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x4004300C at Transform Master Files, DTS.Pipeline: Execute phase is beginning.
Error: 0xC0202009 at Transform Master Files, Excel Source [1]: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E21.
Error: 0xC0208265 at Transform Master Files, Excel Source [1]: Failed to retrieve long data for column "NKoreaPoi".
Error: 0xC020901C at Transform Master Files, Excel Source [1]: There was an error with output "Excel Source Output" (9) on component "Excel Source" (1). The column status returned was: "DBSTATUS_UNAVAILABLE".
Error: 0xC0209029 at Transform Master Files, Excel Source [1]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output "Excel Source Output" (9)" failed because error code 0xC0209071 occurred, and the error row disposition on "component "Excel Source" (1)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047038 at Transform Master Files, DTS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Excel Source" (1) returned error code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Error: 0xC0047021 at Transform Master Files, DTS.Pipeline: SSIS Error Code DTS_E_THREADFAILED. Thread "SourceThread0" has exited with error code 0xC0047038. There may be error messages posted before this with more information on why the thread has exited.
Error: 0xC0047039 at Transform Master Files, DTS.Pipeline: SSIS Error Code DTS_E_THREADCANCELLED. Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. There may be error messages posted before this with more information on why the thread was cancelled.
Error: 0xC0047021 at Transform Master Files, DTS.Pipeline: SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread0" has exited with error code 0xC0047039. There may be error messages posted before this with more information on why the thread has exited.
Information: 0x40043008 at Transform Master Files, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x40043009 at Transform Master Files, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Transform Master Files, DTS.Pipeline: "component "SQL Server Destination" (20)" wrote 0 rows.
Task failed: Transform Master Files
Information: 0x4001100C at Transform Master Files: Aborting the current distributed transaction.
Warning: 0x80019002 at Process Master Files: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (8) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "CR Sec Watch List 2.dtsx" finished: Success
For some reason it tells me that it not able to retrive the long data for a column. The column mentioned in the error contains special characters.
Once again, there seems to be no problem with the package as the package runs fine on my desktop.
Any help or clues as to why this is happening on my server is greatly appreciated.
Regards
Ryan
I have a project which spans multiple servers and aggregates literally billions of rows of data into a much smaller and manageable result set which I store on another server. There are two stored procedures which take up 99.9% of the processing time. Each one of these SPs are estimated to run over 3 hours a piece (gives you an indication of how much data there really is). With in this SSIS package, I have two control flows, one for each SP. When I run the SSIS package from VS2005, I can see that there is a pre-execute phase which takes about an hour abnd a half to complete, and then move on to executing. My question is, what the heck is this, and what is it doing? I know it validates the sql in the procedure, but does it actually run the SP during that pre-execute phase? If it does, is there any way to get around that?
Any help would be appreciated.
I have a fuzzy lookup in a data flow but this data flow never gets pass pre-execute phase. What is the problem?
Thanks
Hi All,
I have a table which consists the leave details of an employee.
I have the columns like paid leaves,sick leaves,personal leaves in the above table
Problem :
For eg: An employee joined on 21 May 2008. After 6 months i.e., 21 Nov 2008 I need to update the above columns (i e., increase the no.of leaves)
So updation is to be done for every 6 months and for every 1 year.
Can anyone say me how to execute the update query based on the duration.
Thanks in advance.
Regards,
Praveen
I have the following SQL query
SELECT
[Req_ID]
,[Service_Name]
,[Req_Started_Date]
,[Task_Name]
,[Task_Status]
,[Performer_Full_Name]
[code]....
Which works fine, but what I need to calculate the total duration of a request based on the duration of the tasks completed in the request based on Req_ID. I would like to use the CASE statement I have to determine the SLA_Mins for each task and add them together to get total request SLA_Mins.
Below is the create table schema and data
GO
/****** Object: Table [dbo].[MidrangeOtherSourceControl] Script Date: 06/03/2015 18:13:15 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[MidrangeOtherSourceControl](
[Req_ID] [float] NULL,
[Service_Name] [nvarchar](255) NULL,
[code]....
Does the time it takes to execute sp_repldone depend on the size of the transaction log it's working on? Basically, I am executing sp_repldone to mark all of the transactions as replicated, so I am wondering it sp_repldone goes through each unreplicated transaction and marks it, or does it just go to the last unreplicated transaction and marks only the last one, and all of the previous ones are automatically consideres replicated? Reason to ask: our transaction log is around 27GB right now, although normally it's approximately 1GB. The sp_repldone command has been executing for the past 30 hours, and since it doesn't give any information as to the status of the execution, I'm wondering if it's doing anything at all... :confused: I also checked the process, and it in not blocked by anything. Any insight would be greatly appreciated.
View 5 Replies View RelatedHere's a little SP to break up those long-running, massively-locking, bring-app-to-a-halt queries. By default it does 500 rows at a time and allows for a maximum SQL query size of 4000 characters; it should be trivial to adjust those.
Cheers
-b
CREATE PROCEDURE p_BatchExecute (@vcSQL varchar(4000)) AS
set nocount on
DECLARE @iRows int
select @iRows=1
SET ROWCOUNT 500
WHILE @iRows>0
BEGIN
print 'Executing batch of 500...'
exec (@vcSQL)
set @iRows=@@ROWCOUNT
END
GO
How can I execute a long running transaction using something similar to the fire and forget pattern?
I intend to start the execution of a very long stored proc from within IIS. I would like to execute a sql script that will start the job and return immediately so that it doesn't hold an IIS thread.
How to implement two pase commit in SQL Server 2000, Is there Database Link link Oracle available here ?
View 5 Replies View RelatedDoe's anyone have a good article on it ?
Thanks
Avi
Hi All,
Server A has been configured as distributor, publisher and push subscription to Server B. Replication works fine from A to B. But B to A gives an error "OLE DB Provider SQLOLEDB does not support distributed transaction".
The other way I tried is :
Server B has been configured as distributor, publisher and push subscription to Server A. Replication works fine from B to A. But A to B gives an error "OLE DB Provider SQLOLEDB does not support distributed transaction".
Regards,
Suresh.
Hello ,My question is:does ms sql support two phase commit protocol?and Why if no.How if yes.Thank you
View 2 Replies View RelatedHi, My package hangs and the log says DTS.Pipeline: Validation phase is beginning. Any ideas why this is happennig? This same package runs fine when I run it without turning on the transaction.
View 4 Replies View RelatedI'm running SQL Server 2008 R2 with latest patch. I'm performing all index maintenance online. How long the final phase of the index operation takes? Does the size of the index matter and if any blocking occurs, does the duration increase because the size of indexes is larger? I've been told by management, we can not have any downtime, its my understanding even with online index there's a chance blocking can occur in the final phase of the index operation. My database and index size is over 1.5 tb and the number of transaction per second are in the 100's.
View 2 Replies View RelatedI posted this to the ms sql clustering news group, but I thought I'dpost it here just in case someone here knew the answser. This problemhas had me baffled for 5 months, and I'm not seeing any supportarticles from Microsoft on it, yet. Anyone here got a clue?About 5 months ago, I posted a query to this group after experiencingfailures with my database maintenance plans (during the verify phase)that run on SQL Clusters on Windows 2003(http://groups-beta.google.com/group...9b?dmode=source)I was re-posting this in hopes that someone knows of a fix besidescoding a home-brew verify step in to the backup job to check the file'sintegrity?Thanks,TW
View 1 Replies View RelatedHello,
I have noticed that for one of my data-flows, the process is really long during the phase "the final commit data insertion has started".
To be accurate, the process is fast until it reaches this phase. It happens often when I load millions of lines.
The extraction is done from a database SQL Server 2005 to a database SQL Server 2005, on the same server (with the SQL Server native provider).
I used a SQL Server destination but I have tried with an OLE DB destination and it is the same situation.
Why the process could be so long during this phase?
There is a way to optimised my package to avoid that?
Any idea is welcome.
Thanks.
Guillaume
If monitoring for duration with sql profiler, what does the number represent ie 2733906 is it milliseconds, thousandths, looked in BOL no clear definition
View 2 Replies View RelatedCan someone please tell me what unit of measurement the Duration column is in when running SQL Profiler? I assume milleseconds, but am not sure.
I'm trying to filter on this and want to be sure I'm not missing anything.
Thanks!
I found this nifty code on stackoverflow that works well but I'm trying to send the results to a text file and the column lengths are huge. I used CAST for the first line and it worked great but I can't seem to make it work with duration. Here's the original code:
SELECT
j.name,
h.run_status,
durationHHMMSS = STUFF(STUFF(REPLACE(STR(h.run_duration,7,0),
' ','0'),4,0,':'),7,0,':'),
[code]....
how to run a CAST on DURATION?
I am trying to get a query that will allow me to report the time taken to complete a certain training module.
The database itself does not have a duration field so I am tring to get the duration by using MIN and MAX. I can get the timing for when the module was opened and the time for the last mouse click on it, from this I need to be able to calculate the time taken to complete.
Query I am using to get the basic info comes from 3 tables so I have only attached the relevent output. Query used is as follow:
SELECT *
FROM PPS_SCOS, PPS_TRANSCRIPTS, PPS_TRANSCRIPT_DETAILS, PPS_PRINCIPALS
WHERE PPS_SCOS.SCO_ID = PPS_TRANSCRIPTS.SCO_ID
AND PPS_TRANSCRIPTS.TRANSCRIPT_ID = PPS_TRANSCRIPT_DETAILS.TRANSCRIPT_ID
AND PPS_TRANSCRIPTS.PRINCIPAL_ID = PPS_PRINCIPALS.PRINCIPAL_ID
AND PPS_SCOS.NAME LIKE 'MTM-106 The Dangers of Smoking'
AND PPS_PRINCIPALS.NAME LIKE 'Nigel Cordiner'
AND PPS_TRANSCRIPTS.TICKET NOT LIKE 'l-%'
ORDER BY PPS_TRANSCRIPT_DETAILS.DATE_CREATED
Output:
pps_scospps_scospps_transcript_detailspps_principalspps_principals
SCO_IDNAME DATE_CREATED PRINCIPAL_ID NAME
136850MTM-106 The Dangers of Smoking08:17:2516287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:17:2516287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:17:4016287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:18:2516287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:18:5716287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:19:1416287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:19:4716287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:20:2116287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:20:4416287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:21:2616287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:22:1316287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:24:5516287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:25:1216287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:25:2916287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:26:4916287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:27:0216287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:27:2916287Nigel Cordiner
136850MTM-106 The Dangers of Smoking08:27:4316287Nigel Cordiner
Have added the column heading and the tables the output comes from.
Relatively new to SQL so any help would be greatly received.
I have a query that displays number of hours duration in a jobs by months matrix. Here is a simplified version of my tables.
CREATE TABLE jobhead (
jobnum varchar (14) NULL,
jobcompletiondate datetime NULL
)
CREATE TABLE labordtl (
jobnum varchar (14) NULL ,
clockindate datetime NULL ,
clockintime decimal(6, 2) NULL ,
)
Here's an abbreviated query.
SELECT h.JobNum,
(CASE WHEN MONTH(h.JobCompletionDate) = 1 THEN datediff(day,MIN(l.ClockInDate),MAX(l.clockInDate)) ELSE 0 END) AS JAN,
(CASE WHEN MONTH(h.JobCompletionDate) = 2 THEN datediff(day,MIN(l.ClockInDate),MAX(l.clockInDate)) ELSE 0 END) AS FEB,
...
FROM JobHead h
INNER JOIN LaborDtl l ON h.JobNum=l.JobNum
WHERE JobCompletionDate>='20070101' AND JobCompletionDate <'20080101'
AND l.ClockInTime<>0
GROUP BYh.JobNum
,h.JobCompletionDate
The query shows, for each job, the month in which the job completed, and the number of hours it took to complete. I€™m calculating the number of days€™ duration by doing a datediff between the oldest and newest clockindates. I need to ignore adjustment transactions in the labordtl table €“ these rows are easily identified as they have clockintime values of 0. So far, so good. Now here€™s my problem.
There are some jobs which have only one €śreal€? labor transaction €“ this could happen if the job only took one day to complete. Other labor transactions may exist for that job, but let's say they are adjustments which we can ignore -- the date they were entered should not extend the duration of the job. In this situation, my datediff between the oldest valid transaction and the newest, returns 0. I don€™t have to count hours between clockintime and clockouttime. The rule is simply that if there is only one "real" labor transaction, I need to count this as a 1 day job.
I thought a nested CASE statement or expression might be the way to go but I didn't make any real progress.
Any ideas to solve this problem would be appreciated.
Hi!
Does anyone know how to query for the job duration of all jobs in a server?
Thanks!
I'm developing a web app that displays the running packages and the total elapsed time. I'm calling GetRunningPackages() method and using the ExecutionDuration property of the returned package. The duration seems to be only for the currently executing container and not the entire package. Is there a way to get the duration of the entire package? Thanks.
View 2 Replies View RelatedHi all, I want to find working duration between two datetimes in c#.i'm using following code... DateTime starttime = Convert.ToDateTime(Session["StartTime"]); DateTime endtime = DateTime.Now; TimeSpan duration = endtime - starttime; DateTime period = new DateTime(duration.Ticks); i want to store this duration in database through stored procedure, i've give datetime datatype to duration but it is giving error in conversion of TimeSpan to DateTime..Please help... Thanks
View 1 Replies View RelatedSQL 6.5 - run duration 6-7 hours
SQL 7.0 - run duration 12-13 hours
175+ columns with total record size=570
4.2M records with tablesize 2.5G
It's a simple 'select into...' with some embedded logic from a work table with all char fields into the actual table converting char fields into various data types (int, datetime, real, etc.) Here is a sample of the code:
SELECT
LoanNum=CASE
WHEN ISNUMERIC(ACCT#)=1 THEN CONVERT(int,ACCT#)
ELSE NULL
END,
PaidToDt=CASE
WHEN PAIDDT = '0001-01-01' THEN NULL
WHEN ISDATE(PAIDDT)=1 AND SUBSTRING(PAIDDT,1,2) = '19' THEN CONVERT(smalldatetime,PAIDDT)
WHEN ISDATE(PAIDDT)=1 AND SUBSTRING(PAIDDT,1,2) = '20' AND SUBSTRING(PAIDDT,3,2) < '79' THEN CONVERT(smalldatetime,PAIDDT)
ELSE NULL
END,
.
.
.
INTO db.owner.tablename
FROM db.owner.wrktablename (NOLOCK)
Hello, everyone:
There is a trigger to monitor the modification on a table, and it turn on. For a special duration, I need to turn off this trigger to modify the table. And then turn on the trigger again.
Any help will be appreciated.
Thanks
ZYT
I have a table called Tickets which contains ticket information for a machine. Each machine can have more than one ticket number opened at the same time. The ticket number contains start date/time and end date/time of the ticket. Thereefore the table looks something like this:
Ticket_No (int)
Machine_No (int)
Description (char)
Start_Time (datetime)
End_Time (datetime)
I want to be able to calculate total duration time(in hours) that EACH MACHINE had a ticket open...but here is the tricky part. The total duration time that a machine had ticket open has to encompas any tickets that may fall in the same time period. For example:
If Machine A has a ticket open at 8:30 and the ticket is closed at 10:00. Meanwhile, Machine A had another separate ticket open at 9:30 which was closed at 10:30. In this case, the total duration time for this machine would be from 8:30 to 10:30 for a total of 2 hrs duration time.
Can anyone help me get started in tackling this problem or provide any examples?