In my data flow, I am reading addresses from a CSV file. Then for each row, I would like to execute a process from the command line which outputs the latitude and longitude for the address, parse the output, and add the latitude and longitude into the pipeline. To call the process, I am using a script component transform. Here's my code:
Dim m_Latitude As Double
Dim m_Longitude As Double
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim street As String
Dim city As String
Dim state As String
Dim zip As String
street = Row.address
city = Row.city
state = Row.state
zip = Row.zip
Dim p As Process = New Process()
p.StartInfo.FileName = "C:\GeoCodeDotNet.exe"
p.StartInfo.Arguments = String.Format("""{0}"" ""{1}"" ""{2}"" ""{3}""", street, city, state, zip)
p.StartInfo.WorkingDirectory = "C:\"
p.StartInfo.UseShellExecute = False
p.StartInfo.CreateNoWindow = True
p.StartInfo.RedirectStandardOutput = True
AddHandler p.OutputDataReceived, New DataReceivedEventHandler(AddressOf ConsoleDataReceived)
p.Start()
p.BeginOutputReadLine()
If p.WaitForExit(10 * 1000) Then
Row.Latitude = m_Latitude
Row.Longitude = m_Longitude
Else
p.Kill()
Row.Latitude = 0.0
Row.Longitude = 0.0
End If
End Sub
Private Sub ConsoleDataReceived(ByVal sender As Object, ByVal e As DataReceivedEventArgs)
Dim output As String() = e.Data.Split(New [Char]() {" "c})
m_Latitude = CDbl(output(0))
m_Longitude = CDbl(output(1))
End Sub
I'm just getting very weird behavior. First of all, at the point where I assign values to Row.Latitude and Row.Longitude, m_Latitude and m_Longitude don't always have valid values (e.g. - they are unassigned). Secondly, after attempting to process the first couple rows, it just stops. In my data flow, the script component is yellow, but execution has ended, and the final step of writing to the output CSV file has not even started. Finally, in the directory where my source CSV file is located, I get a SQL dump file with the following content:
I'm guessing this all has to do with some kind of threading/concurrency thing and how the data flow pipeline works. Could someone please shed some light on this?
By the way, the script component transform is synchronous.
Much thanks.
Dear list Im designing a package that uses Microsofts preplog.exe to prepare web log files to be imported into SQL Server
What Im trying to do is convert this cmd that works into an execute process task D:SSIS ProcessPrepweblogProcessLoad>preplog ex.log > out.log the above dos cmd works 100%
However when I use the Execute Process Task I get this error [Execute Process Task] Error: In Executing "D:SSIS ProcessPrepweblogProcessLoadpreplog.exe" "" at "D:SSIS ProcessPrepweblogProcessLoad", The process exit code was "-1" while the expected was "0".
There are two package varaibles User::gsPreplogInput = ex.log User::gsPreplogOutput = out.log
How do I use the execute process task? I am trying to unzip the file using the freeware PZUnzip.exe and I tried to place the entire command in a batch file and specified the working directory as the location of the batch file, but the task fails with the error:
SSIS package "IngramWeeklyPOS.dtsx" starting.
Error: 0xC0029151 at Unzip download file, Execute Process Task: In Executing "C:ETLPOSDataIngramWeeklyUnzip.bat" "" at "C:ETLPOSDataIngramWeekly", The process exit code was "1" while the expected was "0".
Then I tried to specify the exe directly in the Executable property and the agruments as the location of the zip file and the directory to unzip the files in, but this time it fails with the following message:
SSIS package "IngramWeeklyPOS.dtsx" starting.
Error: 0xC002F304 at Unzip download file, Execute Process Task: An error occurred with the following error message: "%1 is not a valid Win32 application".
The command in the batch file when run from the command line works perfectly and unzips the file, so there is absolutely no problem with the command, I believe it is just the set up of the variables on the execute process task editor under Process. Any input on resolving this will be much appreciated.
I am designing a utility which will keep two similar databases in sync. In other words, copying the new data from db1 to db2 and updating the old data from db1 to db2.
For this I am making use of the 'Tablediff' utility which when provided with server name, database, table info will generate .sql file which can be used to keep the target table in sync with the source table.
I am using the Execute Process Task and the process parameters I am providing are:
The customer.bat file will have the following code: tablediff -sourceserver "LV-SQL5" -sourcedatabase "TC_CTI" -sourcetable "CUSTOMER_1" -destinationserver "LV-SQL2" -destinationdatabase "TC_CTI" -destinationtable "CUSTOMER" -f "c:SQL_bat_Filessql5TC_CTIsql_filescustomer1"
the .sql file will be generated at: C:SQL_bat_Filessql5TC_CTIsql_filescustomer1.
The Problem: The Execute Process Task is working fine, ie., the tables are being compared correctly and the .SQL file is being generated as desired. But the task as such is reporting faliure with the following error :
[Execute Process Task] Error: In Executing "C:SQL_bat_FilesSQL5TC_CTIpackage_occurrence.bat" "" at "C:Program Files (x86)Microsoft SQL Server90COM", The process exit code was "2" while the expected was "0". ]
Some of you may suggest to just set the ForceExecutionResult = Success (infact this is what I am doing now just to get the program working), but, this is not what I desire.
Hi I am currently trying to write a custom transform componet in c# that will take a row of data, perform a look-up via an external system, then if there is a match then send the data from the extranel system down macth ouptut (which will have different columns to the input) and drop the data that was read, else send the data down the unmacthed output which will be the same as the input.
So I would like to write a synchrons transform becuase I don't need read all the rows from the input buffer before I started processing, also I wish have millions of rows load in memory.
Can this be done? also does any have explame code of how to do this? becuse I can't see how to send data down the match output buffer, as this will have the lookup results data which will have diffent columns to the input data and how disgard the input data as well.
I have a requirement to access a lookup table from within an SSIS Transform Script Component
The aim is to eliminate error characters from within the firstname, lastname, address etc. fields by doing a lookup of an ASCII code reference table and making an InStr() type comparison.
I cannot find a way of opening the reference data set from withing the transform.
Would anyone happen to have any pointers or know of any good code examples to either programmatically change the type of an input column when it is passed through the component, or add a new column to the output? I am extracting data from an Oracle database which is in Julian date format (represented within SSIS as a DT_NUMERIC column) and I need to to either transform the input column holding it into a date column, or to dynamically add a new output column holding the transformed data.
I have a master package, which executes child packages that are located on a SQL Server. The Child packages execute other child packages which are also located on the SQL server.
Everything works fine when I execute in process. But when I set the parameter in the mater package ExecutePackageTask to ExecuteOutOfProcess = True, I get the following error
Error: 0xC00470FE at DFT Load Data, DTS.Pipeline: SSIS Error Code DTS_E_PRODUCTLEVELTOLOW. The product level is insufficient for component "Row Count" (5349).
Error: 0xC00470FE at DFT Load Data, DTS.Pipeline: SSIS Error Code DTS_E_PRODUCTLEVELTOLOW. The product level is insufficient for component "SCR Custom Split" (6399).
Error: 0xC00470FE at DFT Load Data, DTS.Pipeline: SSIS Error Code DTS_E_PRODUCTLEVELTOLOW. The product level is insufficient for component "SCR Data Source" (5100).
Error: 0xC00470FE at DFT Load Data, DTS.Pipeline: SSIS Error Code DTS_E_PRODUCTLEVELTOLOW. The product level is insufficient for component "DST_SCR Load Data" (6149).
The child packages all run fine when executed directly, and the master package runs fine if Execute Out of Process is False.
I know this one has shown up here before, and I have tried the various things to get past this error and they are not working for this situation. First of all I have tried setting the ReadBatchSize to 1 and secondly I have made the QueryTimeout paramter 2000 in case that were the problem.
In looking at the Error Details within the Log Reader Agent, the following is the message:
Error Message: "The process could not execute 'sp_replcmds'"
Error Details: "An invalid datetime value was encountered. Value exceeds the year 9999."
I can not seem to find any additional information based on the Error Details. If any of you has a suggestion, I would love to hear them.
Hello all, I have the need to connect to a network drive as a different user id when my SSIS package runs. If I stick the command into a batch script (forums don't play well with Opera, so sorry for code block not working)
@echo off C:windowssystem32cmd /c net use \ServereeShare SecretPassword /usererverSecretUser
If I call that batch script from SSIS, at the conclusion of execution I'll have a share mapped. To ease deployment and maintenance, I want to refrain from using the batch script. So, I set my task as so Executable: cmd.exe Arguments: /C WorkingDirectory: C:windowssystem32 StandardInputVariable: User::FileShareMappingParameters StandardOutputVariable: User::FileShareMappingOutput StandardErrorVariable: User::FileShareMappingError
and the rest are the default values. The value of User::FileShareMappingParameters is "net use \ServereeShare SecretPassword /usererverSecretUser"
My task does not fail nor do the output or errors variables get populated but I do not get my share. Anyone out there run into a similar situation or have a pointer on this one?
We have an SSIS Execute Process Task which calls an executable along with the required parameters. When we run this package, it intermittently gives the error as shown below in red
Executing "ppscmd.exe" "StagingDB /Server http://SERVERNAME:46787 /path OSB_FY08.Planning.dimensionECFuncArea /Operation LoadDataFromStaging" at "", The process exit code was "1" while the expected was "0". End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:05:08 PM Finished: 5:08:40 PM Elapsed: 212.203 seconds. The package execution failed. The step failed.
We are not able to debug this issue. We had a look at the logging information as well but we are not getting any information on this issue.
I am trying to exectue batch file which is located on other server. Can someone explain how to use execute process task? I can see Executable ,Arguments, working directory, standard input v ariable, standard output variable, Fail task if return code is not success, success value when I open execute process task .
Also when I try to execute the batch file I am getting "Publisher could not be verified window and then its working after clicking the RUN button on the message. How can I avoid this in SSIS ?
For example I have the batch files located in some location \112.156.82.9\BATCHFILES\FILENAME
A batch file when being executed using the "execute process task"(DTS package) gives problems whereas the same batch file without any change when executed manually from the explorer runs fine.
The problem in debugging is that sometimes the same task runs fine and sometimes not....
The batch file contains code to extract a zip file(zipped using arj) to a paricular directory.
In the properties sheet for a DTS Execute Process Task, there is a input box for 'Parameters' which allow you to add command-line options to the task exe specified. How can you use the DTS Global variables to be passed in this parameter field? Is there a special format to indicate that the parameters are in fact global variable tokens?
i have designed DTS packages, I have a script component that picks up the 'Path' of a file stored. The Path is a column in database and that obtained from OLEDB source.
now i need to zip the files in the path given above and than store it in some location. How do I do that?
We are attempting to use SSIS execute process task.
In a cmd.exe session the executable runs with normal arguments:
mycmd < myscript.txt > foo.txt 2>error.txt
We have attempted to several permutations of this without success. We declared variables and assigned them to stdin, stdout, stderr. However, mycmd opens in the SSIS session as if no arguments have been supplied.
Is there some expression that is required to connect the stdin/out/err variables in the Arguments within Execute Process Task. The help/examples are terse.
I have stored procedure which executes some exe console application, which creates excel file on the fly and return result synchronously. The main part of code is :
The problem is that now I'm moving SQL database to consolidate server where it is not allowed to run some exe.What would be the best way to execute exe remotely(on remote process) and read return value?This ProcessStartInfo executes exe from remote server but process runs inside current instance.
I have a FOREACHLOOP container, that is set up to loop through files in a directory:
Enumerator: Foreach File Enumerator
Folder: c:myfolder
Files: *.txt
Retrieve: Name and extension
Variable mappings: User::file_name, Index 0
So far, so good...
Then, inside the FOREACHLOOP, I have an Execute Process Task, which is simply a .bat file that I am running, and passing the name of the file to:
For example: myBat.bat file_name
In the Execute Process Task, I have:
RequireFullFileName: false
Executable: c:myFilemyBat.bat
WorkingDirectory: c:myFile
StandardInputVariable: User::file_name
Everything else is set to the defaults.
However, when I execute my container, I get the following error:
Error: 0xC0029151 at Execute Process Task, Execute Process Task: In Executing "c:myFilemyBat.bat" " " at "c:myFile", The process exit code was "1", while the the expected was "0".
Is there anything I am obviously doing wrong here at the package level? When I run the .bat file from the command line, it works fine.
My problem is that I am trying to set up an Execute Process Task to run a C# executable. The .exe captures information from an ftp site and downloads it into a directory on our server. Here is the command line run:
C:SAXOEodinReleaseSaxoEod
I have set the Execute Process Task up as follows:
I have the FailTaskIfReturnCodeIsNotSuccessValue set to False because when it is set to true I get the following error:
Error: 0xC0029151 at Execute Process Task, Execute Process Task: In Executing "C:SaxoEODobjReleaseSaxoEod.exe" "" at "", The process exit code was "-532459699" while the expected was "0".
Setting FailTaskIfReturnCodeIsNotSuccessValue to False does not work either because the files are not downloaded from the FTP site.
I am relatively new to the SSIS arena. I have googled this thing to death and have not found an answer. Any help in the right direction is appreciated. Thanks.
I need help with some C# code. I have build a SSIS package with an Execute Process Task. I need to send dynamic variables in to my C# program so I thought it was a good idea to use the StandardInputVariable.
I am having trouble running a package in SQL Agent. The step involves zipping up a number of files using SQL Agent. I know in the error message it states "Access is denied" but I can run the package manually in BIS. Also I have applied a SQL proxy to the step using my own credentials which have rights to the file location, but still no luck.
I get the following error:
Error: 2008-03-10 09:04:02.04 Code: 0xC002F304 Source: Call ZipFiles Batch Execute Process Task Description: An error occurred with the following error message: "Access is denied". End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 09:04:00 Finished: 09:04:02 Elapsed: 1.219 seconds. The package execution failed. The step failed.,00:00:02,0,0,,,,0
HI, I have some parent parent packages that calls child packages. When I added a bunch of packages, I faced the buffer out of memory error. I then decided to set the child packages property ExecuteOutOfProcess to TRUE. I noticed that the execution time is longer now. Is this a good practice to set the ExecuteOutOfProcess to true? If so, is it normal that the execution time is longer?
On an Execute Process Task, are there properties I can examine to determine the outcome of a process? I know that I can use a Success/Failure constraint to direct my workflow appropriately, depending on the outcome, but I would prefer the workflow continue on to the same task regardless, and then be able to change the flow later on based on the outcome of the process.
I see there is an ExecValueVariable property, and this may be what I'm looking for, but I have no idea what this is and can not find any documentation on it. I tried using it, but kept getting a runtime error trying to set it to a variable.
I know that a process returns a numeric value, and it would be nice if I could store this somewhere.
Hi All, Would you please clarify me how to use or setup Execute Process Task, i have seen the document from microsoft but i need more detail and if possible example how to setup. what does it mean by Executable, Arguments, Working Directory, StandardInputVariabl, StandardOutputVariable and some example respectively. And how do wee set them with example, especially taking taking zip/unzip executable file. Please i need help and make it clear example. I really appreciate for your time in advance.
I am using a Excute Process Task calling a selfwritten EXE and I'm trying to pass a command line argument to it. The code for the EXE is quit simple:
Public Sub Main() Try Dim info As String = My.Application.CommandLineArgs.Item(0) MessageBox.Show("Info:" & info) Catch ex As Exception MessageBox.Show(ex.ToString) End Try End Sub
According to the help files there are two ways to pass an argument, using the property Arguments, which I guess is for hard-coded stuff, and the StandardInputVariable for passing dynamic info.
When I use the Arguments property everything works fine. I get the info messagebox with correct data. But when I leave that blank and instead use the StandardInputVariable property setting it to my package variable User::Info then I crash into an exception meaning that no arguments existed (array is out of bonds).
The variable User::Info was at first filled from a previous SSIS task and using the OnPreExecute breakpoint I verified that it contained a stringvalue. I then hardcoded a string value into the variable, but nothing helps. The task refuses to start my EXE with the data in the StandardInputVariable as an argument.
For the Execute Process task (such as for running .bat files), what is the "Working Directory" for?? Should it be set to the directory that contains the .bat file, for example?
I have a SSIS package that has several Execute SQL Components. One of the first components reurns a Full Result Set of IDs based on a stored procedure call. The stored procedure can return multiple rows. I store the results to an ADO recordset (object variable) to be used later. I want the component to fail, and the package if the return of the stored procedure is zero records. What is the best way to do this? I had a raise error statement if @@rowcount was zero but this did not fail the component. Any other suggestions?
Hello, I have been wrestling with a script component and the post execute method. I have obtained some bits of information from the discussions from previous posting - http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=97494&SiteID=1 and have also tried the suggested approaches. Being new to SSIS and .NET as well I am certain that I do not see the entire picture yet.
I have a Read/Write variable that I am assigning a value to. The variable is then processed through a derived column and subsequently used by a flat file.
I have eliminated the following exception thus far.
Error: 0xC0047062 at Data Flow Task, Script Component [518]: Microsoft.SqlServer.Dts.Pipeline.ReadWriteVariablesNotAvailableException: The collection of variables locked for read and write access is not available outside of PostExecute.
But, my flat file does not contain in it the assigned values to the variable. A brief summary of the process follows:
OLE DB SOURCE ===> followed by the Script Component ===> followed by the Derived Columns ===> followed by the flat file.
Sample Script is:
Public Class ScriptMain Inherits UserComponent
Public NewName As String
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer) ' ' Add your code here ' NewName = Row.BPNM
Me.Variables.ShortName = NewName
End Sub
Public Overrides Sub PostExecute()
MsgBox("In first script...") Me.Variables.ShortName = NewName MsgBox("variable is " & Me.Variables.ShortName) End Sub
Question. What am I not doing that prevents me from extracting and passing the respective values to the derived column for each of the rows processed. When I try to assign the value of NewName to the Short Name variable, In red above, I encounter the post execute exception, again.
What code should I have in place to insure that I am not trying to reference the variable on a PostExecute within the normal process of the script component? Any guidance would be immensly appreciated.
I have several DTS packages that take data from SQL Server and exports them onto an Access DB located on the network. Basically, one Execute Process Task from within my DTS package takes the Access DB and zips it while another such task copies the zipped DB and pastes it onto another location on the network. All this works fine.
This export process happens once a month so each month I have to manually add a datestamp to the end of the Access DB file that’s being exported to distinguish it from prior month's export. For example, this month's export file would have AccessDB_20040727.mdb name and the next month it would be AccessDB_20040820.mdb (date is determined based on the date the output is exported on). AccessDB.mdb is the default name of the export DB and datestamp is added at the end of the file name depending on what date the export was run. As I said, I can do this manually each month and it works fine.
I want to, however, know if there is a way to automatically supply the datestamp to the Execute Process Task's Parameters' text box? Following is what I have right now in the Parameters box: \ghf1-ndc8-sqll$productionoutputlob200406LOB_CF_forDril ldown_All_20040727.zip \ghf1-ndc8-sqll$productionoutputlob200406LOB_CF_forDril ldown_All.mdb
I want to take above text from the parameters box and replace it with something like: \ghf1-ndc8-sqll$productionoutputlob200406FileName_RunDa te.zip \ghf1-ndc8-sqll$productionoutputlob200406LOB_CF_forDril ldown_All.mdb
Where FileName_RunDate is a variable/placeholder for the name of the output with the datestamp the output is exported on.
There are three different Execute Process Tasks that are happening within each of my DTS packages so it's a time consuming job to have to manually add a datestamp to each package every month when data is exported.
Does anyone know if what I am asking is doable? If I can use a variable in the parameters box of each Execute Process Task’s properties and supply current datestamp values to it prior to executing the package each month? If so then what are the ways? How would I do that?
I am having trouble running a package in SQL Agent. The step involves zipping up a number of files using SQL Agent. I know in the error message it states "Access is denied" but I can run the package manually in BIS. Also I have applied a SQL proxy to the step using my own credentials which have rights to the file location, but still no luck.
I get the following error:
Error: 2008-03-10 09:04:02.04 Code: 0xC002F304 Source: Call ZipFiles Batch Execute Process Task Description: An error occurred with the following error message: "Access is denied". End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 09:04:00 Finished: 09:04:02 Elapsed: 1.219 seconds. The package execution failed. The step failed.,00:00:02,0,0,,,,0
I'm just starting to find my way around SSIS, coming from SQL 2000 DTS, but I can't see a way of including a Win32 Console application's output into the logging process.
I've started playing around with SSIS Logs (OnTaskFailed etc..), but I can find no where to allow me to capture the output from a console app. In SQL 2000 DTS I could capture this by specifying an "output file" on the advanced tab of the job step definition. Is there something as straight forward as this in SSIS ?
I'd be grateful for a few pointers in the right direction.