Datafile SSIS Transform XML - Help
May 19, 2007
Hi,
I want to insert the contents of a data file into a SQL Server table with XML datatype column using SSIS. I am fairly new to SSIS and script tasks.
Example:
Salary Data file:
EmpID, Name, Salary
100, John, 100000
200, Scott, 200000
Database Table:
create table xmlTable (id int identity(1,1), xmlColumn XML)
The contents of the file needs to be written to xmlTable. For each row in the Salary Data file there will be a corresponding row in the xmlTable.
xmlTable (Result):
1, <EmpID 100...>
2,<EmpID 200...>
Any help with details is greatly appreciated.
Thanks,
Rob
View 4 Replies
ADVERTISEMENT
Aug 19, 2015
I have seen several problems posted where an SSIS package writes a file successfully when executed manually, but fails when executed via SQL Agent job..I have the opposite problem. I'll try to lay it out succinctly: SSIS writes to a file on a shared folder, specified as HostAShare, for example. I created the share and gave full control to Everyone (out of frustration).I'm working from HostA via RDP, connected to the DB on DBHost via SSMS.If I kick off a SQL Agent job that executes the package, it works fine. (SQL job is running under SQL Server Agent Service Account).If I execute the job interactively (logged in to SSMS with Windows Auth), it fails with "Error: Cannot open the datafile "HostASharefilename.ext""We did find that if I RDP directly to the DBHost, I am able to execute manually.Also, if I try executing an xp_command shell command to write a file to the share, it works. (When RDP's into HostA with an SSMS connection to DBHost under my windows domain auth, as above.)The problem is the same when I RDP to any remote host.
View 2 Replies
View Related
Oct 26, 2007
Hi,
I need some urgent help on a conceptual question.
Requirement:
I have a simple package with one dataflow task. In that I need to read from a sql table and for every row in that table loop through n times and generate new output rows based on certain conditions (which are best evaluated in custom script as they are rather complex). Hence, if I have 100 rows in the table as my input, I may end up with 100*n rows as output.
My Design:
To implement this I have used an OLE DB Source which outputs to a Script Transform (ST). In the ST I intend to loop through in custom code and generate new rows using the .AddRow feature when I need new rows. This ST then feeds into another OLE DB Destination which writes the data to the table. Simple!
I am using the default buffer settings. All I have tweaked is the Synchronous... property on the script transform (otherwise I do not get to the Output0Buffer within the script!).
Problem:
I wish to do as much as possible in parallel. So I would expect the OLE DB Source to provide more than one row at a time to the script transform and that should process more than one input row simultaneously. It seems the script componenet is serializing input, so it seems to take one row at a time from the OLE DB source, loop through and process in the script transform).
AM I RIGHT IN THINKING THAT THE SCRIPT TRANSFORM IS EXECUTING THE INPUT IN A SEQUENTIAL MANNER?
CAN I PARALLELISE THIS?
If so, how?
Thanks,
Tirthankar
View 3 Replies
View Related
Jan 15, 2008
High all,
I have a very simple SSIS package that is moving data from a DB2 database to a Teradata box. I've run it around 10 times, twice it pushed data over, the balance of the time, it executes with no error, but moves nothing over. In the "incomplete" runs, a command line box pops up for half a second, then the package ends.
Does anyone have ideas as to why this behavior is occurring?
Thanks,
Mark
View 1 Replies
View Related
Jun 21, 2005
My first period using SSIS in a real-world application convinced me that there
View 4 Replies
View Related
Oct 2, 2006
I am trying to write a ssis surrogate key data transform, my problem is I can't find an example how to add a column to the incoming columns and add some data to it. If anyone has a sample, can you please post it. I found a script option that works but I would like an actual transform.
Thanks
View 2 Replies
View Related
Jan 30, 2015
I have an SSIS with several data flows I need to do some complex data evaluations so I have used a script as transform in two of the DFT's. If I run these separately everything works great and there are no problems what so ever. If I run them together I notices I was getting an error on the second one. I discovered that this seems to be some kind of namespace problem since both Scripts were using Input_0 buffer. So I changed the name of the second one and retested.
Well I no longer get the error and in fact it seems to run through the entire SSIS just fine. However when I look closer I notice that the second Script task just does not seem to do anything at all. The script task does a lot of evaluation of the incoming data and then does some calculations depending on the value in the service code. however when it runs through this in the second script task all of the define output rows are just empty.
I have gone through and made sure that all input and output buffers are unique names thinking this was a similar problem but no luck. I even changes all column and variable names to unique with no luck. Again If I run them separately everything work fine it is only when I run the entire package that this problem occurs.
View 2 Replies
View Related
Oct 12, 2006
Hello,
I have an OLE DB Source and i want to transform the data type fields of the table before i export the table in an OLE DB Destination.
Is there a way to transform numeric value to float, and numeric to nvchar?
Thank you in advance.
View 5 Replies
View Related
May 3, 2007
I am trying to read in a flat file, transform the fields and store into a destination database.
In DTS, this works using Transform Data Task Properties. I define the columns and then have a VB script on the Transformations tab that changes any bad data.
Is there a way to do this in SSIS that I can define the column transformations and re-use my VB scripts?
Linda
View 16 Replies
View Related
May 8, 2015
I have a Pivot Transform in SSIS (2005) working perfectly, EXCEPT for that the first column of the output (the date) repeats for each of the following columns, which are themselves falling into the correct column, but not on the same line for a particular date as the others. Snipet of result from Data Viewer is:
dbDate site1 site2
1/1/2015 0:00 0.03 NULL
1/2/2015 0:00 0.04 NULL
[code]....
View 2 Replies
View Related
Mar 7, 2008
i have too many DTS packages to migrate to SSIS, and while examining a DTS package in BIDS (converted with the migration utility) i tried to edit the resulting migrated package, which opened the DTS interface with the two connection icons joined by the big fat arrow with a gear on it...not exactly what i had in mind, iow, it looks like SSIS on the outside, but its still DTS on the inside.
So I stripped out a series of components from a more complex package hoping that simplifying it would reveal the contents of old DTS Transformations tab at least partially set up in a Derived Column transformation.
Can i get there from here, or must i recreate every stinking definition in a derived column manually from the ground up?
thanks very much for your help
View 2 Replies
View Related
Oct 26, 2006
I can't figure out how to put nested tables into the Data Mining Model Training Transform (SSIS). I can do a simple case table, but how do you get those nested tables with DM Training Transformation? Any ideas? Samples?
Thanks in advance,
-Young K
View 3 Replies
View Related
Jan 16, 2013
I work in the healthcare area, and am handling the survey data ETL's. There are around 8 different survey areas and based on information received from them for the visit they reference, I want to pull in more info from our invoicing database. My idea is this:
1.)Â Pull in the flat file to an ODBC staging table
2.)Â Cache all invoice records that fall between the MIN(Date of Service) and MAX(Date of Service) from the staging table.
3.)Â First lookup the information needed on patientID, providerID, date of service, and billing location.
4.)Â For the surveys that didn't match on those 4 columns, try looking up based on patientID, date of service, and billing location (since I could be 99% sure this would still return the record I need).
5.) For the remaining surveys, lookup based just on patientID and date of service. These records will be flagged for manual review because clearly, if a patient has multiple appointments in the same day, this will be prone to error.
However, in trying to use only 3 of the columns in the lookup, I get the error saying basically that I need to utilize all 4. Is there a way around this, or is there an entirely different way I should be approaching this? The reason I thought cache transform was the answer is because I will need to run a different package for each lookup, as the data and logic between each survey will vary, but the invoice data "pool" will stay the same regardless.Â
View 5 Replies
View Related
Jun 2, 2011
In my current project i have a requirement to assign value of an aggregate transform to a variable. But i need to accomplish it without using a script task.
View 3 Replies
View Related
Aug 9, 2007
I would like to know what happens when a very large reference data set for a lookup transform with full caching enabled is getting loaded during package execution and the computer memory runs out or is very low.
Does SSIS
a) give an out of memory error of some sort
b) resort to a no caching or partial caching mode
c) maintain the full caching mode but will switch to using the paging file(virtual memory).
I think it will resort to using the page file in which case the benefits of in memory lookups are lost and performance would suffer. If I cannot upgrade the memory or shrink the reference set somehow, i should switch that lookup task to use partial caching or no caching with an indexed lookup table. Would this make sense?
View 1 Replies
View Related
Jul 30, 2015
We are using lookup transformation in SSIS 2012. The lookup transformation queries a table with two date columns. When we hover the mouse over the two columns in the 'columns' tab of the lookup transformation editor, the two columns show as DT_WSTR instead of DT_DBDATE. This causes the SSIS package to fail due to data type mismatch.A similar abandoned thread is available at: URL....
View 2 Replies
View Related
May 7, 2007
Hi JayH (or anyone). Another week...a new set of problems. I obviously need to learn .net syntax, but because of project deadlines in converting from DTS to SSIS it is hard for me to stop and do that. So, if someone could help me some easy syntax, I would really appreciate it.
In DTS, there was a VBScript that copied a set of flat files from one directory to an archive directory after modifying the file name. In SSIS, the directory and archive directory will be specified in the config file. So, I need a .net script that retrieves a file, renames it and copies it to a different directory.
Linda
Here is the old VBScript Code:
Public Sub Main()
Option Explicit
Function Main()
Dim MovementDataDir
Dim MovementArchiveDataDir
Dim MovementDataFile
Dim MovementArchiveDataFile
Dim FileNameRoot
Dim FileNameExtension, DecimalLocation
Dim CurMonth, CurDay
Dim FileApplicationDate
Dim fso ' File System Object
Dim folder
Dim FileCollection
Dim MovementFile
'======================================================================
'Create text strings of today's date to be appended to the archived file.
FileApplicationDate = Now
CurMonth = Month(FileApplicationDate)
CurDay = Day(FileApplicationDate)
If Len(CurMonth) = 1 Then
CurMonth = "0" & CurMonth
End If
If Len(CurDay) = 1 Then
CurDay = "0" & CurDay
End If
FileApplicationDate = CurMonth & CurDay & Year(FileApplicationDate)
'=====================================================================
' Set the movement data directory from the global variable.
MovementDataDir = DTSGlobalVariables("gsMovementDataDir").Value
MovementArchiveDataDir = DTSGlobalVariables("gsMovementDataArchiveDir").Value
fso = CreateObject("Scripting.FileSystemObject")
folder = fso.GetFolder(MovementDataDir)
FileCollection = folder.Files
' Loop through all files in the data directory.
For Each MovementFile In FileCollection
' Get the full path name of the current data file.
MovementDataFile = MovementDataDir & "" & MovementFile.Name
' Get the full path name of the archive data file.
MovementArchiveDataFile = MovementArchiveDataDir & "" & MovementFile.Name
DecimalLocation = InStr(1, MovementArchiveDataFile, ".")
FileNameExtension = Mid(MovementArchiveDataFile, DecimalLocation, Len(MovementArchiveDataFile) - DecimalLocation + 1)
FileNameRoot = Mid(MovementArchiveDataFile, 1, DecimalLocation - 1)
MovementArchiveDataFile = FileNameRoot & "_" & FileApplicationDate & FileNameExtension
If (fso.FileExists(MovementDataFile)) Then
fso.CopyFile(MovementDataFile, MovementArchiveDataFile)
' If the archive file was coppied, then delete the old copy.
If (fso.FileExists(MovementArchiveDataFile)) Then
fso.DeleteFile(MovementDataFile)
End If
End If
Next
fso = Nothing
folder = Nothing
FileCollection = Nothing
Main = DTSTaskExecResult_Success
End Function
View 6 Replies
View Related
Nov 7, 2007
I lost my datafile , that's corrupted and I don't have database backup. Any idea ?:(
View 1 Replies
View Related
Nov 3, 2004
If a database consists of more than one datafile, how does SQL Server use the space in these datafiles ?, does it fill up the first one then move to the next and so forth, or does it use up pages across all the files evenly ?.
View 3 Replies
View Related
May 23, 2007
Hi need to something like this:
"alter database modify file (name=<name>, ONLINE)",
but it isn't possible.
Here what I did:
alter database bdj add file (name ='bdjfg1', filename='d:db
djfg1.ndf' )
to filegroup bdjfg;
alter database bdj modify file (name='bdjfg1', OFFLINE);
alter database bdj modify file (name ='bdjfg1', filename='d:db
ewdestdjfg1.ndf' );
--Msg 5056, Level 16, State 4, Line 1
--Cannot add, remove, or modify a file in filegroup 'bdjfg' because
the filegroup is offline.
alter database bdj modify filegroup bdjfg READWRITE;
--Msg 5056, Level 16, State 3, Line 1
--Cannot add, remove, or modify a file in filegroup 'bdjfg' because
the filegroup is offline.
Yes, yes and should have read the cautions section saying:
"Use this option only when the file is corrupted and can be restored. A file set to OFFLINE can only
be set online by restoring the file from backup. For more information about restoring a single file,
see RESTORE (Transact-SQL)."
But I have not an backup of the datafile, but I have the datafile itself!
What can I do to get it online again, the old location could be fine, but it would be better on an new location (thats is the reason for all the trouble, the original drive has not much space left, so I wanted to move the datafile)
What can I do, HELP
/Bjorn
View 1 Replies
View Related
Mar 12, 2008
Hi,
If I run the package from BIDS, it works fine. If I run the package inside Management Studio it works when I run it as a package.
It does NOT run when I schedule the job.
Error: 2008-03-12 10:51:56.16
Code: 0xC020200E
Source: Data Flow Task Flat File Destination [194]
Description: Cannot open the datafile "D:old_timesheet_reposTimeSheetfilesdate.txt". End Error
Error: 2008-03-12 10:51:56.16
Code: 0xC004701A
Source: Data Flow Task DTS.Pipeline
Description: component "Flat File Destination" (194) failed the pre-execute phase and returned error code 0xC020200E. End Error
DTExec: The package execution returned DTSER_FAILURE (1).
Started: 10:51:55 AM Finished: 10:51:56 AM Elapsed: 0.344 seconds.
The package execution failed. The step failed.,00:00:01,0,0,,,,0
Any idea?
Thanks in advance.
Eric.
View 6 Replies
View Related
Aug 3, 2001
Hi,
Is there any way to change the location for the datafile. I need to change the drive from say c to d because it is filling up. Is there anyway to do this or do I have to recreate the database from scracth. I have a whole lot of data in the database already.
Thank you all.
View 1 Replies
View Related
Dec 14, 2004
Hi All,
I have a database -- MDB -- with datafile for data and transaction log under the folder d:mssqldata . Now i want to move the data file from d: to e:, say e:mssqldata . Can someone let me know if this is possible under SQL server v7.0 and if so, how
View 2 Replies
View Related
Nov 2, 2005
Hi,
i try to shrink and remove one datafile. But i always get following error:
Server: Msg 5042, Level 16, State 1, Line 1
The file 'M1Pdata15' cannot be removed because it is not empty.
***********************************************
use M1K
go
dbcc shrinkfile (M1Pdata15,emptyfile)
go
use master
go
Alter database M1K
remove file M1Pdata15
go
***********************************************
Microsoft SQL Server 2000 - 8.00.760 (Intel X86)
After Shrinking the file has 1.94 MB.
please help.
thx
DocHolliday1108
View 5 Replies
View Related
Nov 10, 2005
Hi,
In SQL Server 2000 how to move a secondary data file to different filegroup without removing and re-creating the secondary data file.
Thanks in advance.
-S
View 1 Replies
View Related
Aug 17, 2006
I obviously did not search the archives on the right terms so what isthe easiest and fastest way to move a 3G database from a nearly full Cdrive to the nearly empty D drive that should have been used.I could back it up, drop it, recreate it using the D drive, and restoreit but it seems like there should be a way to just move the datafileand use if from the new location.I am thinking that detatch/attach is the best method, but I would likeconfirmation or suggestions on how to proceed or things to be aware ofwhen using this method.-- Mark D Powell --
View 8 Replies
View Related
Jan 4, 2007
Hi there,
I have written a program that loads a package (SomePackage.dtsx) from the physical drive and executes that. The package does nothing but imports data from a csv file to the Sql server 2005. But I can see that the package is failing continuously. I meant the package.Execute() method is returning a DTSExecResult.Failure. I investigated the Package.Errors property that contains the error collection and found that there are two DTSError objects into the collection.
The first one€™s description says that
Cannot open the datafile "D:SOME.csv".
And the later one€™s is
component "SOURCE FLAT FILE COMPONENT" (1) failed the pre-execute phase and returned error code 0xC020200E.
But the most interesting thing is if I execute the package through the Execute package Utility (double clicking onto the SomePackage.dtsx file) ships with Sql server 2005 then it executes fine and works as expected. I have checked the permission of the csv file and it has everyone€™s full access.
Can anyone help me on this?
I will appreciate all kind of suggestions.
Thanks
Moim
View 14 Replies
View Related
Nov 28, 2005
Is it possible to convert a SQL2K datafile to SQL2K5? I have a 2K database that I need to easily convert to 2K5, I apprecaite any insight on this issue.
View 1 Replies
View Related
Dec 22, 1999
In order to perform an automatic way
to link a software using .txt database
to our SQL Database, I need some tips.....
I think about the following solution
- maybe using a "data-base extractor" (Access)
and convert the result into a .txt file,
Which can be automaticly refresh using a
.bat file which will open the .txt file created
3 times a day to refesh the data.
If you have some solution less complicated,
Please send it to
atoarlindo@yahoo.com.hk
Arlindo
View 1 Replies
View Related
Jul 16, 2001
Is it possible to move (detach and attach) the physical data and log file (.mdf &.ldf) to another location?
View 1 Replies
View Related
Apr 6, 2005
hi,
I'm trying to calculate how much unused space i have on one datafile. My main goal is to determine the max space i can save by doing a dbcc shrink.
Any help is greatly appreciated.
ThanKs in advance.
Al
View 1 Replies
View Related
Dec 8, 2004
Is there any limit to the maximum size of a datafile or transaction log you can have with SQL Server 2000 on Windows 2000. Also is there a maximum size that should be adhered to for performance and admin reasons ?.
View 4 Replies
View Related
Jan 30, 2004
Hi need lots of help here.
I have a dts package scheduled to run hourly as job since recent November. (Win2000 server, MSSQL2000 standard)
Its been running fine, except last few days, SQL Agent shows it attempted to run it but fails
Checking the history for the job i got this error message each time:
================================================== ======
DTSRun: Executing... DTSRun OnStart: DTSStep_DTSDataPumpTask_2
DTSRun OnError: DTSStep_DTSDataPumpTask_2,
Error = -2147467259 (80004005) Error string: Error creating datafile
mapping: The volume for a file has been externally altered so that the
opened file is no longer valid.
Error source: Microsoft Data Transformation Services Flat File Rowset
Provider Help file: DTSFFile.hlp Help context: 0 Error
Detail Records:
Error: 1006 (3EE); Provider Error: 1006 (3EE) Error string: Error
creating datafile mapping: The volume for a file has been externally altered
so that the opened file is no longer valid.
Error source: Microsoft Data Transformation Services Flat File Rowset
Provider Help file: DTSFFile.hlp Help context: 0 DTSRun
OnFinish: DTSStep_DTSDataPumpTask_2 DTSRun: Package execution complete.
Process Exit Code 1. The step failed.
=================================================
The key information i think would be these few lines:
Error creating datafile mapping: The volume for a file has been externally altered so that the opened file is no longer valid.
I don't think its SQL Agent as its scheduling is running other jobs fine(backup)
The DTS package runs fine manually.
Such i suspect dtsrun.exe itself. But where do i go from here?
Thks.
View 1 Replies
View Related