Ok everybody. I am new to sql. I have ms sql staging database that pulls data from mysql database. Then once a day I run a ssis package that moves the data to a live database and also creates a flat file that is posted to a ftp site then truncates the table. One problem I am running into is if the mssql staging database has no records the flat file is still created. How do I stop it?
View 10 RepliesTrying to figure out the best method of reading in a number of flat files, all with different number of columns and data types and outputting them to a database.
Here's the problem: They are EBCDIC encoded and some of the columns are packed decimal. I've set up one package that takes the flat file, unpacks the decimal (Using UnpackDecimal component) and then sending the rest through a second component to go from EBCDIC -> ASCII.
What I need is a way to do this for every flat file based on the schema for that flat file. One current solution is to write a script/app to create the .dtsx XML file and then execute that for each flat file. It appears like this may be possible, but I haven't gotten far enough to know for sure. So my questions are this:
1) Is there an easier way to do this (ie somehow feed the schema to the package and use it to dynamically set up the column makers and determine which columns get fed to the unpack decimal component.
2) If there isn't a better way, will dynamically creating the .dtsx XML file based on the necessary input/output columns for each flat file work? If so, what is a good source of information on this (information about how the .dtsx XML file is set up, what needs to be changed/what doesn't, etc).
Thanks,
Travis
Hello!
I want to make a very simple package: Export all rows in a table to a flat file.
This package I can create pretty much by only using the wizards.
Now to my problems:
1) I need the output to have this format:
H20070920161522
DS3 Plastpall trippelkrage 40 1
E00000000003
H is a header post, in this case with date and time following.
D is a details post, that is all the rows that was exported.
E is and end post, containing only the number of rows in the file, including H and E posts.
2) I need to set the file name dynamically, preferably using date and time to name the file.
I�ve done this very same thing in T-SQL, like so:
Code Snippet
USE AVK
GO
SET TRANSACTION ISOLATION LEVEL SNAPSHOT;
GO
SELECT *
FROM tempProducts
GO
CREATE VIEW EXPORT_ORDERS
AS
SELECT 1 AS ROW_ORDER, 'H' + REPLACE(CONVERT(char(8), GETDATE(), 112) + CONVERT(char(8), GETDATE(), 108), ':', '') AS Data_Line
UNION ALL
SELECT 2 AS ROW_ORDER, 'D' + COALESCE (CONVERT(char(10), LBTyp), '') + COALESCE (CONVERT(char(50), Description), '') + COALESCE (CONVERT(char(5),
Volume), '') AS Data_Line
FROM dbo.tempProducts
UNION ALL
SELECT 3 AS ROW_ORDER, 'E' + RIGHT('0000000000' + RTRIM(CONVERT(char(13), COUNT(*) + 2)), 11) AS Data_Line
FROM dbo.tempProducts AS tempProducts_1
GO
IF @@ROWCOUNT > 0
BEGIN
BEGIN TRANSACTION
SELECT *
FROM tempProducts
DECLARE @date char(8)
DECLARE @time char(8)
DECLARE @sql VARCHAR(150)
SELECT @date = CONVERT(char(8), getdate(),112)
SELECT @time = CONVERT(char(8), getdate(),108)
SELECT @time = REPLACE(@time,':','')
DECLARE @dt char(14)
SELECT @dt = @date + '_' + @time
SELECT @sql = 'bcp "SELECT Data_Line FROM avk..EXPORT_ORDERS ORDER BY ROW_ORDER" queryout "c:AVK_' + @dt + '.txt" -c -t -U sa -P dalla'
EXEC master..xp_cmdshell @sql
--WAITFOR DELAY '0:00:10';
DELETE
FROM tempProducts
COMMIT TRANSACTION
END
DROP VIEW EXPORT_ORDERS
GO
But I�m sure it can be done in SSIS aswell, giving me some nice options for i.e. error handling aswell.
Pointers please
Hi all,
I am passing flat file source as a variable to Dtexec Utility. (like package.variables[User::varFileName].Value;"D:sourcedata.txt).
Destination table is having one more column.
I want to add custom value in that column at run time by parameter to Dtexec(User::varDate)
I dont know how to do it, please help me.
Madhukar
I have three tables in data base:
customer
product
sales
And i want to use SSIS package dynamically load data from database into three separate flat file, each table into each file.
I know i got to use for each loop task ADO.Net schema row set enumerator, with OLEDB connection manager, select table name or view name variable from access mode list, but the problem comes, as table name is dynamic then flat file connection is also dynamic, i am using visual studio 2013...
I have requirement like to develop dynamic package for inserting data from flat file to table.
Find below points for more clarification :--
1) if I changed the flat file values and name in source variable AND the table name should be also changed based on variable value .
2) it should dynamically mapped with column values with source file as we have to insert data in target table.
See below diagram for more clarification.
Hello Experts,
I am createing one task (user control) in SSIS. I have property grid in my GUI and 2 buttons (OK & Cancle).
PropertyGrid has Properties like SourceConnection, OutputConnection etc....right now I am able to populate Connections in list box next to Source and Output Property.
Now my question to you guys is depending on Source Connection it should read that text file associated with connection manager. After validation it should pick header (first line of text file bases on record type) and write it into new file when task is executed. I have following code for your reference. Please let me know I am going in right direction or not..
What should go here ?
->Under Class A
public override DTSExecResult Execute(Connections connections, VariableDispenser variableDispenser, IDTSComponentEvents componentEvents, IDTSLogging log, object transaction)
{
//Some code to read file and write it into new file
return DTSExecResult.Success;
}
public const string Property_Task = "CustomErrorControl";
public const string Property_SourceConnection = "SourceConnection";
public void LoadFromXML(XmlElement node, IDTSInfoEvents infoEvents)
{
if (node.Name != Property_Task)
{
throw new Exception(String.Format("Invalid task element '{0}' in LoadFromXML.", node.Name));
}
else
{
try
{
_sourceConnectionId = node.Attributes.GetNamedItem(Property_SourceConnection).Value;
}
catch (Exception ex)
{
infoEvents.FireError(0, "LoadFromXML", ex.Message, "", 0);
}
}
}
public void SaveToXML(XmlDocument doc, IDTSInfoEvents infoEvents)
{
try
{
// // Create Task Element
XmlElement taskElement = doc.CreateElement("", Property_Task, "");
doc.AppendChild(taskElement);
// // Save source FileConnection
XmlAttribute sourcefileAttribute = doc.CreateAttribute(Property_SourceConnection);
sourcefileAttribute.Value = _sourceConnectionId;
taskElement.Attributes.Append(sourcefileAttribute);
}
catch (Exception ex)
{
infoEvents.FireError(0, "SaveXML", ex.Message, "", 0);
}
}
In UI Class there is OK Click event.
private void btnOK_Click(object sender, EventArgs e)
{
try
{
_taskHost.Properties[CustomErrorControl.Property_SourceConnection].SetValue(_taskHost, propertyGrid1.Text);
btnOK.DialogResult = DialogResult.OK;
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
#endregion
}
Hi all,
In a foreachloop, I am inserting records into a flat file which is working fine. But the thing is that as the file grows, it takes longer for it to locate the EOF(End of File) of the flat file so as to insert the records.
I have around 70-100 lines written to the file at each loop and there are more than 20k records to be looped. wihich means that at the end I should be having 1400k - 20000k line in the text file.
One solution would be to insert the records at the start of the file itself so that it does not has to lookup the EOF each time before writting.
Another would be to generate separate files and then merge it.
Any idea how can this can be done?
Beside this I have to zip the file and then SFTP to a given address.
Any suggestion or help would be welcome.
Rdgs
David
During my development of a ssis package i've noticed that when creating two control flows that pulling data from seperate tables, each going to its own flat file, that the second keeps the attributes of the column names from the first. So when I create my second flat file, not only does it have the names of its correct columns but has the name of the the first flat file.
I'm hoping that I've explained the correctly. I'll provide more info "OR" I can provide the code to package if anyone would like.
I have a few flat files that will be retrieved from some SFTP server. One of the flat file will act as a terminal file where it will specify the total number of records expected in each other the flat file.
Data in the terminal.txt
FileName TotalRecords
File1 1000
File2 1500
File3 2000
So, before transforming the data from the flat file sources into the target destination, i wish to do a row count checking for each of the flat file source to make sure that the number of records in the flat file source is tally with the number of records specify in the terminal.txt file. I'm able to get the number of records in each of the flat file by using the RowCount component but don't know how to get the data out from the terminal.txt file in order to make a rowcount comparison.
Can any1 help me on this? Or is there any other way we can do to make sure that the flat file source is alright before proceeding with the data transformation task?
Thanks!
I have a package set up basically with two consecutive data flows. The first flow takes data from an OLE DB Source and stores it into a Flat File Destination. The second flow uses this same flat file as a source, alters the data, and stores the data in the same flat file, overwriting the old file. I set DelayValidation to True on the flat file. Still, here are the error messages I am receiving:
Error: 0xC020200E at DO, Flat File Destination [7676]: Cannot open the datafile "C:Temp.txt".
Error: 0xC004701A at DO, DTS.Pipeline: component "Flat File Destination" (7676) failed the pre-execute phase and returned error code 0xC020200E.
I am new to SSIS, so I'm sure I have a setting wrong or something. Is the problem that SSIS is trying to write to a file from which it is simultaneously reading data?
Thank you.
Hi everyone,
There is a small problem encountered while creating a package in sql
server 2005.
Actually i am using a flat file which has 820 rows and 2 columns which
are seperated by line feed(for ROW) and tab(for COLUMN).after
importing i found that ther are only 800 rows imported into the table.
Ather verifying the input file i found out that there are some null
values in the second column so there is no line feed for those
values.
Can anyone please help me how to give multiple delimiters for the same
input flat file.
I am trying to import a flat file into SQL Server 2005 using SSIS. I have never used it before and I am getting confused by the error I am receiving.
I have a link to a flat file, that gets sent through a Derived Column flow where dates in YYYYMMDD are changed to MM/DD/YYYY format. Then the string MM/DD/YYYY is converted to a date in a Data Conversion flow. And finally the data is put into a SQL Server table (currently with no rows).
The problem I am having is with a text field with the email address in it. The error I am getting is:
[Import Allstate Auto Club [1]] Error: Data conversion failed. The data conversion for column "email_source" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
The problem is I can't see where in the flow the problem is. The field length is 20 wherever I look and the codepage is 1252 wherever I look. Does anyone have an insight? Keep in mind, I have never used SSIS before and I consider myself an amateur with SQL Server. It could easily be a data type conflict or something easy. Any help will be appreciated.
Hi,
i need to import a fixed with text file into several tables in SQL 2005.
The file contains records that should go inte different tables and i would like to know the order of the design..
I also get some errors when trying to convert the text to unicode, even with the derviced column data flow task..?
flat file source -> Data conversation/Derived column -> aggregate?
I have not found the way of importing the file into several different tables in the DB.
Thanks for suggestions..
Regards,
Daniel
I want to use SSIS to export to a flat file, for various reasons.
However, my flat file has padded out each column to match the number of characters in the DBase column.
See below for example. The first column is char(3), the second is char(9), the third is char(9) etc
How to I get rid off the excess spaces.
What I get
2*852240 *5006 *MPH00095-02 *200709241200*200709241230
2*692677 *5002 *MPH00180-03 *200701181200*200709241230
What I want2*852240*5006*MPH00095-02*200709241200*200709241230
2*692677*5002*MPH00180-03*200701181200*200709241230
We have begun useing SSIS to export data into a Data warehouse.
For continuity of service and testing perposes I wish to export to flat files.
However, although the export seems to work fine, I get alot od spaces in my text file.
It seems to pad out to the exact number of characters in the data base, i.e. Char(3) outputs 1 character plus an extra 2 spaces, char(9) gives me the six characters + 3 spaces.
I cannot change the DBase.
How do I get rid of the extra spaces?
Requirement.
2*852240*5006*MPH00095-02*200709241200*200709241230
2*692677*5002*MPH00180-03*200701181200*200709241230
What I get
2*852240 *5006 *MPH00095-02 *200709241200*200709241230
2*692677 *5002 *MPH00180-03 *200701181200*200709241230
I have a fixed width flat file I'm trying to insert into an SQL 2005 table using SSIS -- it's a recurring task. One of the columns in the flat file has to go to a column of type Numeric. No matter what I try : a data conversion, defining the field as DT_NUMERIC in the connection,... I always get "The conversion returned status value 2 and status text :The value could not be converted because of a potential loss of data". It is driving me bonkers, up to the point that I find myself wishing for the 'good old' DTS days of SQL 2000. And I dread to think what will happen when I try to port some serious, much more complex DTS packages on my SQL 2000 to SQL 2005.
The data in question represents longitudes and latitudes so quite often there is a leading white space in the data : ex. : " 95.15". Surely that cannot be the cause ?
I've spent hours doing the RTFM-thing and searching the newsgroups, fora...you name it. Apart from ending up running in circles in the MS documentation, the only thing I've really learned so far is that I'm aparently not the only one driven to dispair by the new SSIS thing.
I can think of a number of ways to hack my way around this thing, but that's not the kind of 'progress' I had in mind when I started the move to SQL 2005.
Intelligent suggestions would be most welcome.
Hi All,
I have a requirement wherein I have to read the data from a "|" separated flat file. Each row is specified this way...
<row>Data1|Data2|Data3</row>
<row>Data1|Data2|Data3</row>
<row>Data1|Data2|Data3</row>
<row>Data1|Data2|Data3</row>
.
.
.
.
I am specifying the row delimiter as : </row>{CR}{LF}<row>
When I create Flat File Connection and when I see the preview of columns...in the first row "<row>" remains and in the last row "</row>" remains...
I am trying hard to sort out and remove these extra string but unable to do so...
Please let me know how to approach this? How should I specify the Row Delimiter?
Thanks,
Sapan
Hello Guys,
I am working in one company and currently I am assigned to new project for Data Migration from company X to our company Y using SSIS. I am totally new and i just completed 5 tutorial which was gien on MSDN website.
Basically client is going to send us first flat file with 1 million records with Header, Detail and Trailer records.
I want to create a Package in such a way that it dumps all this first load into 7 to 8 different tables at a time.
we also have to include functionlity for validation and error check.
On successfull load error file should only return Header and Trailer but no detail records.
If there are any errors then error file should contain Header, Detail records which were unable to load plus trailer which we have to sent back to client.
When 2nd file comes that time we have to check whether this is new records or change (update) one depending on Flag which tells it.
This is basically high level idea of my Package what i need to create. If u guys have any question then let me know.
I know you guys are very experienced one. Anyone of you please give me some detail idea on it I would really appricate it.
I have very limited time line for it.
Thanks
Shah
I am having frustration with issues I'm seeing within Development Studio. Here's the problem:
A fixed-width flat file is to be imported into SQL 2005. I am running from within BIDS on my workstation and send the data remotely to the server. It has over 200 fields, and I am attempting to import them into dimension tables, and one fact table. I had to manually create the columns on the flat file connection manager for this process, since the data file is delimited.
The SSIS solution has one project, and several independent data flow tasks within it. After I set up the fcm, I mapped fields using a flat file source, hooked up a derived column and sort transform. The idea was to use a 3rd party tool, TableDifference, to implement a Type 2 dimension. I was able to successfully build three data flow tasks that ran correctly. Later, as was building new data flow tasks, our project people decided to change the data types of several of the fields from Datetime to character, and also rename another 40+ fields. I dutifully kept my fcm synchronized, since column mappings are automatic when the names match; data type changes were also unavoidable on some of the columns.
The problem was noticed after I changed some of the columns in the fcm. My flat file source adapters on all the tasks needed to resolve changed metadata so I had to 'edit' and save them again. The derived column, sort, and downstream transforms also had to be updated. I saw strange behavior on previously tested data flow tasks. Some of them gave me errors even connecting to their sources. Other times I saw erroneous paths being taken which led to data being changed incorrectly on the dimension tables (new records being added even if there was an existing match, records being marked as expired incorrectly, etc.).
Are there known issues with editing a shared file connection manager after it has already been referred to within data flow tasks? It appears that the only way to avoid the problems I've seen is to either:
1. Create separate fcm's for each data flow task. Use 'filler' fields for the ranges of all the columns in between each referenced column. This way if you have to change anything with it, the only rework involved is limited to the data flow task using it.
2. Create one fcm for the entire package and NEVER change anything with it. If a column name is wrong, use a derived column to rename it in the data flow. If a data type changes, try to use a data conversion transform, but be sure to rename its output column so that it is not called 'data conversion.<field name>'. I think that's done in the Advanced Editor for the sort task.
My experience using BIDS has been hot and cold. It's nice to be able to quickly build the tasks you need, but it's as if the metadata (pipeline) is much too sensitive to change and requires a lot of touching up, either upstream at the source adapter level, or across data flow tasks when you make changes to the FCM. Maybe the problem is just limited to flat files...
Are there any tips or lessons learned from anyone on this? I've got another developer willing to create the FCM for other data feeds so I can copy/paste them into other projects. It would just be nice to know there was a bug fix or a good method to avoid all this 'rework' when you have to make changes to the FCM after the fact.
Thanks in advance.
Hi,
I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.
Did someone else faced this issue?
Thanks,
AQ
I have a simple SSIS package -> It reads a local text file which has 10 rows of data ( id, name, telephone # ) and puts it into a table.
It uses the "SSIS Flat File source" to read and a "SQL Command" to insert into the table. I can see that it reads line by line and puts each line into one row in my table.
Now, my production data is over 5 GiG of mainframe data and it seems their data is arranged in some hierarchical form.. so the position or arrangement of data in that file is important.
I pulled the data using my package and as far as I can see , my SSIS package pulled one line at a time ( from the flat file) and pushed it into my table. For each row, I also created an identity column in my table to be able to identify the positional arrangement of the hierarchical data and then use relational mappings to suit our business needs.
In all of this, my assumption is -
"SSIS reads one line at a time, inserts to my table and goes down to the next line .
It does NOT read a snapshot of rows from the flat file so as to write them into the table using internal ordering methods based on that particular snapshot "
My question is .. is my assumption correct ?
hi -
I am totally new to SSIS etc and SQL 2005.
I have a dts task to recreate in SSIS. I have done most of them and muddled my way through, but this basic problem has got me stuck.
When mapping columns from my file to my ole db output table, I want to map one input column onto two output columns, but it will only seem to let me select one destination column for each input?
I have tried shift/alt/ctrl etc to try to get it to map to both columns but it wont have it.
How do I do it?
Also, somehow my Dataflow Sources tab has gone from the toolbox, and I can't seem to get it back any way - I switched on everything I could see and all components etc, but it is not in there as an option. How do I get it back in the toolbox?
New to SSIS and dts. Stumbling along on this one, really looking for resources and help.
I have a flat file, i defined through connection manager
and (for now) a fixed destination excel file I defined in connection manager.
My dataflow, is pretty simple, mapping two fields to each other an amount field and a phone field in a flat file source and excel destinatinon.
the amount column is formated as a number in the excel, and a currency in the connection and both input output properies.
A few questions,
1. why do cells on the excel show up with that green wedge on the upper left? appears to be a formating issue.
2. in the flat file, my amount field does not have the decimal, what would be the best way to apply that? it's should be implied.
3.Everytime I test the SSIS package, it keeps appending to the excel (it actually does not even work right on the second run). What's the best way to have it write to a fresh file? have an ssis script task copy the file from an empty template?
4. Id like to remove the last row? what's the best way to do that?
Thanks for any help or information!
Hi All,
I want to know is it possible to have source as Flat File and destination as XML
Thanks in advance,
Shagun
Firstly, I hope this question isn't asked too frequently but I found no existing reference to this situation....
I had a bunch of stored procedures in SQL 2k which imported and exported data to and from flat files using TEXTPTR, READTEXT, UPDATETEXT etc... The flat files were continuously changing so the filepath was a parameter for the sp.
The reason I used the pointer to flat files is because I didn't want to
load the files in memory before commiting them ie. with TEXTPTR and
UPDATETEXT I can import a 1Gb binary file 80000 bytes at a time and
keep (precious) memory usage down.
I was accessing this procs from a C# application.
Since these methods are going to be phased out by the guys at MS what is the best way of importing/exporting very large binary files in SQL 2005?
As far as I can tell SSIS requires a Flat File Source Manager object which needs a static filepath - not good.
Hope you can help,
Paul
Hi
I am trying to use a conditional split task so that I can check for specific fields. If the value doesn't exist I am piping the records to a derived field task, where I add an error. I then try to send these records to a flat file destination so that I can keep track of them. However, when I execute the SSIS data flow task I get the following error
[Log Invalid Records [5496]] Warning: The process cannot access the file because it is being used by another process.
This file isn't being used by any other process as far as I can tell, and the only process using it is the SSIS task trying to write to it.
If anyone has any ideas, then I would really really appreciate it
Thanks
Darrell
Hi,
Here is my problem :
I work on a SSIS package with SQL SERVER 2005
I need to extract data from a table and put these data in csv files
But... the flat files name should be dynamic and assigned by a variable ...
Here's an example of my table :
Column header :
Id, Name, Number
1 TOM 22
2 TOTO 44
3 SAM 44
4 RADIO 55
I expect to have 3 csv files :
USER_22.csv
USER_44.csv
USER_55.csv
for example : USER_44.csv contains :
2;TOTO;44
3;SAM;44
if there's 50 different number, i expect 50 files
can i do that in a dataflow ?
thanks for answering
Hi,
We have a csv file which contains a date field. The data in the field contains "0" as well as "dd/mm/yyyy". Is it possible to update all "0" to "01/01/1900" on import using SSIS.
Basically when we import the flat file now it falls over due to the destination table data type being datetime.
If this is not clear please let me know and i'll try and explain more?
Thanks for any help.
Slash.
Hello,
I have a DTS package that copies data from a fixed width file and inserts it into a SQL Server 2000 table. These files are generated daily, and usually contain about 200k records.
Sometimes, the format of the file between two days can differ slightly. I can't tell if the problem lies with the length of the record string, or the string terminator. The record string is supposed to begin with 'D1'. One file (as viewed in Notepad) will contain the records each in its own row, and each row will start with 'D1'. Another file will show that the first record starts with 'D1', but then the next record does not begin on a new line. Instead the next record continues at the end of the previous record. In this case, 'D1' is preceded with an unrecognized character as if it to indicate a carriage return.
Anyway, in SSIS, when I configure the row width of the fixed width file in the flat file connection manager to 386, and the beginning of all the records are all on their own row, the data processes without a problem. When the records do not all begin on their own row, the data will process fine if I change the row width to 385.
I don't have to compensate for this in DTS. Between SSIS and DTS, both flat file connections are configured with a row delimiter of LF, and no text qualifier. Why will SSIS and DTS process these files differently?
Thank you for your help!
cdun2
I want to extract data from a table (based on a query) to a flat file.
So I have an OLE DB Source (data access mode SQL command) and then a flat file destination. The extract works finem except it extracts all table rows as one line in the file - whereas i want a separate line per DB record - what am I missing?
Thanks for any help!
Hi,
I have a situation where a tab limited text file is used to populate a sql server table.
The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?
I am transferring data from an OLEDB source to a Flat File Destination and I want the column width for all of the output columns to 30 (max width amongst the columns selected), but that is not refected in the Fixed Width Flat File that got created. The outputcolumnwidth seems to be the same as the inputcolumnwidth. Is there any other setting that I am possibly missing or is this a possible defect?
Any inputs will be appreciated.
M.Shah