I have a package that contains three database tables (Header, detail and trailer record) each table is connected via a OLE DB source in SSIS. Each table varies in the amount of colums it holds and niether of the tables have the same field names. I need to transfer all data, from each table, in order, to a flat file destination.
View 6 RepliesHi,
I want to combine a series of outputs from tsql queries into a single flat file destination using SSIS.
Does anyone have any inkling into how I would do this.
I know that I can configure a flat file connection manager to accept the output from the first oledb source, but am having difficulty with subsequent queries.
e.g. output
personID, personForename, personSurname,
1, pf, langan
***Roles
roleID, roleName
1, developer
2, architect
3, business analyst
4, project manager
5, general manager
6, ceo
***joinPersonRoles
personID,roleID
1,1
1,2
1,3
1,4
1,5
1,6
My Requirement ,In Source Database 5 tables are there ( Emp,Loc,dept,Time,Product ), Destination is Single Excel file.But Dynamically how to load each table information to load into each sheet wise through SSIS Package?
View 3 Replies View RelatedI have a delimited text file with 650+ columns. The sum of the column lengths of a single row, if fully populated, exceeds 30K bytes. The "killer" fields lengthwise are the "Description" fields. If they were removed from the input file, the remainig columns would occupy about 5000 bytes, which is within SQL max row length.
Can SSIS be used to created these two tables? (one without description fields, the other with those field but arranged vertically in the table rows).
The fundamental issue is I can not import a single file row into a sql table because that row length could exceed the max byte count for a row.
I am trying to create a text file from multiple SQL Tables using @BCP_Command. I tried using DTS and SQL but the number of columns in the tables have to be the same size when doing a union. I also not to place a delimeter between each column. I've learned how to use BCP_commands on one file not sure if you can make it work with two or more.
Rich Pezick
Can someone show me the command string required to write data out to 2 or 3 tables at once? How about how to write 2 entries at once? Write now my solution writes 1 record to a table and then I write somewhere around 40-200 records to a second child . Write now I'm writing like this:Open --> Add 1 record in table #1 --> closeOpen --> Add 1 record in table #2 --> closeOpen --> Add 1 record in table #2 --> closeOpen --> Add 1 record in table #2 --> closeOpen --> Add 1 record in table #2 --> closeand I'm just wondering if there is a better approach, such as writing all the data at once. Something like:Open --> Add 1 record in table #1 --> Add 5 records in table #2 --> close
View 1 Replies View RelatedSo here's the issue
16 flat files all fixed width. Some over 350 columns.
Open flat file 1
extract id and go see if its in table 1, if true update table 1 with first 30 columns
otherwise insert into table 1 first 30 columns.
goto table 2, lookup id, insert/update next 30 columns...etc..etc..for 10 different tables
So I've got my flat file source, I do a derived column to convert the dates, i've got a lookup for table 1, then 2 ole db commands, 1 for update if lookup successful, 1 for insert if lookup fails.
How can I pass the id as a param into the update command so it updates where x = 'x'
also I need a pointer on doing the next lookup, eg table 2, would I do this as some sort of loop?.
If you can help great, but, please don't just reply with "I'd use this object"...then no explanation of how
v.v.frustrated newbie to SSIS
Probably a stupid and regularly asked question but I can't seem to find an answer, so here goes,
we have 16 .txt files, some with over 350 columns.
That info from each individual file needs importing to multiple sql tables.
need to look at sql table1 does record exist? if not create new then add in data once its been transformed eg datetime from yyyymmdd into datetime values [managed to get this using derived column] for first 20 columns, otherwise do update for the 20 columns...
then look at sql table2 and repeat for next n columns....
So I was wondering is it going to be better to write this as a dtsx package? if so can you point me to an example
or should I just write the code as part of a code behind page that scrapes the info and does a standard update/insert procedure?
Any help would be welcome.
thanks
We are trying to use SSIS Import export wizrd to import the flat files (CSV format) that we have into MS SQL Server 2005 database tables. We have huge number of CSV files. Is there a way by which we can import these flat (CSV) files in to corresponding SQL server tables in a single shot. I would really appreciate this help as it is painful to convert each and every file using the Import Export wizard.
Thank you very much,
Regards,
Srinivas Raveendra
I need to export data from multiple tables into one single file. The big problem here is that the tables will have different column types.
I am attempting to create something that allows users to be able to send me the contents of their tables's, through either email or ftp. I would prefer to make it easier for them so they only have to deal with one file, instead of the multiple files that bcp and dts create when exporting from multiple tables.
I was thinking of using DTS or BCP and then join (append) the files (either zip them or append the files together in some fashion), but I was hoping that there was an easier method out there.
Any ideas on how I may accomplish this would be greatly appreciated.
Andy
Hi
I have a data flow task where I have to write to a flat file. It works fine for me. But the thing is next timeI run the package it must write the data in the OLDEB source to a different copy. Usually the data is overwritten or appended to already existing data. What I want is everytime the package is run the data must be written to a different copy.
Thanks
Sai Abhiram Bandhakavi
I have a Foreach loop which scans a table, and gets names of a bunch of procedures, and then back in the foreach loop, they get executed. Im trying to figure out how I can create a sort of log file to say the name of the procedure that is getting executed currently and the current date time stamp onto a flat file. I havent been able to figure this out yet..anyone know how to do this? I grab the names of the storedprocedures from the table and store it in a variable and use the name from the variable to actually execute the stored procedure.
I guess in essence, the question is how do i directly write lines of 'text' (from say a variable) into a flat file.
I'm doing a test package which reads a flat file, makes an adjustment using the derived column task and writes to the same flat file. But, the read locks the flat file, so the write can't access it. Any ideas for a resolution?
Thanks,
Dave
Hi,
I am writing to a flat file. When the data is written to a flat file the columns have to be tilde seperated i.e ~.
What I am doing is I am taking a destintaion text file and having all the columns as tild seperated. Is there any way I can
avoid doing this. That is I should not mention couluns in the text file.
Lastly I want the columns to have a width that is fixed.
How I can do this.
Thanks
Sai
This a microsoft SQL 2000 server.
I have a DB with mutliple tables that have a column called "Date_stamp", which is used as a primary ID.
Here is my problem:
Some of tables have a bad datetime entry for the "Date_stamp". The bad entry is '2008-3-18". I need to delete this entry from every single table that has a name similary to 'Elect_Sub%Daily'.
I know how to get the user table names from the DB as follows:
SELECT name
FROM dbo.sysobjects
WHERE xtype = 'U' and name like 'Elect_Sub%Daily'
What I need to do is have a query that will basically scroll through the tables name produced by the above query and search and delete the entries that read '2008-3-18".
delete from tableName where Date_Stamp = '2008-3-18'
I'm writing to a flat file destination (CSV file) which contains 2 header rows, lets call it Col1 and Col2.
For some reason, the header rows seem to get duplicated in the output - i.e.
Col1,Col2
A,B
Col1,Col2
C,D
Is there any way to resolve this?
I don't want the file to be overwritten everytime since its used for record-keeping purposes.
Thanks
I'm unable to figure out how to write a column header to my flat file destination. My source is a OLE DB SQL query and I need the column names as a header row in my text file destination. This seems easy but the closet I can find is hardcoding the column header row in the header property. Is this the only option?
Thanks
Flat file is the source for to load the data into a table. I am using "Derived Column Component" for the data validation.
"Derived Column Component" Fails then i am writing/redirecting the records into the Flat File using "Flat File Destination" component.
It works fine except the following the issue.
Issue:
The derived columun value (that cause an error) is not get inserted into the Flat File
Scenario:
the data comes as "000000" and tring to convert to date format
(DT_DATE)("20" + RIGHT(Check_Date,2) + "/" + SUBSTRING(Check_Date,1,LEN(Check_Date) - 4) + "/" + SUBSTRING(Check_Date,LEN(Check_Date) - 3,2))
The above expression is working fine, except the data 000000 not passed into the Flat File Destination.
Pls advise. Thank you.
I have a set of parent/child records that need to be exported to a space delimited Flat File. Each parent record must be followed by 3 child records, each on their own line with different format.
I have a prototype using the Derived Column component that concatinates the various fields of each record into one "wide" text column. This fools SSIS to think that each row has the same format. Then I merge them together using an artificial sort id. But this seems overly tedious and very brittle.
What would be the best approach to writing these records out? I'm hoping there is a better more maintainable method.
Thanks,
Jon
Hi Folks,
I would like to write my table to a delimited file but I seem to have no choice but to use comma as the delimiter. Is there any way I can choose the delimiter ?
Thanks.
Sid
Hi,
I am trying to write to a fixed format flat file using Flat File Destination Data Flow Component. I have all required information gathered from more than one sources. But when I tried to format the columns to a big string that will make up one line in the flat file, I could not figure out how to do that. Couple of issues that I am facing are:
How to padd different columns? For example, One interger column has could be 1 to 10 character long in my case. When I convert to string, dont know how to padd the remaining characters i.e. if the value of integer is '1234', it should be written to file as '1234 ' . Which transformation is best in this case, if available?
How to convert T-SQL datetime to a specific date and time format to write in the flate file? I have to write these date formats depending upon one of the parameters passed.
Also, I dont want to put a delimiter at the end of each column, just the new line characters at the end of each record.
Some of the columns has some unwanted characters (like new line characters) how to find them and remove them from the string.
Can we directly write columns to a specific position in the flat file? e.g. col 1 a position 1 and col2 starts at postion 20 etc.
Your co-operation will be appreciated.
Thanks,
Paraclete
Hi everyone,
I am new to SSIS and I thought maybe someone would give me tips for solving the problem I am facing.
Overview:
I want to insert data contained in a flat file into several DB tables, which have N-M relations.
For illustration, I would explain the problem on a very simple DB:
1. The database contains the following 3 tables:
EMPLOYEE (EMP_ID, EMP_NAME)
PROJECT (PROJ_ID, PROJ_NAME)
EMP_PROJ (EMP_ID, PROJ_ID) , where EMP_ID and PROJ_ID are foreign keys referencing records in the EMPLOYEE and PROJECT tables respectively.
2. Each entry in the falt file contains the following data:
EMP_ID, EMP_NAME, PROJ_ID, PROJ_NAME
3. In SSIS, I have created a Data Flow Task containing:
- a path from a Falt File Source to an SQL Server Destination (Table: Employee)
- a path from a Falt File Source to an SQL Server Destination (Table: Project)
- a path from a Falt File Source to an SQL Server Destination (Table: Emp_proj)
Note: I used SQL Server Destination, because I need to import a huge amount of data and I read that this component performs better than the OLE DB Destination!
Questions:
1. I would like to eliminate EMP_ID and PROJ_ID from the Flat File Source. Instead, I would like these fields to be generated automatically upon insertion.
a. How can I do this and propagate the generated key among the different paths, which I have explained previously?
b. Can I first generate the two keys somehow then the parallel insertions into the different tables should start using the generated keys?
2. Is my solution correct in the first place? Or is there another better way for inserting data which belong to N-N relations?
Thanks in adavance,
Samar
I have a variable defined as "Country". Based on the value, the header row printed needs to be different.
I've already created a 'HeaderRow' variable that I'm able to set using a script task. But how can you set the Header text value at run time from the variable? There is no expression defined for the Header with the Flat File Destination object, and when I attempt to reference the HeaderRow variable as the Header text, the variable name is printed as the header.
Another approach I tried was to write the Header Row separately through another data flow task, but the issue here is: what is the input source when all you have is a Country variable?
We are using an ADO.NET provider in SSIS to read data from a SQL Server 2000 table that contains DateTime columns to write to a Flat File Destination. When the date values are written to the file they are formatted in TimeStamp to the 10th decimal position; e.g.��2006-04-24 12:00:00.123000000�?. Since SQL Server supports values to Timestamp(3), we need to truncate the last seven zeros to put the data in this format ��2006-04-24 12:00:00.123�? to keep the file as small as possible.
Since we have several hundred DateTime columns in scope for our requirements we are looking for the least logic/effort to accomplish this task. We can do this via Data Conversion and Derived Column transformations to cast the dates and strings but it is very labor intensive. It would be something like singing 99 bottles of beer on the wall eight times in a row with each verse taking 3 minutes each. Yikes.
We have tried casting the DateTime columns to varchar in the SELECT statement but receive this format ��Apr 24 2006 12:22PM�?.
Is there a configuration we've missed that forces timestamp(10) with non significant digits?
Hello all,
I was trying to run a test to write a ebcdic file out with a comp - 3 number (testing this for other people) and have run into a problem writing the string out to the flat file destination. I have the following script component:
Code Block
' Microsoft SQL Server Integration Services user script component
' This is your new script component in Microsoft Visual Basic .NET
' ScriptMain is the entrypoint class for script components
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
Public Class ScriptMain
Inherits UserComponent
Public Overrides Sub CreateNewOutputRows()
'
' Add rows by calling AddRow method on member variable called "Buffer"
' E.g., MyOutputBuffer.AddRow() if your output was named "My Output"
'
Output0Buffer.AddRow()
Dim myByteArray() As Byte = {&H12, &H34, &H56, &H7F}
Output0Buffer.myByteStream = myByteArray
Output0Buffer.myString = "ABCD"
Output0Buffer.myString2 = "B123"
myByteArray = Nothing
End Sub
End Class
I have added myByteStream as a DT_BYTES length 4, myString as (DT_STR, 4, 37) and myString2 as (DT_STR, 4, 37) to the output 0 buffer.
I then add a flat file destination with code set 37 (ebcdic us / canda) with the corresponding columns using fixed width.
When i place a dataviewer on the line between the two the output looks as I expect ("0x12 0x34 0x56 0x7F", "ABCD", "B123"). However, when it gets to the flat file destination it errors out with the following:
Code Block
[Flat File Destination [54]] Error: Data conversion failed. The data conversion for column "myByteStream" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
If i increase the size of the byte stream (say, to 50) the error goes away but I am left with the string "1234567F" instead of the appropriate hex values. Any clues on how to go about this? I obviously don't care if it gets transferred to "readable" text as this is supposed to be a binary stream, thus the no match in target page seems superfulous but is probably what is causing the problems.
NOTE: this is relating to the following thread (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2300539&SiteID=1) in that I am trying to determine why these people are not seeing the "UseBinaryFormat" when importing an EBCDIC file (i see this fine when i use an ftp'd file, but it auto converts to ascii) with comp-3 values. I also see the "UseBinaryFormat" when I am importing a regular EBCDIC file which I create that has no import errors with zoned decimals.
Hi! I have a general SQL CE v3.5 design question related to table/file layout. I have an system that has multiple tables that fall into categories of data access. The 3 categories of data access are:
1 is for configuration-related data. There is one application that will read/write to the data, and a second application that will read the data on startup.
1 is for high-performance temporal storage of data. The data objects are all the same type, but they are our own custom object and not just simple types.
1 is for logging where the data will be permanent - unless the configured size/recycling settings cause a resize or cleanup. There will be one application writing alot [potentially] of data depending on log settings, and another application searching/reading sections of data.
When working with data and designing the layout, I like to approach things from a data-centric mindset, because this seems to result in a better performing system. That said, I am thinking about using 3 individual SDF files for the above data access scenarios - as opposed to a single SDF with multiple tables. I'm thinking this would provide better performance in SQL CE because the query engine will not have alot of different types of queries going against the same database file. For instance, the temporal storage is basically reading/writing/deleting various amounts of data. And, this is different from the logging, where the log can grow pretty large - definitely bigger than the default 128 MB. So, it seems logical to manage them separately.
I would greatly appreciate any suggestions from the SQL CE experts with regard to my approach. If there are any tips/tricks with respect to different data access scenarios - taking into account performance, type of data access, etc. - I would love to take a look at that.
Thanks in advance for any help/suggestions,
Bob
I am getting the following error running a data flow that splits the input data into multiple streams and writes the results of each stream to the same destination table:
"This operation conflicts with another pending operation on this transaction. The operation failed."
The flow starts with a single source table with one row per student and multiple scores for that student. It does a few lookups and then splits the stream (using Multicast) in several layers, ultimately generating 25 destinations (one for each score to be recorded), all going to the same table (like a fact table). This all is running under a transaction at the package level, which is distributed to a separate machine.
Apparently, I cannot have all of these streams inserting data into the same table at one time. I don't understand why not. In an OLTP system, many transactions are inserting records into the same table at once. Why can't I do that within the same transaction?
I suppose I can use a UnionAll to join them back together before writing to a single destination, but that seems like an unnecessary waste and clutters the flow. Can anyone offer a different solution or a reason why this fails in the first place?
Thanks in advance.
I have a couple of hundred flat files to import into database tables using SSIS.
The files can be divided into groups by the format they use. I understand that I could import each group of files that have a common format at the same time using a Foreach Loop Container.
However, the example for the Foreach Loop Container has multiple files all being imported into the same database table. In my case, each file needs to be imported into a different database table.
Is it possible to import each set of files with the same format into different tables in a simple loop? I can't see a way to make a Data Flow Destination item accept its table name dynamically, which seems to prevent me doing this.
I suppose I could make a different Data Flow Destination item for each file, in the Data Flow. Would that be a reasonable solution, or is there a simpler solution, or should I just resign myself to making a separate Data Flow for every single file?
I used the data export wizard to export a single table to a single flat file (multiple wasn't allowed). I saved the package as a *.dtsx file which I'm attempting to edit to add the additional tables.
Creating additional sources is fairly easy copy of the first source and change to the table name.
I've tried copying the destination connection and changing to a new text file, but can't get past having to add each column manually to the new destination.
How can I duplicate the mapping that must be taking place in the wizard in the *.dtsx editing environment?
This seems like a simple / common task, but I've been unable to find a solution.
Thanks, Richard
HiI am trying to write two Select * statements to the same text fileusing bcp (from a stored procedure).But cannot find a way of appending to a file using bcp.Does anyone know if this is possible or is there another way of writingmultiple queries to a file from a stored procedure?ThanksCaro
View 2 Replies View RelatedHow to insert single row/multiple rows into multiple tables by using single insert statement.
View 1 Replies View RelatedHi,
I have searched but not found quite the best way to look at this so far..
I have an application that outputs data to several text files (up to 30). These have commonality by an object name, but then contain completely different column data.
In DTS I had each of the source text file connections going to one OLE DB connection and then individual transform data tasks pointing to the one OLE DB connection.
Looking at SSIS, it would appear that I would need to have one source and one destination for each of these and therefore 30 parallel data flows?
Just wondering if there is a neater way of doing this??
It is a regular data import that happens a few times a day - the text files are named the same as the SQL tables - ie app_userdata.txt goes to app_userdata table.
Hope that explains ok and thanks in advance.
Mike
I have a text file which contains the data that has to be inserted into multiple tables.The columnames of table 1 form the H1 follwed by Details D1,D1,D1...
The column names of table two form the H2 followed by details D2,D2,D2 so on and similarly for Table 3.
Am using a link server to the file directory and schema.ini which defines the column names fofr the text file
Is there any way of defining column names for more than one table through the schema.ini? or is there any other way through I can parse the text file contents to multiple tables?
Sample text file:
H1,JobDate,JobNumber,FileName,
D1,13/02/2008,asdf123,text1.txt
D1,13/02/2008,asdf123,text2.txt
D1,13/02/2008,asdf123,text3.txt
H2,PagesUsed,PagesPrinted,Pages emailed
D2,10,10,9
D2,1000,100,99
D2,50,22,93
Schema.Ini - defined for the first table
[LogConfig.txt]
Format=CSVDelimited
CharacterSet=ANSI
ColNameHeaders=true
Col1=JobDateText Width 20
Col2=JobNumberText Width 20
Col3=FileName Text Width 100
Hoe do i define the column names for the second table. All these contents are in a single text file and need to be parsed only thru sql.
Any help/suggestions are welcome..
Thanks a lot for taking time to read this.