Dynamic Source And Dynamic Destination
Apr 15, 2008
I have a requirment which i have partly accomplished , but could not get through completely
i have a file which comes in a standard format ending with date and seq number ,
suppose , the file name is abc_yyyymmdd_01 , for first copy , if it is copied more then once the sequence number changes to 02 and 03 and keep going on .
then i need to transform those in to new file comma delimited destination file with a name abc_yyyymmdd,txt and others counting file counting record abc_count_yyyymmdd.txt. and move it to a designated folder. and the source file is then moved to archived folder
what i have taken apprach is
script task select source file --------------------> data flow task------------------------------------------> script task to destination file
dataflow task -------------------------> does count and copy in delimited format
what is happening here is i can accomlish a regular source file convert it to delimited destination file --------> and move it to destination folder with script task .
but cannot work the dynamic pick of a source file.
please advise with your comments or solution you have
View 14 Replies
ADVERTISEMENT
Mar 27, 2007
Hi,
I am building SSIS for 3 different files that have identical
schema and mapping logic.
In my OLE DB Source (object name - "OLEDBSource_SourceTable")
Data Access mode is "Variable name".
As soon as I swithced to this Data Acces mode
it started to give me an error:
[OLEDBSource_SourceTable [1]] Warning: The external metadata column collection is out of synchronization
with the data source columns.
The column "DEAL_NUM" needs to be updated in the external metadata column collection.
The "external metadata column "DEAL_NUM_Flag" (34529)" needs to be removed
from the external metadata column collection.
The "external metadata column "recordID" (33740)"
needs to be removed from the external metadata column collection.
View 22 Replies
View Related
Aug 8, 2007
Hello,
What I'm trying to accomplish is to have a variable names "SourceTable" and "DestinationTable". So for each SourceTable, the DestinationTable will have the same columns. All I need is to auto-map these columns between source and destination via code?
Is this possible?
Thanks,
awiora
View 3 Replies
View Related
Jun 1, 2015
I have a requirement to take xml file, in case the number of column changes, it should not fail the package, rather it should load the data in destination table. Destination table could be altered separately depending on xml schema by the DB team in production.
View 3 Replies
View Related
Jan 29, 2004
Hello everybody .
I am building DTS transfer data from
SQL server into Excel file
source query constant ,but destination will be supplied by parameter
At design time I created destination
excel file and saved a copy of it like
C: empl_excel.xls
presently dts work in following order
1. set datasource of destination
from global varaibale(@@X)
2. execute xp_cmdshell to copy
C: empl_execel.xls to file in @@X
3.Run transformation
How to eliminate step 2 ?
If I run step 1 and 3 ,I get error "table does not exist"
How dynamicly create table in excel and map columns for transfer
Thank you
alex
View 3 Replies
View Related
Feb 8, 2007
How to create a new table dynamically in OLE DB destination.
This is what i am doing
I am reading multiple flat files in loop and saving file name to a variable. Then i have a source script component which read and transforms data .Now how can I push the data to SQL table. I want to create a new table with name saved in a variable. I tried using OLE DB destination and assigning table name from variable. Does'nt work.
Thanks in advance for any insight on how to make this work.
-Amar
View 15 Replies
View Related
Dec 1, 2007
Hi
In SSIS i have an execute SQL task where i get the job_id as the output which i set it to a package variable.
Now what I need to do is i have to map this value to one of the columns of the OLEDB destination.
How can I do it? Please help me ASAP.
Thanks
Sai
View 3 Replies
View Related
Dec 23, 2005
Hi,
Is there anyway to dynamically change the Sql Server Destination? Let's imagine i want the data to go to a table based on a specific parameter like year for example...
Imagine tables like table1_2005, table1_2006 and i want to dinamically change the SQL Server Destination so it would insert 2005 or 2006 and so on....
I can't seem to find that option... It was really a good idea for microsoft to implement Expressions in such components... Dynamics it's all about it :)
Best Regards,
View 8 Replies
View Related
Feb 5, 2006
Hello,
I am a beginner for the SSIS and would like to know how to modify the OLDEB Destination connectionString property at run time like using "for each loop container".
My requirement is that I have a single source which would be Sql Server 2005 and my destination is in MS-Access database residing in 100 places. I do not want to manually design in the data flow to these 100 destinations.
I have all the destinations stored in a table and would like to pick these destinations from the table and loop through the same at run time by modifying destination connection string.
I have planned using dts but the for each loop container does not work through as it works with flat file connection manager , but does not go well with OLDEB connection.
Highly appreciate any help in this regard.
Regards
Sameer
View 10 Replies
View Related
May 4, 2007
Hi,
I am trying to access from OLE DB source. And based on one of the columns, I need to write the data to a Flat File Destination.
For Example,
CustID, ProductID, Product Name, Product Description
Say I am going to write to a different Flat File for every product. So if there are 10 products in the data. There should be 10 Flat Files. Also the file name should include the Product Name And Product ID.
It is being done in a single Data Flow Task.
Right now the Property Expression for the File Name is which is not working)
Code Snippet@DestFolder + [Data Conversion].ProductID + @TodaysDate + ".txt"
The ProductIDs are in the ascending order. Any help or guidance?
Thanks
-Leo
View 7 Replies
View Related
Apr 17, 2008
Anyone knows how to give the name of a flat file specifying today's week number??
filename_ww
for today it should be filename_16 (since week number is 16)
View 1 Replies
View Related
Feb 4, 2008
I need to take 5 or 6 select statements to excel. Here are my limitations...
- each of the queries (thankfully) have the same data format
- each of the queries could return more than 65k, so a new worksheet needs to be generated dynamically.
- the names of the excel worksheets need to be custom, but a naming scheme would have to be developed for queries that ran over into multiple worksheets.
What's the smartest way to do this?
I'm having a hard time getting my head around this. I would love any help... I know I'm not breaking any new ground here. I've found pieces of what I'm doing on lots of forums, but never the exact thing. The complexities compound quickly when dealing with dynamic excel worksheets. =)
View 3 Replies
View Related
Jul 10, 2006
I have a database app, and we're implementing various data export features using SSIS.
Basically, it's a couple of straight extracts of various recordsets/views, etc. to CSV (flat files) from our SQL Server 2005 database, so I'm creating an SSIS package for each of these datasets.
So far, so good, but my problem comes here: My requirements call for users to select from a list of available columns the fields that they want to include in their exported file. Then, the package should run, but only output the columns specified by the user.
Does anyone have any idea as to the best way to accomplish this? To recap, at design time, I know which columns the users will have to choose from, but at run time, they will specify the columns to export to the flat file.
Any help or guidance here is greatly appreciated
View 7 Replies
View Related
Apr 26, 2007
Hi,
I am using VS.net 2003 as a front end and SQL server 2005 backend.
i am creating SSIS packages for Datatransformation programically in .NET.
but the package created is compatible to the previous version of SQL server ie SQL server 2000.
So i need to migrate it in SSIS package compatible to SQL server 2005.
it is migrate also using Data Transformation migration wizard.
But i want to migrate my DTS package programically or by using stored procedure.
Is there any stored procedure or any code is there from which i can migrate DTS into SSIS ?
Thank you
View 9 Replies
View Related
Jan 4, 2006
Here's what I want to do -
Dynamically load a flat file from a dynamic source table-
The source table metadata is known via the SYSOBJECTS and SYSCOLUMNS tables- I can pull the column names, type and lengths from these tables based on the table name. (my goal is pretty simple- pull data from a table in our database, and save it down to a flat file)
Would this be enough to dynamically create the destination flat file? If so, how do I do it?
Thanks
-rob
View 3 Replies
View Related
Jul 10, 2007
I created a data flow with complaicated SQL. There is "type" field in the output column.
I would like to created excel files for each "type" value
E.g. If there is 3 "type" values (A, B, C), I would like to create 3 excel files to store type A, type B, and type C data respectively.
Since the number of possibe value of "type" field is various, how can I create the xls destination dynamic and move the correct type to the corresponding excel file?
The conditional split has fixed conditions, it is not suitable for by dynamic number of value
For Loop condition is not a good choice because I need to run the complicated SQL for many time.
Thanks.
View 1 Replies
View Related
Oct 11, 2007
I have searched the archives and haven't found a solution that works.
I started out with a package based on the Excel import wizard and have modified it to include a for each loop for processing more than one Excel source file and also modified it to do some dynamic SQL like dropping and creating the tables with names that are variable based. The package is cabable of processing more than one file, and each file has the same data elements or columns in it, but each file has to go in it's own table. The drop and preperation tasks use string variable SQL that does dynamically create the SQL and runs great. The data flow task and it's OLE destination task (which the wizard did intially create) does not perform as expected. There were some posts in the archives stating that for dynamic destination table names it can be done in the OLE destination but I can't get it to work. I tried using an access mode of table/view name variable and select a variable that is string expression based. But this variable isn't fully populated until run time and so when I'm trying to configure this I get a msg stating the table name object doesn't exist and I can't save the task.
Am I doing something wrong, or trying to do something that wrong way or trying to do something that isn't possible?
View 10 Replies
View Related
Apr 30, 2008
I have a source file name which changes every day , basically the filename like 9500 , the next file would come is 9501 , then next could be anything may be 1000Q. I need the source connection to pick dynamicallly those file name. is their any way i could achieve this in SSIS.
I would not know what is the name of the file i would expect
please advise
View 1 Replies
View Related
Jul 20, 2005
I'm looking for a way to transform the contents of n source tablesinto a single destination table. This by itself is no problem.However, the name of the source tables change, so I'll need to basethe transform task on a global variable that I can update via externalcode. Not sure how to do that. I'm ok with executing the package 10times if there's 10 source tables.The last unknown piece is modifying the query used for the transform.There are 10 columns in the source table, but there are 12 columns inthe destination table. I must provide the 2 missing columns. They willsimply contain a year and month, ie. 05 2003.I'm taking a bunch of source tables (for a given month and year) androlling them together into one destination table, and carrying overthe month and year. I assume the month and year would also be globals.But I'm not sure how to incorporate them into the transform task sinceit wants strict SQL syntax.Any help is appreciated, thanks in advance!
View 2 Replies
View Related
Jun 6, 2005
Sorry if this is a pretty basic question.. but my team here is working on SQL Reporting app for the first time.
View 19 Replies
View Related
Jul 14, 2006
I want to be able to programmatically set the schema location for an XML source. I first thought it would be a simple task using expressions and variables but it doesn't appear to allow anything in the way of setting it at runtime. Is this possible?
View 4 Replies
View Related
Mar 21, 2008
Hi All,
I have a requirement to create a dynamic SQL Command in an OLE DB Source due to the fact that I need to read data from another database based on a date range. For example, the SQL command would look like
SELECT * FROM Table1 WHERE DateField BETWEEN '17/03/2008' and '21/03/2008'
and I need to change the dates - '17/03/2008' and '21/03/2008' to different dates when the package is deployed in production, how do I do that ?
Regards
Ash
View 12 Replies
View Related
May 2, 2006
Hi,
I am building a data warehouse for a customer who has systems located in two different countries.
I need to import that data from four seperate databases, which all share the same structure.
To do this i have created 20 packages to import that data from the source database. What i would like to do, is at run time set which database the SSIS package should get its data from.
In sql 2k this was easy with a global variable that was set, then use a dynamic properties task to set the data source.
How can i achieve the same result in SSIS? the data source is an ODBC connection, with the four ODBC connections having similar names, eg ABC_NZ, ABC_AU
Thanks in Advance!
Truby
View 6 Replies
View Related
Jan 2, 2008
Dear Friends,
I currently have my excel source dynamic for the connectionstring. Using a global variable with the filename and using the expressions of this control to dynamically have the connecttionstring. My problem is that i need tto read always the first workseet and the name changes frequently and generates an erro in SSIS.
how can I set the excel source to goes only for the first sheet independently of the worksheet name?!
Regards!!
View 5 Replies
View Related
Aug 25, 2007
Hi Craig/Kamal,
I got your email address from your web cast. I really enjoyed the web cast and found it to be
very informative.
Our company is planning to use SSIS (VS 2005 / SQL Server 2005). I have a quick question
regarding the product. I have looked for the information on the web, but was not able to find
relevant information.
We are getting Source data from two of our client in the form of Excel Sheet. These Excel sheets
Are generated using reporting services. On examining the excel sheet, I found out that the name
Of the columns contain data itself, so the names are not static such as Jan 2007 Sales, Feb 2007 Sales etc etc.
And even the number of columns are not static. It depends upon the range of date selected by the user.
I wanted to know, if there is a way to import Excel sheet using Integration Services by defining the position
Of column, instead of column name and I am not sure if there is a way for me to import excel with dynamic
Number of columns.
Your help in this respect is highly appreciated!
Thanks,
Hi Anthony, I am glad the Web cast was helpful.
Kamal and I have both moved on to other teams in MSFT and I am a little rusty in that area, though in general dynamic numbers of columns in any format is always tricky. I am just assuming its not feasible for you to try and get the source for SSIS a little closer to home, e.g. rather than using Excel output from Reporting Services, use the same/some form of the query/data source that RS is using.
I suggest you post a question on the SSIS forum on MSDN and you should get some good answers.
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
http://forums.microsoft.com/msdn/showforum.aspx?forumid=80&siteid=1
Thanks
Craig Guyer
SQL Server Reporting Services
View 12 Replies
View Related
Nov 23, 2007
Hi,
I have a need to display on screen AND email a pdf report to email addresses specified at run time, executing the report with a parameter specified by the user. I have looked into data driven subscriptions, but it seems this is based on scheduling. Unfortunately for the majority of the project I will only have access to SQL 2005 Standard Edition (Production system is Enterprise), so I cannot investigate thoroughly.
So, is this possible using data driven subscriptions? Scenario is:
1. User enters parameter used for query, as well as email addresses.
2. Report is generated and displayed on screen.
3. Report is emailed to addresses specified by user.
Any tips on how to get this working?
Thanks
Mark Smith
View 3 Replies
View Related
May 2, 2007
If anyone could confirm...
SQL Server 2000 SP4 to multiple SQL Server 2005 Mobile Edition on PDAs. My DB on SQL2k is published with a single dynamic row filter using host_name() on my 'parent' table and also join filters from parent to child tables. The row filter uses joins to other tables elsewhere that are not published to evaluate what data is allowed through the filter.
E.g. Published parent table that contains suppliers names, etc. while child table is suppliers' products. The filter queries host_name(s) linked to suppliers in unpublished table elsewhere.
First initial sync with snapshot is correct and as I expected - PDA receives only the data from parent (and thus child tables) that matches the row filter for the host_name provided.
However - in my scenario host_name <--> suppliers may later be updated E.g. more suppliers assigned to a PDA for use or vice versa. But when I merge the mobile DB, the new data is not downloaded? Tried re-running snapshot, etc., no change.
Question: I thought the filters would remain dynamic and be applied on each sync?
I run a 'harmless' update on parent table using TSQL e.g. "update table set 'X' = 'X'" and re-sync. Now the new parent records are downloaded - but the child records are not!
Question: I wonder why if parent records are supplied, why not child records?
If I delete existing DB and sync new, I get the updated snapshot and all is well - until more data added back at server...
Any help would be greatly appreciated. Is it possible (or not) to have dynamic filters run during second or subsequent merge?
View 4 Replies
View Related
Apr 11, 2008
I have CSV file as source for SSIS package every time the filename will be changing like trd_1990M1_1990M12.csv,trd_1991M1_1991M12.csv , trd_1992M1_1992M12.csv etc.,
so it will vary as per user selection . i need to run the same SSIS package to execute the different file name with the same structure.
Please let me know the solution for that how to pass the file name dynamically to SSIS package.
View 1 Replies
View Related
Nov 21, 2006
I'm using SSIS to import data from a table (SQL) containing varchar fields. The problem is, that those varchar fields are changing over time (sometimes shrinking and sometimes expanding). I.e. from varchar(16) to varchar(20).
When I create my SSIS package, the package seem to store information about the length of each source-field. At runtime, if the field-length is larger then what the package expects an error is thown.
Is there anyway around this problem?
Oh, yeah... My destination fields are a lot wider then the source fields, so the problem is not that the varchar values doesn't fit in my destination table, but that the package expects the source to be smaller...
Regards Andreas
View 3 Replies
View Related
Sep 11, 2006
hi everyone,
joy mundy alluded in her webcast that it is possible to dynamically specify a table name in a parameterized ole db source query. is this true? if so, how can it be done?
View 6 Replies
View Related
Feb 21, 2006
Hello,
I have an OLEDB Source component with a Oracle OLEDB connection manager.
In my SQL statement I must do something like this ...
SELECT * FROM OracleTable
WHERE convert(datetime, OracleDate) = parameter
I've dynamicaly build the statement like this : http://blogs.conchango.com/jamiethomson/archive/2005/12/09/2480.aspx
Unfortunately this doesn't work because the variable is a datetime value and not a string.
Maybe another way is to store the date in a table in my SQL database and include it in my Oracle SQL statement ? This would mean that there are multiple connection managers being used in one SQL statement ... I have no idea if this would be possible ...
Any other options ?
View 3 Replies
View Related
Oct 20, 2007
I have created an SSIS package that is designed to move data from SQL Server 2000 to an Access db. I have set the package up to accept four parameters. They are:
the name of the SQL Server Database, which is used in an expression to provide the source connection manager connection string;
the full path to the Access db destination database which is used like #1 above.
the SELECT statement used in an OLE DB Source object to get the data from the source table
the table name which is used by an OLE DB Destination object.
I know that the source and destination tables have exactly the same structure and do not require a transformation.
In order to change variables 3 and 4 from above and have it work, I go through the following steps:
I change the variables to appropriate values.
Go to the Advanced Editor for the OLE DB Source object and click on "Refresh". This produces an error in the OLE DB Destination object that is something like "Validation error. Data Flow Task: DTS.Pipeline: input column "strRECTYPESUFFIX" (301) has lineage ID 17 that was not previously used in the Data Flow task.
Go to the Advanced Editor for the OLE DB Destination which brings up the "Restore Invalid Column Reference Editor". I mark all the columns that show up with the option to <Delete invalid column reference> and click OK
I then reopen the Advanced Editor for the OLE DB Destination, go to the 'Column Mappings' tab click 'Refresh', then in the upper pane where the input and output columns list appear I right click and choose "Map Items by Matching Names"
At this point I no longer have the error and the package will execute without any problems.
I am doing this so that I can load the SSIS package in VB.NET (2.0) so that I can then set these variables programmatically and then execute the package. The problem is that this actually performs steps 1 and 4 above. 2 and 3 are left out and the package fails miserably.
I have found some information that would be helpful if I could get my hands on the appropriate object. I realize that the data flow component would return VS_NEEDSNEWMETADATA from the Validate method, and that could be repaired with the ReinitializeMetaData method of the data flow component ( I assume that is the object to be using in this case). But I do not know how to grab the "Data Flow Component" as an object based upon the "Package" object I have loaded so that I can check if it is valid and manipulate it if necessary.
Thank you in advance for your help.
View 4 Replies
View Related
May 1, 2008
I have a problem while creating a dynamic source connection.
i have four files while which comes with different extension , apparently the name of the files are same for example.
9500.txt, the other one is 9500. rtf, 9500.dat, 9500.map, They are all text file but with different formats. I have problem selecting a dynamic source connection for each of this files.
when i create variable for input file name the file connection gets confused and throws me error with input file.
please help
Thank you
View 13 Replies
View Related
