Problem With Dynamic Source For Multiple File
May 1, 2008
I have a problem while creating a dynamic source connection.
i have four files while which comes with different extension , apparently the name of the files are same for example.
9500.txt, the other one is 9500. rtf, 9500.dat, 9500.map, They are all text file but with different formats. I have problem selecting a dynamic source connection for each of this files.
when i create variable for input file name the file connection gets confused and throws me error with input file.
please help
Thank you
View 13 Replies
ADVERTISEMENT
Feb 27, 2008
Hi All,
I have a source files folder where the files generated everyday.
My goal is pick the latest file and copy this single file to another folder.
I used the Foreach loop container and got the latest file and stored the file name to a varible i.e. LatestFile
Then i want to use the File System Task to copy this to the destination.
On the beginning, I could not setup the Latestfile since I don't its name then, so when I setup the Source Connection property of the File system task, it is not allowed to leave the SourceVarible as blank!
Any suggestion?
Thanks
Micor
View 3 Replies
View Related
Mar 28, 2006
Hi,
we have one requirement to run the package daily basis.
The package should run at specific time on that day.
we are using windows schedular for that.
we will have one new flatfile everyday.
Is there any process to attach this file to flat file source dynamically?
The requirement is,
The flat file should be able to read the new flatfile everyday.
we have no option change it manually, the flatfile source should have to take the file automatically at that time.
So that it can take that flatfile and load it into database table.
View 1 Replies
View Related
Dec 4, 2005
How can I run a single SP by asking multiple sales question eitherby using the logical operator AND for all the questions; or usingthe logical operator OR for all the questions. So it's alwayseither AND or OR but never mixed together.We can use Northwind database for my question, it is very similarto the structure of the problem on the database I am working on.IF(SELECT OBJECT_ID('REPORT')) IS NOT NULLDROP TABLE REPORT_SELECTIONGOCREATE TABLE REPORT_SELECTION(AUTOID INT IDENTITY(1, 1) NOT NULL,REPSELNO INT NOT NULL, -- Idenitifies which report query this-- "sales question" is part ofSupplierID INT NOT NULL, -- from the Suppliers tableProductID INT NOT NULL, -- from the Products table, if you choose--a ProductID, SupplierID is selected also by inheritenceCategoryID INT NOT NULL, -- from the Categories tableSOLDDFROM DATETIME NULL, -- Sold from which dateSOLDTO DATETIME NULL, -- Sold to which dateMINSALES INT NOT NULL, -- The minimum amount of salesMAXSALES INT NOT NULL, -- The maximum amount of salesOPERATOR TINYINT NOT NULL -- 1 is logical operator AND, 2 is OR)GOINSERT INTO REPORT_SELECTIONSELECT 1, 1, 2, 1, '1/1/1996', '1/1/2000', 10, 10000, 1 UNION ALLSELECT 1, -1, -1, 1, '1/1/1996', '1/1/2000', 10, 1000, 1You can ask all kinds of sales questions like:1-I want all employees that sold products from supplierID 1(Exotic Liquids), specifically the ProductID 2 (Chang) from theCategoryID 1 (Beverages) between Jan 1 1996 to Jan 1 2000 and soldbetween $10 and $10000 - AND for my 2nd sales question2-I want all employees that sold CategoryID 1 (beverages) betweenJan 1 1996 to Jan 1 2000 and sold between $10 and $1000I want to get the common result of both questions and find outwhich employee(s) are in this list.Here are some of the points:1-I want my query to return the list of employees fitting theresult of my sales question(s).2-If I ask three questions with the logical operator AND, I wantthe list of employees that are common to all three questions.3-If I ask 2-3-4. questions with the logical operator OR, I wantthe list of employees that are in the list of the 1st "successful"sales question (the first question that returns any employee isgood enough)4-You can ask all kind of sales question you want even if theycontradict each other. The SP should still run and returnnothing if that is the case.5-Let's assume you can have the same product name from the samesupplier but under different categories. So entering a ProductIDshould not automatically enter the CategoryID also; whereasentering the ProductID should automatically enter its SupplierID.6-SOLDFROM, SOLDTO, MINSALES, MAXSALES, OPERATOR are mandatoryfields, you can't leave them NULL7-SupplierID, ProductID and CategoryID are the dynamic inputparameters, there can be 5 different combinations to choose from:a-SupplierID onlyb-SupplierID and a ProductID,c-SupplierID and a CategoryIDd-SupplierID, ProductID and a CategoryIDe-CategoryID onlyf-Any time you choose a ProductID, the SupplierID valuewill be filled automatically based on the ProductID'srelationshipg-Any of the three values here that is not chosen by theuser will take a default value of -1 (meaning return ALLfor this Column, in other words don't filter by this column)The major problem I have is I can't use dynamic SQL for choosingthe three dynamic columns as the 2nd row of records would have adifferent selection of dynamic columns (at least I don't know howif the solution is dynamic SQL). The only solution I can think oflooks pretty bad to me. I would use a cursor, run each row at atime, store a TRUE, FALSE value to stop processing or not andstore the result in another detail table. Then if all ANDquestions have ended with TRUE do a union of all the result andreturn the common list of employees. It sounds pretty awful as anapproach. I am hoping there's a simpler method for achieving this.Does anyone know if any SQL book has a topic on this type ofquery? If so I'll definitely buy the book.I appreciate any help you can provide.Thank you
View 7 Replies
View Related
Apr 11, 2008
I have CSV file as source for SSIS package every time the filename will be changing like trd_1990M1_1990M12.csv,trd_1991M1_1991M12.csv , trd_1992M1_1992M12.csv etc.,
so it will vary as per user selection . i need to run the same SSIS package to execute the different file name with the same structure.
Please let me know the solution for that how to pass the file name dynamically to SSIS package.
View 1 Replies
View Related
Aug 23, 2006
Hi,
I've seen a number of posts similar to this but i still cannot figure out what i need to do to get it working. So here goes with a couple of newbie questions.
Question 1:
Once created how do i go about executing a SSIS package. I want to be able to call it from a C# application from which i pass in a couple of parameters?
Question 2:
How do i go about setting the file path of my Excel source to a dynamic value passed at runtime. I want to be able to loop through a number of Excel files and do some processing on them. I've set up a variable (which i think i need to do) after that i get stuck however. Some other posts suggest configuration packages but i cannot get my head around how they work?
Any help on this matter would be gratefully recieved.
Thanks in advance,
Grant
View 5 Replies
View Related
Sep 3, 2007
Hi,
I am migrating one of my DTS package to SSIS.
My task is to read the filename from a database table and transfer the flat file data in to a table.
In SSIS,I am able to fetch the file name using a Data Reader Source; but how to pass this fileName parameter to Flat File Source ?
In DTS I have used ActiveX script to pass filename variable as flatfilecon.Source.
Any help ?
Thanks,
Ravi
View 4 Replies
View Related
May 23, 2007
I have a question regarding Raw Files. I am breaking a large package into more modular components for better processing and debugging.
The process will start with a preparatory dataflow that will create a Raw File(s). This Raw File will then be used as the source in possibly 6 data flows and/or packages.
My question is whether 1 Raw File can be read concurrently by the multiple jobs and how this would affect processing. I'm assuming that this would slow processing.
My other option is to Multicast the writing of the Raw File to 5 other versions of the file. All would be identical except for filename. Obviously this would use more disk space but this is not a concern as we have lots of disk space. Our concern is for speedy processing.
If you have experience with Raw Files, please let me know how you approached this issue. As always, blogs and specific examples are always great!
Thanks in advance.
View 4 Replies
View Related
Sep 6, 2007
Hi everyone,
There is a small problem encountered while creating a package in sql
server 2005.
Actually i am using a flat file which has 820 rows and 2 columns which
are seperated by line feed(for ROW) and tab(for COLUMN).after
importing i found that ther are only 800 rows imported into the table.
Ather verifying the input file i found out that there are some null
values in the second column so there is no line feed for those
values.
Can anyone please help me how to give multiple delimiters for the same
input flat file.
View 9 Replies
View Related
Feb 23, 2008
Hello Guys,
I am working in one company and currently I am assigned to new project for Data Migration from company X to our company Y using SSIS. I am totally new and i just completed 5 tutorial which was gien on MSDN website.
Basically client is going to send us first flat file with 1 million records with Header, Detail and Trailer records.
I want to create a Package in such a way that it dumps all this first load into 7 to 8 different tables at a time.
we also have to include functionlity for validation and error check.
On successfull load error file should only return Header and Trailer but no detail records.
If there are any errors then error file should contain Header, Detail records which were unable to load plus trailer which we have to sent back to client.
When 2nd file comes that time we have to check whether this is new records or change (update) one depending on Flag which tells it.
This is basically high level idea of my Package what i need to create. If u guys have any question then let me know.
I know you guys are very experienced one. Anyone of you please give me some detail idea on it I would really appricate it.
I have very limited time line for it.
Thanks
Shah
View 4 Replies
View Related
Mar 2, 2014
I am trying to create an ssis package with dynamic csv file as output. and out format contains query output.
sample file name:
Unique identifier + query output + systemdate();
The expression is looking like this.
@[User::FilePath] + @[User::FileName] + ".CSV"
-- user filepath is a variable from ssis package. File name is the output from SQL query. using script task i have assigned the values to @[User::FileName] .
When I debugged the script task the value getting properly but same variable am using for Flafile destination. but its not working.
View 3 Replies
View Related
Apr 15, 2008
I have a requirment which i have partly accomplished , but could not get through completely
i have a file which comes in a standard format ending with date and seq number ,
suppose , the file name is abc_yyyymmdd_01 , for first copy , if it is copied more then once the sequence number changes to 02 and 03 and keep going on .
then i need to transform those in to new file comma delimited destination file with a name abc_yyyymmdd,txt and others counting file counting record abc_count_yyyymmdd.txt. and move it to a designated folder. and the source file is then moved to archived folder
what i have taken apprach is
script task select source file --------------------> data flow task------------------------------------------> script task to destination file
dataflow task -------------------------> does count and copy in delimited format
what is happening here is i can accomlish a regular source file convert it to delimited destination file --------> and move it to destination folder with script task .
but cannot work the dynamic pick of a source file.
please advise with your comments or solution you have
View 14 Replies
View Related
Apr 30, 2008
I have a source file name which changes every day , basically the filename like 9500 , the next file would come is 9501 , then next could be anything may be 1000Q. I need the source connection to pick dynamicallly those file name. is their any way i could achieve this in SSIS.
I would not know what is the name of the file i would expect
please advise
View 1 Replies
View Related
Jul 20, 2005
I'm looking for a way to transform the contents of n source tablesinto a single destination table. This by itself is no problem.However, the name of the source tables change, so I'll need to basethe transform task on a global variable that I can update via externalcode. Not sure how to do that. I'm ok with executing the package 10times if there's 10 source tables.The last unknown piece is modifying the query used for the transform.There are 10 columns in the source table, but there are 12 columns inthe destination table. I must provide the 2 missing columns. They willsimply contain a year and month, ie. 05 2003.I'm taking a bunch of source tables (for a given month and year) androlling them together into one destination table, and carrying overthe month and year. I assume the month and year would also be globals.But I'm not sure how to incorporate them into the transform task sinceit wants strict SQL syntax.Any help is appreciated, thanks in advance!
View 2 Replies
View Related
Mar 27, 2007
Hi,
I am building SSIS for 3 different files that have identical
schema and mapping logic.
In my OLE DB Source (object name - "OLEDBSource_SourceTable")
Data Access mode is "Variable name".
As soon as I swithced to this Data Acces mode
it started to give me an error:
[OLEDBSource_SourceTable [1]] Warning: The external metadata column collection is out of synchronization
with the data source columns.
The column "DEAL_NUM" needs to be updated in the external metadata column collection.
The "external metadata column "DEAL_NUM_Flag" (34529)" needs to be removed
from the external metadata column collection.
The "external metadata column "recordID" (33740)"
needs to be removed from the external metadata column collection.
View 22 Replies
View Related
Jun 6, 2005
Sorry if this is a pretty basic question.. but my team here is working on SQL Reporting app for the first time.
View 19 Replies
View Related
Jul 14, 2006
I want to be able to programmatically set the schema location for an XML source. I first thought it would be a simple task using expressions and variables but it doesn't appear to allow anything in the way of setting it at runtime. Is this possible?
View 4 Replies
View Related
Mar 21, 2008
Hi All,
I have a requirement to create a dynamic SQL Command in an OLE DB Source due to the fact that I need to read data from another database based on a date range. For example, the SQL command would look like
SELECT * FROM Table1 WHERE DateField BETWEEN '17/03/2008' and '21/03/2008'
and I need to change the dates - '17/03/2008' and '21/03/2008' to different dates when the package is deployed in production, how do I do that ?
Regards
Ash
View 12 Replies
View Related
May 2, 2006
Hi,
I am building a data warehouse for a customer who has systems located in two different countries.
I need to import that data from four seperate databases, which all share the same structure.
To do this i have created 20 packages to import that data from the source database. What i would like to do, is at run time set which database the SSIS package should get its data from.
In sql 2k this was easy with a global variable that was set, then use a dynamic properties task to set the data source.
How can i achieve the same result in SSIS? the data source is an ODBC connection, with the four ODBC connections having similar names, eg ABC_NZ, ABC_AU
Thanks in Advance!
Truby
View 6 Replies
View Related
Jan 2, 2008
Dear Friends,
I currently have my excel source dynamic for the connectionstring. Using a global variable with the filename and using the expressions of this control to dynamically have the connecttionstring. My problem is that i need tto read always the first workseet and the name changes frequently and generates an erro in SSIS.
how can I set the excel source to goes only for the first sheet independently of the worksheet name?!
Regards!!
View 5 Replies
View Related
Aug 8, 2007
Hello,
What I'm trying to accomplish is to have a variable names "SourceTable" and "DestinationTable". So for each SourceTable, the DestinationTable will have the same columns. All I need is to auto-map these columns between source and destination via code?
Is this possible?
Thanks,
awiora
View 3 Replies
View Related
Nov 21, 2006
I'm using SSIS to import data from a table (SQL) containing varchar fields. The problem is, that those varchar fields are changing over time (sometimes shrinking and sometimes expanding). I.e. from varchar(16) to varchar(20).
When I create my SSIS package, the package seem to store information about the length of each source-field. At runtime, if the field-length is larger then what the package expects an error is thown.
Is there anyway around this problem?
Oh, yeah... My destination fields are a lot wider then the source fields, so the problem is not that the varchar values doesn't fit in my destination table, but that the package expects the source to be smaller...
Regards Andreas
View 3 Replies
View Related
Sep 11, 2006
hi everyone,
joy mundy alluded in her webcast that it is possible to dynamically specify a table name in a parameterized ole db source query. is this true? if so, how can it be done?
View 6 Replies
View Related
Feb 21, 2006
Hello,
I have an OLEDB Source component with a Oracle OLEDB connection manager.
In my SQL statement I must do something like this ...
SELECT * FROM OracleTable
WHERE convert(datetime, OracleDate) = parameter
I've dynamicaly build the statement like this : http://blogs.conchango.com/jamiethomson/archive/2005/12/09/2480.aspx
Unfortunately this doesn't work because the variable is a datetime value and not a string.
Maybe another way is to store the date in a table in my SQL database and include it in my Oracle SQL statement ? This would mean that there are multiple connection managers being used in one SQL statement ... I have no idea if this would be possible ...
Any other options ?
View 3 Replies
View Related
Oct 20, 2007
I have created an SSIS package that is designed to move data from SQL Server 2000 to an Access db. I have set the package up to accept four parameters. They are:
the name of the SQL Server Database, which is used in an expression to provide the source connection manager connection string;
the full path to the Access db destination database which is used like #1 above.
the SELECT statement used in an OLE DB Source object to get the data from the source table
the table name which is used by an OLE DB Destination object.
I know that the source and destination tables have exactly the same structure and do not require a transformation.
In order to change variables 3 and 4 from above and have it work, I go through the following steps:
I change the variables to appropriate values.
Go to the Advanced Editor for the OLE DB Source object and click on "Refresh". This produces an error in the OLE DB Destination object that is something like "Validation error. Data Flow Task: DTS.Pipeline: input column "strRECTYPESUFFIX" (301) has lineage ID 17 that was not previously used in the Data Flow task.
Go to the Advanced Editor for the OLE DB Destination which brings up the "Restore Invalid Column Reference Editor". I mark all the columns that show up with the option to <Delete invalid column reference> and click OK
I then reopen the Advanced Editor for the OLE DB Destination, go to the 'Column Mappings' tab click 'Refresh', then in the upper pane where the input and output columns list appear I right click and choose "Map Items by Matching Names"
At this point I no longer have the error and the package will execute without any problems.
I am doing this so that I can load the SSIS package in VB.NET (2.0) so that I can then set these variables programmatically and then execute the package. The problem is that this actually performs steps 1 and 4 above. 2 and 3 are left out and the package fails miserably.
I have found some information that would be helpful if I could get my hands on the appropriate object. I realize that the data flow component would return VS_NEEDSNEWMETADATA from the Validate method, and that could be repaired with the ReinitializeMetaData method of the data flow component ( I assume that is the object to be using in this case). But I do not know how to grab the "Data Flow Component" as an object based upon the "Package" object I have loaded so that I can check if it is valid and manipulate it if necessary.
Thank you in advance for your help.
View 4 Replies
View Related
Apr 14, 2008
Hi all,
I am able to set dynamic source for the text file(flat file) but i want to set the connection string (file name) to excel source dynamicaly. I have tryied lots of time by taking a variable in foreachloop container . Variable is itself able to pick the file name dynamicaly but whern i am tying to set connectionstring to excel source it gives error.
Steps that i have done: -
1) Drag foreachloop container
2) set directory,FileNameRetrieval,FileSpec
3) Made VariableMapping
4) Now drag a dataflow task in the foreachloop container
5) select excel source
6) When i am selecting varaible as connectin string from properties of excel connectin manager, i am getting this error : -
TITLE: Microsoft Visual Studio
------------------------------
Error at Package3 [Connection manager "Excel Connection Manager 2"]: An OLE DB error has occurred. Error code: 0x80040E4D.
Error at Data Flow Task [Excel Source [1]]: The AcquireConnection method call to the connection manager "Excel Connection Manager 2" failed with error code 0xC0202009.
------------------------------
ADDITIONAL INFORMATION:
Exception from HRESULT: 0xC020801C (Microsoft.SqlServer.DTSPipelineWrap)
------------------------------
BUTTONS:
OK
------------------------------
Please help me, whats the problem? can i set connectionstring via variable to excel source
View 6 Replies
View Related
Apr 14, 2008
Hello,
Kindly give me the solution ASAP how to do Dyanmic Connection in ExcelConnection manager.
Thanks
Thiru
View 1 Replies
View Related
Apr 14, 2008
Hi SSIsians,
I have a package with Excel Destination with dynamic connection.
I did ExcelFilePath = [@user::VarSourceFolder]+[@user::VarSourceFileName]
then i changed the Delayvalidation = True.
When i try to run the package in BIDS it gives the error.
ERROR:
[Excel Source [30501]] Error: An OLE DB error has occurred. Error code: 0x80040E37.
[Excel Source [30501]] Error: Opening a rowset for "DailySheet" failed. Check that the object exists in the database.
It saying there is no sheet in the name of "DailySheet" but when i removed the expression in connection manager property it is working fine.
Please let me know what is the problem OR how to configure the dynamic connection in ExcelSource.
Thanks
Thiru
SE - SSIS
Chennai
View 5 Replies
View Related
Feb 13, 2007
Hi,
I am trying to create a program that transfers tables to flat files.
At this point in time, I have suceeded in created one that creates delimited files.
However, I am now trying to create fixed-width files as you can do with the SSIS designer, but programatically.
Is there a way to programatically determine the width of a column from the source table? I can not seem to find any kind of function or member that stores this information or allows me to retrieve it.
I know what I need to change in order to set a width for a column, but I just don't know how to find the width without just asking the user to provide one.
View 5 Replies
View Related
May 29, 2008
I would like to use a dynamic pivot table in my data source view. It seems that a named query can be only one sql statement. So, I cannot use my multi-statement procedure that creates a dynamic pivot table output.
What is the best course of action here? I could hard-code my pivot table query. I could maintain a redundant table in the pivot format. Do I have any good options?
KenS
Ken
View 1 Replies
View Related
Dec 12, 2007
I am building an SSIS package that loops through a table in SQL Server and dynamically builds a select statement that i would like to use as an ole db source. I have been having a difficult time with this as the select statement that i am generating is over 200,000 characters long so using an sql variable is out of the question.
I ended up placing the select statement into a table where each row of the table represents a piece of the select. I then use an execute_sql task that selects the entire rowset from this table into a variable object. I then use a for each loop to shred the variable and concatenate it into on big string variable called user:: sql_statement that is my select.
After setting up the loop and testing to see if the user:: sql_statement variable populates correctly i then added a data flow transfer with an ole db source and destination. I then go into the advanced editor for the source and set it to accept an sql statement from a variable and use my user:: sql_statement variable. I was forced to set validate external metadata option to false to avoid an error since there is no way to validate the columns until the for each loop runs during run time.
Now thats all fine and good but what is causing my problem is that during run time, when the package gets to the data flow task, the select statement doesn't seem to be populating the input columns of the data source. I have been searching to no avail on a way to tell the data source to update the input columns but every time it gets there, the package bombs out telling me the ole db source has no available output columns.
Specifically the error i get is :
[DTS.Pipeline] Error: "output "OLE DB Source Output" (6616)" contains no output columns. An asynchronous output must contain output columns.
Any help with this would be much appreciated.
View 18 Replies
View Related
Jul 14, 2015
I have a requirement to make all of my KPI goals configurable by the end user - I would like to add the data to my datawarehouse so that it is updated in the cube whenever the cube is processed. I'm thinking I will need a table for each KPI that includes two columns (I will be setting goal by year):
Year
Value
In my cube I then create a measure group for each KPI (for some I will add a calculations to break down the yearly goal to lower levels of detail).I will add each measure to my Dimension Usage and relate to each of my time dimensions at the Year level.I currently have around a dozen KPIs - my above approach seems a bit messy i.e. requiring a table and measure group for each KPI (I guess I could at least put them all into one table in my relational db).
View 2 Replies
View Related
Apr 16, 2008
I am a business user trying to build an incremental ETL package with SSIS. I have a working prototype on SQL Server 2005 where I select the max(ID) from the last successful run and pass that value into a variable. Then, in my Data Flow step, I select an OLE DB source adapter and use this variable in a custom select statement.
Here's my challenge....the live data is actually in a Postgres DB so I have to use a Data Reader Source adapter. When I try to pass my variable to this adapter the job bombs out. Does anyone know how to dynamically update the query text inside a Data Reader source adapter using variables or otherwise?
View 3 Replies
View Related
