Use Query Results To Feed Data Flow

Sep 24, 2007

Greetings,

I need some help determining the best way to accomplish my task. The workflow starts by generating a list of unique ID's from a local table. Then take that list of unique ID's and query an Oracle table for all matching records.

My thought was to first use an Execute SQL task with the following SQL:

select projectid from projectlist group by projectid

with Result Set configured as follows:

Result Name = projectid
Variable Name = varProjectIDList

Then in the Data Flow Task add a DataReader Source to pull the matching data. Here's where I'm getting hung up. I'd like to pass the result set from the Execute SQL task. I tried the following SQL but it doesn't work.

select * from masterlist where projectid = @[User::varProjectIDList]

I'm open to any suggestion on the best way to take my unique list and use it as input for a query against my Oracle DB.

Thank you for your ideas.

Rob


View 4 Replies


ADVERTISEMENT

Query Results SSIS Data Flow Source Adapter

Jun 1, 2006

Hi,
Quick question on how SSIS handles queries from Data Source in a Data Flow. I noticed that when I run a particular query from Query Analyzer it takes forever. But, when I run the same query in SSIS data source in a data flow. The query results are immediate.

The query plan is already cached in SQL.

Is this just something which I am seeing incorrect or is there some bit of optimization in there in SSIS. As per my understanding SSIS does not optimize the source query.

Thanks,
Gaurav



View 3 Replies View Related

HOW? Access Results Of Computed SQL Statement In A Data Flow

Apr 24, 2007

The SQL computed is complex enough that I can't see a way to make it a parameterized query. The obvious approach seems to be to compute the SQL in a CONTROL FLOW SCRIPT TASK and then use it to load a variable to set the VARIABLE SOURCE of a CONTROL FLOW EXECUTE SQL TASK.



I see that I can return a resultset to a variable.



But getting the rows of the results into a dataflow is not obvious. I have heard mentione that a Derived Column can do this. I can see using a dummy SCRIPT COMPONENT as DATA SOURCE with nothing in it to then drop into DERIVED COLUMN. But when setting up DERIVED COLUMN I don't see how to pull the columns out of the RESULTSET variable.



If it makes a difference I think the columns of the resultset will always be the same in this scenario.



Maybe this is totally the wrong approach? Any clues would be appreciated.

View 1 Replies View Related

Should You Save Results Of Intermediate Data Flow Steps To Temporary Tables Or Raw Files?

Jun 2, 2006

Hi,

I'm just starting off in SSIS and have a question that I can't find an answer to...

I'm loading in a number of files (in separate Data Flows) and performing some transformations on them before merging them back together. What I'm not sure about is what I should be doing with the data at the end of each of my "Import Data From XXXX Flat File" Data Flows. Am I better off using OLE DB Destinations (or SQL Server Destinations) and saving this intermediate data to temporary tables, or am I better off using a Raw File Destinations and saving this intermediate data to files? Or is there, perhaps, a better option that I'm currently unaware of?

If the Raw File Destination is the way to go, then isn't there a maintenance issue with cleaning up all the files created? And will there not be a management issue to ensure that there is sufficient disc space available on the drive you are saving to?

I'm a bit confused and overwhelmed by SSIS at the moment, so any help would be much appreciated!

Thanks in advance,
Lawrie.

View 3 Replies View Related

Bulk Insert Vs. Data Flow Task (different Row Results Using Flat File Source)

Nov 2, 2006

I'm importing a large csv file two different ways - one with Bulk Import Task and the other way with the Data Flow Task (flat file source -> OLE DB destination).

With the Bulk Import Task I'm putting all the csv rows in one column. With the Data Flow Task I'm mapping each csv value to it's own column in the SQL table.

I used two different flat file sources and got the following:

Flat file 1: Bulk Import Task = 12,649,499 rows; Data Flow Task = 4,215,817 rows
Flat file 2: Bulk Import Task = 3,403,254 rows; Data Flow Task = 1,134,359 rows

Anyone have any guess as to why this is happening?

View 9 Replies View Related

Is There A Way To Hold The Results Of A Select Query Then Operate On The Results And Changes Will Be Reflected On The Actual Data?

Apr 1, 2007

hi,  like, if i need to do delete some items with the id = 10000 then also need to update on the remaining items on the with the same idthen i will need to go through all the records to fetch the items with the same id right?  so, is there something that i can use to hold those records so that i can do the delete and update just on those records  and don't need to query twice? or is there a way to do that in one go ?thanks in advance! 

View 1 Replies View Related

Reuse Existing Data Flow Components In A Custom Data Flow Component

Aug 29, 2007

Hello,

Is it possible to use existing data flow components (Merge Join, aggregation,...) in a custom data flow component?

Thanks,

Yoann

View 15 Replies View Related

Insert Line Feed In Query

Aug 13, 2007

Hi,
    I want to insert line feed. I have used char(10) and char(13) function. But I could not  insert line feed.    Ex: set error1= 'error description' + char(10)+ 'Employee'
Output must be: error description                Employee
Any suggestions are welcome.

View 8 Replies View Related

What Is Data Feed?

Mar 12, 2004

Hello Friends,

I would like to know what Data Feed means while working as an Administrator on Ms SQL Server 2000?
Also what do you mean by integrating feed of data?
I would appreciate it if anyone could help me out with this queries...
Also I was wondering which would be best book to refer while learning about ms sql server 2000.

View 6 Replies View Related

XML Data Feed

Dec 29, 2005

Hello,I have an XML data feed that I would like to use to create tables inSQL Server. The xml data feed consists of a large amount ofinformation that changes on a regualar basis. Is there a way toautomatically create SQL Server tables using the data feed?ThanksBilly

View 7 Replies View Related

SSIS Data Flow Parameterize Query

Jun 1, 2006

Hi,

I am using SSIS in SQL Server 2005 and want to have a query like this in my data flow task

Select a.*
from abc as a
inner join (Select max(b.id) as ID from xyz as b inner join pqr as c on b.id = c.id and b.id > ?) as t1
on t1.id = a.id


SSIS fails to detect the parameter (?) for the inner query and gives message.

"
Parameters cannot be extracted from the SQL command. The provider might not help to parse parameter information from the command. In that case, use the "SQL command from variable" access mode, in which the entire SQL command is stored in a variable.", so assuming this is your problem, then you can workaround.

"


The idea is to parameterize the inner query ,,,
(so if the above query doesnt make sense ignore it )

View 1 Replies View Related

Using Result From One Query For Data Flow Task

Oct 13, 2006

I have a table with five entries. For each row read in that table I want to pass the value

to a data flow task to be used to set the filename in a flat file connection.



Thanks.

David

View 4 Replies View Related

How To Execute A SQL Query Only Once In A Data Flow Task

Mar 13, 2007

Hi Everyone,

In the data flow task, i have thosands of rows flowing, now just before inserting these rows into a table, i want to delete some rows in the destination table. For this, if i use the oledb command, then it will run several times. I think a script can do it but I want to avoid it because it would be an inefficient affair.

Please post your suggestions on this.

Regards,

Manu





View 13 Replies View Related

Data Feed From DB2/AS400 To SQL Server 2000

Jun 13, 2001

I need to have an automated process to read data from DB2/AS400 and feed it to SQL Server 2000. Has anyone done this before? Any suggestions how it may be done? I know my company doesn't want to spend a lot to do this.

Thanks for your time.

View 2 Replies View Related

Data Feed / Make / Model Catalog

Mar 23, 2007

Does anyone know where to start if I want an exported copy of homeappliances, cars, cell phones etc.I just need to populate my database and I want to get some data feedfrom all the different types of manufactures.I think there are services out there. But I have not seen one.I want a CSV file that has all the make/models for lets say: TVs

View 1 Replies View Related

How To Pass Parameter Froon Control Flow To Data Flow

Feb 14, 2006

Hi, All,

I need to pass a parameter from control flow to data flow. The data flow will use this parameter to get data from a Oracle source.

I have an Execute SQL task in control flow to assign value to the Parameter, next step is a data flow which will need take a parameter in the SQL statement to query the Oracle source,

The SQL Looks like this:

select * from ccst_acctsys_account

where to_char(LAST_MODIFIED_DATE, 'YYYYMMDD') >?

THe problem is the OLE DB source Edit doesn€™t have anything for mapping parameter.

Thanks in Advance





View 2 Replies View Related

HELP: How Do I Pass Variables From Control Flow To Data Flow

Mar 9, 2007

I have an Execute SQL Task that returns a Full Rowset from a SQL Server table and assigns it to a variable objRecs. I connect that to a foreach container with an ADO enumerator using objRecs variable and Rows in first table mode. I defined variables and mapped them to the columns.

I tested this by placing a Script task inside the foreach container and displaying the variables in a messagebox.

Now, for each row, I want to write a record to an MS Access table and then update a column back in the original SQL Server table where I retreived data in the Execute SQL task (i have the primary key). If I drop a Data Flow Task inside my foreach container, how do I pass the variables as input to an OLE DB Destination on the Data Flow?

Also, how would I update the original source table where source.id = objRects.id?

Thank you for your assistance. I have spent the day trying to figure this out (and thought it would be simple), but I am just not getting SSIS. Sorry if this has been covered.

Thanks,

Steve

View 17 Replies View Related

Handle Tasks In Control Flow Tab From Data Flow Tab

Jan 17, 2008

Dear All!
My package has a Data Flow Task. In Data Flow Task, I use a Script Component and a OLE BD Destination to transform data from txt file to database.
Within Data Flow Task, I want to call File System Task to move file to a folder or any Task of "Control Flow" Tab. So, Does SSIS support this task? Please show me if any
Thanks

View 3 Replies View Related

SSIS Variables Between Data Flow And Control Flow... How To????

May 17, 2007

Hi everyone,

Primary platform is 64 bit cluster.

How to move information allocated in SSIS variables from Data Flow to Control Flow layers??

We've got a SSIS package which load a value into a variable inside a Data Flow. Going back to Control Flow how could we retrive that value again????

Thanks in advance and regards,

View 4 Replies View Related

Is There A Way To Set A Variable In A Data Flow From A SQL Statement (like In Control Flow)

Jan 12, 2006

I'm currently setting variables at the package level with an ExecuteSQL task.  This works fine.  However, I'm now starting to think about restartability midway through a package.  It would be nice to have the variable(s) needed in a data flow set within the data flow so that I only have to restart that task. 

Is there a way to do that using an SQL statement as the source of the value in a data flow? 

OR, when using checkpoints will it save variable settings so that they are available when the package is restarted?  This would make my issue a moot point.

View 2 Replies View Related

SQL 2012 :: Count Records From Each Separate Data Table Corresponding To Feed

Nov 5, 2014

I have 1 table that is just a list of feeds. A, B, C, D etc (15 rows in total) and each feed has other information attached to it such as Full name, location etc etc. I am only interested in the Feed column.

Each feed then has a corresponding data table which contains a list of records. Eg Feed A data is contained in TableA, Feed B data is contains in TableB and so on.

Basically what I need is a combined table that shows the list of Feeds in the 1st Column ( So A, B, C, D…..) and then a second column which counts the records from each separate data table corresponding to that feed.

So the end result would look something like this:

Feed------No of Records
A----------4 (from TableA)
B----------7 (from TableB)
C----------8 (from TableC)
D----------1 (from TableD)

Possible?

View 2 Replies View Related

T-SQL (SS2K8) :: Import Data From XML Feed To Get Daily Exchange Rates

Aug 26, 2015

I'm trying to find a way to import data from this data xml feed to get daily exchange rate. I' tried:

select *
from
openrowset(bulk 'http://www.bankofcanada.ca/stats/assets/xml/fx-noon.xml',single_blob) as x

Which is a feeble attempt at a start; however, am getting this error message:

Cannot bulk load because the file "http://www.bankofcanada.ca/stats/assets/xml/fx-noon.xml" could not be opened. Operating system error code 123(The filename, directory name, or volume label syntax is incorrect.).

How to parse this file using SQL.

View 1 Replies View Related

Please Advise: Big Control Flow Or Big Data Flow

Jul 22, 2007

Hi all! I recently started working with SSIS and one of the things that is puzzling me the most is what's the best way to go:



A small control flow, with large data flow tasks
A control flow with more, but smaller, data flow tasksAny help will be greatly appreciated.
Thanks,
Ricardo

View 7 Replies View Related

Transact SQL :: FOR XML To Write Data With Carriage Return / Line Feed At End Of Elements?

Sep 30, 2015

I am trying to use FOR XML under SQL Server 2014 to write out a large XML data set. I want it to look like

<CVS_Member_Add_Change>
    <RecordType>3</RecordType>
    <Carrier>1266</Carrier>
    <MultiBirthCode>0000000</MultiBirthCode>
    <MemberType></MemberType>

[Code] ....

That's how it looks when you click on the results of a small subset of the query.  Just what I want.  Unfortunately when you try to right click and save it you get 

<dataroot><CVS_Member_Add_Change><RecordType>3</RecordType><Carrier>1266</Carrier<MultiBirthCode>0000000</MultiBirthCode><MemberType></MemberType<LanguageCode>1</LanguageCode><DURFlag></DURFlag><DURKey></DURKey><SocialSecurityNumber>000000000</SocialSecurityNumber</CVS_Member_Add_Change>

Everything being on one line blows up the translator application that reads the data.

The FOR XML statement copied out of the query is below.

FOR XML RAW ('CVS_Member_Add_Change'), ROOT('dataroot'), ELEMENTS 
GO

Is there a way in the T-SQL to force it to break lines neatly?

Is there a way to force it to a specific file name or directory?

View 3 Replies View Related

Lookup Task Data Flow Transformation Causes Data Flow Task To Hang?

Dec 28, 2007

Hi,
I'm trying to implement an incremental data pull (Oracle to SQL) based on Andy's blog:
http://sqlblog.com/blogs/andy_leonard/archive/2007/07/09/ssis-design-pattern-incremental-loads.aspx

My development machine is decent: 1.86 GHz, Intel core 2 CPU, 3 GB of RAM.
However it seems the data flow task gets hung whenever I test the package against the ~6 million row source, as can be seen from these screenshots. I have no memory limitations on the lookup transformation. After the rows have been cached nothing happens. Memory for the dtsdebug process hovers around 1.8 GB and it uses 1-6 percent of CPU resources continuously. I am not using fast load to insert new records into my sql target table. (I am right clicking Sequence Container 3 and executing this container NOT the entire package in the screenshots)

http://i248.photobucket.com/albums/gg168/boston_sql92/1.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/2.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/3.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/4.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/5.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/6.jpg


The same package works fine against a similar test table with 150k rows.
http://i248.photobucket.com/albums/gg168/boston_sql92/7.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/8.jpg

The weird thing is it only takes 24 minutes for a full refresh of the entire source table from Oracle to the SQL target table.
Any hints,advice would be appreciated.

View 18 Replies View Related

MDX Query Results Loses Format After Data Transformation

Dec 28, 2007

Hi,

My first post here. I have an Analysis Services Cube and I need to export some of its data into a flat file (a semicolon separated value file) with fixed length columns.

What I have in Integration Services now is:

DataReader Source (ADO.Net conection)
Data Transformation (NTEXT -> WSTR)
Derived Column (WSTR -> STR, and some ISNULL validation)
Flat File Destination (Delimited by ";")

the thing is I have all the measures and calculated measures in the cube formated with format strings and they work fine when I do a query through the SQL server Managment Studio, but in Integration Services before the columns are written into the file they lose its format.

for example, I have a calculated member called "Deuda Total Nacional" that returns something like this +0001234,00 and I need to take off the decimal separator so it is written into the file like this +000123400 , I'm doing it this way

(ISNULL([Deuda Total Nacional]) ? "+00000000000" : REPLACE([Deuda Total Nacional],",",""))

instead it is written like this 1234.

I hope you can help me with this.

Thanks

View 14 Replies View Related

Stored Procedure | Selects Some Data |query The Results It Created And Output

Dec 2, 2005

I can create 1 view of a database then use that views results to query

but i want do this with a stored procedure
pass some data to it, so it selects some data

query the results it created and output this data.

ie


pass 2 values to the procedure
(
@FirstValue int,
@secondValue int output

)


select from a database with the firstpassed value

Select *
From TableName
Where ID = @FirstValue

Then
using the results from the above select

Select *
From theResultsOfAbove
Where ID = @SecondValue

any ideas would be fabo !

View 6 Replies View Related

Sampling Data Set Via Integration Services Data Flow For Data Mining Models Without Saving Training And Test Data Set?

Nov 24, 2006

Hi, all here,

Thank you very much for your kind attention.

I am wondering if it is possible to use SSIS to sample data set to training set and test set directly to my data mining models without saving them somewhere as occupying too much space? Really need guidance for that.

Thank you very much in advance for any help.

With best regards,

Yours sincerely,

View 5 Replies View Related

Need To Display Results Of A Query, Then Use A Drop Down List To Filter The Results.

Feb 12, 2008

Hello. I currently have a website that has a table on one webpage. When a record is clicked, the primary key of that record is transfered in the query string to another page and fed into an sql statement. In this case its selecting a project on the first page, and displaying all the scripts for that project on another page. I also have an additional dropdownlist on the second page that i use to filter the scripts by an attribute called 'testdomain'. At present this works to an extent. When i click a project, i am navigated to the scripts page which is empty except for the dropdownlist. i then select a 'testdomain' from the dropdownlist and the page populates with scripts (formview) for the particular test domain. what i would like is for all the scripts to be displayed using the formview in the first instance when the user arrives at the second page. from there, they can then filter the scripts using the dropdownlist.
My current SQL statement is as follows.
SelectCommand="SELECT * FROM [TestScript] WHERE (([ProjectID] = @ProjectID) AND ([TestDomain] = @TestDomain))"
So what is happening is when testdomain = a null value, it does not select any scripts. Is there a way i can achieve the behaivour of the page as i outlined above? Any help would be appreciated.
Thanks,
James.

View 1 Replies View Related

Need To Save A Value From Data Flow A To Use In Data Flow B

Mar 20, 2007

Good morning, all,

I am working on importing an Excel workbook, saved as multiple CSV flat files, that has both group level data and related detail row on the same sheet. I have been able to import the group data into a table. As part of the Data Flow task, I want to be able to save the key value for the group, which I will use when I insert the detail rows.

My Data Flow has the following components: The flat file with the data, which goes to a derived column transformation to strip out extraneous dashes, which leads to the OLEDB Destination component.

I want to save the value as a package level variable, so that I can reference it in another dataflow.

Is this possible, and if so, at what point do I save the value?

Thanks,
Kathryn

View 1 Replies View Related

Extract Outlook Contacts Data(on Public Directory) Directly In Data Flow

Jun 13, 2006

Hi everyone,

I have to extract, dayly a list of contacts on a exchange server in a table on our EDW on sql server 2005. Is it possible to get the information directly from a dataflow or i will have to developpe a script task ?

Need help desperatly !!!

View 3 Replies View Related

Data Flow Stuck In Phase The Final Commit Data Insertion Has Started

Jun 19, 2007

Hello,



I have noticed that for one of my data-flows, the process is really long during the phase "the final commit data insertion has started".

To be accurate, the process is fast until it reaches this phase. It happens often when I load millions of lines.



The extraction is done from a database SQL Server 2005 to a database SQL Server 2005, on the same server (with the SQL Server native provider).

I used a SQL Server destination but I have tried with an OLE DB destination and it is the same situation.



Why the process could be so long during this phase?

There is a way to optimised my package to avoid that?



Any idea is welcome.



Thanks.

Guillaume

View 6 Replies View Related

Data Flow Task Error To Extract Data From Sql Server To Excel

Mar 28, 2008

Hi All,

I want to export data from SQL Server2005 to an Excel spreadsheet thru "Data Flow Task". I am using OLE DB for SQL Server for the source connection and a Connection To Excel as my destination source. The Excel spreadsheet (2003) exists and has the first row with column names. I don't have any warnings before trying to execute.

The SQL datable fileds are
i) ID - Int

ii) RefID
iii) txtRemarks - nvarchar(MAX)
iv) ddlWaterLevel - nvarchar(50)

While executing the tasks, I got the error
Error: 0xC0202025 at Data Flow Task, Excel Destination [427]: Cannot create an OLE DB accessor. Verify that the column metadata is valid.
Error: 0xC004701A at Data Flow Task, DTS.Pipeline: component "Excel Destination" (427) failed the pre-execute phase and returned error code 0xC0202025.


After analysing I found in the DataFlow --> Excel destination --> Advanced Editor for Excel Destination, the default data type for txtRemarks shows as "Unicode string [DT_WSTR]". But this is supposed to be "Unicode text stream [DT_NTEXT]". Even if I change the data type in the design time, It doesn't accept.

Please do help me out.

thanks
Sanra

View 4 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved