Is it possible to set up at dataflow with for example an excel source that points to a sheet with varying number of colums.
In the file there is some information about number of columns. But the question is here how you use that information to define the columns in the dataflow ?
Dear Friends,
I need to execute a SQL query, inside a dataflow (not in controlFlow) and need the records returned to continue the dataflow... In my case I cant use lookup and OLE DB COmmand and nothing else...
I need to execute a query and need the records for dataflow... with OLE DB command I cant see the fields returned... :-(
How can I do it? Using a script? Can I use a Script Component? That receive 2 parameters for input and give me the fields returned from query as output?
Thanks!!
I have the below CTE that I just can't seem to get to give me the right results. Basically what im trying to do is use the first query to show the "sources" that are involved in each inquiry and the second query to show which of those have became "admissions" the thing is the counts of the sources when the CTESource query is ran alone is different than my query to join the two tables.
Code:
With CTESource(Total, ID, Source, Program) AS
(
SELECT count(Inquiry.ID) as Total, Referral.InquiryID_fk, Source, Inquirer.Program from Referral
Inner Join Inquiry on Inquiry.ID = Referral.InquiryID_fk
Inner Join Inquirer on Inquirer.ID = Inquiry.InquirerID_fk
[code]....
The total inquiries can be higher than the source totals since a source isnt required in the system as well as there does not have to be admissions regardless of inquiry count.
HI all,
I implemented Pull snapshot replication about 3 weeks ago on a live server. Everything has been working very well and I really have no complaints. However, I am currently working towards optimization, which brings me to my question.
I have noticed that the snapshot agent will vary in the duration of time it takes to generate the snapshot.
Here is the job history:
9/25 - 3.48min
9/24 - 17min
9/23 - 17min
9/22 - 3.3min
9/21 - 17min
9/20 - 17min
9/19 - 3.39min
The distribution agent currently retains history for 12 hours and the snapshot agent runs once every 24 hours so I don't think this variation in performance could be related to a cache issue.
There are no sql jobs that are scheduled during this timeframe and both severs, Publication and Subscription, are decent boxes with very little load during the replication window.(currently 1am - 2am)
Any ideas why this variation in snapshot generation exists?
Thanks,
Michael
I have the following table:
Policy Sequence LossDate Description Amt
123 1 9-1-2012 property 100
123 2 10-1-2012 property 200
345 1 9-1-2012 property 100
345 2 10-1-2012 property 100
678 1 9-1-2012 property 100
I want to select the data as follows:
Policy LossDate1 Description1 Amt1 LossDate2 Description2 Amt2
123 9-1-2012 property 100 10-1-2012 property 200
345 9-1-2012 property 100 10-1-2012 property 100
678 9-1-2012 property 100 NULL NULL NULL
Is there a way to do this using T-SQL?
The max sequence number can vary, so far the max has been 12, but may be higher in the future.
trivial example, i have a tabular model for orderlines.Each line has a product.A product fits under a hierarchy Category > Product.I need to display the top N rows by quantity under each category. My can break down as follows: Warehouse, Â Product category, Product, PurchaserID
If im to do the top N in excel i use pivot table functions to get the top N by whatever measure i pick from the dropdown:I cant do this in powerbi builder though, as it works very differently, or powerview which again is done differently.Do i have to take a different approach depending on the report platform, Â or is there a way to do this in the tabular model that will work on any platform?
Hello, all
We are trying to work out a memory problem associated with a SQLCLR procedure we have developed.
This procedure will run properly on a SQLExpress box, but fails with an Insufficient Memory error (701) when run on our 32-bit enterprise edition server on W2K3 Enterprise Server. The server has 3.25 GB of RAM, and does *not* have the /3GB switch enabled; AWE is *not* enabled on SQL Server.
I initially suspected the problem was due to the idiosyncracies of how the BPool is allocated on SqlExpress and its 1GB buffer pool limitation, giving a 1GB MemToLeave region for SQLCLR allocations -- substantially larger than the default available on the server with it's default 256MB allocation (+128MB for thread stacks). That seemed to explain why a memory problem might not manifest itself on the client, but would on the server. I then altered the server to include an explicit "-g" parameter at startup, reserving 512MB for MemToLeave, but the query still fails with the same memory errors. I've tried values as high as 1GB (-g1024), but none have worked Our DBA has installed a recent hotfix associated with certain memory errors, to no avail.
It appears that the server will max out the CLR allocations at 102MB, regardless of the presence of the -g parameter. Could it be that the MemToLeave region really is that severely fragmented? There are still log messages indicating failures to reserve virtual memory...
One thing I notice is that the buffer's VM Reserved value never seems to change regardless of the value of the "-g" option. I would expect to see it decrease by the amount specified as reserved, eg -g 768 should leave, for example, something like 1.2GB VM Reserved for buffer pool allocation, but it doesn't. That tells me I'm not really establishing a larger MemToLeave area. Is this expectation incorrect?
Bottom line, we don't know why this procedure is running on the server, but not in SqlExpress. It is a procedure that connects to a remote Oracle database and collects about 14,000 records, but dies after sending about 7,000 or so records when run on the server. Again, the same query running against the same code on a SqlExpress box works perfectly.
Any thoughts or suggestions welcomed and greatly appreciated.
David
I am using CROSS APPLY instead of UNPIVOT to unpivot > one column. I am wondering if I can dynamically replace column names based on different tables? The example code that I have working is based on the "Allergy" table. I have thirty more specialty tables to go. I'll show the working code first, then an example of another table's columns to show differences:
select [uplift specialty], [member po],[practice unit name], [final nomination status]
,[final uplift status], [final rank], [final uplift percentage]
,practiceID=row_number() over (partition by [practice unit name] order by Metricname)
,metricname,Metricvalue, metricpercentilerank
[code]....
Rheumatology Table:The columns that vary start with "GDR" and [GDR Percentile Rank] so I'm just showing those:
GDR (nvarchar(255), null)
GDR Percentile Rank (nvarchar(255), null)
GDR PGS (nvarchar(255), null)
GDR Rank Number (nvarchar(255), null)
PMPM (nvarchar(255), null)
[Code] ....
These are imported from an Excel Workbook so that's why all the columns with spaces for now.
Hi,
I'm wondering if it is possible to make an update of a specifics rows in a database table using dataflow tasks
thanks
When i use tablename or viewname variable in datasource component and data determination component ,
how can i manage the output columns and the input columns?
Yes,i can use default value init the columns ,but when variable value changed,error occurs.
Thanks a lot!
Hi Pals,
Here is my scenario in my ETL process, I have one DataFlow task.
Assuming that i have 10 clean records in my source database and i need to load all the 10 recs into my target table.
IS there any means of cross checking the no of rows from source table and number of rows loaded into my target table.
Any suggestions are greatly appreciated.
Thanks & Regards.
Hi all,
In my DataFlow i set the "OLEDB Source" which is a table in my Extract Server and need to do some transformations and stage the table which will be a Dimension in the staging DB,
Q1-Now i need only 3 columns from the Source table, which transformation do i need to use to just extract the the 3 columns?
Q2- Two Columns of 3,which i will need to transform as it is-no changes at all and One of the column which has values like "BOSTON...."
(I have a vague idea of what i need to do,need something solid suggestions/advices to kickoff,plan is to use this city column with a Replace function (as one of the forum member's Spirit1 adviced..thanks..!!))to take out the dots and then need to write a condition if BOSTON then Assign Code "BOS" which will be City_Code and this "City_Code" will have to be looked in City_Dimension to get the "City_Key_Number" for "Boston" and lastly the City_Code and City Key Number both have to be transformed to the destination Dimension.
Any ideas /suggestions will be appreciated.
Thanks in advance...!!
ravi
Greetings!
I am attempting to implement the following case statement BEFORE getting the data in to my destination table but I don't know how to create an expression for it.
In the mapping section of my OLE DB destination component I can only do mapping but I can't actually manipulate the data before it gets to the destination table.
What do I have to do to implement :
case
when SOPD.PRICE_TOP_NUMBER is NULL then -8
else SOPD.PRICE_TOP_NUMBER
end AS price_top_number,
before it goes to the destination column?!
Thanks for your help in advance.
Hello every one,
I have a simple dataflow from source to target.
My source is raw file and target is oracle table.
The problem I€™m facing in SSIS is its talking more than 10 minutes to load almost 30-40k records, I do not have any transformation activity this is just a simple dump from source to target.
Please do inform me of any setting to be talk care for fasten the process.
Thank you
I know the idea was to seperate workflow and dataflow, but I have come across a scenario where it would be useful for a branch of a dataflow to wait until another branch has finished.
I have some transactional data which records events for the start and end of a session. I want to build a list of unique sessions with the start and end date. I currently have the list of events sorted by time, followed by a conditional split for the start and end events. I can then insert all of the start events and would like to wait until all of the starts are inserted before updating them with their relevant end times.
Is this achievable?
Does anyone else think it would be a good idea to be able to set precendence across multiple branches of a data flow?
Does anyone have a better solution?
I know this is the wrong forum, but is there a way to model this against the transactional data in SSAS, I will move this question to the SSAS forum if anyone can think this would work!
Philip Coupar
This error seem to be very silly.did anyone come across this error.
I have been transferring data from textfile to a table using oledb destination.
The number of records in the text file are 2,091,650
Its was running just fine couple of days ago when the incoming data was little small then this...(arround 300,000).Now it seem to have a problem.
Here is the flow
1.File System task ->I copy the file to different location
2.Execute sql task->truncate tables
3.DataFlow task->I check for only the error files in this data flow.and all valid rows i transfer to a different text file.
4.Dataflow->filesource i connect to new text file created earlier.Here i convert fields to repective datatype and i insert if new or update record.
I dont know whats going on...
When i run my package it runs through the first three perfectly fine.When it comes to fourth step it sits there.....it dosent go to the tasks within this dataflow at all...and begining i have flat files source...
What could be the reason...
when i look at progress tab...i was able to look at the progress of other tasks but when it comes to this task it shows start>>>>>time and it sits there...
I am transfering data from a textfile to sql server.I use a data flow task for trasfering my text files.
Here is what i do.
1.Add text file source
What i want to achieve here is if the text file countains the column name in the first row i should delete them and if it does not contain column name in the first row just transfer it.
how can this be achieved???
2.add one more column to my text file which should contain the status(insert or update).
how can this be done??
3.before transfering data ot destination i want to know if the record exists if exists i just want to update it instead of insert.and if new record i want to insert it .and the status in the above new column need to change.
please help...
Hi all,
I have encountered a SQL 2005 deadlock issue while executing dataflow in a SSIS package. The deadlock happens when I have indexed two columns. If I don't have index, deadlock does not happen.
Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.".
The above causes the rest of the dataflow execution to be terminated.
What my dataflow does is to extract data from 14 flat files and then insert the records into a single table (no primary key, but with two columns indexed).
Can anyone please advise how I can avoid deadlock with indexes in a table?
Thank you and much appreciated!
Greetings!
I am painfully learning SSIS and just when I think I have a better understanding of how things work and go down the path that I should take, I hit a wall. Frankly, discouraging and demoralizing.
I am performing a Select of rows that are then passed on to a ForEachLoop with an ADO Enumerator with a FullResultSet.
Variable mappings for the columns used are:
Variable IndexAccountNumber 0
AccountName 1
AccountAddress 2
CountryCode 3
A ScriptTask in the foreach loop allows me to confirm that I am processing rows.
A DataFlow in the ForEachLoop has in it a DERIVED COLUMN item that allows me to assign the four variable fields above to their respective DERIVED COLUMN.
An attempt to use a Flat File Destination to process the derived columns is being made. As I monitor the execution of the package I notice the following message.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "Flat File Destination" (382)" wrote 0 rows.
Can anyone PLEASE tell me why is it that I am able to see the data being processed through the script task? And yet not see any rows being written to the Flat file. Why?
Any information you may provide would be immensely appreciated.
Thank you and God Bless.
HI, I have a lookup that find a specific row. If that row does not exist, I need to create a new one. If it exists, I need to raise an error and trap it with the "on_error" event of the dataflow in order to log it. Is there a way to raise an error and specify the error message from the dataflow? I was thinking using the conditional split and verify if the value returned from the lookup (by set the error config to ignore the error) is not or not. But how can I raise an error when the value is not null?
Thank you,
Ccote
Hi:
I am getting the following error when I start debugging my Package, I am not sure what this is related to, but basically, input (datatype is a int, and its mapped to a column which is also int), so I am not sure whats happening here. The input column is actually a derived column, and its set as a 4 byte un-signed int, please advice on where should I start looking to troubleshoot this issue. This loanapplicationid is actually a user variable that is utilized by other tasks in my control flow as well:
Error: 0xC020901C at InsertApplicationCL5, OLE DB Destination [16]: There was an error with input column "LoanApplicationID" (1161) on input "OLE DB Destination Input" (29). The column status returned was: "The value violated the integrity constraints for the column.".
The point is, i want to calculate the max id of a table using Aggregate Transformation, then insert some rows with a OLEDB Command and finally , with another OLEDB Command select those rows with id >(max_id) calculated before.
How can i get a value that was calculated before? Can i store it in a variable?
Many Thanks!
Hello
Trying to figure out a clever solution for splitting multivalued columns out into n-columns.
For that I've build a custom component in SSIS using
ms-help://MS.VSCC.v80/MS.VSIPCC.v80/MS.SQLSVR.v9.en/dtsref9/html/4dc0f631-8fd6-4007-b573-ca67f58ca068.htm as an example.
I need to be able to add columns to the OutputCollection in designtime, but the designer returns an error:
Error at Data Flow Task [Uppercase [5910]]: The component "Uppercase" (5910)
does not allow setting output column datatype properties.
How do I enable the designer to accept designtime changes in the
columncollection?
Kind regards
HI, I know that we can get package informations from msdb.dbo.sysdtspackages90 once packages are published into SQL Server. Is it possible to retreive informations for dataflows also from msdb tables?
Thank you,
Ccote
Hello all,
I got a text file with two columns. and I need to generate a integer key automatically with the row number (or any distinct number, I thought row number will be OK). and when I make the data flow task to import this text file into a raw file I need to get the unique rownumber as Id.
How can I make this in the data flow tak??
regards,
I was transfering more that 100,000 records from flat file to sql table
It took about 1 hour.Is this the way it is?????i used oledb command.
As the data passes by i got to insert to several table.Like i insert some of incoming data to one table then get the key from that table and insert rest of the data with the key field from previous table to another table.
In this case i felt OLedb would be best as we can use query.
I cannot use oledb destination as it has only error output(to insert some of incoming data and i want to have a look up to get the key but oledb des has only error output)
i cannot use sql destination as the database is sql server 2000.It dosent let me.
How can i increase the performance????Please let me know
I created a dataflow that transferred about 1 million records from a SQL database on one server to a differend SQL database on the same server. The processing took about 30 minutes. I used the Fast Load option.
I then created a "Execute SQL Task" and wrote a "SELECT * INTO TABLE" and this processing took about 30 - 60 seconds.
Can someone tell me why creating a Data Flow Tak would take so much longer or give differences between the two options above? Can someone give some pointers on how to make a Data Flow task more efficient?
Thanks.
I was wondering what has worked for all of you in regards to using a sproc as a source within a dataflow. I have had limited success doing this, but maybe there is a workaround I'm unaware of. Basically, using a SQL command in an OLE DB Source, I run an EXEC statement that returns a resultset from a stored procedure. I've noticed that depending on how the sproc is structured, I will either get metadata info in the columns tab of the OLE DB Source or not. Without this metadata of course I can not link it with a destination, since the destination believes that no data is being returned, even if this is not the case. This all seems to depend on the "final" select statement in the sproc being at the very top of the sproc. If it is not at the top, the columns tab will not be populated. Has anyone else had similar issues? Is there a workaround other than populating a temp table outside of the dataflow?
View 14 Replies View RelatedI am trying to recode a SQL2K DDQ into an SSIS Dataflow. I have no issue recoding it in SSIS except I am not sure that I am doing it correct way.
My DDQ has source,destination, update/delete/insert statements and few lookups. I have used activex script for the transformation because I have atleasst 10 If conditions. For every "if condition" the destination columns are popuated with different lookups/source columns and constant values.
Now When I start doing it with SSIS I have to use at least 10 Conditional split. and then at least one lookup,one OLEDBCommand and one OLEDB Destination for each of them . that brings my count of DF objects to 30-40
It makes my data flow to complex with two many objects. Earleir I could do whole of this in one sngle DDQ. It makes me think if I am doing it the correct way. should I be using Activex Script Task to these kind of activity.
Any advice would be appreciated.
Cheers,
siaj
I written a SSIS package to import a table from one database to another database. I used dataflow task with oledb source and oledb destination with fastload. For 2 million records its taking 5 min . The same import using DTS I am getting in 2 mins. DTS package is more faster than SSIS package ?. any reasons why SSIS is taking more time?
View 4 Replies View RelatedNeed help regarding ssis dataflow task
I need to create a ssis package. I want to import the data from a flat file to a table.
Lets say, the table has 5 columns -- col1, col2, col3, col4 , col5.(Assume that all columns can be NULLABLE) The datafile contains the data related to only three columns say col1, col2, col3. So when I use dataflow task to import the data from the file to the table, I will only get three columns, col1, col2, col3. Columns col4, col5 will be NULL.
However, I want to populate columns col4, col5 with some values which are stored in the variable.
IS there any way to do this??
Any help would be appreciated.
Thanks
Does anyone have a nice script that can "Propercase" the values of a column in the dataflow. I was thinking of a Transform Script Task, with a Input and Output Column, which can be transformed.
With Propercase i think:
STEVEN SPIELBERG becomming Steven Spielberg
Maybe also with some exceptions soo that "DELIVERY EU" stays "Delivery EU"
I have issue where based up a value in a column i need to do some processing of the previous and current row. The dataflow is also already sorted. I tried creating a Script Data Flow Transformation to do this but it isn't working right and the debugging of it sucks. Would anyone know of the best way to do this? or some helpful pointers? I tried "firing" information to help debug but doesn't help when the error message i get back is a stack overflow message.
An example of what I'm trying to do is process the sorted incoming rows for each person. Each person can have multiple rows. Based upon a "status" column in each row do some different processing on the previous or current row. Some Psuedo code:
if prev.PersonID = current.PersonID
if status = 1
change prev.PersonDate to today + 60 days
if status = 2
change current.PersonDate to prev.PersonDate
change prev.PersonDate to today + 1 day
else
send rows to output
Any comments or suggestions or helpful advice/critique would be MUCH appreciated!