I'm trying to replicate a SQL join across two seperate data sources in SSIS. If I were to write SQL to do this, it would be as follows:
SELECT Costs.CostRateEntryId,
Costs.UserId,
Costs.HourlyRate * 8 AS DailyCostRate,
Dates.DateKey,
Dates.ActualDate,
FROM Costs
INNER JOIN Dates ON Dates.ActualDate >= Costs.EffectiveDate AND Dates.ActualDate <= Costs.EndDate
Unfortunately, as the tables 'Dates' and 'Costs' are in two seperate SQL2005 systems, I can't really do this. I was hoping that it could be achieved in SSIS, but I cant seem to find any way that I can do a join that's <= or >=.
I need to periodically import a (HUGE) table of data from an external data source (not SQL Server) into SQL Server, with the following scenarios: Some of the records in the external data source may not exist in SQL.Some of the records in the external data source may have a different value at different imports, but this records are identified univocally by the same primary key in the external datasource and in SQL Server.Some of the records in the external data source may be the same in SQL.
Due to the massive volume of the import, I would like to import only the records which are different from what I have in SQL Server (cases 1 and 2 above). In fact case 2 is the most critical.
I thought of making a query with a left outer join between the data in the external data source table (SOURCE) and the data in the SQL Server table (DESTIN). The join is done on the respective primary keys (composed keys of up to 10 columns) and one of the WHERE conditions will be that the value in SOURCE is different from the value in DESTIN.
The result of this query would be exactly what I need to import. How to do this in SSIS??? I couldn't figure out how to join tables in different data sources yet.
In fact I cannot write a stored procedure to do that, since one of the sources is in a datasources not SQL Server. I have seen the Lookup transformation in this article http://www.sqlis.com/default.aspx?311 but this is not exacltly what I want to do. Another possibility is to use the merge join, but due to the sorting I believe its performances would be terrible!
I am attempting to use the foreach loop structure in an SSIS package toloop through however many Excel files are placed in a directory andthen perform an import operation into a SQL table on each of thesefiles sequentially. The closest model for this that I was able to findin the MS tutorial used a flat file source rather than Excel. Thatinvolved adding a new expression to the Connection Manager that set theconnection string to the current filename, as provided by the foreachcomponent. That works just fine, but when I attempt to apply the samemethod to an Excel source, rather than a flat file source, I cannot getit to work. I see the following error associated with the Excel sourceon the Data Flow page: "Validation error. Data Flow Task: Excel Source[1]: The AcquireConnection method call to the connection manager "ExcelConnection Manager 1" failed with error code 0xC020200." I think thatit's just a matter of getting the right expression, and I thought thatperhaps I should be constructing an expression for ExcelFilePath ratherthan the Connection String, but I have fiddled with it for hours andhaven't come up with something that will be accepted. Has anybody outthere been able to do this, or can perhaps refer me to somedocumentation that contains an example of what I am trying to do?Thanks for any help you can give.
I've encountered a few problems using SSIS against non-SQL Server data sources and was hoping that others might have some experience. Google searches and browsing MSDN hasn't led me to a solution, so any advice on the following is appreciated:
1) When using the "Data Source Views" wizard to add a data source view from an ODBC data source, only tables appear in the object listing. Views do not. (I've also observed this with SQL Server 2005 databases as well, but it's a bigger issue when you can't use the native SqlClient, as is the case for many ODBC-only databases.) ODBC traces show that both table and view metadata is being returned to SSIS correctly, so it appears as though SSIS is filtering out views from the object listing in the wizard.
2) When creating named queries against (non-SQL Server) ODBC data sources, SSIS appears to use the SQL Server SQL syntax for referencing schemas/objects (e.g. "SELECT * FROM [schema].[table]", rather than "SELECT * FROM schema.table"). This isn't valid SQL in many databases. Am I missing something? Is there a way to change this through some configuration setting?
>> In Integration Services, data sources are design-time objects that you use when you build packages in SSIS Designer.
This is indeed the case but why? I would like to see it changed or provide some way of including the data source in some kind of a "solution/project configuration", the same way one can create a "package configuration" right now.
If one can build a project, and create a deployment utility to deploy the entire project (all packages), it is incoceivable why one shouldn't be able to set configurations at a "project" level.
Right now I am having to create package configurations to configure my connection setting for dev/test/prod etc environments. But the trouble is, if I have 10 packages each involving 3 connections in my project, I have only 3 data sources in my project, but come deployment time, I'm having to create 10 package configurations, and set my ConnectionString (which is different for dev/test/prod etc) 30 times during deployment.
If there is a better way to do it, I would like to know. Otherwise, this is what I would like to see in the next SP.
I am new to SSIS. I would like to know if I want to transfer data from one Oracle schema to another Oracle schema and also to do scheduling of the packages, can I still use SSIS? If yes, what are the components that need to be installed on the database server and the development environment? I hope I don't need the full SQL Server database installation in order to use SSIS.
I am wanting to change from using a query to move this data to using an SSIS Data Flow. I am familiar with using Merge Join to combine the two tables (H & S in this case), but I'm not sure where I can use the ISNULL in the manner described above. Is there a way to do it in the Merge Join? Do I have to do it after the Merge Join?
I have the following scenario: N identical Databases (corresponding to different Fiscal Years, with names <Company Name>.<YEAR>). We want to consolidate the N DBs to a New Datawarehouse.
In SSIS we have designed a Dataflow that reads through a OLE DB Source (Connected to one of the N Databases) and maps to a OLE DB Destination (Connected to the NEW DB).
The question is, how we loop in SSIS through the N identical Connections, so to repeatedly execute the designed Dataflow, each time with a different Connection?
I'm just beginning to use SSIS (bracing for a steep learning curve due to lack of helpful documentation) and am starting out trying use the Import and Export Wizard. On the "Choose a Data Source" page there is a dropdown for the Data Source. I see a list of possible data providers, but not one of "Microsoft OLE DB Provider for ODBC drivers," which is the one I wanted to use because I'm trying to connect to an obscure database. So I figured that I need to use ".Net Framework Data Provider for Odbc." Unfortunately, regardless of what I enter for the Connection string or the Dsn or the Driver I invariably get an error, although it's somewhat dependent on that I have entered for those three items.
Either this (when I type in a DSN)
Cannot get the supported data types from the database connection "Dsn=Terrascan_Okanogan_WA".
or this (if I enter a full connection string and a driver)
So I have a couple questions. First, why doesn't "Microsoft OLE DB Provider for ODBC drivers" appear in the list of data sources, and secondly, when using the ".Net Framework Data Provider for Odbc" data source what inputs are expected because whatever I'm doing doesn't seem to work?
I manage to get the SSIS working. Now I would need to do these tasks.
I first want to get data from 2 different sql servers. What would be the best method to accomplish this? Both are in Sql Server 2005.
Secondly I want to make sure if any of the servers couldn't be found on the network or in any case the getting data task failed for any one of them the package won't continue and an email should be send to an email address.
Thirdly If everything is ok then I should combine both and generate one sequence no for them and save them on to another location and then generate a file with modified values.
script in SSIS to check data connections/sources and send an email if it fails validation. I found this script online, but I'm not real good with scripting and only VB scripting at that.How can I modify this script or is there another one here that will do what I need. The data connection in question links to Salesforce and will fail validation if the password/security token is invalid.
The reason I need this script is that we have no control over the password changes and need to know if the salesforce team changes the password without informing us.
/* The Script Task allows you to perform virtually any operation that can be accomplished in * a .Net application within the context of an Integration Services control flow. * * Expand the other regions which have "Help" prefixes for examples of specific ways to use * Integration Services features within this script task. */ #endregion #region Namespaces using System;
Ok I have a complex query that works great, but now I need to join yet another table but the problem is, the value that links it to the rest of the query is a value that is returned from a case statement earlier in the query...
For Example...
SELECT var1, var2, var3, CASE T1.Number WHEN 0 THEN T2.Result ELSE T3.Result END as var4 FROM TABLE1 T1 LEFT OUTER JOIN TABLE2 T2 on T2.Value=T1.Value LEFT OUTER JOIN TABLE3 T3 on T3.Value=T1.Value2
Above is a generalized, much smaller example of my working Query. Now what I need to do is join TABLE4 but the value I need to connect it to the rest of the query is var4, which can come from 2 different places depending on the value of T1.Number
Basically the logic I need to make it work is to either have a case statement in my JOIN. Like, 'LEFT OUTER JOIN TABLE4 on T4.Value=(CASE T1.Number WHEN 0 THEN T2.Result ELSE T3.Result END)'
Is this even possible and if so, how? I need to keep the logic in the main query to preserve performance...
I have a multiple table dataset that needs to be returned, with at least 5 joins, some inner, some left outer.
Currently, this is done via a parameterized stored prodedure, which is used fairly frequently. The parameters only affect the where clause, not the joins.
Would it be better to create the view with the joins already done, then pass in the parameters with the stored procedure? Which is better for overall performance? Which is better for quicker response times to the calling asp.net application?
Need a guidance on consuming the WCF service with complex type in the SSIS package. I have a WCF service with complex type inside the complex type (Nested complex type) in the web method as an argument. When I try to use this WCF service in the SSIS web service task, I get an error "The web method has unsupported arguments".
I am able to consume the WCF service with the web method having Complex type and simple/prmitive type in side that as argument.
For example:
The web method in WCF service which accepts the argument as comlex type say "Employee" whose structure is:
Employee { FName String; LName String; Age int }
It is possible to consume this WCF service and pass the arguments.
But when the Employee complex type is changed to have one more complex type in side it it give the above mentioned error. The Employee type is modified as:
Employee { FName String; LName String; Age String Type EmployeeType; }
EmployeeType { type string; }
Now I get the error Web Method has unsupported arguments.
The code below is from a nested view, which I've read should be avoided. I've also noticed GETDATE() is used, which I believe causes GETDATE() to be executed for every record selected (correct me if I'm wrong). I'm also guessing a JOIN containing a UNION against a SELECT statement is not a good idea. What other problems do you notice?
SELECT trans.Entry_Code, trans.D_C, trans.ADP_Security_# , trans.TRID, trans.Batch_Code, trans.Last_Money, null as Shares, Settle_date as Process_Date, null as Closing_Price, trans.Dwnld_Date, trans.Acnt, null as Mktval, cast(Null as varchar(20)) as Cusip_#, ACT.dbo.account.account_key AS account_key FROM (SELECT * FROM ADPDBBOOK.dbo.YTD05B WHERE (DATEDIFF(mm, Process_Date, GETDATE()) <= 15) UNION SELECT * FROM ADPDBBOOK.dbo.YTD06B) trans INNER JOIN ACT_DATA.dbo.account ON ACT_DATA.dbo.account.account_key = RIGHT(trans.Acnt, 5) INNER JOIN tbl_Accounts_TransactionalData ON trans.Acnt = tbl_Accounts_TransactionalData.Acnt
Hey all, Here is what I would like to do.. if any can give some advice or direction with this i would appreciate it.
We have a "master database" (MS SQL) at our main office. Then a smaller office with only a dial-up connection. Can I have my db application write all the data input to a local DB. Then write something to connect with the master DB, through a Terminal server, etc., and update it with the data from the other smaller db?
I've got a customer who thinks their data should be sepeated and reside on two seperate databases in case one is compromised. Now I've never heard of anyone doing this and would like comments from the users here as to why this should/(shoult not) be done.
I'm assuming that they want to keep their general info e.g. name and address on one database and keep other (medical info) on the other db.
None of our db's are outside of the firewalls so to me this makes no sense but I would like feedback.
I have a project were I will have it so that users can sign in and change information on an SQL server. The catch is that this site will be from a different domain name and from a different hosting company then where the SQL database is located. Sorry if this is a dumb question but how can I utilize asp.net to change and view an SQL database that is located else-where. For example: a user logs into www.something.com and he/she can view and edit SQL tables from www.somethingelse.com's database. Thanks in advance.
I'd like to create a view on server x which references tables on an entirely seperate server. Is this possible? Is seems strange to have to copy the tables over just to create a view. In the view wizard I can't seem to 'browse' to the tables on the other server.
The code I'm working with would conceptually be something like this:
select server name.database instance.owner.table.field
Hello everyone, Here's my situation... I'm running a web service which involves 51 seperate servers and databases. There are fifty licensee servers (One for each US state) and one corporate server. Each night I need to upload sales and membership data from the licensee's databases to the corporate database to compile reports. The application platform I'm using is ASP.NET 2.0 and the the database is SQL2005 express. I want this process to be run automatically, so I believe it's a scheduled windows service I need to setup up in .NET to make the data transfers. If anyone has already set something up like this, or knows the steps to take? I would love to have your input. Thanks in advance, Robert
I am pretty new to SSIS. I am trying to create a package which can accept data in any of several formats. i.e. CSV, Excel, a SQL Server database/table and import the data into my destination database.
So far i've managed to get this working OK. However I am now TOTALLY stuck. I'm currently trying to just concentrate on the data sources being a CSV (using a Flat File Data Source) and/or an Excel Spreadsheet.
I can get the data in and to my destination using a UNION ALL component and mapping the data sources to it so long as both the CSV file and the Excel spreadsheet exist.
My problem is that I need my package to handle the possibility that only the CSV file might exist and there is no Excel spreadsheet. In which case i'd like the package to ignore the Excel datasource completely. Currently either of my data sources do not exist I get errors and the package terminates.
Is there any way in SSIS that I can check all my data sources to see which ones exist (i.e. are valid). If they exist I want to use them. If it doesn't exist i'd like to disgard it (without error - as long as there is a single datasource the package should run)
I've tried using the AcquireConnection method in a script task on each of my connections, hoping that it would error if the file/datasource did not exist. It doesn't though (in the case of an Excel datasource it just creates a empty excel file for me).
The only other option I can come up with are to have seperate packages depending on the type of data we want to import and then run a particular package depending on the format of the source data. This seems a bit long winded. I am pretty sure I must be able to do what I want to achieve but I can't work out how.
I'll be grateful to anyone who can send me any tips/hints/links on how I can achieve this.
Hi, i'm wondering which is the best way to search data in a SQL Server. I reach data using Data Sources and Data Views and also with OLE DB Source with a Data access mode: Named query. I have to write the data into a Flat File. So, does any one knows which is the best practice for this? Or any one of the two are good choices? Thanks for your help.
In my project i want a report. In that report data is getting from more than one data sources(systems). While creating data source view i used named query for both primary and secondary data source. But at the time of crating "Report Model" i am getting below error.
An error occurred while executing a command. Message: Invalid object name 'Table2'. Command: SELECT COUNT(*) FROM (SELECT SerialNum, ModelNum AS com_model FROM Table2) t
Is there any way to create a report with multiple data sources?
I'm wondering if SSIS will be the solution to the problem I'm working on.
Some of our customers give us an Excel sheet with data they want to insert or update in the database.
I've created a package that will take an Excel sheet, do some data conversion so the data types match up and after that I use a Slowly Changing Data component to create the insert/update commands.
This works great. If a customer adds a new row to the Excel sheet or updates an existing row changes are nicely reflected in the database.
But now I€™ve got the following problem. The column names and the order of the columns in the Excel sheet are not standard and in the future it could happen a customer doesn't even use an Excel sheet but something totally different.
Can I use SSIS for this? Is it possible to let the user set the mappings trough some sort of user interface? I€™ve looked at programmatically creating the package but I€™ve got to say that€™s quit hard to do€¦ It would be easier to write the whole thing myself than to create the package trough code ;)
If not I thought about transforming the data in code before I pass it on to the SSIS package in something like XML. That way I can use standard column names and data types.
So how should I solve this problem? Use SSIS or not?
Hi i have data on a Server in a different database which i like to access from within my ssis job.
I just need to look up information from one table on this database so i can references it. Is there a way of doing this is SSIS. Rather then me having to load the data from one database to another as the data may change.
i tried having 2 sources of data feed into a look up but that does not work..
On my home page I have several different folders to reports which require different data sources. the problem is that within these folders there are multiple copies of the same datasource. is it possible to store all of the datasources in one folder, one location? it would certainly be easier when changes to usernames and passwords need to be modified!
I searched and read about Data Sources and I'm seeing that there is no advantage in using it, which is what I found from playing around with it.
I expected that you would set a global connection in Data Sources and somehow link this to the things in your Connection Manager, giving you one place to switch from one environment to another. But reading the discussions here and playing around with it, this is not the case.
So, why is it there?
Next question.... another thing I gathered so far is something called "Configurations" that will do what I was describing above. Where do I do this?
I'm trying to combine two reports I've created into a form letter. The goal is to have a letter that indicates the students maximum UBSCT score for Math, Reading and Writing. Also in the letter we want to include the student's total number of credits awarded and overall GPA. I not sure if I need to try and combined the two queries or use two different data sources in Reporting Services. I've tried creating one sql statement but the grouping are giving me fits. I fairly new to Reporting Services.
Report 1:
704 Dixie High 11 311880 Student_1 UBSCT Math 176 704 Dixie High 11 311880 Student_1 UBSCT Reading 182 704 Dixie High 11 311880 Student_1 UBSCT Writing 173 704 Dixie High 11 311881 Student_2 UBSCT Math 168 704 Dixie High 11 311881 Student_2 UBSCT Reading 172 704 Dixie High 11 311881 Student_2 UBSCT Writing 165
SELECT track.schoolc as School_Code, school.schname as School, stustat.graden as Grade, studemo.ident as Ident, RTRIM(studemo.lastname) + ', ' + RTRIM(studemo.firstname) AS [Student], stutscors1.testc as Test_Code, left(zsubtest1.descript,20) as [Subject], max(stutscors1.testscore) as Score
FROM stutscors1 INNER JOIN stutests1 ON stutscors1.ststuniq = stutests1.ststuniq INNER JOIN studemo ON stutests1.suniq = studemo.suniq INNER JOIN stustat ON studemo.stuuniq = stustat.stuuniq INNER JOIN track ON stustat.trkuniq = track.trkuniq INNER JOIN school ON track.schoolc = school.schoolc INNER JOIN testdef1 ON stutests1.testuniq = testdef1.testuniq INNER JOIN zsubtest1 ON stutscors1.subtestc = zsubtest1.subtestc and stutscors1.testc = zsubtest1.testc
WHERE stutscors1.testscore <> ' ' and stutscors1.testscore <> '0' and school.schname = 'Dixie High'and stutscors1.testc = 'UBSCT' and stustat.graden = 11
GROUP BY track.schoolc, school.schname, studemo.ident, RTRIM(studemo.lastname) + ', ' + RTRIM(studemo.firstname), stustat.graden, stutscors1.testc, zsubtest1.descript
704 Dixie High 11 Student_1 311880 23.2500 23.2500 87.50300000 3.763569 704 Dixie High 11 Student_2 311881 20.2500 20.2500 76.84300000 3.794716
select stugrp_active.schoolc as School_Code, school.schname as School, stugrp_active.graden as Grade, rtrim(stugrp_active.lastname) + ', ' + rtrim(stugrp_active.firstname) as Student, trnscrpt.suniq as Ident, sum(trnscrpt.gpacratt) as CreditAtt, sum(trnscrpt.gradcrawd) as CreditAwd, round(sum(case when Trnscrpt.GpaCrAtt is null then 0 else Trnscrpt.GpaCrAtt end * gpamarks.gpavallvl0),3) AS TrmGpaPts, (round(sum(case when Trnscrpt.GpaCrAtt is null then 0 else Trnscrpt.GpaCrAtt end * gpamarks.gpavallvl0),3))/(sum(trnscrpt.gradcrawd)) as GPA
from dbo.trnscrpt inner join dbo.stugrp_active on (trnscrpt.suniq = stugrp_active.suniq) INNER JOIN school ON stugrp_active.schoolc = school.schoolc INNER JOIN gpamarks ON trnscrpt.marksetc1 = gpamarks.marksetc AND trnscrpt.markawd1 = gpamarks.mark
where school.schname = 'Dixie High' and stugrp_active.graden = 11 and trnscrpt.graden >= 9
group by stugrp_active.schoolc, school.schname, stugrp_active.graden, rtrim(stugrp_active.lastname) + ', ' + rtrim(stugrp_active.firstname), trnscrpt.suniq
