I've two SQL databases, the first one having the schema named sch1 and the second one with the schema named sch2. I need to copy data of more tables belonged to the sch1 into the sch2 of the second SQL Server db and I'd like to create a SSIS 2012 pkg in order to achieve this goal.
View 2 Replies1. Take a subset of data from about 100 tables that have multiple references to other tables in this group of 100 from a first DB.
2. Insert the above data into a second DB, a database that already has data in the 100 tables, while maintaining the correct references.
As a general approach, the best way I can think of doing this is as follows:
1. Create mapping tables for every ID that is referenced in a different table (OldID NewID)
2. Insert the old data into the new table and output the OldID and NewID into the mapping table.
3. Use that mapping data to make sure all tables that use those IDs have the new IDs in DB2.
This approach is extremely labor intensive both on initial implementation and would require a fairly substantial amount of work to maintain going forward.
Hi,
I have the following scenario: N identical Databases (corresponding to different Fiscal Years, with names <Company Name>.<YEAR>). We want to consolidate the N DBs to a New Datawarehouse.
In SSIS we have designed a Dataflow that reads through a OLE DB Source (Connected to one of the N Databases) and maps to a OLE DB Destination (Connected to the NEW DB).
The question is, how we loop in SSIS through the N identical Connections, so to repeatedly execute the designed Dataflow, each time with a different Connection?
Thanks in Advance,
--Dimitris Doukas
I am trying to copy the data from Oracle to SQL, it is taking 10 mins to load only 50K records of data. I am using only one DFT task.
In the DFT task I am using 2 tasks oracle Source and OLEDB destination .
what can I do to improve the ETL process and reduce the load time ?
We've had a new server set up with SQL 2012 and I'm in the process of moving data to it from a 2008 (SP2) server.
Details are as follows:-
2012 instance:- Microsoft SQL Server 2012 - 11.0.5058.0 (X64)
May 14 2014 18:34:29
Copyright (c) Microsoft Corporation
Standard Edition (64-bit) on Windows NT 6.3 <X64> (Build 9600: ) (Hypervisor)
2008 instance:- Microsoft SQL Server 2008 (SP2) - 10.0.4000.0 (X64)
Sep 16 2010 19:43:16
Copyright (c) 1988-2008 Microsoft Corporation
Standard Edition (64-bit) on Windows NT 6.1 <X64> (Build 7600: ) (VM)
I don't want to do a backup/restore routine as there are collation conflicts on the 2008 server.I've created the database and tables on the 2012 instance and now I want to transfer the data from the 2008 instance to the 2012 one.
The 2012 instance has a linked server to the 2008 instance.I was trying to use sp_MSForEachTable (I know, it's old and will probably disappear shortly) but that doesn't seem to work properly because some of the columns have an Identity field set up.
Some of the tables have upwards of 10 million records in them and are quite sizeable.how I can achieve the transfer without a back-up/restore?
Greetings,
I have two SQL Server tables on the same server and in the same database. I'll call them table A and table B. They have identical schemas. I need to insert all rows in table A into table B. (Don't laugh - this is just for testing and long run the tables will reside on different servers.)
Can someone please tell me the correct task to use for this and the connection type I need for both the source and destination?
Thanks,
Black Cat Bone
I have a SQL Server 2012 Enterprise Edition install that I am needing to do the following:
- Provide db_owner access to a single user. Have the db_owner privilege apply for the user in all Schemas of the database (this is a generic AD account that is being used for an application).
They are wanting me to do this because they do not want to get so granular as to tell me all of the objects that the user will need specific access to in any of the given Schemas of the database(s).I have 8 databases, and each one has additional Schemas. These are all contained in a Default instance.I am finding that without explicitly applying permissions to the various objects in the Schemas that are not the default Schema for this user, I am a bit stuck.Is there any way to have the db_owner permission apply to the single user for all the Schemas in a given database (without making them a Sysadmin)?
Currently, I am using T-SQL to combine some data and using an ISNULL function like this:
,ISNULL(H.Last_Name, S.PatientLastName)
,ISNULL(H.First_Name, S.PatientFirstName)
I am wanting to change from using a query to move this data to using an SSIS Data Flow. I am familiar with using Merge Join to combine the two tables (H & S in this case), but I'm not sure where I can use the ISNULL in the manner described above. Is there a way to do it in the Merge Join? Do I have to do it after the Merge Join?
We have SQL Server 2012 running on Windows 2008 Server. We need to copy five databases from our 'sandbox' to our test server and then to our production server. The database backup file sizes are 3 MB, 20 MB, 344 MB, 645 MB and 17 GB. We are planning on using the backup and restore method since we already have full backups and the scripts to recreate the logins/users/permissions. We believe this method provides more flexibility and control over the process. However, we have a few jobs, maintenance plans and ssis packages.
To get the jobs to the new server instance, the plan is to script out the jobs on the 'sandbox' and execute the scripts on the test server instance. Is this the best or only way to handle the jobs?How to get these maintenance plans to the new server instance? (There is no 'script out' maintenance plan option.) We may have to just recreate them on the new server instance. Is this the best or only way to handle the maintenance plans?
We have a few ssis packages. How to handle getting the ssis packages over to the new server instance (using the backup restore method)? These packages use the Project Deployment Model. Therefore, should we restore the SSISDB or open up the package file using VSS on the new server instance and change the connection information to point to the new server instance.
Just wondering if there is any reason or advantage to use the Detach and Attach method or Copy Database Wizard method? I have read where the Copy Database Wizard method handles the database's dependent objects like logins, jobs, maintenance plans, user-defined error messages and shared objects from the master database. Are there any other move/copy database methods to consider? Just trying to make sure we have thought out everything and are using the best method to copy our databases over to another server instance.
Using server 2012 on local machine, I created an SSIS package that will execute in integrated services and Visual Studio solution but will not work when creating a job. Other solutions work well except when exporting data. The program pulls data from query and exports into .csv file. The messages I get are -
Data flow task 1:error:destination- Stage.csv failed the pre-execute phase and returned code 0xC020200E
and
Data flow task 1:error:Cannot open the datafile "pathStage.csv".
Version-
Microsoft SQL Server Management Studio11.0.3128.0
Microsoft Analysis Services Client Tools11.0.3128.0
Microsoft Data Access Components (MDAC)6.1.7601.17514
Microsoft MSXML3.0 6.0
Microsoft Internet Explorer9.11.9600.17041
Microsoft .NET Framework4.0.30319.18444
Operating System6.1.7601
I have requirement of sending table data or result of an query as an email with SSIS package.
View 6 Replies View RelatedI'm looking at building an SSIS workflow to integrate new XML product data from our supplier (they're switching to XML instead of the Current MS Access database they use). They're using a schema based on the OAGIS structure. When I try to load a Data Flow Task with the XML file, to attempt to parse it, and I specify the XSD schema they have provided, SSIS errors out by telling me:
TITLE: Microsoft Visual Studio
------------------------------
Error at Data Flow Task [Data Import [1]]: The component "Data Import" (1) was unable to process the XML data. Substitution Groups not supported. See element "Dimensions" with "substitutionGroup=http://www.openapplications.org/oagis/9: Dimensions".
------------------------------
ADDITIONAL INFORMATION:
Pipeline component has returned HRESULT error code 0xC02092B0 from a method call. (Microsoft.SqlServer.DTSPipelineWrap)
------------------------------
BUTTONS:
OK
------------------------------
Is there any way at all to get around this, or am I screwed? As I said, this data is provided to us by our supplier - we have no control over it, but I am looking to pull the data from XML and put it into a regular SQL table which I am in the process of defining - we have a large traffic e-commerce site and obviously cannot use XML as the "back-end".
I need to copy all the data from all the tables in a database to a copy of this database on another server.
What feature of SSIS should I take advantage of to accomplish this?
We have an SLA for 8am, most times the data warehousing jobs complete at 8:05am. Adding an additional process/set of tasks to this package would obviously make matters so I'm trying to update/copy/replicate the data in the fastest manner. Typically we're talking 2 marts (10-20GB) with 2 large tables (5-10 mill records) and 20 marts (0.5 - 5 GB) with many more smaller tables (~40 tables with record count ranging from 1 to a million)
Additionally please indicate if the design/feature you suggest can handle (pushing schema changes and additions to the target server) schema changes or new tablesviews added to the source database.
My only idea so far...is using the import wizard (in Management Studio) to create an SSIS package (top copy all the tables from one server to another) and saving it to the server, Then executing this package after the job is complete. However this would not work if the schema of a table changed, or if a a table is added. Moreover I don't think I can edit this package in visual studio.
I am transferring data from oracle and getting below error message.
I using 4 data flow tasks with in a single control flow and all the 4 tasks quueries same table but populates data in to different sql tables based on the where contidion
[OLE DB Source 1 [853]] Error: An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Oracle" Hresult: 0x80004005 Description: "ORA-01652: unable to extend temp segment by 64 in tablespace TEMP ".
What iam trying to do is put the table name from one db into a variable and another one into another variable and pass them into my statement basicaly trying to bulk copy data from one table in a db and insert it into another db on the same server based on a condition
if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[ClientComment]')
and OBJECTPROPERTY(id, N'IsUserTable') = 1)
Declare @BDFR varchar(20), @BDTO varchar(20), @EQID varchar(20)
set @BDFR = 'Commander' + '.dbo.' + 'ClientComment'
set @BDTO = 'Test_Commander' + '.dbo.' + 'ClientComment'
set @EQID = '80_300_005'
insert into @BDTO
select * from @BDFR where Eqid = @EQID
Hi there;
i've recently designed a new db for my portal website, and wanna move
the old data from former db to the new one. The problem is that
for the 'News' table, for instance i have two different schemas:Old >>News(id,title,summery,text,date, ... , (old columns) , ...)New >> News(id, title , summery , text , date, lastUpdated, ... , (new columns) , ...) i
want to copy the bolded columns, and want to have 'lastUpdated' &
'date' in the new table to be the same as 'date' in the former one.as i googled, i just found materials on copying dbs to production servers, or cross db scenarios. thanks in advance
Hello I am developing a web application that will allow users to upload a .mdb file and from that file I need to populate an SQL database. I know the table name of the .mdb file, but I am unclear how to structure my data access layer correctly. How do I pull data from the .mdb file and once I have it how do i populate the SQL database?Any advice would be greatly appreciated.thanks!
View 1 Replies View RelatedI'm having an issue with a job that calls a stored procedure. The SP is fairly complicated, but basically creates a temp table, then inserts the records from the temp table into a table on a linked server (via VPN). The basic premise here is that I'm trying to update a web-accessible server from the production server. The funny thing is that the process works correctly for 4 out of 5 companies (company is the only parameter fed to the SP). The company that fails is by far the largest, but the process has worked in the past.
I'm not strong with all the tools available in SQL Server that might help me figure out what's going wrong. In the SQL Server Agent, it simply tells me that the job failed. If I use Query Analyzer and run the SP in Debug mode (if I'm stating that correctly), I get this error for any company I try it for, even companies that run fine (line 51 is the DELETE line):
Server: Msg 7391, Level 16, State 1, Procedure procStmtDataToReno, Line 51
[Microsoft][ODBC SQL Server Driver][SQL Server]The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction.
So I suppose my long-winded question is, how can I determine why this SP fails for the one company? If relevant, the job is simply:
EXEC procStmtDataToReno 'B'
and some snippets of the SP:
CREATE PROCEDURE [dbo].[procStmtDataToReno]
@CoCode as varchar(2)
AS
--various declarations and such
DELETE
FROM [10.1.1.4].OnlineAR.dbo.StatementData
WHERE CompanyID = @CoCode AND (Line1Date < Convert(varchar(10), GETDATE() - 60, 101) OR Line1Date >= Convert(varchar(10), GETDATE() - 5, 101) )
--code to build temp table
INSERT INTO [10.1.1.4].OnlineAR.dbo.StatementData
SELECT * FROM #Statement
GO
TIA
I need to see inside a SSIS 2012 project a new SSIS installed component, but in the SSDT 2010 I cannot see the SSIS Data Flow Items tab for adding data source/data destination respect to the choose toolbox items pane.
View 4 Replies View RelatedHi all,
I got a situation here.....
From a source table (in SERVER1) I get ids of candidates and from another source (in SERVER2) I get their CVs (text files stored in various Folders). My destination table (in SERVER3) has two fields, CandidateId & CandidateCV.
I have to transfer the data in above fashion for nearly 1 million records.
How can I write a DTS package which picks up the text file from SERVER2 based on the CandidateId which comes from SERVER1? Probably I need some kind of looping mechanism which changes the candidate id & his CV file.
Can anyone help???
Thanks...
An SSIS package to transfer data from a DB instance on SQL Server 2005 to SQL Server 2000 is extremely slow. The package uses an OLEDB Source to OLEDB Destination for data transfer which is basically one table from sql server 2005 to sql server 2000. The job takes 5 minutes to transfer about 400 rows at night when there is very little activity on the server. During the day the job almost always times out.
On SQL Server 200 instances the job ran in minutes in the old 2000 package.
Is there an alternative to this. Tranfer Objects task does not work as there is apparently a defect according to Microsoft. Please let me know if there is any other option other than using a Execute 2000 package task or using an ActiveX Script to read records from one source and to insert them into the destination source, which I am not certain how long it might take and how viable will that be?
Any inputs will be much appreciated.
Thanks,
MShah
When copying data to a remote SQL2K5 destination from a SQL2K5 source database, both using mixed sql server security mode, my job generates the following error:
[Transfer SQL Server Objects Task] Error: Execution failed with the following error: "Cannot apply value null to property Login: Value cannot be null..".
This occurs after the destination database tables have been truncated and replacement data from the source would begin to copy.
The same process can be successfully completed from the Management studio with a simple data export process. However, when I run the saved package again from the BI interface, I get this error.
My search engine searches have yielded numerous hits of others having the same problem with one microsoft rep indicating it was a bug and would be resolved in sp1. I am working with sp1. Oddly, there is only mention of this in the forums. No KB article from MS addresses the problem and I do not see it addressed elsewhere at sqlservercentral.
It appears that others have switched to Integrated Security and resolved the problem. However, I do not have that option with a remotely hosted database.
Does anyone have any information concerning this problem?
I have 2 SQL server 2000 machines, I need to take a table from each one and combine them together based on a date time stamp. The first machine has a database that records information based on an event it is given a timestamp the value of variable is stored and a few other fields are stored in Table A. The second machine Table B has test data entered in a lab scenario. This is a manufacturing facility so the Table A data is recorded by means of a third party software. Whenever a sample is taken in the plant the event for Table A is triggered and recorded in the table. The test data may be entered on that sample in Table B several hours later the lab technician records the time that the sample was taken in Table B but it is not exact to match with the timestamp in Table A. I need to combine each of these tables into a new SQL server 2005 database on a new machine. After combining the tables which I am assuming I can based on a query that looks at the timestamp on both Tables A & B and match the rows up based on the closest timestamp. I need to continuously update these tables with the new data as it comes in. I havent worked with SQL for a couple of years and have looked at several ways to complete this task but havent had much luck. I have researched linked servers, SSIS, etc Any help would be greatly appreciated.
View 10 Replies View RelatedI'm working on a project where I need to build a small database and then copy it to a server at the client's site. I can't connect directly, so I have to use a VPN connection and use Remote Desktop, copy the database backup from my machine to the cloud, then download it to the client machine. The project is still in the early stages, and the client is still sending me data in CSV files and Excel spreadsheets. I'm periodically needing to do a complete refresh of the database at the client. I've hacked my way through it a couple of times, but I need to know the proper way to do it. I get errors on the restore step, telling me the file is in use.
View 8 Replies View RelatedSource System : Oracle
Target System : Ms Sql Server 2012
ETL Tool Used : SSIS
My source data is present in XML File which is stored in CLOB column Of Oracle. CLOB column is compressed.I need to Migrate data by Uncompressing XML to SQL 2012 .
Do I need to define XML column in SQL Server 2012 for storing Uncompressed CLOB values ?
How to uncompress the clob and extract the required data from XML using SSIS .
In cases when the SSIS package has to be deployed in SQL Server, when we use ProtectionLevel ="server storage" while creating SSIS Catalog, does it use TDE? or Database Standard Encryption? if the database is using TDE?
View 0 Replies View RelatedIn cases when the SSIS package has to be deployed in SQL Server, when we use ProtectionLevel ="server storage" while creating SSIS Catalog, does it use TDE? or Database Standard Encryption? if the database is using TDE?
View 5 Replies View RelatedI am a complete newbie to SSIS. I can create a simple package to transfer data between SQL instances and thats about it.
I have tableA (source data) and tableB (Destination data). TableA has 4 column and tableB has 5. I want to transfer all of the columns from tableA into TableB, but the 5th column in tableB needs to be populated with the ServerInstance name of the server TableA sits on. Do I need to have multiple data sources to achieve this? I have tried but no matter how I set it up, the Column in the destination is set to ignore.
I would like to know how I can add the following sample code to my Source data on Data Flow on SSIS, or what other options there are. The main issue is time as we have talking about 100's of millions of rows
select Sample,
CASE
WHEN Sample IS NULL
THEN NULL
WHEN SUBSTRING(Sample, 1, 6) IS NULL
THEN ' '
ELSE RTRIM(SUBSTRING(Sample, 1, 6))
END AS [Sample_1_6]
from TestTable
what I have done at this stage is just to Create a SQL task with a Insert into
INSERT INTO [dbo].[TestTable1]
([Sample]
,[Sample_1_6])
select Sample,
CASE WHEN Sample IS NULL =THEN NULL
WHEN SUBSTRING(Sample, 1, 6) IS NULL THEN ' '
ELSE RTRIM(SUBSTRING(Sample, 1, 6))
END AS [Sample_1_6]
from TestTable
If there is a way adding this to a dataflow so I van use fast load that would really be the best solution. I know there are derived columns, but would this really be faster than the straight insert into in a SQL Task? If this is the way to go what is the code I would use in the derived column or any other option.
Hi All,
I created a SSIS package in my local computer to load data from flat file and convert data type and then load them into another table.
Then I checked in the package into a shared server so that another person can access it. Then now he wants to chage source and destination file locations. Only the location is different. But when he changes the file location, It does not execute. Other machine SQL Express is running instead of SQL Developer version.
Where we went wrong? Is there any special way to copy SSIS packages?
If so, can you explain it step by step?
Is there any way to copy SSIS packages from one mirrored server to another? I'm considering using robocopy, but is there another solution? I'd then like to use this solution with the Transfer jobs task.
View 4 Replies View RelatedIs there a way to create a table in SQL Server Management Studio under a different schema? I've tried setting my default schema and adding a table, but that didn't work. I know I could do it with a SQL statement, but wanted to know if I could do it with the designer.
View 1 Replies View RelatedI am using VS2012 and creating a package on a 64bit machine to import some data from a .xlsx file. My question is that I am getting an error for the Excel connection manager, do I need to install some kind of excel drive or excel itself on the machine in order to be able to import the data?
View 6 Replies View Related