Copy Files Dynamically
Jan 11, 2008
hi , I am trying to copy files dynamically from a remot folder by using File System task ,
the files are going to be generated daily with f1yyyy-mm-dd.txt
I created 3 file connections :
- Existing remote folder (RemotFolder) and set that to var v_remotefolder
- Existing destination folder (DestFolder)
- existing file (filename) and set the expression to var (string ) =v_fname
in the File System task I set the the
Destination connection = DestFolder
Operation =Copy file
Source = filename
expression >>Property (source) = @[User::v_remotefolder] + @[User::v_fname]
I know I need the date too ,but I 'm getting error on this step.
Can you please tell me if you know how to resolve this without using For each loop since I don't have control on the remote folder and I don't want to copy unneeded files (the files are 10 GB daily) and get delete it once every month.
-I know this not uncommon task but I can't find any example.
Thanks
View 3 Replies
ADVERTISEMENT
Feb 28, 2008
Hi!
I did:
alter database mydb set single_user with rollback immediate;
exec sp_detach_db @dbname='mydb', @keepfulltextindexfile='true';
then I tried to copy files to new location on other drives, same server but got
>>Cannot copy <myfile>: Access is denied
Make sure the disk is not full or write-protected and that the file is not currently in use<<
I also tried rename of file without success.
I also tried with db service stoppet (not preferred) without success.
How to find out, which process locks the files?
Best regards
View 7 Replies
View Related
Nov 27, 2014
Need to scheduled to copy table data from one database to another database daily @7:00 AM by creating the table with date and time stamp.
Example :
Test1DB ---> Table1
Test2DB -----> Table1_281120140700 in Day1 and Table2_291120140700 in Day2.....and so.
SELECT * INTO Test2db.dbo.new_table_name FROM Test1db.dbo.old_table_name in this format ??
script and scheduled to run daily @ 7:00AM?
View 1 Replies
View Related
Oct 27, 2006
I need to get a list of customer ids and then use them as a parameter to select from a transaction table and then create a file for each customer. I have used an execute SQL task to get the list of customers and have put the result set into a variable.
How do go through the recordset to create a file for each customer?
View 1 Replies
View Related
Mar 10, 2006
I've just started using SQL Server 2005 Integration Services and
I've come up against a situation where I need to name output files
dynamically (i.e. based on a timestmap). Looking through this
newsgroup, and other web resources it looks like my only option is to use a Script Task/ActiveX Script Task to rename the file after it has
been created with a generic file name.
I was wondering if there was a different approach or if there was a way to pass in a variable to the file connection manager that I could append to the file name at run-time.
Thanks
Bill
View 7 Replies
View Related
Nov 29, 2007
I am pretty new to SQL Server 2005 and SSIS. I am trying to develop a package that will dynamically load files into SS2005 based on the contents of a configuration table. The configuration table (see below for example) contains the path to the file, a flag indicating whether or not to process the flag, the type of file (specifies the nature of data -- financial, order, etc.) and some parameters specific to each file.
FileName
ProcessFlag
Type
ExcelTab
Param1
C:File1.xls
TRUE
1
Sheet1$
C:File2.xls
TRUE
1
Sheet1$
C:File3.xls
TRUE
1
Sheet1$
C:File4.xls
TRUE
2
Sheet1$
C:File5.xls
FALSE
2
Sheet1$
C:File6.csv
TRUE
3
C:File7.txt
TRUE
4
Right now I basically have a seperate sequence for each of the file types. The task in each sequence are virtually identical with the exception of the the data flow source in the data flow task (since the source file could be .xls, .txt, .csv). The first sequence ran fine in isolation, but when I linked a second sequence I started getting a Package Validation Error:
DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER
I use the same variables (FileName, ExcelTab) for each of the sequences, so I am not sure if that causes te error. I also tried changing the ValidateExternalMetadata setting to false, since the connection variable wont be applicable until that sequence is being processed. I am not sure where to go here ... should I rearchitect how the package all together?
Is there a better/most efficient way to to architect this package to handle the different file types (develop a package for each type which is called by a master package, create one package with a different sequence container for each type, etc).
Any help would be greatly appreciated!!
View 5 Replies
View Related
Nov 12, 2007
I am pretty new to SQL Server 2005 and SSIS. I am trying to dynamically load files into SQL Server 2005 using files/paths contained in tables. I have a key table in my SSIS package that defines which files should be processed, some default values that will be associated with each file (ie CompanyKey and PeriodKey) and the file path (see below for example):
CompanyKey
PeriodKey
ProcessFlag
FileName
2
61
FALSE
2005TB200501.xls
2
62
FALSE
2005TB200502.xls
2
63
FALSE
2005TB200503.xls
2
64
TRUE
2005TB200504.xls
2
65
TRUE
2005TB200505.xls
2
66
TRUE
2005TB200506.xls
2
67
TRUE
2005TB200507.xls
2
68
TRUE
2005TB200508.xls
2
69
TRUE
2005TB200509.xls
2
70
TRUE
2005TB200510.xls
2
71
TRUE
2005TB200511.xls
2
72
TRUE
2005TB200512.xls
The package I am developing is supposed to loop through this table checking for files with ProcessFlag = TRUE. For those files, it will load all of the records/columns (prefaced with the company and period keys) into a common table.
Do I have to manually create data sources for each file or is there a way to dynamically define the connection and process the connection?
Any assitance you could provide would be greatly appreciated!
Thanks.
Vik
View 3 Replies
View Related
May 22, 2008
Hi,
There is one manual process done by my friend. The process is " In query analyser, she runs a sql query and the resultset of Sql query, she copies and pastes into a excel workbook".
I wanted to write a SSIS package to automate it and providing a UI, in which she can give connection details and paste the SQL query and the package will in turn generate excel workbook in the target path". I thought of trying ExecuteSQL task, but the output resultset of ExecuteSQL task cannot be mapped dynamically to excel source(correct me if I am wrong). If I use Dataflow task, then also whether dynamic mapping of SQL output to Excel destination input is possible?
Can you please tell me the best approach for achieving the above functionality?
Thanks,
Venkataraman R
View 5 Replies
View Related
Nov 16, 2011
Every day an application creates new tables and dumps static info into them.Â
I would like to create a package to dynamically export those database tables to raw files for long term archive, one file per table. Here is what I have so far and the issue I am having.
1)Â Â Â Â Â Get a list of un-archived tables.
2)Â Â Â Â Â Foreach table do the following.
a. Export the table into raw file.
b. Zip the raw file.
c. Update archive tracking table.
As long as the metadata for each table is the same this package seems to work fine. However, I have many tables with different metadata. How can I dynamically get the package update the metadata column collection when it hits a new table? When it hits a table with different metadata I am getting warnings like this:
The column "some_column" needs to be added to the external metadata column collection.
The "external metadata column "someother_column" (103)" needs to be removed from the external metadata column collection.
Then I get this error:
Error: 0xC004706B at dump the table into a raw file, DTS.Pipeline: "component "OLE DB Source" (1)" failed validation and returned validation status "VS_NEEDSNEWMETADATA"
View 7 Replies
View Related
Sep 29, 2015
I have developed an SSIS package which extracts and creates 5 flat files and finally using Process Extraction task zip the folder. On my Dev environment everything is working fine but when I am moving to SIT and UAT, not able to set up jobs dynamically by importing XMLConfig file.I created variables and assigned values but still it doesnt take.Below are varaibles I created for flat file destination, Arguments and Working Directory (for zipping)On UAT when I go to SQLAgentJobs to set, import .dtsx file, XML config file....the new values doesnt appear. why ?DataSource is taking always dev location....why ? How can I set it up to take dynamic values what I mentioned in config file ?
View 14 Replies
View Related
May 29, 2015
how do you load the multiple flat files  to into destination dynamically?
View 9 Replies
View Related
Mar 24, 2007
I copied a database using backup and restore from Sql server 2000 to Sql server 2005. I noticed the data(mdf) and log(ldf) files did not copy. when I try to do a copy and past it gives me an error: cannot copy lof: it is being used by another person or program. Close all programs and try again. Does this mean I have to stop both servers to copy between them? Please help.
View 1 Replies
View Related
Dec 20, 2007
Hello, everyone:
I want setup a scheduled job to copy txt files from FTP to local hard drive. How to open FTP and copy files from SQL Server Query Analyzer?
Thanks
ZYT
View 5 Replies
View Related
Feb 5, 2008
I have been looking for a way to copy only new flat files from an FTP server. I can find how to copy all of the files into another directory, but I only want to copy in files that are not already in my target directory.
So, is there a way to compare the files in the FTP directory to a list of files in a SQL table and use this to control the transfer, and how do I get those FTP file names into a SQL table? Or is there some other way?
Thanks in advance for any help.
Graham
View 4 Replies
View Related
Nov 18, 2007
Howdy, hope someone can help me out with this.I want to run a job each night that copies files from one server to another.I cant even get a simple copy one file from one directory to another,statement to work.When I try :@cmd 'copy c: empfile1.txt c:ackupsfile1.txt'master.dbo.xp_cmdshell @cmdI get the msg: 'c' is not recognised as an internal command, program orbatch file, Null.What I ideally want to do is pass xp_cmdshell variables for the 2 file'spath+names. I have tried numerous variations of single and double quoteswithout success, which is why I decided to work up form the simple 'copyc: empfile1.txt c:ackupsfile1.txt', but can't even get that to work.Any advice much appreciated.
View 3 Replies
View Related
Jul 20, 2005
Hi,I'm having some trouble finding any information on this topic - can MS SQLserver 2000 copy/delete files on the host computers disk.Any info on this would be great.Thanks,Eirik
View 2 Replies
View Related
Mar 24, 2007
I copied a database using backup and restore from Sql server 2000 to Sql server 2005. I noticed the data(mdf) and log(ldf) files did not copy. when I try to do a copy and past it gives me an error: cannot copy lof: it is being used by another person or program. Close all programs and try again. Does this mean I have to stop both servers to copy between them? Please help.
View 5 Replies
View Related
Jun 19, 2000
I was trying to copy backup files from production domain to test domain nightly using xp_cmdshell. I
View 3 Replies
View Related
Aug 10, 1999
I connect to remote servers using PPTP connection. There are 4 web servers and 1 production server on the other side of the firewall. From my staging server, I am supposed to archive the NT event logs(Application and system) for each of the servers.
To solve this problem, I have mapped the C drives of each of the remote servers to my local staging server. I then created a batch file, webevnt.bat containing code,
G:
Cd winnt
Cd system32
Cd config
Copy sysevent.evt c:eventarcweb01sysevent.evt
Copy appevent.evt c:eventarcweb01appevent.evt
Then I used SQL Agent to create a job of type, Operating system command(cmdexec) and typed the following command :
C:atch_~1eventl~1webevnt.bat
When I run the command, C:atch_~1eventl~1webevnt.bat through the command prompt, the files are copied with the correct modified date.
The output is,
1 file(s) copied
1 file(s) copied
But, when I use SQL Agent and start the job containing the same command, the job runs successfully but, the files have a wrong modified date. Moreover, when I view the job history for this job, it says, specified drive not found. 1 file(s) copied. 1 file(s) copied.
Please let me know why this happens.
Thank you.
Praveena
View 1 Replies
View Related
Jan 29, 2006
hi.
how can I copy any mdf file to another folder without stopping sql server?
View 8 Replies
View Related
Dec 14, 2005
Hi,
I'm fairly new to SQL, waiting on a course on how to use it, but wanting to get stuck it.
I was wondering how I can get SQL to copy a file, say in Access, from one location to another.
This would be part of my job with delivering data extracts users.
Many thanks in advance
View 5 Replies
View Related
Jul 21, 2015
I was wondering if there is a way to copy files from a network to an ip site where you would need to log in at. What I am trying to do is make a SP that would take all .csv files from the network drive and place them in the other ip address folder however you would need to use the login information.
View 5 Replies
View Related
May 15, 2007
I have a situation from where I need to loop through different folders and files in these folders. After processing these files, I need to archive these folders to different location.
e.g., C:MainFolderMar01 ==> Multiple files in Mar01 folder
C:MainFolderMar02 ==> Multiple files in Mar02 folder
Does any one know the best way to do this in SSIS?
Thanks in advance.
BC
View 1 Replies
View Related
Mar 14, 2008
Hi,
How do I copy files from several subdirectories into the same directory? I have about 80 subdirectories of a parent directory from which I want to copy the files and place the copies into the same, flat destination directory. I tried using the copy file and copy directory file system tasks, but this always seems to recreate the subdirectory structure under the destination directory.
How can I get around this in SSIS?
Thanks in advance,
Bruce.
View 6 Replies
View Related
May 8, 2007
In SSIS, I was to:
take files from c:directory1
copy them to cirectory2 with a different name (concatenated date on the end)
delete them from c:directory1
Should I be using the Script Task for this? I am wondering if I should be using the File System Task which copies directories or files, but I don't know if I could rename the files during the copy.
Also, I want the directory names to be in the configuration file. I am not sure how to do this. If I set up a flat file source connection, I need to specify a file name and I just want the directory names. How do I get them in the config file and how to I read them into my script from the config file?
Linda
View 25 Replies
View Related
Apr 27, 2006
This is a question of whether or not to use SSIS to solve a problem.
I need to copy SQL Server database backup files from a server in the DMZ to a fileserver inside the firewall. The SQL Server is not allowed to write it's backup files directly to the fileserver, so they are written to local disk. A connection can be made from inside the firewall to the SQL Server to copy the files off.
So, I'm considering SSIS for the job. Is it possible to use SSIS to perform the file copies from one remote server to another? If so, is the FTP task required, or can File System tasks be used?
An alternative would be Windows scripting, xcopy, robocopy, etc. but I like the features of SSIS and would like to take advantage of it's flow control, error handling, database scheduling, etc.
Any tips, sample code, etc. would be appreciated.
Thanks,
david
View 4 Replies
View Related
Mar 20, 2008
I am looking to implement a system whereby a given user can 'authorise' changes to the system itself and promote from test to live.
For instance a user might be running an application which has a SQL 2005 database DataA opena nd runnign stored procedures from that. I'd like a Stored procedure in that able to copy over a given stored procedure from a different database - DataB and copy into DataA. Is that possible?
Also is it possible from T-SQL to copy files on the servers underlying filesystem? Ie to copy some .aspx files from one location to another.
Regards
Clive
View 12 Replies
View Related
Sep 23, 2014
I've an emergency requirement to copy Source server database backup files to destination Server through xcopy command. Backup job on source server runs daily, so once this job get completes all databases backups needs to be moved to destination server. But here the main concern is "the backup files on destination server shouldn't be overwritten, they should be placed separately as Source server job runs daily".
We've a command which overwrites backups on destination server. But we need to keep backups on destination at-least for 4 weeks (means : retention should be 4 Weeks).
View 5 Replies
View Related
Feb 4, 2015
I have one job for copy files from another server to my computer using Robocopy.but when I execute my job in sql server I get this error:
Accessing Source Directory 192.168.101.191Upload
Access is denied.
this is my job:
USE [msdb]
GO
/****** Object: Job [Test] Script Date: 02/04/2015 13:08:00 ******/
BEGIN TRANSACTION
DECLARE @ReturnCode INT
SELECT @ReturnCode = 0
/****** Object: JobCategory [[Uncategorized (Local)]]] Script Date: 02/04/2015 13:08:00 ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)
[Code] ....
Should I create Credential and proxy?How can I do this?Would you mind giving me required scripts for this?
View 2 Replies
View Related
Nov 29, 2007
I need help with better solution to copy my bak and trn files to a file server. This is the background. I use to dump the backup files directly to the file server; this was not €œbest practices€? according to some books and forums. Back to dump the file first to local disc and after that letting robocopy copy the files to the file server. And I end up with this problem.
This is error from the backup log
Failed-1073548784) Executing the query "BACKUP DATABASE [BPROJCT] TO DISK = N'F:\backup\BPROJCT\BPROJCT_backup_200711281700.bak' WITH NOFORMAT, NOINIT, NAME = N'BPROJCT_backup_20071128170005', SKIP, REWIND, NOUNLOAD, STATS = 10
" failed with the following error: "Write on "F:\backup\BPROJCT\BPROJCT_backup_200711281700.bak" failed: 112(There is not enough space on the disk.)
This is the error from the robocopy log
New File 0 BPROJCT_backup_200711201700.bak
2007/11/20 17:02:11 ERROR 32 (0x00000020) Copying File F:ackupBPROJCTBPROJCT_backup_200711201700.bak
The process cannot access the file because it is being used by another process.
Waiting 30 seconds... Retrying...
Robocopy tries to copy the file before its completely written to disk. I have made a batch file that looks like this
robocopy
"F:ackup" "\sefwg-sqlbackackupACON" *.* /MIR /TBD /MON:1 /MOT:5 /LOG+:"F:ackupackup.log" /NP
Monitor the file folder and copy all files every 5 min, the file is started with scheduled Task.
Have I missed some switch? Can start robocopy from SQL Serer Agent in a xp_cmd_shell?
View 1 Replies
View Related
Sep 19, 2007
Hello,
We're new users of SQL Server 2005. I created two maintenance plans...one to backup the database at 2 AM daily and one to back up transaction logs every 30 minutes. These maintenance plans write to a local disk. What we want to do, within the maintenance plan, is copy the files as soon as they are written to a remote server. Is that possible?
Thanks in advance.
View 7 Replies
View Related
Feb 10, 1999
We are having a problem with trying to backup the database device and log DAT files located in the MSSQLData directory.
The Seagate Backup Exec. states that the files are busy and skips them during its backup cycle. It skips all the devices in the
directory.
Any suggestions?
View 3 Replies
View Related
Sep 27, 2007
Hi All!
Microsoft SQL Server 2005 - 9.00.3159.00 (Intel X86)
Mar 23 2007 16:15:11
Copyright (c) 1988-2005 Microsoft Corporation
Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2)
40 subsribers.
Adding new article.
On any aubscribe need regenerate snapshot agent agent.
The tables are very big. With filters.
Regenerating process anew copy data in in ...ReplDataame.bcp, during 30-40 minutes for any subscriber!
Full work to small correct of replication flow during 2-3 days....
Is it possible to initialize subscription without copying bcp data?
View 5 Replies
View Related
