Database Or File System?
Mar 11, 2008
Hello all,
I've been asked to build a .net web application, where users will be able to upload, search and view images. There will probably be about 2000 - 10000 users who will upload and view photos each day.
Each image that is uploaded, is saved as a thumbnail, a small image, and the original. So considering the possible amount of data being uploaded, stored, searched for and viewed, whats the best way to store the images? Database or File system? Whats would be the fastest option for a web application? Should I even be looking at a different solution?
cheers!
Matty
View 2 Replies
ADVERTISEMENT
Jun 19, 2007
I have recently designed and built my first database using SQL server 2005 express. I have included an image (BLOB) column in one of the database tables. This is a bad idea according to some experts, and some say it is OK!
I am currently carrying out a trial with just 3 pictures via Visual Basic 2005 express forms, and there is no problem so far as the images are displayed for each record. But I anticipate between 300 - 1000 images for the table, and this could pose real problems for SQL server 2005 express and Visual Basic 2005 express, I guess.
I have just been reading that the cost of storing large images in the database is too high! I have also read it's better to store images (BLOB) into the file system because it is cheaper to store them no matter how many there are.
But the question is how I can reference an image in this path: C:PictureProductGrocery 0052745.jpg in the database table, so that when I select a record Visual Basic 2005 forms the image is displayed accordingly, similar as when stored directly in the database table? Your help very much appreciated.
View 3 Replies
View Related
Mar 4, 2007
i am creating/uploading a new file on the webserver, and if it is successfully i want to insert a record in the database (with the filename).is there a way to create a transaction for this so that if either operation fails they both fail?
View 2 Replies
View Related
Jun 2, 2015
I have a well-structured but also very large binary data-set that is generated by a C++ application every five minutes. The data needs to be accessed by SQL applications. Since data is generated every five minutes, performance is key, both for write and read. The data set is about 500MB.If data is written to the file system, the write performance doesn't involve SQL server. For reading it, I have a CLR to read the portions of the data that I need based on offset and length. That works and is very fast. The problem is that data is stored in the file system, so it is not self-contained within the database.
A second option that I haven't explored yet, is to write the data into a table as VARBINARY(MAX). I would read the data using SUBSTRING with appropriate offset and length. Performance of SQL write/read of binary data of this size, and whether there is a third option I haven't thought off. I'm using SQL Server 2014.
View 5 Replies
View Related
Sep 2, 2007
Greetings, I have just arrived back into the country (NZ) and back into ASP.NET.
I am having trouble with the following:An attempt to attach an auto-named database for file (file location).../Database.mdf failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.
It has only begun since i decided i wanted to use IIS, I realise VWD comes with its own localhost, but since it is only temporary, i wanted a permanent shortcut on my desktop to link to my intranet page.
Anyone have any ideas why i am getting the above error? have searched many places on the internet and not getting any closer.
Cheers ~ J
View 3 Replies
View Related
Dec 29, 2006
Hello there,I just want to ask if storing data in dbase is much better than storing it in the file system? Because for one, i am currenlty developing my thesis which uploads a blob.doc file to a web server (currently i'm using the localhost of ASP.NET) then retrieves it from the local hostAlso i want to know if im right at this, the localhost of ASP.NET is the same as the one of a natural web server on the net? Because i'm just thinking of uploading and downloading the files from a web server. Although our thesis defense didn't require us to really upload it on the net, we were advised to use a localhost on our PC's. I'll be just using my local server Is it ok to just use a web server for storing files than a database?
View 6 Replies
View Related
May 14, 2015
I'm copying files to a folder with the naming convention as follows in the source folder:
CM_ABC_MY_TEST.txt
In the destination folder, this filename needs to appear as:
CM_XYZ_MY_TEST.txt
In my File System Task, I'm pretty sure I'm going to need an expression with a replace, substring, etc. But am having a hard time nailing down the exact syntax.
View 10 Replies
View Related
Aug 18, 2006
Does anyone know how to do this using variables? Everytime I try it, I get the
Error: Failed to lock variable for read access with error 0xc00100001.
I also tried it writing a script and still the same error. If I hard code the values into the variables it works fine but I will be running this everday so that it will pull in the current date along with the filename. So the value of the variables will change everyday. Here is my expression:
@[User::Variable] +(DT_WSTR,4) YEAR( GETDATE() )+"0"+(DT_WSTR,2) MONTH( GETDATE() ) + (DT_WSTR,2) DAY( GETDATE() )
The result:
C:Documents and SettingsmroushDesktopOSU20060818
the 20060818 part will change everyday ie.(tomorrow will be 20060819, next day 20060820 and so on.)
View 6 Replies
View Related
Oct 9, 2006
I am having an issue with the File System Task.
I was wondering if there is a way to 'Move File' with the File System Task inside of a For Each Loop container but to dynamically set the Destination path variable.
Currently, this is what I have:
FileDestinationPath variable - set to C:TestFiles
FileSourcePath variable - set to C:TestFiles
FileNameAndLocation variable - set to blank
For Each Loop Container €“ Iterates through a folder C:TestFiles that has .txt files in it with dates in the file name. Ex: Test_09142006.txt. Sets the file path (fully qualified) to the Variable Mapping FileNameAndLocation.
Script Task (within For Each Loop, first step) €“ Sets the FileDestinationPath to the correct dated folder within C:TestFiles. For example, if the text files I want to move are for the 14th of September, it takes FileDestinationPath and appends the date folder to the end of it. The text files have a date in the file name (test_09142006.txt) and I am picking this apart (from FileNameAndLocation in the For Each Loop) to get the folder date. (dts.Variables(€œUser::FileDestinationPath€?).Value = dts.Variables(€œUser::FileDestinationPath€?).Value & €œ€? Month & €œ_€? & Day & €œ_€? & Year & €œ€?) which gives me €œC:TestFiles 9_14_2006€?.
File System Task (within For Each Loop, second step) €“ This is where the action is supposed to occur. I want it to take the FileDestinationPath and move the FileNameAndLocation file (from the For Loop) into this folder for each run.
Now as for my problem. I want this package to run everyday but it has to set the FileDestinationPath variable dynamically according to that day€™s date. Basically, how do I get this to work since I can€™t hard code the destination path variable from the start? I have the DestinationVariable on the File System Task set to the FileDestinationPath variable, after the script task builds it. However, using FileNameAndLocation as the SourceVariable on my File System Task tells me that the €œVariable €œFileNameAndLocation€? is used as a source or destination and is empty.€?
Let me know if I need to clarify further€¦...I may be missing something very simple. Any help would be greatly appreciated!
View 10 Replies
View Related
Jun 8, 2007
I am able to run SSIS packages as SQL Server Agent jobs with a Control Flow items "File system task", if I move a file (test.txt) from a drive (c on the server (where SQL Agent jobs run) to a subdirectory on the same drive. But, if I try to move a file on a network drive, the package fail.
What I can do to solve this issue.
Bye!
Daniel
View 1 Replies
View Related
Jun 19, 2015
Historically I've always written a VB script to copy a file from a sharepoint library. I don't like this method because I have to input a username & password in the script and maintain a config file.
Yesterday I was playing around with using a file system task. The sharepoint file has a UNC path so why not? I created a simple test package with a single file system task that copies the sharepoint file (addressed via UNC) to another network location. Package runs fine locally.
I try running on our utility server but am getting a "The file name [SHAREPOINT UNC PATH] specified in the connection was not valid" error. Package is running with a proxy on the server and the proxy account has the same permissions to the sharepoint site (so far as I can tell) as me.
View 0 Replies
View Related
Sep 5, 2006
I have used the copy database wizard, but I realized I had forgotten to shrink the transaction log file. So I canceled the wizard. My database, detached by the wizard, has now disappeared. The mdf file is still there, but when I try to attach it manually I get the "create file encountered operating system error 5 while attempting to open the physical file..." error.
Any way I can recover it?
Thanks.
View 4 Replies
View Related
Nov 9, 2015
I have created a File System task which is contained in a Foreach Loop Container. I have .bak files that are populating a directory from a maintenance backup plan.
There is a point where I need to delete the .bak file's after I've zipped them all up.
How do I set the SourceVariable to read through the directory and pick up just the .bak file's in the directory to delete.
View 3 Replies
View Related
Jan 31, 2007
Hi,
I use the DTS 2000 Migration Wizard to migrate one of the DTS 2000 packages to SSIS. The migration failed with the following error message:
LogID=17
#Time=6:31 PM
#Level=DTSMW_LOGLEVEL_ERR
#Source=Microsoft.SqlServer.Dts.MigrationWizard.Framework.Framework
#Message=Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException: Failed to save package file "C:Documents and SettingsfuMy DocumentsVisual Studio 2005ProjectsKORTONKORTONProcessCubesMF.dtsx" with error 0x80070002 "The system cannot find the file specified.".
---> System.Runtime.InteropServices.COMException (0xC001100E): Failed to save package file "C:Documents and SettingsfuMy DocumentsVisual Studio 2005ProjectsKORTONKORTONProcessCubesMF.dtsx" with error 0x80070002 "The system cannot find the file specified.".
at Microsoft.SqlServer.Dts.Runtime.Wrapper.ApplicationClass.SaveToXML(String FileName, IDTSPersist90 pPersistObj, IDTSEvents90 pEvents)
at Microsoft.SqlServer.Dts.Runtime.Application.SaveToXml(String fileName, Package package, IDTSEvents events)
--- End of inner exception stack trace ---
at Microsoft.SqlServer.Dts.Runtime.Application.SaveToXml(String fileName, Package package, IDTSEvents events)
at Microsoft.SqlServer.Dts.MigrationWizard.DTS9HelperUtility.DTS9Helper.SaveToXML(Package pkg, String sFileLocation)
at Microsoft.SqlServer.Dts.MigrationWizard.Framework.Framework.StartMigration(PackageInfo pInfo)
Looking at the call stack, it looks like COM wrapper fails on SaveToXML. Can someone tell me how I should workaround this problem?
Thanks,
Bobby Fu
View 1 Replies
View Related
Dec 4, 2014
I am trying to create and later read a data file from a package deployed in SSISDB, but it is not reading it while it is successfully creating the file. The same package when run from the file system package, runs successfully. Generating ispac and deploying in SSISDB is running for infinite time. Is it a permission issue?
View 7 Replies
View Related
Dec 21, 2006
Hi,
I am facing a problem on a server which has raid 5 solution (3 disks), the raid controller went down 2 of the disks were off in the Bios.
We added the 3 disks to a different server identical in brand and architecture, the raid controller was able to reconfigure the virtual drive H:.
All files were there, we installed sql server 2005 on the new server, but when we tried to attach the database we got the error below:
TITLE: Microsoft SQL Server Management Studio
------------------------------
Attach database failed for Server 'myserver'. (Microsoft.SqlServer.Smo)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=9.00.1399.00&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Attach+database+Server&LinkId=20476
------------------------------
ADDITIONAL INFORMATION:
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
------------------------------
The operating system returned error 38(Reached the end of the file.) to SQL Server during a read at offset 0x00000000af0000 in file 'C:Datamylog_log.LDF'. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.
Operating system error 38(Reached the end of the file.) on file "C:Datamylog_log.LDF" during ReadFileHdr.
Could not open new database 'mydb'. CREATE DATABASE is aborted. (Microsoft SQL Server, Error: 823)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&EvtSrc=MSSQLServer&EvtID=823&LinkId=20476
------------------------------
BUTTONS:
OK
------------------------------
I tried the following steps but it always failed:
- create a new db with the same name of the lost db;
- put the db in emergency mode;
- stop sql service and replace the mdf file;
- start sql service;
- Run Dbcc checkdb('mydb')
we got the error below:
Msg 945, Level 14, State 2, Line 1
Database 'ism0506' cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details.
Any HELP please ?
Thanks,
Tarek Ghazali
Sql Server MVP
View 5 Replies
View Related
Feb 7, 2007
Hi
This should be incredibly simple and easy, but I can't find any examples of how to do this.
I just want to make a File System Task move a file, and have the destination be filename + date and time. For example \serversharefilename02072007.txt
What syntax do I use in a variable to make this work?
Thanks
View 16 Replies
View Related
Jun 29, 2015
Ok so I have some dynamic sql to delete a file that is created via sql earlier on. It is to provision a copy of a database to an instance on link server. Everything works great and the files used to delete. Now, with no code changes it is throwing a syntax error. I do a print of what the dynamic sql is creating before executing and then I copy / paste what was generated into command prompt and guess what!! The file deletes.
Here is the result on screen:
@DeleteBackupFileStatement: DEL adas16.clients.advance.localwip$AvionteAP_Template_893.bak /Q
Msg 102, Level 15, State 1, Line 1
Incorrect syntax near ''.
Msg 102, Level 15, State 1, Line 1
Incorrect syntax near ''.
Here is the code that creates the statement:
SET @DeleteBackupFileStatement = NULL
BEGIN
SET @DeleteBackupFileStatement = 'DEL ' + LTRIM(RTRIM(@BackupFile)) + ' /Q'
END
PRINT '@DeleteBackupFileStatement: ' + cast(@DeleteBackupFileStatement as varchar(400))
BEGIN
EXEC adasdb.master.sys.Sp_executesql
@DeleteBackupFileStatement
END
END
The value of @BackupFile is simply the path of the file with the file name, everything in the prepared statement with the exception of DEL and the switch at the end.
View 9 Replies
View Related
Jul 12, 2006
Hello
I want to move and rename a file and embed the date/time into it, so that each time the package runs a new file is created. For example MyFile_20060712_150000.doc.
Can someone give me a hint how to do this with the File Systen Task SSIS Control Flow Item?
Thanks for an early reply
Regards
Chaepp
View 12 Replies
View Related
Feb 27, 2008
Hi All,
I have a source files folder where the files generated everyday.
My goal is pick the latest file and copy this single file to another folder.
I used the Foreach loop container and got the latest file and stored the file name to a varible i.e. LatestFile
Then i want to use the File System Task to copy this to the destination.
On the beginning, I could not setup the Latestfile since I don't its name then, so when I setup the Source Connection property of the File system task, it is not allowed to leave the SourceVarible as blank!
Any suggestion?
Thanks
Micor
View 3 Replies
View Related
Jul 13, 2006
Could someone please instruct me on how to use the File System Task Editor to rename a file? I place control on control flow tab, change the operation to rename, from there I am not sure what to do.
View 29 Replies
View Related
Jan 19, 2008
I have the following directories
\servershare
\serversharearchvie
I have a for each file loop and inside it a data flow that pulls from one of the flat files in the directory and then a file system task. If I choose the "Move File" option in the file system task to move the file to the archive directory, it fails with an access denied message. The access denied message occurs after a message saying file was successfully deleted. I am running this from BIDS right now and my local user can write delete etc in both the above directories. However, if I do a "copy file" in the file system task it seems to work. I think what is happening is it is deleting the file first and then trying to move it, but it no longer exists because it has been deleted--is this possible? Is this a bug of some sort?
For now I am going to workaround by putting in another file system task that deletes the files after they were copied and see how that goes, but would prefer just to do the "move" option.
Thanks,
Kayda
View 3 Replies
View Related
May 10, 2006
I'm having trouble working this out in SSIS. I am trying to use a File System task to rename a file using an expression so that file.zip will be renamed to filemmyy.zip at the end of every month (for instance this month would be file0506.zip).
I am using the destination expression variable. But I'm not sure what to put for DestinationConnection. It seems to want a file name, but the file name is going to be variable, so I'm not sure what to put.
Any ideas?
View 2 Replies
View Related
Mar 19, 2007
Hi there,
Can some one tell me how to rename the file Dynamically using file system Task.
I could able to rename the file and couldnt do it dynamical renaming.
Please let me know if anyone have an idea.
Thanks and Best Regards
View 3 Replies
View Related
May 19, 2006
Hi All,
I am having a problem using the file system task, what I am trying to achieve is to move a file after it has been processed . I am using a For each loop container to process bunch of files but I want to remove the files that have been processed after every loop.To achieve this I added a File System Task after my data flow task and was using the same variable used in the for each loop container as my source variable but the package is not being validated and is gives the following error
"variable used as the source or the destination is empty"
Appreciate your help
Thanks
View 6 Replies
View Related
Nov 7, 2006
I have a File System Task that uses variables to resolve the destination and source paths of a document. When I select the 'copyfile' operation...the document is copied from the source to the destination without error.
However when I change the property from 'copyfile' to 'movefile' I get an error and the document is not moved.
The source and destination variables contain a valid file path name since the copy commmand is working as expected. However when I alter the properties of the File System task to move the document. I get the following error:
Could not find a part of the path 'G:CommonInformation SystemsDropFilesrtNRT_ConfirmationOrder Confirmation Report_11062006.xlsOrder Confirmation Report_11062006.xls
It seems a little nonsensical since the document file paths are valid when performing a copy. For some reason the error log is showing that that the file path is the document name "Order Confirmation Report_11062006.xls" and adding it twice to the the directory path called "'G:CommonInformation SystemsDropFilesrtNRT_Confirmation" as you can see in the above error message.
To replicate the 'move' action...I added an extra File System Task that deletes the document once the copy has been performed. I would like some insight into why this doesn't seem to work.
Thank you.
...cordell...
View 1 Replies
View Related
Oct 9, 2007
yes,I have an error, like 'The database file may be corrupted. Run the repair utility to check the database file. [ Database name = SDMMC Storage Cardwinpos_2005WINPOS2005.sdf ]' .I develope a program for Pocket Pcs and this program's database sometimes corrupt.what can i do?please help me
View 4 Replies
View Related
Jan 8, 2002
Anyone reading XML disk-files into SQL Server?
I have a process that I may want to do this with.
It would be a stored procedure that would read the XML attributes into 2 tables, the number of attributes could be 1-N, so I thought XML would be a good choice. Also, one of the attributes could be up to 4000 characters. I think this may limit our options, can 100-150 4000 character strings be passed in a standard call to a query/proc in SQL?
Currently the client application makes round-trip network calls to save upwards of 100 pairs of data. 1 header row, and many detail rows. All within a transaction.
I think If we move an XML file to the SQL box, then do all the import/save work on the "Server" side it would be much better. Cutting the transaction time down a lot by not doing so many round-trips at network speed...
Thoughts?
View 1 Replies
View Related
Nov 16, 2006
I have setup a test IIS box and would like to use that as my development platform. I have installed sql express 2005 for my database on the box as well.
I can setup new pages on the file server just fine.. But when I try and setup a login page and use the asp.net configuration web page to setup security which create a user DB it will not let me create it...
I get the following:
The following message may help in diagnosing the problem: An error occurred during the execution of the SQL file 'InstallCommon.sql'. The SQL error number is 5110 and the SqlException message is: The file "J:INETPUBNBSRECAPAPP_DATAASPNETDB_TMP.MDF" is on a network path that is not supported for database files. CREATE DATABASE failed. Some file names listed could not be created. Check related errors. Creating the ASPNETDB_4a9331c5903148ca92e3ffbee7d29976 database...
Can someone let me know.
View 1 Replies
View Related
Nov 15, 2001
When I try to open a DTS package I get the message above. These packages were written by other developers.
Other packages on this server state that the Parameter is incorrect when I try to open them.
Any ideas?
Jim
View 2 Replies
View Related
Nov 8, 2001
I am developing a new application and am trying to find the best way to deliver XML documents to the app. Here's the scenario: each set of xml docs will have a unique identifier. The app needs to retrieve a known xml doc by using this identifier. So the dilema I am having is as follows: I could create a folder for each identifier and load the xml docs relevant to each id into the folder. For the app to retrieve the document, it would simply construct a file path and the windows B-tree search would find it and return it very quickly. Alternatively, I could create a table with a primary key that stores the unique identifier, cluster it, and store the xml in a blob field.
Does anyone have any experience, references, or thoughts about which would be the best way to do this? I am an avid database developer and the app will be using sql 2000 anyway, however it seems in this situation the database might not be the best option in terms of speed, performance, and resource useage. Thoughts?
Thanks,
Mike
View 5 Replies
View Related
Apr 4, 2007
I'm running SQL2005. The application running under SQL are a small accounting software and Microsoft WSUS. The file that is over 28G is the system model log file (modellog.ldf). I tried shrinking it and that only shrunk it by 2G. Is the model log file supposed to get that big, because the primary model data file is only 1Mg. How do I fix this? Can I delete the model log file?
Thanks
Willie
View 5 Replies
View Related
Aug 20, 2007
hi friends
I PLACED A FILE SYSTEM OBJECT WICH IS USED TO (COPY FILE/MOVE FILE SO ON )
Once copy file working fine second time copy file gives an errors
we need to check the condition if that folder contrain the dest.txt file we dont require to copy a file
other wise we need to copy
so i need a controle for checking a folder contrain the dest.txt file or not
regards
koti
View 2 Replies
View Related
