Correct Way To Upload Large Data File
Nov 25, 2013
Our client has a user facing web application running on Jboss. There is a separate admin application (in its own ear) but deployed on same Jboss server on which user facing web application is running.
They need a screen to upload large amount of data into database. Their original files were in excel with size > 60 mb. We suggested following to them:
a. Change upload format to CSV - this brought down file sizes to 25-30 mb
b. Upload process will be MDB - asynchronous processing of data so that admin web app does not stop responding
We also suggested following to them:
a. Host admin app on a different machine so that user facing site does not respond slow during data processing
b. We can provide incremental upload feature and they should upload files in the chunks of 4-5 mb, specifically if they have user a web page to upload such files - they don't buy this argument though.
c. Data processing can be a separate script instead of a part of admin web application. They can FTP files to a designated location and this script will process those files.
I have following questions:
Q1 - Have you seen upload of such large datafiles to a web application? I see sites like Zoho CRM or Salesforce do not support such data imports and mostly fail or not respond.
Q2 - Is there a set of guidelines/best practices to upload large data files of this nature? How do insurance companies or others with enormous set of data accomplish such tasks (what is the architecture of such programs)?
View Replies
ADVERTISEMENT
Sep 3, 2009
I have a requirement where a large file (100 -200MB) is uploaded from the client to a content management system. I am using a servlet with Apache Commons File Upload API. Apache FileUpload has 2 ways of handling files,
1) Non-Streaming
2) Streaming
Currently I use the Non-Streaming approach where the servlet stores the file in a temp location and upload the same into the content management system - This is taking lot of time so I am trying to implement Streaming API.
Content Management API supports streaming in 2 methods,
a) SetContent - Takes the file's ByteArrayOutputStream as input -> This gives OutOfMemoryException because the file being large
b) AppendContent - Takes the file's ByteArrayOutputStream as input -> This method can be called multiple times to upload the large file but I dont know how to do this. The Apache File Upload gives InputStream of the file and I need to split that into chuncks and append into the content management system.
How to convert InputStream to 4KB ByteArrayOutputStream so that I can use the AppendContent method in content management API?
View Replies
View Related
Apr 9, 2015
I would like to understand how does multipart/form-data works during file upload scenario's, Does it chunks the data from client to server while transferring the files ?
View Replies
View Related
Jan 25, 2014
Iam trying to upload jpeg image.
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
[Code] ....
In servlet the below code
DataInputStream din = new DataInputStream(request.getInputStream());
byte[] data = new byte[0];
byte[] buffer = new byte[50];
int bytesRead;
while ((bytesRead = din.read(buffer)) > 0 ) {
byte[] nData = new byte[data.length + bytesRead];
System.arraycopy(data, 0, nData, 0, data.length);
System.arraycopy(buffer, 0, nData, data.length, bytesRead);
data = newData;
}
What i see is servletinput stream does't have any values.It does't even enter while loop.
View Replies
View Related
Dec 1, 2014
In my web application i want to upload file to drop-box. I am getting file name from browser.Is it possible to upload file to drop-box with only file name.
Below the drop-box upload code with java.
File inputFile = new File("New Text Document.txt");
System.out.println("inputFile.getAbsoluteFile(): " + inputFile);
FileInputStream inputStream = new FileInputStream(inputFile);
try {
DbxEntry.File uploadedFile = client.uploadFile("/magnum-opus.txt",
DbxWriteMode.add(), inputFile.length(), inputStream);
System.out.println("Uploaded: " + uploadedFile.toString());
} finally {
inputStream.close();
}
In the above code the place New Text Document.txt we have to provide total path of file.
View Replies
View Related
Oct 10, 2013
I have an application which has 10 million rows and 1000 columns in Oracle. Each value has a different set of calculations that are stored in User Defined PLSQL functions.
Data is displayed in form of data grid. When a user updates any value, the calculation is performed using plsql function and value is stored in database. Is there an easy way through which calculation is performed on the fly and i get maximum performance ?
View Replies
View Related
Dec 4, 2014
I'm programming an android app where I want to send large sets of data from an MySQL Table to the app and write this data to an SQLite database.So I know i have send an HttpRequest and I can send data with json, but how can I handle large datasets like tables?I think its no good idea to just concatinate a string together in php and then send it?
View Replies
View Related
Nov 14, 2014
I upload a file and save it like the following
byte[] uploadedFileBuf = new byte[(int) uploadedFile.length()];
saveFileSystem(excelUpload + getLocalFileName(acad_year, stuid), uploadedFileBuf);
private boolean saveFileSystem(String filename, byte[] data){
try{
FileOutputStream fos = new FileOutputStream(filename);
fos.write(data);
fos.close();
[Code]...
I can not read it use
HSSFWorkbook WorkBook = new HSSFWorkbook(new FileInputStream(filename));
View Replies
View Related
Jun 19, 2015
I have a huge data set 10000+ rows which I need to show in the tableview. It take a lot of time to render the UI and is slow.
View Replies
View Related
Jan 21, 2015
I need to upload a file in R, I can do it with a CSV file or excel ect. The file I received is already in R format .Rdata.It is a file called surveydata.rdata.The two data frames are surveydata.4.num.frame and surveydata.4.lab.frame.The surveydata.4.num.frame is what I need.I can click on it in Rstudio to see the data but I can't load it to run an analysis.How can I upload this?
View Replies
View Related
Apr 19, 2015
This first number of the output is meant to be a timestamp for the data (formatted to show only minutes). The second part of data, after the ':', is the data being read from the serialport. This number is not correct, and is changed within this bit of code. The data is read as "Solar: A.AA", where A.AA is a number read from a sensor. I am able to read the data fine right before this if statement, but after that the data output changes. StringProperty is not printing out the correct data:
Java Code:
serialPort.addEventListener(event -> {
if(event.isRXCHAR()) {
try {
sb = new StringBuilder();
sb.append(serialPort.readString(event.getEventValue()));
String str = sb.toString();
if(str.endsWith("
[code]....
View Replies
View Related
Mar 22, 2014
I'm having some problems styling the PrimeFaces upload tag <p:fileUpload /> in my application.
In my xhtml code I have the following:
<p:fileUpload id="fileUpload" fileUploadListener="#{filters.upload}"
allowTypes="#{filters.uploadTypes}" invalidFileMessage="#{filters.uploadBadType}"
sizeLimit="#{filters.uploadSize}" invalidSizeMessag="#{filters.uploadBadSize}"
update="fileUpload fileTable filterTab uploadMessage hiddenNum hiddenPhoto uploadError footer
namePhotSystem checkOverwrite"
description="Select Text File" disabled="#{filters.fileuploadDisabled}" fileLimit="1"
fileLimitMessage="You are only allowed to upload one file at a time"
styleClass="fileUploadClass" label="Choose the File to Upload" auto="true" />
This all works correctly, both the AJAX update and on the server, but I'm unable to get the button or buttons displayed in the way I need.
By including the following in my CSS:
.ui-fileupload.fileUploadClass {width:428px;}
.ui-icon-plusthick {display:none;}
.ui-button-text {width:200px; height:20px;}
the first sets the width of the button bar, and by looking at the HTML code generated by PrimeFaces, I discovered that the second hides the "+" to the left of the "Choose" button, which here has been replaced by the text "Choose the File to Upload". However, this text is displayed on two lines, with the word "Upload" on the second. I want the whole text to be on one line, but no matter what I do, I cannot get this to work properly. The third line of my CSS code does indeed reduce the height of the button, but the width does not work at all, and the last part of the text is simply lost. How do I put all the text on one line and adjust the width of the button?
Another question is that when I remove the auto="true" attribute from the tag, which I had originally, the "Upload" button is displayed to the right of the "Choose" button, and when a file is chosen but not yet uploaded, it is displayed below the button. This is a capability that I might still want, however, I am unable to style the display, and in particular the progress window to the right of the file name and size. All the files I want to upload are relatively small text files, and I want to adjust this window or even hide it completely. How is this done?
Incidentally, because the files are all quite small, I have disabled the display of the "Cancel" button, so either only the "Choose" button is shown, or the "Choose" and "upload" buttons.
View Replies
View Related
Apr 16, 2014
I am trying to implement a file upload functionality. One of the requirement is to display a 'Please Wait!!!' message to the user while the upload is going on. My project currently uses JSF 1.2, Tomahawk, JQuery 1.9, JQuery UI 1.10. I am aware that this can be accomplished using a4j with richfaces, however I am not supposed to add any new library to the existing project. how this can be done?
View Replies
View Related
Mar 6, 2014
i have to write more than 100000 rows in a excel sheet (file size more than 20 MB) via java.
when i use XSSF, i am getting below Error.
java.lang.OutOfMemoryError: Java heap space
at org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1592)
at org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1223)
at org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1144)
[Code]....
when i use HSSF , i am getting the below Error.
java.lang.OutOfMemoryError: Java heap space
I have tried increasing the java heap size , by giving upto -Xms1500m -Xmx2048m
View Replies
View Related
May 20, 2014
I'm using jsf 2 to upload file, first I upload the file in a system directory, then trying to store the path to database with other information, my stuck is that when submitting I upload the file successfully, find it in the right place, find the other information such as description, file name ... in database but don't find the path. this is my managed bean :
package mbeans;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.util.List;
[code]...
View Replies
View Related
Apr 1, 2014
I have a JSF page (called MainPage) with a commandButton: clicking on it, I open a modal panel with a <Rich:fileUpload> component, related to a listener in corresponding bean.
When I start to upload a file, page MainPage starts to refresh, but I would like to prevent this because it's not necessary.
I tried to "play" with <Rich:fileUpload> property values, but nothing seems to work and I don't know what to do anymore.
Here is file upload component
<rich:fileUpload fileUploadListener="#{pannelloUploadBean.uploadListener}"
id="#{cid}_input"
ajaxSingle="true"
immediateUpload="true"
listHeight="200px"
listWidth="458px">
[Code] .....
View Replies
View Related
Apr 16, 2015
I am not all secure on the execution of the fileupload Primefaces.
The uploading execution occurs normally, but I would like to have a counter that control's the received files. The problem is that sending files go by so fast by this method that appears to be running in parallel. My question is whether this behavior is normal while sending files.
If it is not, where may be the error. If it is normal what can i do to make the counter really take the right value?
I'm using the scope of the "bean" type of view, but when im trying to run the session the problem also occurs depending on the file size.
If my files are very small, the execution is very fast and I cant follow the counter. If I clean the code the execution usually occurs no matter the size of the files.
@ManagedBean (Name = "bean")
@ViewScoped
public class BeanUpload {
private int counter;
@PostConstruct
public void init () {
counter = 0;
[Code] .....
The result on the console for 3 files is:
COUNTER EX: 1
COUNTER EX: 1
COUNTER EX: 1
If I run the code in debug mode on the console this is the result:
COUNTER EX: 1
COUNTER EX: 2
COUNTER EX: 3
Versions: Mojarra 2.1. Primefaces 5.1.15
View Replies
View Related
Dec 4, 2014
How to upload file to google drive using java. my java code is.
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
httpTransport, jsonFactory, CLIENT_ID, CLIENT_SECRET, Arrays.asList(DriveScopes.DRIVE))
.setAccessType("online")
.setApprovalPrompt("auto").build();
[Code] ....
So in that file object(java.io.File fileContent = new java.io.File("document.txt");) asking complete file path. But in file upload we can get only file name not path.
View Replies
View Related
Oct 22, 2014
Creating a file upload servlet to accept a CSV and parse for insert into a database, however, whenever I click submit, it always seems to open a new tab/window. Below is the method I have that builds the upload form: (Using GWT 2.4)
private void buildUpload(){
LayoutContainer headerContainer = new LayoutContainer(new ColumnLayout());
headerContainer.setStyleAttribute("padding", "5px");
add(headerContainer);
NamedFrame hiddenFrame = new NamedFrame("uploadFrame");
final FormPanel form = new FormPanel(hiddenFrame);
[Code] .....
Is there something I'm missing? or something I've added that makes it open a new tab/window?
View Replies
View Related
Jun 30, 2014
I have a code that uploads files in server after browsing folders and files then get the paths of files but I have a problem in getting the paths
List items = upload.parseRequest(request);
Iterator iterator = items.iterator();
while (iterator.hasNext()) {
FileItem item = (FileItem) iterator.next();
if (!item.isFormField())
{ fileName = item.getName();
root = getServletContext().getRealPath("/");
path = new File(root + "/uploads");
[Code]...
list1 must has paths that I want but I do not get the paths of upload files
View Replies
View Related
Jan 5, 2014
reading a large json file which is really huge and it contains pair of id and URL in a array as shown below.. i need to get this all id and URL from this file in my java code and need to perform operations . how to read such a big file and store the contents in java code to process it further.
{
"Array": [
{
"id": "1test",
"URL": "http://servername/test/1test.html"
},
{
"id": "2test",
"URL": "http://servername/test/2test.html"
}
....................many id and URL in the same Array which makes the file Huge..
]
}
View Replies
View Related
Apr 7, 2015
I need to built a file upload service which should be memory effective. I should avoid loading the entire file into memory,Since I may have multiple http request which will pile up the Heap memory. Any effective way to upload a large file(For ex:1GB file) using http Streaming. I need to do the file upload on a single http call. Let's consider a scenario where 1 GB file to be uploaded using 512MB Heap memory. Not Sure If practically I can achieve this or not.
View Replies
View Related
Aug 13, 2014
I am using java process to start a system command in windows
Runtime r = Runtime.getRuntime();
Process pr = r.exec(cmdString);
I want to get the prompt out put from cmdString = "cmd /c type fileSmallSize"->>>> It is ok the have the content of the file when file is small.
However, for a large file java process will hang and no Exception occurred, what is the problem?
The easiest testing you can try on the logging.properties file in java.
public static String executeCmdAndReturnPromptResult(String cmdString)
throws Exception {
LOGGER.entering(CLASSNAME,
"Entering executeCmdAndReturnPromptResult()", cmdString);
String cmd = cmdString;
[Code] ....
It seemed to me that the bufferSize is limited so that I can only have it less than a default one, how to increase it?
My question now is how to increase the size of buffer in order to read a larger InputStream ?
BufferedInputStream() default size is
private static int defaultCharBufferSize = 8192;
private static int defaultExpectedLineLength = 80;
How to make it larger and working? I tried to increase the defaultCharBufferSize to 500000000 but it did not work!
View Replies
View Related
Feb 18, 2014
I have a large text file of 1 GB size. I need to print the line when a matching word is found in a particular line. Below is the code I am using. But if there are many lines that has the matching word, it's taking lot of time. Any solution to print the lines much faster.
Scanner scanner = new Scanner(file);
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
if(line.contains("xyz")) {
System.out.println(line);
}
}
View Replies
View Related
Mar 21, 2015
By using FileReader, FileWriter and their constituents, I am creating a file to contain employee information ( name, age, hours, etc. ). The user is to input all of the data on a single line and we were asked to implement StringTokenizer to assign that data to the file. I have never used the StringTokenizer before, but I have a rough idea how it is to function. I used pw.println to test what I have so far, now I would like to let the user build the initial file with the "first employees" of the company, and then view the file, and then go back and append new employee data to that same file. My question is, how can I take the user input as a StringTokenizer and add that to the file?
In the for loop below, I thought I would see if it would work, but it does not. The loop only executes once and does not allow me to enter data.
public class Records {
public static void main(String [] args) throws IOException {
Scanner input = new Scanner(System.in);
FileWriter fw = new FileWriter("dbs3.java");
BufferedWriter bw = new BufferedWriter(fw);
PrintWriter pw = new PrintWriter(bw);
System.out.println("NEW EMPLOYEE DATA SHEET");
System.out.print("Number of new employees: ");
int number = input.nextInt();
[Code] ....
View Replies
View Related
Apr 23, 2015
I am trying to remove a line based on user input. myFile.txt looks like:
Matt
Brian
John
However when I enter "Brian" (to remove this line), It is deleted on the temp file (myTempFile.txt), but not renamed back to the original file (myFile).
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Scanner;
[code]....
View Replies
View Related
