Java Heap Memory Error While Writing Large Data To Excel
Mar 6, 2014
i have to write more than 100000 rows in a excel sheet (file size more than 20 MB) via java.
when i use XSSF, i am getting below Error.
java.lang.OutOfMemoryError: Java heap space
at org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1592)
at org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1223)
at org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1144)
[Code]....
when i use HSSF , i am getting the below Error.
java.lang.OutOfMemoryError: Java heap space
I have tried increasing the java heap size , by giving upto -Xms1500m -Xmx2048m
I need to process 10000 xml files and verify and insert the data into database. I am loading all the files in the file object and iterating one by one. I am getting the memory issue. How to handle this?
Given a string, compute a new string where identical chars that are adjacent in the original string are separated from each other by a "*". My implementation :
package com.tcs.dash; public class StringBuild { public String edit(String userIp){ StringBuilder builder = new StringBuilder(userIp); String replaceText = ""; for(int i = 0; i < builder.length() - 1; i++){ if(builder.charAt(i) == builder.charAt(i+1)){ replaceText = builder.charAt(i) + "*" + builder.charAt(i+1); builder = builder.replace(i, i+1, replaceText); } } return builder.toString(); } }
I am getting error at line 13. An exception actually.
I/P given = aaaa
Console:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source) at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) at java.lang.AbstractStringBuilder.replace(Unknown Source) at java.lang.StringBuilder.replace(Unknown Source) at com.tcs.dash.StringBuild.edit(StringBuild.java:13) at com.tcs.dash.StringBuildExample.main(StringBuildExample.java:14)
I was wondering where is the memory allocated for an applet; by the browser; by the JVM; some applet specific java option? I get an out of memory error when running my applet (loading pictures).
I have two unix systems in which on one system I installed sun solaris jdk and on another system I installed IBM jdk.
Java programs which consume more heap memory are getting failed on sun solaris jdk system where as same programs are successfully getting executed on IBM jdk system .
My question is does sun solaris 64 bit jdk needs more heap than IBM 64 bit jdk ??
I have a simple JavaFX Application that open a Browser and shows google page. After exit the Application and free all objects, I can see that the JavaFX objects like Scene, Stage, WebView and WebEngine are alive in the heap memory after call GC. I can see this objects with JProfiler and other Profiler tools.
This is my Test code: import java.awt.FlowLayout; import java.awt.event.ActionEvent; import java.awt.event.ActionListener; import javafx.application.Application; import javafx.application.Platform;
To test the application click on Start Button to show google web page, click on Stop Button to stop the application, run a Profiler tool, and call gc, the JavaFX classes are alive. I am using java version "1.7.0_51" and windows 8.1 Is there something wrong in my code? Or this is the normal behavior?
I am not a java developer, but I am using a java code that was available online to convert a large XML file to CSV file. The input file size is big, it is around 3GB. I got an error that it is out of memory, it is expectedly due to the large input file that i am trying to convert. Splitting of this file is not possible,
This is what I ran : xml2csv-conv data.xml data.csv
Error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at com.sun.org.apache.xerces.internal.dom.DeferredDoc umentImpl.createChunk(Unknown Source) at com.sun.org.apache.xerces.internal.dom.DeferredDoc umentImpl.ensureCapacity(Unknown Source) at com.sun.org.apache.xerces.internal.dom.DeferredDoc umentImpl.createNode(Unknown Source) at com.sun.org.apache.xerces.internal.dom.DeferredDoc umentImpl.createDeferredTextNode(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.Abstrac tDOMParser.character
[code]....
Additional information: I am running this from a Windows8 64 bit machine with 8GB physical RAM.
I have a question regarding best practice in using local variables as my method return variable. I have a method like this:
myReturnObject getMyObject(String input) { myReturnObject myObject = null; try { myObject = helperObject.someOtherMethod().getObject(input); //getObject has return type myReturnObject } catch (Exception e) { //log any problems } return myObject; }
And I'm wondering if I rewrite like this if I'll see some performance optimization benefit:
myReturnObject getMyObject(String input) { try { return helperObject.someOtherMethod().getObject(input); //getObject has return type myReturnObject } catch (Exception e) { //log any problems } return null; }
myObject can be quite large -- so I'm wondering if I can omit the myReturnObject local variable instance if it'll save some work from the garbage collector.
I've been tasked with processing a large dataset as part of a class assignment. One of the fields is a 24-digit unsigned hex number. I realized that, rather than storing the field verbatim in a char array of length 24, I could store the actual value of the hex number in an array only 6 chars long (I chose char over int because chars are unsigned). To do this, I wrote the following simple class to accept the hex string and convert it so that it can be stored in that manner:
private class Hex24 { private final char[] hexAsInt = new char[6]; public Hex24(String h) { for(int index = 0; index < 6; ++index) hexAsInt[index] = (char)Integer.parseUnsignedInt(h.substring(index * 4, (index + 1) * 4));
[Code] ....
Assuming all this works (haven't tested it, but I think it should), what I'm wondering is how much memory this will save me (if any) compared to just throwing everything into a char[24] array. The underlying char[6] is obviously quite a bit smaller, but objects must take up more space than just their fields since Java needs to know what kind of object it is so it can know what methods it has, etc..How to accurately compare the size of a Hex24 object to the size of a 24 character array.
I guess another option/thing I might want to compare size efficiency for is putting the char[6] along with the conversion/comparison logic directly into the classes (as fields/member methods) where I'm currently using Hex24 fields. I'd guess this is the most memory-efficient option I've come up with, but it would lead to a lot of code duplication.
Another thing I'd like to compare is the size of a String vs. the size of the equivalent char array for shortish text fields. If this difference is big enough it might be worth storing those fields as arrays rather than Strings.
What I'm tasked to do, is to make a simple Java class that forms a "V" based on whatever height the user would desire, made out of stars "*", and spaces " ".
For example, if a user desires a "V" with a height of 3, it would look print out something like;
* * * * *
Where a "V" with a height of 5 would look something like:
* * * * * * * * *
(That one didn't look too good, but you get the point, it's suppose to be 5 "high" and shaped like a "V"). The problem I have, is that I don't see what loops within loops within loops I would need to build something like this.
All the easy stuff like asking the user what height they want and such, I can handle, but I don't see how this thing is suppose to be coded, to print out a decent-looking and right-sized "V" in the console.
public static void main(String[] args) { int height = 3; for (int i = 0; i < height; i++) { for (int j = 0; j < 2/(height+1)+1; j++) { if(j == i) {
[Code] ....
Looked like something of a good start, and it drew me half (!) of the "V" in the size I wanted. Am I on to it here, or am I on the moon in terms of progress? I need the entire "V", not just a nice "".
I am trying to get the excel spreadsheet data and converting it in someway to java. I'm looking for something that will print out the java code itself that way I can embed it into future projects.
I am reading Excel data using java apache. I got format issue while reading double value such as 869.87929 (in excel) into 869.8792899999999 (in java).
if(type.equals("Double")){ String str = content[i-1]; //System.out.println(str); BigDecimal d = new BigDecimal(str); listObjects.add(d); }
Note: type from schema.csv & content [] value from file.xls If I print **str**, it shows value as 869.8792899999999. But i need to get **str** value as 869.87929. How can I get it?
I need to print contents of database to excel file.Apache POI is the solution but I am not finding a way to print the DB column names/headers into the excel.
E.g.:
incase csv we have opencsv CSVWriter writer = new CSVWriter(new FileWriter(fileName)); System.out.println("writer"); writer.writeAll(rs, true);
will print the db contents with the column headers dynamically whatever the query may be.
Any solution where we can print data and column names without know the query previously as we are getting query at the run time and each time the query is different.
I have a program which consist of several classes. The program reads a pom file and parses the data and then writes the data to a static table. The issue I'm having is with writing my LIB file data to my table. In the HtmlDataTable Class Im trying to write the files that are read in the lib directory to the table. My table currently consist of 3 columns (missing jar files,Lib Directory files, and POM file data) currently Im only able to write the missing jar files data to my table. In the HtmlDataTable class there is a for statement where I write the missing jar file data to my table. Im also trying to write the contents of the Lib directory within this statement as well. This is where I'm having my issue. My other classes consist of a SAX parser which parses the xml file, a class that creates my static table and a class that compares the jar files in my lib directory to the jar files in my pom file. . Theres a lot of code so I included the parts I felt were useful. If needed I can include the other classes as well.
public class ReadPomFile extends DefaultHandler { public static void main(String[] args) { try { // obtain a SAX based parser to parse XML document
I have an application which has 10 million rows and 1000 columns in Oracle. Each value has a different set of calculations that are stored in User Defined PLSQL functions.
Data is displayed in form of data grid. When a user updates any value, the calculation is performed using plsql function and value is stored in database. Is there an easy way through which calculation is performed on the fly and i get maximum performance ?
I'm programming an android app where I want to send large sets of data from an MySQL Table to the app and write this data to an SQLite database.So I know i have send an HttpRequest and I can send data with json, but how can I handle large datasets like tables?I think its no good idea to just concatinate a string together in php and then send it?
Our client has a user facing web application running on Jboss. There is a separate admin application (in its own ear) but deployed on same Jboss server on which user facing web application is running.
They need a screen to upload large amount of data into database. Their original files were in excel with size > 60 mb. We suggested following to them:
a. Change upload format to CSV - this brought down file sizes to 25-30 mb b. Upload process will be MDB - asynchronous processing of data so that admin web app does not stop responding
We also suggested following to them:
a. Host admin app on a different machine so that user facing site does not respond slow during data processing b. We can provide incremental upload feature and they should upload files in the chunks of 4-5 mb, specifically if they have user a web page to upload such files - they don't buy this argument though. c. Data processing can be a separate script instead of a part of admin web application. They can FTP files to a designated location and this script will process those files.
I have following questions:
Q1 - Have you seen upload of such large datafiles to a web application? I see sites like Zoho CRM or Salesforce do not support such data imports and mostly fail or not respond. Q2 - Is there a set of guidelines/best practices to upload large data files of this nature? How do insurance companies or others with enormous set of data accomplish such tasks (what is the architecture of such programs)?
We are getting "Code too large" compilation error for one of our class. This class contains public String fields for label ID and value. We use this class for localization, except for English all other language labels come from .properties files.
The reason we are getting this error is because we have a static block in which using reflection we are populating a HashMap with all public fields and their value. The number of fields have gone up to the extinct where we are crossing the 64K limit for a static method. One of the most feasible solution was to use .properties files for English labels as well.
I will be calling this class MyLabels. We defined a super class for MyLabels called MyLabelsExt. And now we are adding labels into the super class instead of the MyLabels. By running some tests we confirmed that the map that we initialize in MyLables class contains all the fields from both MyLabels and MyLabelsExt class.
How is the 64K limit error not coming if the labels are defined in a super class. Does that mean Java is able to identify that some of the fields are coming from parent class, and that is being treated as separate from the child class. And how is the map that we initialize having all the value.
I have a requirement to download the data from DB to CSV/Excel. What I did is retrieving all the records from DB and store in the List and pass that list to the jsp page. In JSP page, I am using the below setting to download it to CSV/EXCEL.
It was working fine. But I have an issues in it. Excel is capable of storing it only 65,536 rows. Even all the data getting downloaded to CSV/excel (> 65,536), while saving I am getting error from excel saying that it can't save more than 65,536 rows and the below data won't be saved.
This workbook contains data in cells outside of the row and column limit of the selected file format.
Data beyond 256 (IV) columns by 65,536 rows will not be saved. Formula references to data in this region will return a #REF! error.
I am not opt for POI apache third party at this point of time. Is it possible to download it in other sheets if it exceeds the max limit.
Example: Having data of around 100,000.
Download 65,536 - Sheet 1 Remaining in Sheet 2,3.. of the same excel.
