How To Improve File Reading Efficiency And Its Data Insertion In Java
Feb 8, 2014
We have an autosys job running in our production on daily basis. It calls a shell script which in turn calls a java servlet. This servlet reads these files and inserts the data into two different tables and then does some processing. Java version is 1.6 & application server is WAS7 and database is oracel-11g.
We get several issues with this process like it takes time, goes out of memory etc etc. Below are the details of the way we have coded this process.
1. When we read the file using BufferedReader, do we really get a lot of strings created in the memory as returned by readLine() method of BufferedReader? These files contain 4-5Lacs of line. All the records are separated by newline character. Is there a better way to read files in java to achieve efficiency? I couldnt find any provided the fact that all the record lines in the file are of variable length.
2. When we insert the data then we are doing a batch process with statement/prepared statement. We are making one batch containing all the records of the file. Does it really matter to break the batch size to have better performance?
3. If the tables has no indexes defined nor any other constraints and all the columns are VARCHAR type, then which operation will be faster:- inserting a new row or updating an existing row based upon some matching condition?
I am creating a program where it reads the data inside a file and then places this data into arrays. The file I created has numbers 1-30 in it, file named, testing1.txt .
I can read the data to a monitor perfectly. But, I'm having problem reading data from an external file into an array of class objects. Here's my attempt at writing the method:
private void openFile() //This method asks the user to enter a file name(including the file extension) and then //Sets the data to an array of Product type { String fileName, storeName="", emptyLine1="", emptyLine2="", name="", productName=""; int demandRate=0; double setupCost=0.0, unitCost=0.0, inventoryCost=0, sellingPrice=0; Scanner inputStream = null;
I have to read data from a text file and print it in a new text file. An example of one line is like this: Johnson 85 98 75 89 82
I then have to take the average of all the numbers and assign a "grade" to the numbers for each line of the text file and make a new file so it looks like this for 10 lines:
Name 1 2 3 4 5 Average Grade
Johnson 85 98 75 89 82 85.80 B
My problem is extracting the data from the file so I can use it.
I'm trying to insert data into an excel sheet with the below Servlet.
/* * To change this license header, choose License Headers in Project Properties. * To change this template file, choose Tools | Templates * and open the template in the editor. */
But it is giving me the below Exception and stacktrace
java.text.ParseException: Unparseable date: "2-Apr" at java.text.DateFormat.parse(DateFormat.java:357) at Serv1.doPost(Serv1.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:644) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
[Code] ....
i found that the problem is within the date field(, if the problem is with the date, how i can fix it and where am i going wrong.
I'm a complete beginner in Java programming and I'm interested to learn more about its concepts.
Recently, I've been given an exercise which instructs me to display two versions of a picture. The picture to be displayed is provided in the form of a data file of 40,000 digits that are arranged in rows (although there is no marker between rows) and it starts from the top of the picture. So the first digit represents the top left corner of the picture and the last is the bottom right.
Basically, what the exercise wants me to construct a program that plots a dot in one of two colours for each digit. If the digit is in the range 0 to 3 the output should be one colour and for digits in the range 4 to 9 the dot should be in the other colour.
I understand I have to use arrays and also loops to perform this. I'm familiar with the fillEllipse, drawEllipse, drawRectangle and fillRectangle but this exercise is nothing I've attempted before.
I'm having a bit of trouble with using the Scanner and the Printwriter. I start with a file like this (1 = amount of Houses in the file)
1 FOR SALE: Emmalaan 23 3051JC Rotterdam 7 rooms buyprice 300000 energylevel C
The user gets (let's say for simplicity) 3 options:
1. Add a House to the file, 2. Get all Houses which fullfil requirements (price, FOR SALE / SOLD etc.) and 3. Close the application.
This is how I start:
Scanner sc = new Scanner (System.in); while (!endLoop) { System.out.println("Make a choice); System.out.println("1) Add House"); System.out.println("2) Show Houses"); System.out.println("3) Exit"); int choice = sc.nextInt();
Then I have a switch for all of the three cases. I keep the scanner open, so Java can get the user input (house = for sale or sold, price = ... etc). If the user chose option 1, and all information needed is inputted and scanned, the House will be written to the file (which looks like what I typed above).
For this, I use try (PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("Makelaar.txt", false)))). This works perfectly (at least so it seems.)
If the user chose option 1, and all requirements are inputted and scanned, the Houses will be read (scanner) from the file and outputted. For this I use the same Scanner sc. This also works perfectly (so it seems atleast).
My problem is as follows: If a House has been added, I can only read the House(s) which were already in the file. Let's say I have added 2 houses, and there were from the start 3 houses. If option 2 is chosen, the first 3 houses will be scanned perfectly. An exception will be caught for the remaining 2 (just added) Houses. How can I solve this? I tried to close the Scanner, and reopening it, but apparently Java doesn't agree with this
A method in Java returns a schema. In .net, we use DataTable to get the data from the schema. How to write the following code in Java:
/* Result r = method(); r.DataSchema; */ using (DataTable tbl = new DataTable()) { using (MemoryStream ms = new MemoryStream()) using (StreamWriter sr = new StreamWriter(ms))
I am reading Excel data using java apache. I got format issue while reading double value such as 869.87929 (in excel) into 869.8792899999999 (in java).
if(type.equals("Double")){ String str = content[i-1]; //System.out.println(str); BigDecimal d = new BigDecimal(str); listObjects.add(d); }
Note: type from schema.csv & content [] value from file.xls If I print **str**, it shows value as 869.8792899999999. But i need to get **str** value as 869.87929. How can I get it?
I have an battleship program that I would like to improve in performance. I know that using certain algorithms it can be improve however I am beginner in Java. I would like any recommendations and or hint in how to improve this game. The are 5 ships per game and the computers plays 10000. I just want to sink all ships in as many shots as possible.
Im trying to read a file called m1 and put that information into a 2-D array with the first two numbers being the size of the array i was able to set the size of the array but how do i populate array?
import java.util.Scanner; import java.io.*; public class test{ public static void main(String [] args)throws IOException{ Scanner fr = new Scanner(new File("m1.txt")); String line = fr.nextLine();
reading a large json file which is really huge and it contains pair of id and URL in a array as shown below.. i need to get this all id and URL from this file in my java code and need to perform operations . how to read such a big file and store the contents in java code to process it further.
{ "Array": [ { "id": "1test", "URL": "http://servername/test/1test.html" }, { "id": "2test", "URL": "http://servername/test/2test.html" } ....................many id and URL in the same Array which makes the file Huge.. ] }
I am trying to make a bank program that reads to .txt files, one that has the right format and one that has the wrong format. This is part of my problem I do not know how to do this. I also do not know why my code works if I input "account" rather than "account.txt" which is supposed to be the right thing.
Let me present my code that I have so far:
// File: BankAccount.java import java.util.Scanner; import java.io.IOException; public class BankAccount { private int accountNumber; private double balance;
I have to write the Insertion Sort Algorithm using Java codes and at the same time find the time of execution for different sizes of array, filled with random numbers. If I try to show the numbers inserted into the array randomly, they don't appear at the console.
import javax.swing.JOptionPane; public class Insertion { public static void main(String[]args){ int SizeArr = new Integer(JOptionPane.showInputDialog("Enter the size of teh array")).intValue(); int [] r= new int [SizeArr]; {for(int d=0; d<r.length; d++)
I am advised to use a while loop the Scanner method hasNextLine() and in the while loop body, call the Scanner method nextLine(), and add the returned String to the ArrayList of Strings. what I have gotten so far:
Scanner input = new Scanner(new File("")); while(input.hasNextLine()) { String line = input.nextLine(); System.out.println(line);
Where usually to instantiate and declare variables to be used in a loop. If you declare it outside to be used in the loop it will still be there when the loop is done, never to be used again and is just sitting there taking up memory. However, if you declare it inside the loop, you have the issue of it constantly creating space for said variable, but once it's out of scope it's gone. Is there any advantage performance wise to doing it either way?
I have a case in which I want to sort two types of ArrayLists (using quicksort) and the method originally coded only accepts a String ArrayList. The problem is that now I want to sort an ArrayList of type int but couldn't . . . so I decided to overload the method. Since it looks very ugly to copy and paste the same chunk of code only to change the method signature I wondered if there is a better way to make this method more dynamic and be able to take in different types of ArrayLists.
My code:
private ArrayList<String> sort(ArrayList<String> ar, int lo, int hi){ if (lo < hi){ int splitPoint = partition(ar, lo, hi); sort(ar, lo, splitPoint); sort(ar, splitPoint +1, hi);
I am making an android racing game. I have a method that draws the track to the screen, the map is bigger than the screen so i clip the bitmap and draw it to the screen hence to draw the track I have to get the visible track points and then draw them. I have noticed that when I enable drawing the track the game lags quite a bit but when i disable it is smooth as expected the code is below :
public void drawTrack(Canvas canvas) { if (this.trackPoints != null && this.trackPoints.size() > 0) { // get points, convert back to screen coords and add them to array, // then // use that arraylist<point> below getCurrentVisibleTrack(); counter++; if (this.visibleTrackPoints.size() > 0) {
I have two programs that talk to each other over a TCP connection. When I write the data "STX+1234" where STX is the Ascii character STX or Ctrl B and I expect the written String length to be 6 which it is. On the other side of the socket I create the ServerSocket and use the client socket's InputStream to create a BufferedReader. When I receive the string it now has 12 characters where each original character has been replaced by NUL and the character. How do I read the string as it is originally specified without the conversion? And is the problem on the reader side or the writer side?
I tried working with XML today, since that's what my book teaches. First, I read some data about Product objects that are suppose to represent products with a code, description, and price. That worked fine. Then, I tried creating a writeProducts(ArrayList<Product> products) method, and man it did not go well.
How it works is that there is an ArrayList instance variable in the class that keeps track of all of my products after the readProducts method returns that ArrayList from the XML file. When I ran the method I had a lot of stack traces printed. I then checked the file on a web browser and an editor, and saw it was completely empty. Here's the code:
import java.util.ArrayList; import java.io.*; import javax.xml.stream.*; // StAX API public class XMLTesterApp { private static String productsFilename = "products.xml"; public static void main(String[] args) { System.out.println("Products list:");
I am trying to create a second class in a single java file (first time trying this) and want to use data from the first class in the second class but I am not able to do it. What am I missing :
package simplecommissioncalculation; import java.util.Scanner; // import java.util.Scanner public class SimpleCommissionCalculation { public static void main(String[] args) {
