Massive Data Processing Using Unified Model
Dec 11, 2014
I have a requirement where we have an ETL jobs that runs in the backend which will get the up to date data from various sources , here we have both structured and unstructured data and the data capacity is massive which is in petabytes. Based on the data that is fetched we need to come up with a consolidated data view for the data which we retrieved. My queries are:-
1) How to create a consolidate data view , in what form it will be in (XML, JSON, etc)? whether we need to come up with a unified data model as the data is both structured and unstructured format? If so how we can go about it?
2) Since the data is really massive, how we can construct the consolidate data view? whether we need to create it in small chunks or whether we need to store these data in a cache or DB from where we will retrieve it and form the consolidated data view?
3) To process these data whether we need to have any Hadoop clusters to process these data parallel? As we are talking about massive data in the form of structured and un structured format?
4) Whether there is a need for NoSQL database to support unstructured data? If we are supposed to store the data?
View Replies
ADVERTISEMENT
Sep 9, 2014
I'd like to create my own Table Model to handle data from an SQLite database using JSwing, but I'm having difficulty. How to confirm the following:
-- A table model is an object that contains methods for manipulating data in the table, right?
If that's the case, then how should I create a Table Model to handle data coming from a database. From what I understand...
My custom Table Model needs to subclass AbstractTableModelI then override 3 methods from AbstractTableModel (getRowCount(), getColumnCount(), getValueAt())
As it relates to drawing data from a database, how should I be thinking about this problem (i.e. creating a Table Model that can work with a database)?
View Replies
View Related
May 24, 2014
This is a design question is the same problem in any language.as you do to map the controller to the domain model?We have situations in general larger than ... consider the example objects .
situation.1 - We have a request that has all the parameters of the account ;{ " id" : " 1 " , "name " : "test " , "some " : " xxx " } ............. and other fields .
situation.2 - can request that has to have a certain account parameters , for example in the case of an update;{" id" , " 1" , "name " , " testUpdated "}
situation.3 - We have a request that has some parameters of the account , others have more like id as user together;{ " id" : " 1 " , "user " : " xxx " , "service " : " yyy " } in which case each piece of the request will turn an object .
Java Code:
public class Account {
private Long id;
private String name ;
private String some ;
} mh_sh_highlight_all('java');
I see a few options ;
1 - Can I get AccountForm in the controller and set the properties for the Account object and others in CONTROLLER ;
+ ok for situation.1 situations 2, and situation.3
+ Separates the requisition of the object domain
- Pollutes the controller with code conversion
- Controller is full of setters .. if a higher class as a large object as a request is very confusing .
Java Code:
controller ( AccountForm from ) {
Account account = new Account ( )
account.setNome form.getNome = ();
account.setSome form.getSome = ();
Other outher = new Other ( ) ;
other.setSome ( form.getSome ( ) ) ;
} mh_sh_highlight_all('java');
2 - Can I get AccountRequest in the controller and have a method in itself as AccountRequest.getAccount ( ) to return a mapped model , in this case the mapping is at own Request object .
+ Separates the requisition of the object domain
+ Encapsulates the conversion in a place with easy access .
+ Meets situation.1 situation.2 and situation3 ;
- Request object has two responsibilities represent the request and map to a valid model .
Java Code:
controller ( AccountForm accountRequest ) {
Account account = accountRequest.getAccount ( ) ;
Outher outher accountRequest.getOther = ( )
} mh_sh_highlight_all('java');
3 - Can I get the controller Direct Account which had been filled with nulls .
+ Eliminate object request
- Serves only situation.1 situation.2 .
Java Code:
controller (Account account ) {
account.someMethod ();
} mh_sh_highlight_all('java');
4 - Outsource this mapping request parameters to another object mapper for request ..
+ Isolates logic mapping
- Until complexity for simpler cases are used as standard for all such a find by id .
- One more class for each request ;
In the case of API gets worse response has two further classes. speaking in terms of request for response .... AccountRequest, AccountRequestMapper, Account, AccountResponseMapper, AccountResponse .....I'm doing more testing the Hybrid option 3 for simple cases (find ID or updates) .... with option 2 for example for more complex cases ..
View Replies
View Related
Jan 21, 2015
I have observed a strange behaviour from Resultset object. My application fetches 400 records from a table and processes these records every 10secs. By default the resultset has a fetchsize of 10 from the database cursor. As I understand if the query returns 400records, the resultset will fetch 40times, in multiple of 10 to get all these 400 records from database cursor.
Query : SELECT * FROM ( SELECT * FROM TestTable
WHERE STATE_ACTION = 0 ORDER BY ROP_TIME DESC )
WHERE ROWNUM <= 250
Observation : Under a normal operation, the resultset fetches all the 400 records on query execution from database cursor, but under unknown conditions the same resultset object fetches only 10 records from the database cursor and exits. Please refer page 297 in document below for the result fetch size details. JDBC developer guide for oracle 10g : [URL] .... This condition self-heals itself in few hours or restarting the database or restarting the server. The root cause of this behavior is unknown.
View Replies
View Related
Aug 15, 2014
I am working on an Eclipse based project that uses WTP Eclipse plugin. Since I have switched security on in that plugin (by checking "Enable Security" check box on) I'm having multiple problems. I get a stack of security errors - attached in this post. I also show my java.policy file. Because the of the mass exceptions I list only the top, although I have attached a docx file with full thing.
Here is my java.policy file:
grant codeBase "file:${catalina.base}/wtpwebapps/App/WEB-INF/lib/hibernate-core-4.3.5.Final.jar" {
permission java.util.PropertyPermission "*", "read, write";
};
grant codeBase "file:${catalina.home}/lib/catalina.jar" {
permission java.lang.RuntimePermission "*";
[Code] ....
Exceptions:
15/08/2014 10:12:43 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
C:Javajdk1.6.0_33in;C:WINDOWSSunJavain;
C:WINDOWSsystem32;C:WINDOWS;
[Code] .....
View Replies
View Related
May 5, 2015
I'm just wondering whether it is possible to use images as a button in Java. I have two images that I want to use to create a rollover effect, is this possible? And then I would like to reset my java program when the button is clicked, is this also possible?
View Replies
View Related
Jan 26, 2012
I'm relatively new to java and to image processing ... What I've done so far is make a simple application which opens the webcam and can take and save a picture. Ive then loaded the image pixel data into a 2d array. How then do I do what is known as 'fast fourier transform' on it from here?
View Replies
View Related
Jan 24, 2014
How can i use multiple textures on a model? I have the .OBJ loader and .MTL loader and everything is working fine. I can use a single texture. But i have a model that has 2 textures. How can i use both of the textures?
View Replies
View Related
Oct 13, 2014
Boolean retrieval model using skiplist. I don't know how to code it. I have modified it but I'm not sure it work out. What are the modifications I can made.
/**My requirement is to implement two methods for Boolean Retrieval:
● index(String dir)
○ index()supposed to go over all files under "dir". There will be no subdirectories inside it.
● retrieve()
○ retrieve() supposed to return name of all the documents under "dir" that satisfies the given query. Note that, only basename of the files are to be returned, not the full path.
○ Query can be of two forms:
■ OR: returned doc should contain at least one term from the query.
■ AND: returned doc should contain all the terms from the query. **/
import java.util.HashSet;
import java.util.Vector;
public class BooleanRetrievalModel implements DocSearch {
// begin private class
private class SkipList {
// Node in skip-list
[Code] ....
View Replies
View Related
Apr 11, 2015
I have some paired values stored as a hashmap in my model, for example sake, we'll say the 'key' is a Manager's ID and the 'value' is the name of their department, so 'betty123' could be a key to a value 'IT', and 'dave345' could be a key for the value 'Finance'. This hashmap is populated from a database. In my form, I have two "Select" dropdown fields, one for "Department" (with fields which match the values in the hashmap), and one for "Manager ID". I want to autopopulate the "Manager ID" field when the "Department" field is changed, based on the values in the Hashmap. I understand that I can't access dynamically elements of the hashmap in a script, so, for example, I can't do this:
var specificDepartment= $("#department");
var managerId = ${departmentHash[specificDepartment]};
$("#manager").val(managerId);
Because this code is dynamic based on the value of "department" at the time it's run, whereas the departmentHash object is generated when the page is loaded.
Is there some other way I can do this? Am I approaching it the wrong way? FWIW, I'm using Spring 4.1.1 and jQuery 1.8. I understand the conceptual block with code I wrote above, but I can't for the life of me think of an obvious solution although I'm sure this is a fairly commonplace problem in web development.
View Replies
View Related
May 25, 2014
I am new to Java/OOP in general, and am trying to implement a multi-threaded system that contains a master thread, and a set of worker threads that are heterogeneous in the work they do. Once they complete the work, the workers indicate to the master by posting the result on to its queue. Here is the problem. The results of each type of work is different, and the master has to process each differently. In C (which I'm familiar with), this can be achieved by having a message type that is a union of all the expected messages, and by using a switch statement.
I thought of doing something similar in Java, by using instance of on each incoming message (each individual message class having been subclassed from a super message class) , and doing switch on that, but it doesn't seem to be the OO way to do things. The only other way I could think of was to implement an abstract method to get the type of each message, and then use the type in a switch statement, or if-then-else. Is there some other Java idiom to do this kind of processing? Also, if this is an acceptable method, why is it superior to using the reflection to find out the message type (instead of using the abstract getType())?
The message types look similar to the code below:
abstract class Message {
abstract String getType();
} class Result1 extends Message {
ResultType1 content;
String getType() {
[Code] ....
View Replies
View Related
Sep 23, 2014
I understand how vectors work I'm currently using one to store my id's from my txt file but how do you put them in a defaultComboBoxModel?
DefaultComboBoxModel defaultComboBoxModel = new DefaultComboBoxModel();
//declare a vector object that will hold a string (only thing that works with comboboxmodel
Vector<String> myVector=new Vector<String>();
//try statement
try{
FileReader fr = new FileReader(file);
[Code]...
Any example of how a vector is used with a defaultComboBoxModel so I can then use that to populate my JComboBox?
View Replies
View Related
Nov 14, 2014
public PurchaseTableModel() {
try{
//establish connection
conn = DriverManager.getConnection("jdbc:odbc:SupplierDS");
stmt = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
[code]....
I get the value from database and display in table , price value will show 6 decimal and date value will show date and time.What to do to show only 2 decimal places and only date e.g. 1/1/2014 in the table ?
View Replies
View Related
Feb 13, 2015
I am trying to process an array of pixels, checking each within a 5*5 mask. Ive declared my mask as a 2d array and get the pixel values within a nested for loop. Im getting an ArrayIndexOutOfBoundsException at the line:
mask[0][4] = dsIm.red[i-1][j+3];
Have I declared my mask wrong for a 5*5 mask. Is my logic wrong?
I won't give all the code, but basically Im trying to check all 25 pixel values within the mask:
int [][] mask = new int[5][5];
System.out.println("We are in the method just written");
//loop one loop
for(int i = 1; i <= h-2; i++)
{
for(int j = 1; j<=w-2; j++)
[Code] .....
View Replies
View Related
Feb 12, 2015
can we use java programming for image processing. if yes then how do we do it?
View Replies
View Related
Dec 18, 2014
I have 2 different bespoke table models.
When one is needed I simply add it to the table. In this case however whilst updating the model via table update in tableChanged method I have the impression of 2 identical models because the update happens twice (use of audio).
Is it not true that adding a new model cancels out the other ? I test for which model is attached to table and add the correct one depending on the data required.
assert (!(this.getTableModel() instanceof KeywordBeanTableModel));
this.initKeywordBeanTableModel();
View Replies
View Related
Mar 3, 2013
I am looking for the ability, on the server side, to run programs or "jobs" in a job queue, where the jobs are processed as first in first out. If you are familiar with the IBM iSeries, they have a built in job queue mechanism which accomplishes what I am looking for. The primary purposes for this would be to process and update large amounts of data in a thread safe environment.
View Replies
View Related
May 9, 2014
It seems we have abandoned Dice/Die(s), and are now working on something completely foreign. I don't even have a code to start with because I haven't the faintest clue what is going on here (no notes given on this topic, as usual). We are given 4 half-written programs to work with.
The instructions are:
"Examine the FormLetterEntry abstract class, and create the two derived classesTextEntry and DataItemEntry. Be sure to implement all the abstract methods in each derived class."
This is the code we were given:
package homework5;
import java.util.Properties;
/**
* Abstract class representing the entries in a form letter.
**/
public abstract class FormLetterEntry {
/**
* Retrieve the template string for this entry
* @return the value of this entry in a template
**/
[code].....
I understand (and correct me if I am wrong) that a derived class is a class that is created from a base class via inheritance. What I don't see are any notes on how to write a derived class. I see some notes online on how to do so, but they don't fit with what he's written above. What he means by "Be sure to implement all the abstract methods in each derived class."
View Replies
View Related
Oct 9, 2013
class mmm {
public static void main(String[] args) {
OcrEngine ocr = new OcrEngine();
ocr.setImage(ImageStream.fromFile("pp.tif"));
// ocr.setImage((IImageStream) new File("pp.tif"));
[Code] ....
In above code giving error .
Exception in thread "main" com.aspose.ms.System.IO.FileNotFoundException: Can't find file: pp.tiff.
Where i am wrong. I am trying to pass all formt(.png,.gif,.jpg) images. I am giving proper path ....
View Replies
View Related
Oct 8, 2014
I am writing a program that solves sudoku puzzles. It has a Swing GUI with a "solve" button. I have written an actionListener class with an actionPerformed method that is invoked when the user presses this button. It is the following:
public void actionPerformed(ActionEvent event) {
try {
worker = new SudokuSolveWorker();
worker.addPropertyChangeListener(this);
worker.execute();
SolveFrame sf = new SolveFrame();
} catch (Exception exc) {
System.out.println(exc.getMessage());
exc.printStackTrace();
}
}
The code creates a worker, a new instance of SudokuSolveWorker. The worker.execute() statement causes the doInBackground() method of this class to be called. This solves the Sudoku. A property change listener "picks up" the result:
public void propertyChange(PropertyChangeEvent event) {
try {
System.out.println("" + javax.swing.SwingUtilities.isEventDispatchThread());
SudokuModel solvedModel = (SudokuModel) worker.get();
} catch (Exception exc) {
System.out.println(exc.getMessage());
exc.printStackTrace();
}
}
As I wrote, this works without freezing up the user interface in the sense that the program seems unstable. However, the user interface does not respond to "commands" (mouse clicks) anymore until the worker thread has finished.
In the first code fragment I create an instance of SolveFrame which is an extension of a JFrame. It is a simple frame with a "cancel" button. It is drawn on the screen, even though it is called after the worker.execute() statement. I'd like the user to be able to click this "cancel" button, after which the solving of the sudoku puzzle should be stopped. However, since the program does not respond to mouse clicks anymore, the "cancel" button cannot be pressed.
View Replies
View Related
Oct 11, 2014
/*
*
* You need to implement BooleanRetrievalModel using SkipList.
* You should have everything within this file.
*/
import java.util.HashSet;
import java.util.Vector;
public class BooleanRetrievalModel implements DocSearch {
// begin private class
private class SkipList {
// Node in skip-list
[Code] ....
View Replies
View Related
Feb 18, 2014
I want to develop a Java EE application for the following scenario.
Webapp takes a file from a user and analyze the file. This analysis could take hours. User should be able to check if the analysis is finished via AJAX. When the analysis is finished user should be able to view the analysis report that has been generated by the analyzer.
I checked what are the possibles ways I could achieve this but couldn't get a clear idea. I heard about JMS, Work Manager API and servlet asynchronous processing. But still not sure what to use and how to use.I'm not very much familiar with EJB.
View Replies
View Related
Sep 1, 2007
Why is it simple for a word processing software (one that reads number of words, words greater that x characters etc) to deal with text file(.txt) and not with a doc file (.doc) file?? Is there any special requirements or what are the points taken into consideration while developing the software for a .doc file?
View Replies
View Related
Oct 1, 2014
I want to create a new instance of a Java model class for each new request coming to a servlet.
How to do that without doing that in doGet() or doPost().
View Replies
View Related
May 6, 2014
I am working on an project in which some data needs to be received through socket , do some processing on it and then store on the database. After that it will display on GUI. GUI should be updated automatically.
I am able to implement basic model in which data is storing in the database and displayed then on the GUI. I tried to implement MVC design patter as well.
Now i need to create a GUI which can be used by several users at a time means several jvm session for the GUI. I got struck at this point.
Some detail: i have one jvm session in which i am receiving , processing and storing the data. Now i want to fire an alert whenever any new entry is made into the database. Also each time new GUI is opened it should be registered with the model so that model send updated data alert to all this users.
When i am trying to open several GUI and trying to register with the Model then it is not increasing count of registered user. For each time it is showing 1 count only and whenever a new entry is made count is showing 0. Is it because both are running in different jvm sessions ?
Some code for the model which is running in different package and jvm
GUI..
private void runApp() {
//some gui code
WindowListenerClass windowListener = new WindowListenerClass(this);
frame.addWindowListener(windowListener)
//registering viewer with the model
Model.setNewAlarmEvent(this);
tableModel = windowListener.getDefaultTableModel();
[Code] .....
View Replies
View Related
Mar 12, 2014
Here is a link to the program: [URL] ....
I am trying to model a galaxy so that I have multiple stars certain distances away from each other and with different masses.
The accretion and than cylinder expansion happens with just about every galaxy.
I have a few questions.
First My computer only runs this if the number of particles is in the 500s. If that's the case How can I increase the probability that I will when I get to it have a red giant?
Second how can I modify this so that I have a timescale during which white dwarfs form from some stars and you see in the single stars either no supernova or nova just degeneration or a type 2 supernova and in binaries a type 1a supernova which is like a type 2 supernova of 1 star when it is a red giant, that giant becoming a white dwarf, the other star becoming a red giant, and then the white dwarf sucking gas from the giant until it explodes and ejects its companion?
Third how can I make this compacted and taking up less space in case I want to expand it to a few galaxies or even start with a big bang?
Basically what I am asking is how can I make this like a universe?
View Replies
View Related
