Massive Data Processing Using Unified Model

Dec 11, 2014

I have a requirement where we have an ETL jobs that runs in the backend which will get the up to date data from various sources , here we have both structured and unstructured data and the data capacity is massive which is in petabytes. Based on the data that is fetched we need to come up with a consolidated data view for the data which we retrieved. My queries are:-
 
1) How to create a consolidate data view , in what form it will be in (XML, JSON, etc)? whether we need to come up with a unified data model as the data is both structured and unstructured format? If so how we can go about it?

2) Since the data is really massive, how we can construct the consolidate data view? whether we need to create it in small chunks or whether we need to store these data in a cache or DB from where we will retrieve it and form the consolidated data view?

3) To process these data whether we need to have any Hadoop clusters to process these data parallel? As we are talking about massive data in the form of structured and un structured format?

4) Whether there is a need for NoSQL database to support unstructured data? If we are supposed to store the data?

View Replies


ADVERTISEMENT

Creating Table Model For Data From Database

Sep 9, 2014

I'd like to create my own Table Model to handle data from an SQLite database using JSwing, but I'm having difficulty. How to confirm the following:

-- A table model is an object that contains methods for manipulating data in the table, right?

If that's the case, then how should I create a Table Model to handle data coming from a database. From what I understand...

My custom Table Model needs to subclass AbstractTableModelI then override 3 methods from AbstractTableModel (getRowCount(), getColumnCount(), getValueAt())

As it relates to drawing data from a database, how should I be thinking about this problem (i.e. creating a Table Model that can work with a database)?

View Replies View Related

Design - How Map Request Parameters Data To Domain Model

May 24, 2014

This is a design question is the same problem in any language.as you do to map the controller to the domain model?We have situations in general larger than ... consider the example objects .

situation.1 - We have a request that has all the parameters of the account ;{ " id" : " 1 " , "name " : "test " , "some " : " xxx " } ............. and other fields .

situation.2 - can request that has to have a certain account parameters , for example in the case of an update;{" id" , " 1" , "name " , " testUpdated "}

situation.3 - We have a request that has some parameters of the account , others have more like id as user together;{ " id" : " 1 " , "user " : " xxx " , "service " : " yyy " } in which case each piece of the request will turn an object .

Java Code:

public class Account {

private Long id;
private String name ;
private String some ;

} mh_sh_highlight_all('java');

I see a few options ;

1 - Can I get AccountForm in the controller and set the properties for the Account object and others in CONTROLLER ;

+ ok for situation.1 situations 2, and situation.3

+ Separates the requisition of the object domain

- Pollutes the controller with code conversion

- Controller is full of setters .. if a higher class as a large object as a request is very confusing .

Java Code:

controller ( AccountForm from ) {
Account account = new Account ( )
account.setNome form.getNome = ();
account.setSome form.getSome = ();
Other outher = new Other ( ) ;
other.setSome ( form.getSome ( ) ) ;
} mh_sh_highlight_all('java');

2 - Can I get AccountRequest in the controller and have a method in itself as AccountRequest.getAccount ( ) to return a mapped model , in this case the mapping is at own Request object .

+ Separates the requisition of the object domain

+ Encapsulates the conversion in a place with easy access .

+ Meets situation.1 situation.2 and situation3 ;

- Request object has two responsibilities represent the request and map to a valid model .

Java Code:

controller ( AccountForm accountRequest ) {
Account account = accountRequest.getAccount ( ) ;
Outher outher accountRequest.getOther = ( )
} mh_sh_highlight_all('java');

3 - Can I get the controller Direct Account which had been filled with nulls .

+ Eliminate object request

- Serves only situation.1 situation.2 .

Java Code:

controller (Account account ) {
account.someMethod ();
} mh_sh_highlight_all('java');

4 - Outsource this mapping request parameters to another object mapper for request ..

+ Isolates logic mapping

- Until complexity for simpler cases are used as standard for all such a find by id .

- One more class for each request ;

In the case of API gets worse response has two further classes. speaking in terms of request for response .... AccountRequest, AccountRequestMapper, Account, AccountResponseMapper, AccountResponse .....I'm doing more testing the Hybrid option 3 for simple cases (find ID or updates) .... with option 2 for example for more complex cases ..

View Replies View Related

JDBC :: Resultset Data Processing - Default Fetchsize Returned

Jan 21, 2015

I have observed a strange behaviour from Resultset object. My application fetches 400 records from a table and processes these records every 10secs. By default the resultset has a fetchsize of 10 from the database cursor. As I understand if the query returns 400records, the resultset will fetch 40times, in multiple of 10 to get all these 400 records from database cursor.

Query : SELECT * FROM ( SELECT * FROM TestTable
WHERE STATE_ACTION = 0 ORDER BY ROP_TIME DESC )
WHERE ROWNUM <= 250

Observation : Under a normal operation, the resultset fetches all the 400 records on query execution from database cursor, but under unknown conditions the same resultset object fetches only 10 records from the database cursor and exits. Please refer page 297 in document below for the result fetch size details. JDBC developer guide for oracle 10g : [URL] .... This condition self-heals itself in few hours or restarting the database or restarting the server. The root cause of this behavior is unknown.

View Replies View Related

Switching Security On In WTP Eclipse Plugin Causing Massive Errors

Aug 15, 2014

I am working on an Eclipse based project that uses WTP Eclipse plugin. Since I have switched security on in that plugin (by checking "Enable Security" check box on) I'm having multiple problems. I get a stack of security errors - attached in this post. I also show my java.policy file. Because the of the mass exceptions I list only the top, although I have attached a docx file with full thing.

Here is my java.policy file:

grant codeBase "file:${catalina.base}/wtpwebapps/App/WEB-INF/lib/hibernate-core-4.3.5.Final.jar" {
permission java.util.PropertyPermission "*", "read, write";
};
grant codeBase "file:${catalina.home}/lib/catalina.jar" {
permission java.lang.RuntimePermission "*";

[Code] ....

Exceptions:

15/08/2014 10:12:43 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
C:Javajdk1.6.0_33in;C:WINDOWSSunJavain;
C:WINDOWSsystem32;C:WINDOWS;

[Code] .....

View Replies View Related

Using Images As Buttons In Processing

May 5, 2015

I'm just wondering whether it is possible to use images as a button in Java. I have two images that I want to use to create a rollover effect, is this possible? And then I would like to reset my java program when the button is clicked, is this also possible?

View Replies View Related

Image Processing In Java

Jan 26, 2012

I'm relatively new to java and to image processing ... What I've done so far is make a simple application which opens the webcam and can take and save a picture. Ive then loaded the image pixel data into a 2d array. How then do I do what is known as 'fast fourier transform' on it from here?

View Replies View Related

How To Use Multiple Textures On A Model

Jan 24, 2014

How can i use multiple textures on a model? I have the .OBJ loader and .MTL loader and everything is working fine. I can use a single texture. But i have a model that has 2 textures. How can i use both of the textures?

View Replies View Related

Boolean Retrieval Model Using SkipList

Oct 13, 2014

Boolean retrieval model using skiplist. I don't know how to code it. I have modified it but I'm not sure it work out. What are the modifications I can made.

/**My requirement is to implement two methods for Boolean Retrieval:

● index(String dir)
○ index()supposed to go over all files under "dir". There will be no subdirectories inside it.

● retrieve()
○ retrieve() supposed to return name of all the documents under "dir" that satisfies the given query. Note that, only basename of the files are to be returned, not the full path.
○ Query can be of two forms:
■ OR: returned doc should contain at least one term from the query.
■ AND: returned doc should contain all the terms from the query. **/

import java.util.HashSet;
import java.util.Vector;
public class BooleanRetrievalModel implements DocSearch {
// begin private class
private class SkipList {
// Node in skip-list

[Code] ....

View Replies View Related

JSP :: Use Hashmap From The Model In Script On The Page?

Apr 11, 2015

I have some paired values stored as a hashmap in my model, for example sake, we'll say the 'key' is a Manager's ID and the 'value' is the name of their department, so 'betty123' could be a key to a value 'IT', and 'dave345' could be a key for the value 'Finance'. This hashmap is populated from a database. In my form, I have two "Select" dropdown fields, one for "Department" (with fields which match the values in the hashmap), and one for "Manager ID". I want to autopopulate the "Manager ID" field when the "Department" field is changed, based on the values in the Hashmap. I understand that I can't access dynamically elements of the hashmap in a script, so, for example, I can't do this:

var specificDepartment= $("#department");
var managerId = ${departmentHash[specificDepartment]};
$("#manager").val(managerId);

Because this code is dynamic based on the value of "department" at the time it's run, whereas the departmentHash object is generated when the page is loaded.

Is there some other way I can do this? Am I approaching it the wrong way? FWIW, I'm using Spring 4.1.1 and jQuery 1.8. I understand the conceptual block with code I wrote above, but I can't for the life of me think of an obvious solution although I'm sure this is a fairly commonplace problem in web development.

View Replies View Related

Processing Queue Containing Dissimilar Messages

May 25, 2014

I am new to Java/OOP in general, and am trying to implement a multi-threaded system that contains a master thread, and a set of worker threads that are heterogeneous in the work they do. Once they complete the work, the workers indicate to the master by posting the result on to its queue. Here is the problem. The results of each type of work is different, and the master has to process each differently. In C (which I'm familiar with), this can be achieved by having a message type that is a union of all the expected messages, and by using a switch statement.

I thought of doing something similar in Java, by using instance of on each incoming message (each individual message class having been subclassed from a super message class) , and doing switch on that, but it doesn't seem to be the OO way to do things. The only other way I could think of was to implement an abstract method to get the type of each message, and then use the type in a switch statement, or if-then-else. Is there some other Java idiom to do this kind of processing? Also, if this is an acceptable method, why is it superior to using the reflection to find out the message type (instead of using the abstract getType())?

The message types look similar to the code below:

abstract class Message {
abstract String getType();
} class Result1 extends Message {
ResultType1 content;
String getType() {

[Code] ....

View Replies View Related

Swing/AWT/SWT :: Vector Is Used In ComboBox Model?

Sep 23, 2014

I understand how vectors work I'm currently using one to store my id's from my txt file but how do you put them in a defaultComboBoxModel?

DefaultComboBoxModel defaultComboBoxModel = new DefaultComboBoxModel();
//declare a vector object that will hold a string (only thing that works with comboboxmodel
Vector<String> myVector=new Vector<String>();
//try statement
try{
FileReader fr = new FileReader(file);

[Code]...

Any example of how a vector is used with a defaultComboBoxModel so I can then use that to populate my JComboBox?

View Replies View Related

Swing/AWT/SWT :: How To Format Value In Table Model

Nov 14, 2014

public PurchaseTableModel() {
try{
//establish connection
conn = DriverManager.getConnection("jdbc:odbc:SupplierDS");

stmt = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);

[code]....

I get the value from database and display in table , price value will show 6 decimal and date value will show date and time.What to do to show only 2 decimal places and only date e.g. 1/1/2014 in the table ?

View Replies View Related

Image Processing - Array Of Pixels

Feb 13, 2015

I am trying to process an array of pixels, checking each within a 5*5 mask. Ive declared my mask as a 2d array and get the pixel values within a nested for loop. Im getting an ArrayIndexOutOfBoundsException at the line:

mask[0][4] = dsIm.red[i-1][j+3];

Have I declared my mask wrong for a 5*5 mask. Is my logic wrong?

I won't give all the code, but basically Im trying to check all 25 pixel values within the mask:

int [][] mask = new int[5][5];
System.out.println("We are in the method just written");
//loop one loop
for(int i = 1; i <= h-2; i++)
{
for(int j = 1; j<=w-2; j++)

[Code] .....

View Replies View Related

Use Java Programming For Image Processing

Feb 12, 2015

can we use java programming for image processing. if yes then how do we do it?

View Replies View Related

Swing/AWT/SWT :: Table Model Removal

Dec 18, 2014

I have 2 different bespoke table models.

When one is needed I simply add it to the table. In this case however whilst updating the model via table update in tableChanged method I have the impression of 2 identical models because the update happens twice (use of audio).

Is it not true that adding a new model cancels out the other ? I test for which model is attached to table and add the correct one depending on the data required.

assert (!(this.getTableModel() instanceof KeywordBeanTableModel));
this.initKeywordBeanTableModel();

View Replies View Related

Enterprise JavaBeans :: First In First Out Job Queue Processing

Mar 3, 2013

I am looking for the ability, on the server side, to run programs or "jobs" in a job queue, where the jobs are processed as first in first out. If you are familiar with the IBM iSeries, they have a built in job queue mechanism which accomplishes what I am looking for. The primary purposes for this would be to process and update large amounts of data in a thread safe environment.

View Replies View Related

Polymorphic Containers And Text Processing In Java?

May 9, 2014

It seems we have abandoned Dice/Die(s), and are now working on something completely foreign. I don't even have a code to start with because I haven't the faintest clue what is going on here (no notes given on this topic, as usual). We are given 4 half-written programs to work with.

The instructions are:

"Examine the FormLetterEntry abstract class, and create the two derived classesTextEntry and DataItemEntry. Be sure to implement all the abstract methods in each derived class."

This is the code we were given:

package homework5;
import java.util.Properties;
/**
* Abstract class representing the entries in a form letter.
**/
public abstract class FormLetterEntry {
/**
* Retrieve the template string for this entry
* @return the value of this entry in a template
**/

[code].....

I understand (and correct me if I am wrong) that a derived class is a class that is created from a base class via inheritance. What I don't see are any notes on how to write a derived class. I see some notes online on how to do so, but they don't fit with what he's written above. What he means by "Be sure to implement all the abstract methods in each derived class."

View Replies View Related

OCR In Image Processing - Cannot Find TIFF File

Oct 9, 2013

class mmm {
public static void main(String[] args) {
OcrEngine ocr = new OcrEngine();
ocr.setImage(ImageStream.fromFile("pp.tif"));
// ocr.setImage((IImageStream) new File("pp.tif"));

[Code] ....

In above code giving error .

Exception in thread "main" com.aspose.ms.System.IO.FileNotFoundException: Can't find file: pp.tiff.

Where i am wrong. I am trying to pass all formt(.png,.gif,.jpg) images. I am giving proper path ....

View Replies View Related

Swing/AWT/SWT :: How To Do GUI Input Processing After Firing Off SwingWorker

Oct 8, 2014

I am writing a program that solves sudoku puzzles. It has a Swing GUI with a "solve" button. I have written an actionListener class with an actionPerformed method that is invoked when the user presses this button. It is the following:

public void actionPerformed(ActionEvent event) {
try {
worker = new SudokuSolveWorker();
worker.addPropertyChangeListener(this);
worker.execute();
SolveFrame sf = new SolveFrame();
} catch (Exception exc) {
System.out.println(exc.getMessage());
exc.printStackTrace();
}
}

The code creates a worker, a new instance of SudokuSolveWorker. The worker.execute() statement causes the doInBackground() method of this class to be called. This solves the Sudoku. A property change listener "picks up" the result:

public void propertyChange(PropertyChangeEvent event) {
try {
System.out.println("" + javax.swing.SwingUtilities.isEventDispatchThread());
SudokuModel solvedModel = (SudokuModel) worker.get();
} catch (Exception exc) {
System.out.println(exc.getMessage());
exc.printStackTrace();
}
}

As I wrote, this works without freezing up the user interface in the sense that the program seems unstable. However, the user interface does not respond to "commands" (mouse clicks) anymore until the worker thread has finished.

In the first code fragment I create an instance of SolveFrame which is an extension of a JFrame. It is a simple frame with a "cancel" button. It is drawn on the screen, even though it is called after the worker.execute() statement. I'd like the user to be able to click this "cancel" button, after which the solving of the sudoku puzzle should be stopped. However, since the program does not respond to mouse clicks anymore, the "cancel" button cannot be pressed.

View Replies View Related

Boolean Retrieval Model Implementation Using SkipList

Oct 11, 2014

/*
*
* You need to implement BooleanRetrievalModel using SkipList.
* You should have everything within this file.
*/

import java.util.HashSet;
import java.util.Vector;
public class BooleanRetrievalModel implements DocSearch {
// begin private class
private class SkipList {
// Node in skip-list

[Code] ....

View Replies View Related

EJB / EE :: Long Running Task Processing In Java

Feb 18, 2014

I want to develop a Java EE application for the following scenario.

Webapp takes a file from a user and analyze the file. This analysis could take hours. User should be able to check if the analysis is finished via AJAX. When the analysis is finished user should be able to view the analysis report that has been generated by the analyzer.

I checked what are the possibles ways I could achieve this but couldn't get a clear idea. I heard about JMS, Work Manager API and servlet asynchronous processing. But still not sure what to use and how to use.I'm not very much familiar with EJB.

View Replies View Related

Text Files And Word Processing Software

Sep 1, 2007

Why is it simple for a word processing software (one that reads number of words, words greater that x characters etc) to deal with text file(.txt) and not with a doc file (.doc) file?? Is there any special requirements or what are the points taken into consideration while developing the software for a .doc file?

View Replies View Related

Servlets :: New Instance Of Java Model For Each New Request

Oct 1, 2014

I want to create a new instance of a Java model class for each new request coming to a servlet.

How to do that without doing that in doGet() or doPost().

View Replies View Related

Implementation Of MVC With Several Viewers And Single Model / Controller?

May 6, 2014

I am working on an project in which some data needs to be received through socket , do some processing on it and then store on the database. After that it will display on GUI. GUI should be updated automatically.

I am able to implement basic model in which data is storing in the database and displayed then on the GUI. I tried to implement MVC design patter as well.

Now i need to create a GUI which can be used by several users at a time means several jvm session for the GUI. I got struck at this point.

Some detail: i have one jvm session in which i am receiving , processing and storing the data. Now i want to fire an alert whenever any new entry is made into the database. Also each time new GUI is opened it should be registered with the model so that model send updated data alert to all this users.

When i am trying to open several GUI and trying to register with the Model then it is not increasing count of registered user. For each time it is showing 1 count only and whenever a new entry is made count is showing 0. Is it because both are running in different jvm sessions ?

Some code for the model which is running in different package and jvm

GUI..

private void runApp() {
//some gui code
WindowListenerClass windowListener = new WindowListenerClass(this);
frame.addWindowListener(windowListener)
//registering viewer with the model
Model.setNewAlarmEvent(this);
tableModel = windowListener.getDefaultTableModel();

[Code] .....

View Replies View Related

Model Galaxy - Multiple Stars With Different Masses

Mar 12, 2014

Here is a link to the program: [URL] ....

I am trying to model a galaxy so that I have multiple stars certain distances away from each other and with different masses.

The accretion and than cylinder expansion happens with just about every galaxy.

I have a few questions.

First My computer only runs this if the number of particles is in the 500s. If that's the case How can I increase the probability that I will when I get to it have a red giant?

Second how can I modify this so that I have a timescale during which white dwarfs form from some stars and you see in the single stars either no supernova or nova just degeneration or a type 2 supernova and in binaries a type 1a supernova which is like a type 2 supernova of 1 star when it is a red giant, that giant becoming a white dwarf, the other star becoming a red giant, and then the white dwarf sucking gas from the giant until it explodes and ejects its companion?

Third how can I make this compacted and taking up less space in case I want to expand it to a few galaxies or even start with a big bang?

Basically what I am asking is how can I make this like a universe?

View Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved