Hadoop ?

What is Hadoop?

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. Hadoop makes it possible to run applications on systems with thousands of nodes involving thousands of terabytes. Its distributed file system facilitates rapid data transfer rates among nodes and allows the system to continue operating uninterrupted in case of a node failure. This approach lowers the risk of catastrophic system failure, even if a significant number of nodes become inoperative.

Hadoop was inspired by Google's Map Reduce , a software framework in which an application is broken down into numerous small parts. Any of these parts (also called fragments or blocks)

can be run on any node in the cluster. Doug Cutting, Hadoop's creator, named the framework after his child's stuffed toy elephant. The current Apache Hadoop ecosystem consists of the Hadoop kernel, MapReduce, the Hadoop distributed file system (HDFS) and a number of related projects such as Apache Hive, HBase and Zookeeper. The Hadoop framework is used by major players including Google, Yahoo and IBM, largely for applications involving search engines and advertising. The preferred operating systems are Windows and Linux but Hadoop can also work with BSD and OS X.

Why Hadoop? What is BigData?

Big data is a general term used to describe the voluminous amount of unstructured and semi-structured data a company creates, data that would take too much time and cost too much money to load into a relational database for analysis. (Big data doesn't refer to any specific quantity, the term is often used when speaking about petabytes and exabyte’s of data). A primary goal for looking at big data is to discover repeatable business patterns. It’s generally accepted that unstructured data, most of it located in text files, accounts for at least 80% of an organization’s data. If left unmanaged, the sheer volume of unstructured data that’s generated each year within an enterprise can be costly in terms of storage. Unmanaged data can also pose a liability if information cannot be located in the event of a compliance audit or lawsuit. Big data analytics is often associated with cloud computing because the analysis of large data sets in real-time requires a framework like Map Reduce to distribute the work among tens, hundreds or even thousands of computers.

Read More

Spring REST web service Test

We can test the Spring REST web service by following :

1. By java Junit class.
2. By SoapUI.
3. By Firefox browser plugin REST CLIENT.
4. By Chrome browser plugin POST MAN.


1. By java Junit class :
We can write the java junit class to test REST service. its simple test class in ur maven web application.



2. By SoapUI : We can test by using Open Source SoapUI tool, it very simple configure and test the web services.
http://www.soapui.org/REST-Testing/getting-started.html

3. By Firefox browser plugin REST CLIENT : we can use REST CLIENT plugin
https://addons.mozilla.org/en-US/firefox/addon/restclient/

4. By Chrome browser plugin POST MAN: we can use POST MAN plugin
https://chrome.google.com/webstore/detail/fdmmgilgnpjigdojojpjoooidkmcomcm

Read More

WARN (DefaultHandlerExceptionResolver.java:183) - Request method 'POST' not supported. For Spring REST

When Spring RESTful web service post call are throwing following exception, then we need to check 'Content-type' is set in the header/request.

set the content-type in the testing client like SoapUI.
if the REST controller request/response of 'json' then set the content-type as
"Content-Type" : "application/json"
in the request headers.

Read More

java.lang.classnotfoundexception org.springframework.web.context.contextloaderlistener / ClassNotFoundException when running a Spring + Maven2 project on Tomcat from within Eclipse

I had a similar problem when running a spring web application in an Eclipse managed tomcat. I solved this problem by adding maven dependencies in the project's web deployment assembly.

1) Open the project's properties (e.g.right-click on the project's name in the project explorer and select "Properties")
2) select "Deployment Assembly"
3) Click the "Add..." button on the right margin
4) Select "Java Build Path Entries" from the menu of Directive Type and click "Next"
5) Select "Maven Dependencies" from the Java Build Path Entries menu and click "Finish".
You should see "Maven Dependencies" added to the Web Deployment Assembly definition.

Read More

CopyOnWriteArrayList difference with normal ArrayList

CopyOnWriteArrayList was introduced in JDK5, Which will be used to avoid the java.util.ConcurrentModificationException while modifying List by multiple threads or inside loop.

below sample will show the difference.

Read More

MySql DB commands

for mysql :

create database test;
grant all on test.* to test@'localhost' identified by 'test';
grant all on test.* to test@'%' identified by 'test';

to run scripts in mysql

mysql> source <path to sql file>
MySql export schema without data
mysqldump -u root -p --no-data dbname > schema.sql

Read More

JVM_PermGen-space

Java virtual machine has Four generations :


eden, young, old and permanent.

In the eden generation, objects are very short lived and garbage collection is swift and often.

The young generation consists of objects that survived the eden generation (or was pushed down to young because the eden generation was full at the time of allocation), garbage collection in the young generation is less frequent but still happens at quite regular intervals (provided that your application actually does something and allocates objects every now and then).

The old generation, well, you figured it. It contains objects that survived the young generation, or have been pushed down, and garbage collection is even less infrequent but can still happen.

And finally, the permanent generation. This is for objects that the virtual machine has decided to endorse with eternal life - which is precicely the core of the problem. Objects in the permanent generation are never garbage collected; that is, under normal circumstances when the jvm is started with normal command line parameters.

set JAVA_OPTS=-Xms512m -Xmx512m
-XX:PermSize=128m
-XX:MaxPermSize=512m
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
 

Read More