Design Pattern: Strategy

 Strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. It lets the algorithm vary independently from clients that use it.

Strategy Design Pattern
Strategy Design Pattern

Abstract Class: Duck.java

We create objects  after sub classing it into a concrete class from an abstract type. In our case, is the Duck Class. This provides the basis for all the XxxDuck objects that are created. Two things to be mentioned;

  1. We are programming the different “behavior” to an interface. And this could be an abstract class as well. References to these instance variables are provided as instance variables in the Class Definition.
  2. Public “Setters” are provided so that the behavior of the object can be set or changed dynamically as shown in this example.
package com.npk.strategy;

public abstract class Duck {

FlyBehavior flyBehavior;
QuackBehavior quackBehavior;

public abstract void display();

public void setFlyBehavior(FlyBehavior flyBehavior){
this.flyBehavior = flyBehavior;
}

public void setQuackBehavior(QuackBehavior quackBehavior){
this.quackBehavior = quackBehavior;
}

public void performFly(){
flyBehavior.fly();
}

public void performQuack(){
quackBehavior.quack();
}

public void performSwim(){
System.out.println("All ducks can swim!");
}
}

Interface: FlyBehavior.java and QuackBehavior.java

Any variable behavior of an object are to be programmed to an interface.
This way, you get two things:

  1. Compostion
  2. Loosely coupled objects.

Both of which are good OO Designs aspects.


package com.npk.strategy;

public interface FlyBehavior {

public void fly();
}

package com.npk.strategy;

public interface QuackBehavior {

public void quack();
}

Concrete Implementation: DummyDuck.java

We subclass the abstract Duck class to get a concrete DummyDuck class where we can set the behavior in the class constructor ( optional )


package com.npk.strategy;

public class DummyDuck extends Duck {

public DummyDuck(){
this.setFlyBehavior(new FlyNoWay());
this.setQuackBehavior(new SquekQuack());
}

@Override
public void display() {
System.out.println("This is a dummy duck.");

}

}

Concrete classes: FlyNoWay.java and SquekQuack.java

These form the actual implementation of the FlyBehavior and the QuackBehavior interfaces respectively.


package com.npk.strategy;

public class FlyNoWay implements FlyBehavior {

@Override
public void fly() {
System.out.println("Cannot fly.");

}

}

package com.npk.strategy;

public class SquekQuack implements QuackBehavior {

@Override
public void quack() {
System.out.println("Duck squeking.");

}

}

The Application Launcher We show two things here.

  1. Setting the behavior using the constructor.
  2. Changing the set behavior using the setters provided.

package com.npk.strategy;

public class Launcher {

public static void main(String[] args) {
Duck duck = new DummyDuck();
duck.display();
duck.performFly();
duck.performQuack();

System.out.println("---------------------");
duck.setFlyBehavior(new FlyWithWings());
duck.setQuackBehavior(new MuteQuack());

duck.performFly();
duck.performQuack();
}

}

The final output


This is a dummy duck.
Cannot fly.
Duck squeking.
---------------------
Duck flying.
Mute quack.

Advertisements
Design Pattern: Strategy

Linked Lists in Java

You might want to check out these before continuing:

https://linuxjunkies.wordpress.com/2010/11/18/data-structures-using-java-part-6-linked-lists-part-1/

https://linuxjunkies.wordpress.com/2010/11/26/data-structures-using-java-part-7-linked-lists-part-2-doubly-linked/

Linked List

ListNode Class


public class ListNode {

	Object data;
	ListNode nextNode;
	
	ListNode (Object object, ListNode node){
		data = object;
		nextNode = node;
	}
	
	ListNode (Object object){
		this(object,null);
	}
	
	Object getObject(){
		return data;
	}
	
	ListNode getNext(){
		return nextNode;
	}
}

List Class


public class List {

	private ListNode firstNode;
	private ListNode lastNode;
	private String name;
	
	public List(){
		this("list");
	}
	
	public List(String listname){
		name = listname;
		firstNode = lastNode = null;
	}
	
	public void insertAtFront(Object item){
		if (isEmpty())
			firstNode = lastNode = new ListNode(item);
		else
			firstNode = new ListNode(item, firstNode);
	}
	
	public void insertAtBack(Object item){
		if (isEmpty())
			firstNode = lastNode = new ListNode(item);
		else
			lastNode = lastNode.nextNode = new ListNode(item);
	}
	
	public Object removeFromFront() throws EmptyListException{
		if (isEmpty())
			throw new EmptyListException(name);
		Object removed = firstNode.data;
		
		if( firstNode == lastNode)
			firstNode = lastNode = null;
		else
			firstNode = firstNode.nextNode;
		
		return removed;
	}
	
	public Object removeFromBack() throws EmptyListException{
		if (isEmpty())
			throw new EmptyListException(name);
		
		Object removed = lastNode.data;
		
		if( firstNode == lastNode)
			firstNode = lastNode = null;
		else{
			
			ListNode current = firstNode;
			
			while(current.nextNode != lastNode)
				current = current.nextNode;
			
			lastNode = current;
			current.nextNode = null;	
		}
		return removed;
		
	}
	
	public boolean isEmpty(){
		return firstNode == null;
	}
	
	public void print(){
		if (isEmpty()){
			System.out.println("List is empty");
			return;
		}
		
		ListNode current = firstNode;
		while (current != null){
			System.out.printf("%s --> ", current.data);
			current = current.nextNode;
		}
		System.out.println();
	}
	
	public int length(){
		int length = 0;
		
		if (isEmpty()){
			return 0;
		}
		
		ListNode current = firstNode;
		while (current != null){
			current = current.nextNode;
			length++;
		}
		return length;
	}
	
	public int recursiveSize(){
		return recursiveLength(firstNode);
	}
	
	public int recursiveLength(ListNode current){
		
		if (isEmpty()){
			return 0;
		}
		
		if (current == null)
			return 0;
		
		return (1 + recursiveLength(current.nextNode));
	}
	
}
	

ListTest Class


public class ListTest {

	public static void main(String[] args){
		
		List list = new List();
		
		list.insertAtFront( "four" ); 
		list.print();             
		list.insertAtFront( "two" ); 
		list.print();            
		list.insertAtBack( "six" );  
		list.print();          
		list.insertAtBack( "nine" );  
		list.print();            
		
		System.out.printf("Length: %d%n", list.recursiveSize());
		System.out.printf("Length: %d%n", list.length());
		
		try{
		    System.out.printf( "%s removed\n", list.removeFromFront());
		    list.print();
		    System.out.printf( "%s removed\n", list.removeFromFront());
		    list.print();
		    System.out.printf( "%s removed\n", list.removeFromBack());
		    list.print();
		    System.out.printf( "%s removed\n", list.removeFromBack());
		    list.print();
		}catch ( EmptyListException e ){
			e.printStackTrace();
		}
		
		System.out.printf("Length: %d%n", list.recursiveSize());
		System.out.printf("Length: %d%n", list.length());
		
		       
	}
}

This is what you get

four --> 
two --> four --> 
two --> four --> six --> 
two --> four --> six --> nine --> 
Length: 4
Length: 4
two removed
four --> six --> nine --> 
four removed
six --> nine --> 
nine removed
six --> 
six removed
List is empty
Length: 0
Length: 0
Linked Lists in Java

String Reversal in Java using StringBuffer, toCharacterArray and Recursion

Using StringBuffer Class

new StringBuffer(name).reverse().toString()

Using CharArray and StringBuilder

        public static String charReverse(String data){
		StringBuilder sb = new StringBuilder();
		char[] ca = data.toCharArray();
		
		for (int i=data.length()-1; i>=0;i--){
			sb.append(ca[i]);
		}
		return sb.toString();
	}

Using Recursion

public static String recursiveReverse(String data){
		
		if (data.length() < 2)
			return data;
		
		return recursiveReverse(data.substring(1)) + data.charAt(0);
	}

StringReversal Class

public class StringReversal {

	public static void main (String[] args){
		String name = "linuxjunkies";
		
		System.out.println("StringBuffer: " + new StringBuffer(name).reverse().toString());
		System.out.println("CharacterArray: " + charReverse(name));
		System.out.println("Recursion: " + recursiveReverse(name));
	}
	
	public static String charReverse(String data){
		StringBuilder sb = new StringBuilder();
		char[] ca = data.toCharArray();
		
		for (int i=data.length()-1; i>=0;i--){
			sb.append(ca[i]);
		}
		return sb.toString();
	}
	
	public static String recursiveReverse(String data){
		
		if (data.length() < 2)
			return data;
		
		return recursiveReverse(data.substring(1)) + data.charAt(0);
	}
}

The Output

StringBuffer: seiknujxunil
CharacterArray: seiknujxunil
Recursion: seiknujxunil
String Reversal in Java using StringBuffer, toCharacterArray and Recursion

A HDFSClient for Hadoop using the native JAVA API, a tutorial

I’d like to talk about doing some day to day administrative task on the Hadoop system. Although the hadoop fs <commands> can get you to do most of the things, its still worthwhile to explore the rich API in Java for Hadoop. This post is by no means complete, but can get you started well.

The most basic step is to create an object of this class.

HDFSClient client = new HDFSClient();

Of course, you need to import a bunch of stuff. But if you are using an IDE like Eclipse, you’ll follow along just fine just by importing these. This should word fine for the entire code.

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;

1. Copying from Local file system to HDFS.
Copies a local file onto HDFS. You do have the hadoop file system command to do the same.

hadoop fs -copyFromLocal <local fs> <hadoop fs>

I am not explaining much here as the comments are quite helpful. Of course, while importing the configuration files, make sure to point them to your hadoop systems location. For mine, it looks like this:

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

This is how the Java API looks like:

public void copyFromLocal (String source, String dest) throws IOException {

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

Path dstPath = new Path(dest);
// Check if the file already exists
if (!(fileSystem.exists(dstPath))) {
System.out.println("No such destination " + dstPath);
return;
}

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

try{
fileSystem.copyFromLocalFile(srcPath, dstPath);
System.out.println("File " + filename + "copied to " + dest);
}catch(Exception e){
System.err.println("Exception caught! :" + e);
System.exit(1);
}finally{
fileSystem.close();
}
}

2.Copying files from HDFS to the local file system.

The hadoop fs command is the following.

hadoop fs -copyToLocal <hadoop fs> <local fs>

Alternatively,

hadoop fs -copyToLocal
public void copyFromHdfs (String source, String dest) throws IOException {

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

Path dstPath = new Path(dest);
// Check if the file already exists
if (!(fileSystem.exists(dstPath))) {
System.out.println("No such destination " + dstPath);
return;
}

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

try{
fileSystem.copyToLocalFile(srcPath, dstPath)
System.out.println("File " + filename + "copied to " + dest);
}catch(Exception e){
System.err.println("Exception caught! :" + e);
System.exit(1);
}finally{
fileSystem.close();
}
}

3.Renaming a file in HDFS.

You can use the mv command in this context.

hadoop fs -mv <this name> <new name>
public void renameFile (String fromthis, String tothis) throws IOException{
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path fromPath = new Path(fromthis);
Path toPath = new Path(tothis);

if (!(fileSystem.exists(fromPath))) {
System.out.println("No such destination " + fromPath);
return;
}

if (fileSystem.exists(toPath)) {
System.out.println("Already exists! " + toPath);
return;
}

try{
boolean isRenamed = fileSystem.rename(fromPath, toPath);
if(isRenamed){
System.out.println("Renamed from " + fromthis + "to " + tothis);
}
}catch(Exception e){
System.out.println("Exception :" + e);
System.exit(1);
}finally{
fileSystem.close();
}

}

4.Upload or add a file to HDFS

public void addFile(String source, String dest) throws IOException {

// Conf object will read the HDFS configuration parameters
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

// Create the destination path including the filename.
if (dest.charAt(dest.length() - 1) != '/') {
dest = dest + "/" + filename;
} else {
dest = dest + filename;
}

// Check if the file already exists
Path path = new Path(dest);
if (fileSystem.exists(path)) {
System.out.println("File " + dest + " already exists");
return;
}

// Create a new file and write data to it.
FSDataOutputStream out = fileSystem.create(path);
InputStream in = new BufferedInputStream(new FileInputStream(
new File(source)));

byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
out.write(b, 0, numBytes);
}

// Close all the file descripters
in.close();
out.close();
fileSystem.close();
}

5.Delete a file from HDFS.

You can use the following:

For removing a directory or a file:

hadoop fs -rmr <hdfs path>

If you want to skip the trash also, use:

hadoop fs -rmr -skipTrash <hdfs path>

 

public void deleteFile(String file) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(file);
if (!fileSystem.exists(path)) {
System.out.println("File " + file + " does not exists");
return;
}

fileSystem.delete(new Path(file), true);

fileSystem.close();
}

6.Get modification time of a file in HDFS.

If you have any idea on this let me know. 🙂

public void getModificationTime(String source) throws IOException{

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

// Check if the file already exists
if (!(fileSystem.exists(srcPath))) {
System.out.println("No such destination " + srcPath);
return;
}
// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

FileStatus fileStatus = fileSystem.getFileStatus(srcPath);
long modificationTime = fileStatus.getModificationTime();

System.out.format("File %s; Modification time : %0.2f %n",filename,modificationTime);

}

7.Get the block locations of a file in HDFS.

public void getBlockLocations(String source) throws IOException{

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

// Check if the file already exists
if (!(ifExists(srcPath))) {
System.out.println("No such destination " + srcPath);
return;
}
// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

FileStatus fileStatus = fileSystem.getFileStatus(srcPath);

BlockLocation[] blkLocations = fileSystem.getFileBlockLocations(fileStatus, 0, fileStatus.getLen());
int blkCount = blkLocations.length;

System.out.println("File :" + filename + "stored at:");
for (int i=0; i < blkCount; i++) {
String[] hosts = blkLocations[i].getHosts();
System.out.format("Host %d: %s %n", i, hosts);
}

}

8.List all the datanodes in terms of hostnames.
This is a neat way rather than looking up the /etc/hosts file in the namenode.

public void getHostnames () throws IOException{
Configuration config = new Configuration();
config.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fs = FileSystem.get(config);
DistributedFileSystem hdfs = (DistributedFileSystem) fs;
DatanodeInfo[] dataNodeStats = hdfs.getDataNodeStats();

String[] names = new String[dataNodeStats.length];
for (int i = 0; i < dataNodeStats.length; i++) {
names[i] = dataNodeStats[i].getHostName();
System.out.println((dataNodeStats[i].getHostName()));
}
}

9.Create a new directory in HDFS.
Creating a directory will be done as:

hadoop fs -mkdir <hadoop fs path>

 

public void mkdir(String dir) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(dir);
if (fileSystem.exists(path)) {
System.out.println("Dir " + dir + " already exists!");
return;
}

fileSystem.mkdirs(path);

fileSystem.close();
}

10. Read a file from HDFS

public void readFile(String file) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(file);
if (!fileSystem.exists(path)) {
System.out.println("File " + file + " does not exists");
return;
}

FSDataInputStream in = fileSystem.open(path);

String filename = file.substring(file.lastIndexOf('/') + 1,
file.length());

OutputStream out = new BufferedOutputStream(new FileOutputStream(
new File(filename)));

byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
out.write(b, 0, numBytes);
}

in.close();
out.close();
fileSystem.close();
}

11.Checking if a file exists in HDFS

public boolean ifExists (Path source) throws IOException{

Configuration config = new Configuration();
config.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem hdfs = FileSystem.get(config);
boolean isExists = hdfs.exists(source);
return isExists;
}

I know this is no way complete. But this is a rather long post. I hope it is useful. Responses appreciated!
And here is the complete code for HDFSClient.java. Happy Hadooping! 🙂

/*
Feel free to use, copy and distribute this program in any form.
HDFSClient.java
https://linuxjunkies.wordpress.com/
2011
*/

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.BlockLocation;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;

public class HDFSClient {

public HDFSClient() {

}

public static void printUsage(){
System.out.println("Usage: hdfsclient add" + "<local_path> <hdfs_path>");
System.out.println("Usage: hdfsclient read" + "<hdfs_path>");
System.out.println("Usage: hdfsclient delete" + "<hdfs_path>");
System.out.println("Usage: hdfsclient mkdir" + "<hdfs_path>");
System.out.println("Usage: hdfsclient copyfromlocal" + "<local_path> <hdfs_path>");
System.out.println("Usage: hdfsclient copytolocal" + " <hdfs_path> <local_path> ");
System.out.println("Usage: hdfsclient modificationtime" + "<hdfs_path>");
System.out.println("Usage: hdfsclient getblocklocations" + "<hdfs_path>");
System.out.println("Usage: hdfsclient gethostnames");
}

public boolean ifExists (Path source) throws IOException{

Configuration config = new Configuration();
config.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem hdfs = FileSystem.get(config);
boolean isExists = hdfs.exists(source);
return isExists;
}

public void getHostnames () throws IOException{
Configuration config = new Configuration();
config.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
config.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fs = FileSystem.get(config);
DistributedFileSystem hdfs = (DistributedFileSystem) fs;
DatanodeInfo[] dataNodeStats = hdfs.getDataNodeStats();

String[] names = new String[dataNodeStats.length];
for (int i = 0; i < dataNodeStats.length; i++) {
names[i] = dataNodeStats[i].getHostName();
System.out.println((dataNodeStats[i].getHostName()));
}
}

public void getBlockLocations(String source) throws IOException{

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

// Check if the file already exists
if (!(ifExists(srcPath))) {
System.out.println("No such destination " + srcPath);
return;
}
// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

FileStatus fileStatus = fileSystem.getFileStatus(srcPath);

BlockLocation[] blkLocations = fileSystem.getFileBlockLocations(fileStatus, 0, fileStatus.getLen());
int blkCount = blkLocations.length;

System.out.println("File :" + filename + "stored at:");
for (int i=0; i < blkCount; i++) {
String[] hosts = blkLocations[i].getHosts();
System.out.format("Host %d: %s %n", i, hosts);
}

}

public void getModificationTime(String source) throws IOException{

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

// Check if the file already exists
if (!(fileSystem.exists(srcPath))) {
System.out.println("No such destination " + srcPath);
return;
}
// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

FileStatus fileStatus = fileSystem.getFileStatus(srcPath);
long modificationTime = fileStatus.getModificationTime();

System.out.format("File %s; Modification time : %0.2f %n",filename,modificationTime);

}

public void copyFromLocal (String source, String dest) throws IOException {

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

Path dstPath = new Path(dest);
// Check if the file already exists
if (!(fileSystem.exists(dstPath))) {
System.out.println("No such destination " + dstPath);
return;
}

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

try{
fileSystem.copyFromLocalFile(srcPath, dstPath);
System.out.println("File " + filename + "copied to " + dest);
}catch(Exception e){
System.err.println("Exception caught! :" + e);
System.exit(1);
}finally{
fileSystem.close();
}
}

public void copyToLocal (String source, String dest) throws IOException {

Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path srcPath = new Path(source);

Path dstPath = new Path(dest);
// Check if the file already exists
if (!(fileSystem.exists(srcPath))) {
System.out.println("No such destination " + srcPath);
return;
}

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

try{
fileSystem.copyToLocalFile(srcPath, dstPath);
System.out.println("File " + filename + "copied to " + dest);
}catch(Exception e){
System.err.println("Exception caught! :" + e);
System.exit(1);
}finally{
fileSystem.close();
}
}

public void renameFile (String fromthis, String tothis) throws IOException{
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);
Path fromPath = new Path(fromthis);
Path toPath = new Path(tothis);

if (!(fileSystem.exists(fromPath))) {
System.out.println("No such destination " + fromPath);
return;
}

if (fileSystem.exists(toPath)) {
System.out.println("Already exists! " + toPath);
return;
}

try{
boolean isRenamed = fileSystem.rename(fromPath, toPath);
if(isRenamed){
System.out.println("Renamed from " + fromthis + "to " + tothis);
}
}catch(Exception e){
System.out.println("Exception :" + e);
System.exit(1);
}finally{
fileSystem.close();
}

}

public void addFile(String source, String dest) throws IOException {

// Conf object will read the HDFS configuration parameters
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

// Get the filename out of the file path
String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

// Create the destination path including the filename.
if (dest.charAt(dest.length() - 1) != '/') {
dest = dest + "/" + filename;
} else {
dest = dest + filename;
}

// Check if the file already exists
Path path = new Path(dest);
if (fileSystem.exists(path)) {
System.out.println("File " + dest + " already exists");
return;
}

// Create a new file and write data to it.
FSDataOutputStream out = fileSystem.create(path);
InputStream in = new BufferedInputStream(new FileInputStream(
new File(source)));

byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
out.write(b, 0, numBytes);
}

// Close all the file descripters
in.close();
out.close();
fileSystem.close();
}

public void readFile(String file) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(file);
if (!fileSystem.exists(path)) {
System.out.println("File " + file + " does not exists");
return;
}

FSDataInputStream in = fileSystem.open(path);

String filename = file.substring(file.lastIndexOf('/') + 1,
file.length());

OutputStream out = new BufferedOutputStream(new FileOutputStream(
new File(filename)));

byte[] b = new byte[1024];
int numBytes = 0;
while ((numBytes = in.read(b)) > 0) {
out.write(b, 0, numBytes);
}

in.close();
out.close();
fileSystem.close();
}

public void deleteFile(String file) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(file);
if (!fileSystem.exists(path)) {
System.out.println("File " + file + " does not exists");
return;
}

fileSystem.delete(new Path(file), true);

fileSystem.close();
}

public void mkdir(String dir) throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

FileSystem fileSystem = FileSystem.get(conf);

Path path = new Path(dir);
if (fileSystem.exists(path)) {
System.out.println("Dir " + dir + " already exists!");
return;
}

fileSystem.mkdirs(path);

fileSystem.close();
}

public static void main(String[] args) throws IOException {

if (args.length < 1) {
printUsage();
System.exit(1);
}

HDFSClient client = new HDFSClient();
if (args[0].equals("add")) {
if (args.length < 3) {
System.out.println("Usage: hdfsclient add <local_path> " + "<hdfs_path>");
System.exit(1);
}
client.addFile(args[1], args[2]);

} else if (args[0].equals("read")) {
if (args.length < 2) {
System.out.println("Usage: hdfsclient read <hdfs_path>");
System.exit(1);
}
client.readFile(args[1]);

} else if (args[0].equals("delete")) {
if (args.length < 2) {
System.out.println("Usage: hdfsclient delete <hdfs_path>");
System.exit(1);
}

client.deleteFile(args[1]);
} else if (args[0].equals("mkdir")) {
if (args.length < 2) {
System.out.println("Usage: hdfsclient mkdir <hdfs_path>");
System.exit(1);
}

client.mkdir(args[1]);
}else if (args[0].equals("copyfromlocal")) {
if (args.length < 3) {
System.out.println("Usage: hdfsclient copyfromlocal <from_local_path> <to_hdfs_path>");
System.exit(1);
}

client.copyFromLocal(args[1], args[2]);
} else if (args[0].equals("rename")) {
if (args.length < 3) {
System.out.println("Usage: hdfsclient rename <old_hdfs_path> <new_hdfs_path>");
System.exit(1);
}

client.renameFile(args[1], args[2]);
}else if (args[0].equals("copytolocal")) {
if (args.length < 3) {
System.out.println("Usage: hdfsclient copytolocal <from_hdfs_path> <to_local_path>");
System.exit(1);
}

client.copyToLocal(args[1], args[2]);
}else if (args[0].equals("modificationtime")) {
if (args.length < 2) {
System.out.println("Usage: hdfsclient modificationtime <hdfs_path>");
System.exit(1);
}

client.getModificationTime(args[1]);
}else if (args[0].equals("getblocklocations")) {
if (args.length < 2) {
System.out.println("Usage: hdfsclient getblocklocations <hdfs_path>");
System.exit(1);
}

client.getBlockLocations(args[1]);
} else if (args[0].equals("gethostnames")) {

client.getHostnames();
}else {

printUsage();
System.exit(1);
}

System.out.println("Done!");
}
}
A HDFSClient for Hadoop using the native JAVA API, a tutorial

The entire Server code in all its Glory!


import java.io.*;
import java.net.*;
import java.util.*;

public class Server {

 ArrayList clientOutputStreams;

 //String[] output;

 public class ClientHandler implements Runnable{

 BufferedReader reader;
 Socket sock;

 //constructor
 public ClientHandler(Socket clientSocket){

 try{
 sock = clientSocket;
 InputStreamReader isReader = new InputStreamReader(sock.getInputStream());
 reader = new BufferedReader(isReader);
 }catch(Exception ex){
 ex.printStackTrace();
 }
 }

 public void run(){
 String message;
 try{

 while((message = reader.readLine()) != null){
 System.out.println("Read " + message);
 tellEveryone(message);
 }
 }catch(Exception e){
 e.printStackTrace();
 }
 }

 }


 public void go(){
 clientOutputStreams = new ArrayList();


 try{
 ServerSocket serverSock = new ServerSocket(5000);

 while(true){
 Socket clientSocket = serverSock.accept();
 PrintWriter writer = new PrintWriter(clientSocket.getOutputStream());
 clientOutputStreams.add(writer);
 //writer.println(clientOutputStreams);
 Thread t = new Thread(new ClientHandler(clientSocket));
 t.start();
 System.out.println("Got a connection!");
 }

 }catch(Exception exx){
 exx.printStackTrace();
 }
 }

 public void tellEveryone(String messages){
 Iterator it = clientOutputStreams.iterator();
 while(it.hasNext()){

 try{

 PrintWriter writer = (PrintWriter) it.next();
 writer.println(messages);
 writer.flush();
 }catch(Exception e){
 e.printStackTrace();
 }
 }
 }

 public static void main(String[] args){
 new Server().go();
 }

}

The entire Server code in all its Glory!

The entire Chat Application in its glory!

import java.io.*;
import java.util.*;
import javax.swing.*;
import java.net.*;
import java.awt.*;
import java.awt.event.*;

public class Client {

	JTextArea incoming;
	JTextField outgoing;
	JButton sendbutton;

	BufferedReader reader;
	PrintWriter writer;

	Socket sock;

	public void go(){
		JFrame frame = new JFrame("Client");
		JPanel mainpanel = new JPanel();

		incoming = new JTextArea(15,50);
		incoming.setLineWrap(true);
		incoming.setWrapStyleWord(true);
		incoming.setEditable(false);

		JScrollPane qScroller = new JScrollPane(incoming);
		qScroller.setVerticalScrollBarPolicy(ScrollPaneConstants.VERTICAL_SCROLLBAR_ALWAYS);
		qScroller.setHorizontalScrollBarPolicy(ScrollPaneConstants.HORIZONTAL_SCROLLBAR_ALWAYS);

		outgoing = new JTextField(20);
		sendbutton = new JButton("Send");

		sendbutton.addActionListener(new SendButtonListener());

		mainpanel.add(qScroller);
		mainpanel.add(outgoing);
		mainpanel.add(sendbutton);

		setUpNetworking();

		Thread readerThread = new Thread(new IncomingReader());
		readerThread.start();

		frame.getContentPane().add(BorderLayout.CENTER, mainpanel);
		frame.setSize(400, 400);
		frame.setVisible(true);

	}

	public static void main(String[] args){

		Client client = new Client();
		client.go();
	}

	private void setUpNetworking(){

		try{

			sock = new Socket("127.0.0.1", 5000);
			InputStreamReader streamReader = new InputStreamReader(sock.getInputStream());
			reader = new BufferedReader(streamReader);
			writer = new PrintWriter(sock.getOutputStream());
			System.out.println("Networking established!!");

		}catch(IOException e){
			e.printStackTrace();
		}
	}

	public class SendButtonListener implements ActionListener{

		public void actionPerformed(ActionEvent e){
			try{
				writer.println(outgoing.getText());
				writer.flush();
			}catch(Exception ex){
				ex.printStackTrace();
			}

			outgoing.setText("");
			outgoing.requestFocus();
		}
	}

	public class IncomingReader implements Runnable{

		public void run(){
			String message;
			try{
				while((message = reader.readLine()) != null){
					System.out.println("read" + message);
					incoming.append(message + "\n");
					//incoming.setText(message + "\n");
				}
			}catch(Exception e){
				e.printStackTrace();
			}
		}
	}

}

The entire Chat Application in its glory!

A Simple Chat Server in Java using threads Part 1

The Client
———

The client side first. We import the necessary stuff  needed for this program. The java.net.* contains all
the necessary packages for the networking part. The swing and awt packages are for the GUI.

import java.io.*;
import java.util.*;
import javax.swing.*;
import java.net.*;
import java.awt.*;
import java.awt.event.*;

public class Client {

The JTextArea will be used to display all the chat proceedings in real time. The objective of this program is to have mutiple clients connect to a server and receive the information passed to them in real time. We make use of threads in this case.

JTextField is used for passing the chat comments.
JButton is ofcourse used for sending the message.

JTextArea incoming;
JTextField outgoing;
JButton sendbutton;

BufferedReader is used to buffer the inputstream from another source.
We use the PrintWriter to print to the socket. It is similar to using the system println function.

BufferedReader reader;
PrintWriter writer;

Sockets
——–
Sockets form the important part of this program. Simply put, it is an IP addreass with a port number. Now, port numbers vary from about 0 to 65535. It should be noted that the port numbers from 0 to 1023 are reserved and should not, unless for a reason, be used. In this example we will be using the port number 5000 to connect to the server.

Socket sock;

The function go()
—————–
This is the first method to be called. This does the following:
1. Sets up a GUI environment like I mentioned in previous posts.
2. Calls the setUpNetworking() function which will be explained later. Basically, it sets up the networking environment.
3. Most importantly, we start the threads, which I will explain.

public void go(){
JFrame frame = new JFrame("Client");
JPanel mainpanel = new JPanel();

incoming = new JTextArea(15,50);
incoming.setLineWrap(true);
incoming.setWrapStyleWord(true);
incoming.setEditable(false);

JScrollPane qScroller = new JScrollPane(incoming);
qScroller.setVerticalScrollBarPolicy(ScrollPaneConstants.VERTICAL_SCROLLBAR_ALWAYS);
qScroller.setHorizontalScrollBarPolicy(ScrollPaneConstants.HORIZONTAL_SCROLLBAR_ALWAYS);

outgoing = new JTextField(20);
sendbutton = new JButton("Send");

sendbutton.addActionListener(new SendButtonListener());

mainpanel.add(qScroller);
mainpanel.add(outgoing);
mainpanel.add(sendbutton);

setUpNetworking();

Now what this  does is set up a mainpanel which is JPanel. On it we add a JTextArea, which itself is added onto a JScrollPane as seen from the code. The different paramenters of this are set accordingly. The JButton is added and ofcourse, the ActionListener() for the button.

Thread readerThread = new Thread(new IncomingReader());
readerThread.start();

This is how a thread is created. Here we pass a new object of the class IncomingReader which implements the Runnable class, which I will explain in a moment.

frame.getContentPane().add(BorderLayout.CENTER, mainpanel);
frame.setSize(400, 400);
frame.setVisible(true);

}

These are just basic frame housekeeping stuff.

public static void main(String[] args){
Client client = new Client();
client.go();
}

The program begins here ofcourse. An object of the type client is created and the go() function is called.

private void setUpNetworking(){

try{

sock = new Socket("127.0.0.1", 5000);

This step creates a socket at the address “127.0.0.1” which is the localhost, ie, the same computer. And we select arbitary socket number of 5000. This number is known to the server.

InputStreamReader streamReader = new InputStreamReader(sock.getInputStream());

The InputStreamReader reads the bytes of data coming from the socket inputstream.

reader = new BufferedReader(streamReader);

This is then passed on to the BufferedReader object.

writer = new PrintWriter(sock.getOutputStream());

The PrintWriter object is finally responsible for putting the message onto the output stream.

System.out.println("Networking established!!");

}catch(IOException e){
e.printStackTrace();
}
}

Before I mention about the networking function, I will explain the concept of the try — catch block of code. Now, while running a code block, exceptions are bound to happen. If we know in advance, that a particular error is likely to happen at a place, we can place it in a try block. So what happens is that if any error occurs in this part will be caught by the catch() statement and then the housekeeping stuff inside the catch block executes. The whole idea of this is to make the program stop gracefully in case of an error.
We make the exception object print out the StackTrace in this case.

public class SendButtonListener implements ActionListener{

public void actionPerformed(ActionEvent e){
try{
writer.println(outgoing.getText());
writer.flush();

We push the content typed into the JTextField into the outputstream of the socket using the writer and
clear the buffer after every step.

}catch(Exception ex){
ex.printStackTrace();
}

outgoing.setText("");
outgoing.requestFocus();
}
}

Once you send the message, you clear the JTextField and then give the focus back to the JTextfield so that the user can type in agagin.

This is the ActionListener for the JButton Send.

public class IncomingReader implements Runnable{

public void run(){
String message;
try{
while((message = reader.readLine()) != null){
System.out.println("read" + message);
incoming.append(message + "\n");

}
}catch(Exception e){
  e.printStackTrace();
  }
 }
 }
}

This is an important part of the program. This class is passed into the thread as an object. This class implements the Runnable interface. Now, each time a thread is created, this class is called. For each Client object created, new copies or threads of execution of this same program is spawned from the existing main thread, which is the only case in a sequential program and they run independently of each other till they complete or the mother thread stops.

Here what we do is to read from the BufferedReader as and when it is not empty and then post in onto the console as well as onto the JTextField using the append function.

A Simple Chat Server in Java using threads Part 1