org.apache.streams.hdfs

Class WebHdfsPersistReader

java.lang.Object

org.apache.streams.hdfs.WebHdfsPersistReader

All Implemented Interfaces:

Serializable, DatumStatusCountable, StreamsOperation, StreamsPersistReader, StreamsProvider

public class WebHdfsPersistReader
extends Object
implements StreamsPersistReader, DatumStatusCountable

WebHdfsPersistReader reads from hdfs.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.fs.FileSystem	client
protected DatumStatusCounter	countersCurrent
protected DatumStatusCounter	countersTotal
protected static char	DELIMITER
protected HdfsReaderConfiguration	hdfsConfiguration
protected LineReadWriteUtil	lineReaderUtil
protected com.fasterxml.jackson.databind.ObjectMapper	mapper
protected org.apache.hadoop.fs.Path	path
protected Queue<StreamsDatum>	persistQueue
protected org.apache.hadoop.fs.FileStatus[]	status
static String	STREAMS_ID
protected StreamsConfiguration	streamsConfiguration

Constructor Summary

Constructors

Constructor and Description
WebHdfsPersistReader() WebHdfsPersistReader constructor - resolves HdfsReaderConfiguration from JVM 'hdfs'.
WebHdfsPersistReader(HdfsReaderConfiguration hdfsConfiguration) WebHdfsPersistReader constructor - uses supplied HdfsReaderConfiguration.

Method Summary

Modifier and Type	Method and Description
void	cleanUp()
DatumStatusCounter	getDatumStatusCounter()
org.apache.hadoop.fs.FileSystem	getFileSystem() getFileSystem.
String	getId()
URI	getURI() getURI from hdfsConfiguration.
boolean	isConnected() isConnected.
boolean	isRunning()
void	prepare(Object configurationObject)
StreamsResultSet	readAll()
StreamsResultSet	readCurrent()
StreamsResultSet	readNew(BigInteger sequence)
StreamsResultSet	readRange(org.joda.time.DateTime start, org.joda.time.DateTime end)
void	startStream()
protected void	write(StreamsDatum entry)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STREAMS_ID

public static final String STREAMS_ID

See Also:

Constant Field Values

DELIMITER

protected static final char DELIMITER

See Also:

Constant Field Values

client

protected org.apache.hadoop.fs.FileSystem client

path

protected org.apache.hadoop.fs.Path path

status

protected org.apache.hadoop.fs.FileStatus[] status

persistQueue

protected volatile Queue<StreamsDatum> persistQueue

mapper

protected com.fasterxml.jackson.databind.ObjectMapper mapper

lineReaderUtil

protected LineReadWriteUtil lineReaderUtil

hdfsConfiguration

protected HdfsReaderConfiguration hdfsConfiguration

streamsConfiguration

protected StreamsConfiguration streamsConfiguration

countersTotal

protected DatumStatusCounter countersTotal

countersCurrent

protected DatumStatusCounter countersCurrent

Constructor Detail

WebHdfsPersistReader

public WebHdfsPersistReader()

WebHdfsPersistReader constructor - resolves HdfsReaderConfiguration from JVM 'hdfs'.

WebHdfsPersistReader

public WebHdfsPersistReader(HdfsReaderConfiguration hdfsConfiguration)

WebHdfsPersistReader constructor - uses supplied HdfsReaderConfiguration.

Parameters:

hdfsConfiguration - hdfsConfiguration

Method Detail

getURI

public URI getURI()
           throws URISyntaxException

getURI from hdfsConfiguration.

Returns:

URI

Throws:

URISyntaxException - URISyntaxException

isConnected

public boolean isConnected()

isConnected.

Returns:

true if connected, false otherwise

getFileSystem

public final org.apache.hadoop.fs.FileSystem getFileSystem()

getFileSystem.

Returns:

FileSystem

getId

public String getId()

Specified by:

getId in interface StreamsOperation

prepare

public void prepare(Object configurationObject)

Specified by:

prepare in interface StreamsOperation

cleanUp

public void cleanUp()

Specified by:

cleanUp in interface StreamsOperation

readAll

public StreamsResultSet readAll()

Specified by:

readAll in interface StreamsPersistReader

startStream

public void startStream()

Specified by:

startStream in interface StreamsProvider

readCurrent

public StreamsResultSet readCurrent()

Specified by:

readCurrent in interface StreamsProvider

write

protected void write(StreamsDatum entry)

readNew

public StreamsResultSet readNew(BigInteger sequence)

Specified by:

readNew in interface StreamsPersistReader

Specified by:

readNew in interface StreamsProvider

readRange

public StreamsResultSet readRange(org.joda.time.DateTime start,
                                  org.joda.time.DateTime end)

Specified by:

readRange in interface StreamsPersistReader

Specified by:

readRange in interface StreamsProvider

isRunning

public boolean isRunning()

Specified by:

isRunning in interface StreamsProvider

getDatumStatusCounter

public DatumStatusCounter getDatumStatusCounter()

Specified by:

getDatumStatusCounter in interface DatumStatusCountable