org.apache.nutch.indexer.field
Class AnchorFields.Collector

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.nutch.indexer.field.AnchorFields.Collector

All Implemented Interfaces:

Closeable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable,org.apache.hadoop.io.Text,FieldWritable>

Enclosing class:

AnchorFields

public static class AnchorFields.Collector
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable,org.apache.hadoop.io.Text,FieldWritable>

Collects and creates FieldWritable objects from the inlinks. Inlinks are first sorted by descending score before being collected.

Constructor Summary

AnchorFields.Collector()

Method Summary

void	close()
void	configure(org.apache.hadoop.mapred.JobConf conf) Configures the jobs.
void	map(org.apache.hadoop.io.Text key, org.apache.hadoop.io.Writable value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable> output, org.apache.hadoop.mapred.Reporter reporter) Wraps values in ObjectWritable
void	reduce(org.apache.hadoop.io.Text key, Iterator<org.apache.hadoop.io.ObjectWritable> values, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,FieldWritable> output, org.apache.hadoop.mapred.Reporter reporter) Aggregates and sorts inlinks.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AnchorFields.Collector

public AnchorFields.Collector()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Configures the jobs. Sets maximum number of inlinks and whether to tokenize and store.

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

close

public void close()

Specified by:

close in interface Closeable

map

public void map(org.apache.hadoop.io.Text key,
                org.apache.hadoop.io.Writable value,
                org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable> output,
                org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Wraps values in ObjectWritable

Specified by:

map in interface org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable>

Throws:

IOException

reduce

public void reduce(org.apache.hadoop.io.Text key,
                   Iterator<org.apache.hadoop.io.ObjectWritable> values,
                   org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,FieldWritable> output,
                   org.apache.hadoop.mapred.Reporter reporter)
            throws IOException

Aggregates and sorts inlinks. Then converts up to a max number to FieldWritable objects.

Specified by:

reduce in interface org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,org.apache.hadoop.io.ObjectWritable,org.apache.hadoop.io.Text,FieldWritable>

Throws:

IOException