thelabdude

Analyzing the National Solar Radiation Database (NSRDB) with Hadoop

2011-07-08T16:57:00.000-07:00

Introduction

In this posting, I provide a basic framework and some ideas for analyzing the National Solar Radiation Data Base (NSRDB) provided by OpenEI using MapReduce, Hadoop, and the Amazon cloud. First, a little introduction of OpenEI is in order; from the openei.org Web site,

Open Energy Information (OpenEI) is a knowledge-sharing online community dedicated to connecting people with the latest information and data on energy resources from around the world. Created in partnership with the United States Department of Energy and federal laboratories across the nation, OpenEI offers access to real-time data and unique visualizations that will help you find the answers you need to make better, more informed decisions. And because the site is designed on a linked, open-data platform, your contributions are both welcomed and encouraged.

Sounds pretty cool ... let's roll up our sleeves and see what we can do with this data using Hadoop. If you need a refresher on Hadoop, have a look at Yahoo's Hadoop Tutorial.

The current NSRDB data set contains >191M observations collected from Solar stations around the US from 1991-2005. Of course this data set is tiny by Hadoop standards, but the current data set is only a sub-set of available solar data. Also, there are other larger solar-related data sets that can use the same approach described in this posting to distribute expensive computations across large numbers of commodity servers using Hadoop. Moreover, this data can be joined with other data sets and analyzed using various Hadoop sub-projects, such as Hive or Pig. So this posting is mostly about how to get OpenEI data into a format that can be analyzed using MapReduce jobs (and other tools, e.g. Hive or Pig) in a Hadoop compute cluster. Specifically, I provide examples with source code for the following topics:

File size considerations for storing data in HDFS and S3
Loading static metadata into Hadoop's DistributedCache and special handling needed to make this work with S3 and Elastic MapReduce.
Running a MapReduce job locally in pseudo-distributed mode and on Amazon's Elastic MapReduce
Using MRUnit to unit test the MapReduce jobs before deploying to Hadoop
Using Hadoop Counters to help you diagnose unexpected job results and make writing unit tests easier

Building the Application

First, you need to download the source code from github/thelabdude/nsrdb (note: I don't reproduce all the source code in this blog post, so you'll have to download it to get the full details). Next, build the project with Maven 2.2.x using:

mvn clean package

I use the Maven assembly plug-in to create an executable JAR with all dependencies (nsrdb-0.0.2-jar-with-dependencies.jar) and the Hadoop Job JAR with dependencies (nsrdb-0.0.2-hadoop-jobs.jar).

<build>
    <plugins>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.2</version>
        <executions>
          <execution>
            <id>make-exe-jar</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
            <configuration>
              <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs>
              <archive>
                <manifest>
                  <mainClass>thelabdude.nsrdb.GetData</mainClass>
                </manifest>
              </archive>
            </configuration>
          </execution>
          <execution>
            <id>make-jobs-jar</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
            <configuration>
              <descriptors>
                <descriptor>src/main/assembly/hadoop-jobs-jar.xml</descriptor>
              </descriptors>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Download the Data

Next, we need to download the data from the OpenEI.org site using some good ol' screen scraping. The data is contained in 1,454 GZipped TAR files (tar.gz) on http://en.openei.org/datasets/files/39/pub/. The process I used is as follows:

Parse the contents of the data listing HTML page to extract links matching a specified regular expression, in this case ^\d.*\.tar\.gz$.
Download each data file and save locally so that I don't have to return to the site; build in a 2 sec delay between requests so that OpenEI doesn't get overloaded by the app.
Uncompress each GZipped TAR file; each TAR contains 15 CSV files containing data for a station for each year in the data set
Append the contents of each entry in the TAR file into a single Hadoop SequenceFile that is stored in HDFS.

Motivation for Using a Single Hadoop SequenceFile

My current goal is to store the data in a format that makes writing MapReduce jobs easy. Hadoop works better with a small number of large files that can be split and distributed by HDFS vs. a large number of small files. By small, I mean smaller than the HDFS block size (64MB is the default). Since the NSRDB is not actually that big, I chose to merge all data into a single, compressed SequenceFile. Internally, Hadoop will split the SequenceFile into replicated blocks that are distributed around the HDFS cluster. This means that each Map task will work on a sub-set of data that is most likely stored on the same server that the task is running on (i.e. data locality). For example, on Elastic MapReduce, the default block size is 128MB, so Hadoop will run 18 map tasks for our 2.1GB of input data. I should mention that if you store this SequenceFile in S3, then you do not benefit from data locality automatically, so I'll return to this discussion below.

I use a custom class for the key in my SequenceFile: thelabdude.nsrdb.StationYearWritable. This class holds a long station ID and an int year and implements Hadoop's Writable interface for fast Serialization. The value for each key in the file is simply the entire contents of each TAR entry stored in a BytesWritable object. In other words, I'm storing the CSV data in a large BLOB for each year and station (1,454 stations * 15 years = 21,810 records). We'll save the line-by-line parsing of each CSV file for the MapReduce jobs so we can distribute the processing across a cluster.

GetData in Action

So now that you understand the basic process I'm using to get the data into Hadoop, let's see how to run it. First, make sure Hadoop is setup in your environment. For now, I recommend using the pseudo-distributed configuration (see Hadoop 0.20.2 QuickStart). Once Hadoop is running, run the GetData application, output is saved to the HDFS you configured in conf/core-site.xml:

(java -jar nsrdb-0.0.2-jar-with-dependencies.jar \
  -nsrdbIndexUrl http://en.openei.org/datasets/files/39/pub/ \
  -downloadDir /mnt/openei/nsrdb/input \
  -hadoopDataPath hdfs://localhost:9000/openei/nsrdb/nsrdb.seq &>getdata.log &)

I'll refer you to the source code for thelabdude.nsrdb.GetData for all the details. This application took about 2 hours to download the 1454 tar.gz files from OpenEI.org and merge them into a single SequenceFile stored in HDFS (tail the getdata.log file to see the application's progress). While this is slow, you should only have to do this once. When it finishes, you can make sure it is available in HDFS using:

$ bin/hadoop fs -lsr /openei/nsrdb

You should see the following output:

/openei/nsrdb/nsrdb.seq (2346388001 bytes)

Note: Don't delete the raw data files from the download directory just yet as you may want to re-run the GetData application (with -skipDownloadStep) to create more than one SequenceFile if you plan to use Amazon S3 and Elastic MapReduce (see below).

Analyzing NSRDB using MapReduce

Now that the data is in HDFS, it can be used as input to MapReduce jobs. Let's take a look at a few simple jobs that illustrate how to process the data.

Maximum Radiation Measured by Year

Which station is the sunniest in the winter months of each year? While this probably doesn't mean much to energy researchers, and I'm using the term "sunniest" very loosely, it does give us a simple example to exercise the NSRDB data using MapReduce. Here's how to run the job in your local Hadoop:

bin/hadoop jar nsrdb-0.0.2-hadoop-jobs.jar thelabdude.nsrdb.MaxValueByYearJob \
  /openei/nsrdb/nsrdb.seq /openei/nsrdb/maxValueByYear \
  /home/ubuntu/NSRDB_StationsMeta.csv 1

Notice that you need to pass the location of the NSRDB_StationsMeta.csv file to the job. Download the file from OpenEI and save it to the local filesystem where you run the job. The job uses Hadoop's DistributedCache to replicate the static station metadata CSV to all nodes in the cluster. Using Hadoop 0.20.2 in pseudo-distributed mode on a small Amazon EC2 instance (32-bit Ubuntu), the job finished in about 50 minutes (of course, we'll improve on this time in a real cluster). Once it completes, look at the results using:

hadoop fs -text /openei/nsrdb/maxValueByYear/part-r-00000

Let's look at the Map and Reduce steps in detail; the MapReduce job flow looks like this:
Map(k1,v1) → list(k2,v2) where k1 is Station ID + Year, v1 is a CSV file stored as a array of bytes, k2 is a year, such as 2001, and v2 is the station ID and sum of a solar radiation field. Using the new Hadoop API, the mapper class is defined as:

static class MaxValueByYearMapper
 extends Mapper<StationYearWritable, BytesWritable, IntWritable, StationDataWritable> {
  ...
}

Reduce(k2, list (v2)) → list(v3) where k2 and v2 are the same as what was output from the mapper and v3 is metadata for the station the measured the most radiation per year. All v2 values for the same key k2 are sorted by Hadoop during the shuffle and sent to the same reducer. The reducer class is defined as:

static class MaxValueByYearReducer
  extends Reducer<IntWritable, StationDataWritable, IntWritable, Text> {
  ...
}

Map Step

We need to sum up all the radiation measured by each station during the winter each year. In the end, we will have 15 stations identified in our reduce output (one for each year in our dataset). Recall that keys in our SequenceFile are StationYearWritable objects and values are BytesWritable objects. Thus, the map function signature is:

public void map(StationYearWritable key, BytesWritable value, Context context)
        throws IOException, InterruptedException

Now is the time to process the CSV data line-by-line, since presumably our Map task is distributed in a Hadoop cluster. Before processing the CSV data for a station, the Map step consults the station metadata stored in Hadoop's DistributedCache to see if the station is of interest to this job. For this job, I'm only interested in class 1 or 2 stations in the lower 48 US states.

Next, the Map step parses each line from the stored CSV file.
IMPORTANT: Be careful when reading from BytesWritable objects because the byte[] returned from getBytes() may have more bytes than expected. You need to read only up to index getLength()-1:

ByteArrayInputStream bytesIn = new ByteArrayInputStream(value.getBytes(), 0, value.getLength());
    reader = new BufferedReader(new InputStreamReader(bytesIn, UTF8));

For this job, I'll sum up the values in the Modeled Global Horizontal field to find my maximum. I'm also using a common MapReduce technique called a Combiner where I'm doing some of the summarization work in the Map step since we know that each value coming into the map function contains all the data we need for a given station per year. Combiners help reduce the amount of disk I/O needed between the Map and Reduce steps by reducing the number of key/value pairs output by your mapper. Hadoop allows you to configure a separate Combiner class for your job, but since the logic for this job is so easy, I can just do the summarization in the mapper.

A few words about Hadoop's DistributedCache

Hadoop's DistributedCache replicates static data needed by your job at runtime to all DataNodes in the cluster, which addresses my job's need to get access to station metadata nicely. However, using the DC can be tricky as the current documentation is not clear on how to set it up correctly, especially when working with S3 and Elastic MapReduce. In the job's run method, I have the following code to add the station metadata CSV file to Hadoop's DistributedCache:

URI localUri = stationMetadataLocal.toUri();
    if ("s3".equals(localUri.getScheme()) || "s3n".equals(localUri.getScheme())) {
      copyFromS3File(conf, stationMetadataLocal, stationMetadataHdfs);
    } else {
      FileSystem fs = FileSystem.get(conf);
      fs.copyFromLocalFile(false, true, stationMetadataLocal, stationMetadataHdfs);
    }
    DistributedCache.addCacheFile(stationMetadataHdfs.toUri(), conf);

You'll notice that I have to handle files in S3 differently than files on the local file system where the job is run. Essentially, if your static data is in S3, you need to copy it to HDFS using two separate file systems: S3N and HDFS, see my copyFromS3File method.

Reduce Step

The reduce step is simple: it receives a list of values per year and sums them for each station. The framework collects and sorts the values for each year before invoking the reducer during an internal phase called the "shuffle". Once all station data has been summed up, the station with the maximum sum is written to the output. As with the Map step, my Reducer consults the DistributedCache to get station metadata for its output, which is easier to interpret than just a station ID.

Group by Year

I've also included a simple job to transform the data to group values by year (see thelabdude.nsrdb.GroupByYearJob). This example shows how you do not need a Reducer for your job if you just want to transform the input data. The output of this job is useful for using Hadoop's streaming interface.

Running in the Cloud

Now, let's run this same job an Amazon Elastic MapReduce cluster. You'll need a few command-line tools to do the work in this section.

s3cmd: http://s3tools.org/s3cmd
elastic-mapreduce-ruby: http://aws.amazon.com/developertools/2264?_encoding=UTF8&jiveRedirect=1

On Debian-based systems, you can do:

sudo apt-get install s3cmd
sudo apt-get install openssl
sudo apt-get install ruby1.8
sudo apt-get install libopenssl-ruby1.8
sudo apt-get install libruby1.8-extras
wget http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip

Once installed, you'll need your Amazon Access Key and Secret Access Key values to configure s3cmd using: s3cmd --configure. Follow the instructions for elastic-mapreduce to get that setup as well. If you need a refresher on Amazon access credentials, please read: Understanding Access Credentials for AWS/EC2.

Be aware of the implications of the region setting in the elastic-mapreduce credentials.json file as it determines which region your Elastic MapReduce instances launch in. If you store your data in a US Standard S3 bucket, then I recommend launching your EMR instances in the "us-east-1" region as there are no data transfer fees between your EMR instances and S3. On the other hand, if you launch your cluster in the "us-west-1" region, you will incur data transfer fees if your data is in a US Standard S3 bucket, so only use "us-west-1" if your S3 bucket is in Northern California.

File Size Revisited for Amazon S3

Previously, I made the statement that Hadoop works better with a small number of large files vs. a large number of small files. While this is still true with EMR, we need bend this rule slightly when working with files stored in S3 as using only one large file will lead to a major degradation in performance. Your job may actually run slower than it does in pseudo-distributed mode on a single machine since you'll also have Hadoop job management and S3 latency involved. Intuitively, this makes sense because Hadoop sets the number of map tasks based on the number of splits. So no matter how big your cluster is, only one node can process the input from one file on S3. The solution is split the data into smaller chunks, such as double the number of nodes in your cluster. The GetData tool I developed allows you to specify the number of SequenceFiles to create using the chunkSize parameter. For a cluster with 4 nodes, I used chunkSize=256 which resulted in 9 SequenceFiles instead of 1. The key take-away here is that now I can have 9 map tasks running in parallel instead of only one.

Copy the SequenceFile(s) created by GetData from your local HDFS to an S3 bucket you have access to using Hadoop's distcp:

hadoop distcp /openei/nsrdb s3n://ACCESS_KEY:SECRET_KEY@BUCKET[/PREFIX]

Notice that I pass my Amazon ACCESS_KEY and SECRET_KEY in the S3 URI and that I'm using s3n as the protocol (native S3), see: Hadoop and Amazon S3. The distcp command uses MapReduce to distribute a large copy around your Hadoop cluster. The files copied from HDFS will be the input to your MapReduce jobs on EMR.

Next, you need to put the Jobs JAR file in S3, for this I like to use s3cmd:

s3cmd put s3://BUCKET[/PREFIX]

For example:

s3cmd put nsrdb-0.0.2-hadoop-jobs.jar s3://thelabdude/openei/

Now, you're all set for running the job in EMR. The approach I like to take is to start a job flow without any jobs configured (actually, this is pretty much the standard approach when working with EMR). Once the cluster is running, I add job flow steps to it, which is more efficient because if the job fails due to a invalid or missing parameter, I can correct and re-start the step without waiting for the cluster to restart. To start a job flow, do the following:

elastic-mapreduce --create --alive \
  --log-uri s3n://BUCKET/openei/logs/ \
  --key-pair YOUR_KEY_PAIR_NAME \
  --slave-instance-type m1.small \
  --master-instance-type m1.small \
  --num-instances 9 \
  --name openei.nsrdb

This will create a new Job Flow named "openei.nsrdb" with 1 master node and 8 data nodes (m1.small instances); it can take a few minutes before the job flow is started. Take note of the Job Flow ID returned as you will need this to access logs and add job steps. You can add a new "step" (ie. Hadoop job) once you have the Job Flow ID:

elastic-mapreduce --jar s3n://BUCKET/openei/nsrdb-0.0.2-hadoop-jobs.jar \
  --main-class thelabdude.nsrdb.MaxValueByYearJob \
  --arg s3n://BUCKET/openei/nsrdb-chunks \
  --arg s3n://BUCKET/openei/maxValueByYear/ \
  --arg s3n://BUCKET/openei/NSRDB_StationsMeta.csv \
  --arg 8 \
  -j JOB_ID

Notice that I'm now requesting 8 reducers; Hadoop recommends a range of ((0.95 to 1.75) * cluster size). Also, I'm reading the input data directly from s3n://BUCKET/openei/nsrdb-chunks. If you're going to run multiple jobs on the data in the same EMR JobFlow, then I recommend using distcp to copy the data from S3 to the local HDFS for your Elastic MapReduce cluster using:

elastic-mapreduce --jar s3n://elasticmapreduce/samples/distcp/distcp.jar \
  --arg s3n://BUCKET/openei/nsrdb-chunks \
  --arg /openei/
  -j JOB_ID

This will help your jobs run much faster because the input data is already loaded into the local HDFS of your EMR cluster (data locality). To view the logs for your Job Flow, simply do:

elastic-mapreduce --logs -j JOB_ID

With a 8 node cluster of EC2 small instances, the job completed in roughly 8 minutes (after using distcp to copy the data from S3). The output from the job is included here:

2011-07-08 23:15:45,796 INFO org.apache.hadoop.mapred.JobClient (main): Job complete: job_201107082258_0002
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main): Counters: 19
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):   Job Counters 
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):     Launched reduce tasks=8
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):     Rack-local map tasks=5
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):     Launched map tasks=19
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):     Data-local map tasks=14
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):   thelabdude.nsrdb.MaxValueByYearJob$StationDataCounters
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):     STATION_IGNORED=10095
2011-07-08 23:15:45,804 INFO org.apache.hadoop.mapred.JobClient (main):   FileSystemCounters
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     FILE_BYTES_READ=131196
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     HDFS_BYTES_READ=2347266412
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     FILE_BYTES_WRITTEN=276357
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     HDFS_BYTES_WRITTEN=1307
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):   Map-Reduce Framework
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Reduce input groups=15
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Combine output records=0
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Map input records=21810
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Reduce shuffle bytes=141561
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Reduce output records=15
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Spilled Records=23430
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Map output bytes=281160
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Combine input records=0
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Map output records=11715
2011-07-08 23:15:45,805 INFO org.apache.hadoop.mapred.JobClient (main):     Reduce input records=11715

Notice that the job ignored 673 stations (thelabdude.nsrdb.MaxValueByYearJob$StationDataCounters.STATION_IGNORED: 10095 / 15 years). The reducer output looks like:

1991    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,350400.0
1992    722010,1,false,KEY WEST INTL ARPT,FL,24.55,-81.75,1,-5,24.55,-81.75,1,332856.0
1993    722010,1,false,KEY WEST INTL ARPT,FL,24.55,-81.75,1,-5,24.55,-81.75,1,356962.0
1994    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,343829.0
1995    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,354387.0
1996    722010,1,false,KEY WEST INTL ARPT,FL,24.55,-81.75,1,-5,24.55,-81.75,1,377977.0
1997    722010,1,false,KEY WEST INTL ARPT,FL,24.55,-81.75,1,-5,24.55,-81.75,1,354811.0
1998    722695,2,false,LAS CRUCES INTL,NM,32.283,-106.917,1393,-7,32.283,-106.917,1393,354944.0
1999    722735,2,false,DOUGLAS BISBEE-DOUGLAS INTL A,AZ,31.467,-109.6,1249,-7,31.467,-109.6,1249,370871.0
2000    722735,2,false,DOUGLAS BISBEE-DOUGLAS INTL A,AZ,31.467,-109.6,1249,-7,31.467,-109.6,1249,371146.0
2001    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,374057.0
2002    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,354146.0
2003    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,363305.0
2004    722010,1,false,KEY WEST INTL ARPT,FL,24.55,-81.75,1,-5,24.55,-81.75,1,362566.0
2005    722016,2,false,MARATHON AIRPORT,FL,24.733,-81.05,2,-5,24.733,-81.05,2,373423.0

Not surprisingly, all the values are in either Arizona or Florida!

IMPORTANT: Be sure to terminate your job flow when you're finished working with it so that you don't incur unnecessary expenses.

elastic-mapreduce --terminate -j JOB_ID

For more tips about running jobs in Elastic MapReduce, check out my contributions to the Mahout wiki based on some benchmarking work I did for Taming Text. See Building Vectors for Large Document Sets on Mahout on Elastic MapReduce. Alternatively, you can also spin-up your own Hadoop cluster instead of EMR using the notes I compiled for Mahout on Amazon EC2 > Use an Existing Hadoop AMI

Testing Hadoop Jobs with MRUnit

Lastly, I've included a test case (thelabdude.nsrdb.MaxValueByYearMRUnitTest) to test my MR job using MRUnit. The MRUnit library provides the mock objects and job drivers that simulate the Hadoop runtime environment for testing your job outside of Hadoop. MRUnit drivers allow you to specify expected output from Mapper and Reducer classes given specific inputs. Notice that my job uses Hadoop Counters to keep track of decisions it makes, such as skipping a station or when it encounters an error when parsing the data. Counters can help you diagnose unexpected job results and make writing unit tests easier since you can assert expected counts based on your test input data. For example, the following unit test checks to see if the job is handling invalid input records correctly:

@Test
    public void testMapperIgnoreParseError() throws Exception {
      // data that produces parse errors
      String csv = ...

      BytesWritable value = new BytesWritable(csv.getBytes(MaxValueByYearJob.UTF8));              
      mapDriver.withInput(new StationYearWritable(690190,2005), value)
               .runTest();      

      Counters counters = mapDriver.getCounters();
      Counter counter = counters.findCounter(MaxValueByYearJob.StationDataCounters.DATE_PARSE_ERROR);
      assertTrue("Expected 1 date parse rror", counter.getValue() == 1);              
      
      counter = counters.findCounter(MaxValueByYearJob.StationDataCounters.DATA_PARSE_ERROR);
      assertTrue("Expected 1 data parse rror", counter.getValue() == 1);              
    }

Both CSV formatted input records contain data formatting errors. Thus, I expect the MaxValueByYearJob.StationDataCounters.DATE_PARSE_ERROR and DATA_PARSE_ERROR Counters to report 1 parse error each. Try to avoid overloading the meaning of each counter and have a specific counter for each possible error / event in your job as it will help you troubleshoot data problems when running in a large distributed cluster. For information on Hadoop Counters, see Cloud9: Working with counters

Conclusion

So now we have a basic framework for loading OpenEI data into a Hadoop cluster and have seen how to run MapReduce jobs against it. As you can tell, there are many moving parts and some non-trivial Java coding work to do some simple analysis on the data with Hadoop. While energy researchers may benefit from being able to analyze huge Solar data sets using MapReduce and Hadoop, they will need a higher-level of abstraction than what was presented here. Moreover, 8 minutes really isn't very fast for having run such a simple job on 8 machines with such a small data set, so I'll also need to address performance issues. Consequently, I plan to follow-up this posting with an example of working with this data using Apache Hive and Pig to see if these data warehouse frameworks make it easier to analyze the NSRDB. I also plan to compare the results with Hive / Pig to using a MySQL database for analyzing this data.

VinWiki Part 4: Making Recommendations with Mahout

2010-09-23T20:44:00.000-07:00

Introduction

This is the final post in a four part series about a wine rating and recommendation Web application built using open source Java technology. The purpose of this series is to document key design and implementation decisions that can be applied to other Web applications. Please read the first, second, and third posts to get up-to-speed. You can download the project (with source) from here.

In this posting, I lay the foundation for making recommendations using Apache Mahout v. 0.3. For a thorough introduction to Mahout, I recommend Mahout in Action.

Collaborative Filtering in Mahout

Mahout's main goal is to provide scalable machine-learning libraries for classification, clustering, frequent itemset mining, and recommendations. Classification assigns a category (or class) from a fixed set of known categories to an un-categorized document. For example, some feed readers assign articles to broad categories like Sports or Politics using classification techniques. Clustering assigns documents to groups of similar documents using some notion of similarity between the documents in the group. For example, Google News uses clustering to group articles from different publishers that cover the same basic story. Frequent itemset mining determines which items, such as products in a shopping cart, typically occur together.

In this posting, I leverage the collaborative filtering features of Mahout to make wine recommendations based on ratings by VinWiki users. Collaborative filtering produces recommendations based on user preferences for items and does not require knowledge of the specific properties of the items. In contrast, content-based recommendation produces recommendations based off of intimate knowledge of the properties of items. This implies, of course, that content-based recommendation engines are domain-specific, whereas Mahout's collaborative filtering approach can work in any domain provided it has sufficient user-item preference data to work with.

For VinWiki, I experimented with three basic types of Mahout Recommenders:

User Similarity
Item Similarity
SlopeOne

Check out the Mahout Web site for information about other more experimental recommenders, such as one based on Singular value decomposition (SVD).

To decide which one of these recommenders is best for your application, you need to consider four key questions:

How to represent a user's preference for an item?
What is the ratio of items to users?
How do you determine the similarity between users or between items?
If using UserSimilarity, what is the size of a user neighboorhood?

As I'll demonstrate below, Mahout provides a framework to allow you to answer these questions by analyzing your data.

User Preferences

What constitutes a user preference for an item in your application? Is it a boolean "like" or "dislike" or does the preference have a strength, such as "I like Chardonnay but like Sauvignon Blanc better"? The structure of user-item preference data used in Mahout is surprisingly simple: userID, itemID, score, where score represents the strength of the user's preference for the item, see org.apache.mahout.cf.taste.model.Preference. There are two concrete implementations of the Preference interface in Mahout: BooleanPreference and GenericPreference. For VinWiki, I use GenericPreference because I chose to allow users to give a score for a wine.

Basic Structure of a UserSimilarity Recommender

Let's take a look at the basic approach Mahout takes to make a UserSimilarity based recommendation using VinWiki nomenclature:

1: For all wines W that user A has NOT expressed a preference for
2:   For every other user B (in A's neighborhood) that has expressed a preference for W
3:     Compute the similarity S between user A and B
4:       Add the User B's preference X for W weighted by S to a running average preference
5: Sort Wines by weighted average preference
6: return top R wines from sorted collection as recommendations

Intuitively, this approach makes sense. From the pseudo-code above, it should be clear that we need a way to calculate the similarity S between two Users A and B, which is represented in Mahout as a org.apache.mahout.cf.taste.similarity.UserSimilarity. Also, notice that the algorithm weights recommendations by user similarity, which means that the more similar a user is to you, the more heavily their preferences count in making recommendations. Consequently, the selection of the similarity calculation is very important to making good recommendations. Mahout provides a number of concrete implementations if the UserSimilarity interface, see the org.apache.mahout.cf.taste.impl package.

In practice, most systems that need to produce recommendations have many users and calculating a similarity between all users is too computationally expensive. Thus, Mahout uses the concept of a user neighborhood to limit the number of similarity calculations to a smaller subset of similar users. This introduces another question that needs to be answered when building your recommender: What is the optimal size of the user-neighborhood for my data?

Mahout also allows you to make recommendations based on similarity between Items. Don't confuse Mahout's Item-based recommender with content-based recommenders since it is still based on user-item interactions and not the content of items.

Using Mahout in VinWiki

The main service for creating recommendations at runtime is the MahoutWineRecommender, which is an application-scoped Seam component. The MahoutWineRecommender has two dependencies injected during initialization:

DataModelProvider
RecommenderConfig

DataModel

The org.vinwiki.recommender.DataModelProvider Seam component (configured in components.xml) provides a Mahout DataModel to the recommender. For now, I'm using Mahout's FileDataModel, which as you might have guessed, reads preference data from a flat file. During startup, if this file doesn't exist, then the DataModelProvider reads wine ratings from the database and writes them to a new file.

Sample Wine Ratings Data

As this is just an example Web application, I don't have real wine ratings data. Consequently, I generated some fake data that recommends wines to sample users based on the first letter of the user name. For example, the data will cause Mahout to recommend wines that start with the letter "A" to the "A_test0" user. Here is some log output to demonstrate how the sample ratings data works:

NearestNUserNeighborhood[3,0.6,0.8,EuclideanDistanceSimilarity] 
  recommended [1887, 286, 1120, 1350, 520, 1905] wines to A_test0
     Neighbor(43) A_test30
         rated Wine 1120 91.0 pts
         rated Wine 1350 87.0 pts
     Neighbor(33) A_test20
         rated Wine 1887 88.0 pts
         rated Wine 1350 88.0 pts
     Neighbor(63) A_test50
         rated Wine 1350 90.0 pts

Notice that A_test0's neighbor's user names also start with "A_". When I created the sample ratings data, I had users rate wines that begin with the same letter a little higher than they rated other wines. You can try this yourself after deploying the application to JBoss 4.2.3 by logging in with username "A_test0" and password "P@$$w0rD" (without the quotes of course).

Refreshing the Recommender

When I first started working with Mahout, it wasn't clear how to handle data model changes at runtime because most of the built-in Mahout examples work with static, pre-existing datasets. In VinWiki, rating wines is a primary activity, so preference data will be changing frequently. Moreover, if a user provides several new ratings in a session, then they'll expect to have some recommendations based on those new ratings or they will think the site is broken and probably not return. Consequently, it's very important for this application to incorporate recent user activity into recommendations in near real-time.

Whenever a user rates a wine, the ratingHome component will raise the App.WINE_RATED_BY_USER event. The MahoutWineRecommender component observes this event and passes it to the DataModelProvider.

@Observer(App.WINE_RATED_BY_USER)
@Asynchronous
public void onWineRatedByUser(Rating r) {
    // Let the model provider know that data has changed ...
    if (dataModelProvider.updateDataModel(r.getUser().getId(), r.getWine().getId(), r.getScore())) {
        // provider indicates that we should refresh the recommender
        recommender.refresh(null);
    }
}

In response to this event, the DataModelProvider component can choose to update its internal state to reflect the change. In my current implementation, the DataModelProvider uses a nice feature provided by Mahout's FileDataModel by writing updates to a smaller "delta" file. The FileDataModel will load these additional "delta" files when it is refreshed. So that covers updating the DataModel, but what about the Recommender and its other dependencies, such as UserSimilarity and UserNeighborhood? In my implementation, the DataModelProvider makes the decision of whether the Recommender should be refreshed. This allows a more sophisticated DataModelProvider implementation to batch up changes so that the recommender is not refreshed too often as refreshing a recommender and its dependencies can be an expensive operation for large data sets.

Accounting for User Preferences

Users can de-select wines they are not interested in using the Preferences dialog. Changes to a user's preferences should be reflected in recommendations. For example, if a user indicates that they are not interested in white wine, then we should not recommend any white wines to them. Mahout allows you to inject this type of filtering on recommendations using an org.apache.mahout.cf.taste.recommender.IDRescorer.

In VinWiki, filtering recommendations by preferences is provided by the org.vinwiki.recommender.PreferencesIDRescorer class. If you revisit the pseudo-code above, then it should be obvious that the IDRescorer may need to evaluate the filter on a large number of wines. Thus, the IDRescorer should be implemented in an efficient manner; I used the Lucene native API to iterate over all wines to build and cache a Mahout FastIDSet of wine Ids that can be recommended to the current user.

// Using Lucene to initialize a Mahout FastIDSet for rescoring
int maxDoc = reader.maxDoc();
for (int docId = 0; docId < maxDoc; docId++) {
    if (reader.isDeleted(docId))
        continue;

    try {
        doc = reader.document(docId, getFieldSelector());
    } catch (Exception zzz) {
        ...
    }
    if (doc == null)
        continue;

    Long wineId = new Long(doc.get(ID));
    String type = doc.get(TYPE);
    String style = doc.get(STYLE);
    Long regionId = new Long(doc.get(REGION));

    // ask the User's Preferences object if this wine is enabled
    if (prefs.checkWineFilter(wineId, type, style, regionId)) {
        idSet.add(wineId);
    }
}

There is one subtle aspect to the current implementation in that it does not refresh during the user's session as new wines are added to the search index. In other words, you are not going to see any code that tries to update the rescorer after new wines are added to the system. Remember that our recommendations are based on user-item interactions and new wine objects are not going to have enough (if any) ratings to impact the current user's session. However, the rescorer is refreshed if the user changes their preferences.

Tip: using an IDRescorer is a simple form of content-based recommendations in that we're using specific properties of the wine objects to influence our recommendations.

RecommenderConfig

org.vinwiki.recommender.RecommenderConfig is an application-scoped Seam component (configured in components.xml) that supports common options for configuring the behavior of a Recommender.

At startup, the MahoutWineRecommender uses the DataModel and RecommenderConfig to initialize a Recommender. The Recommender is held in application-scope because it is expensive to build and should be re-used for all recommendation requests from FetchRecommended objects (see Server-side Pagination from the first posting in this series). The following code snippet gives you an idea of how to construct a User-based recommender with Mahout:

// see RecommenderConfig.java
    UserSimilarity userSimilarity = createUserSimilarity(dataModel);
    UserNeighborhood neighborhood = createUserNeighborhood(userSimilarity, dataModel);
    return new GenericUserBasedRecommender(dataModel, neighborhood, userSimilarity);

Here is an example configuration from components.xml. NOTE: You must set the fileDataModelFileName to a valid path on your server before running the sample!

<component name="dataModelProvider" auto-create="true" scope="application" 
        class="org.vinwiki.recommender.DataModelProvider">
    <property name="fileDataModelFileName">/home/thelabdude/thelabdude-blog-dev/jboss-4.2.3/bin/recommender/ratings.txt</property>
    <property name="updateFileSizeThresholdKb">10</property>
  </component>

  <component name="recommenderConfig" auto-create="true" scope="application" 
        class="org.vinwiki.recommender.RecommenderConfig">
    <property name="recommenderType">USER_SIMILARITY</property>
    <property name="similarityClassName">org.apache.mahout.cf.taste.impl.similarity.PearsonCorrelationSimilarity</property>
    <property name="neighborhoodSize">2</property>
    <property name="minSimilarity">0.7</property>
    <property name="samplingRate">0.2</property>    
  </component>

Looks easy enough, but what about all those parameters to the recommenderConfig? Thankfully, Mahout provides a powerful tool to help you determine the correct values to use for each of these settings for your data - RecommenderEvaluator.

Evaluating a Mahout Recommender

It should be clear that the optimal recommender for your data requires experimentation with how you represent preferences, calculate user or item similarity, and the size of user neighborhood. Mahout provides an easy way to compare the results for different configuration options using a RecommenderEvaluator. Currently, there are two concrete RecommenderEvaluator implementations:

AverageAbsoluteDifferenceRecommenderEvaluator - computes the average absolute difference between predicted and actual ratings for users.
RMSRecommenderEvaluator - computes the "root mean squared" difference between predicted and actual ratings for users

I chose to use the RMSRecommenderEvaluator because it penalizes bad recommendations more heavily than the AverageAbsoluteDifference evaluator. When doing evaluations, the lowest score is best. Notice in the code snippet below how the RecommenderConfig (part of VinWiki) helps you run evaluations:

RecommenderConfig config = new RecommenderConfig();
config.setRecommenderType(RecommenderType.USER_SIMILARITY);
config.setSimilarityClassName(simClass.getName());
config.setNeighborhoodSize(c);
config.setMinSimilarity(minSimilarity);
config.setSamplingRate(samplingRate);

RecommenderBuilder builder = config.getBuilder();
double score = evaluator.evaluate(builder, 
                 null, // no DataModelBuilder
                 recommenderDataModel, 
                 0.8, // training data pct
                 1); // use all users

For VinWiki, I developed a Seam ComponentTest to run evaluations. At this point, the output is not as important as the process, since the results are based on simulated ratings data (VinWiki is not yet a live application with real users). This is a problem facing any new application that uses machine-learning algorithms that require real user input. One idea to get real user input is to use Amazon's Mechanical Turk service to hire users to create real user-item interactions for your application. Regardless of how you seed your application with real user data, the approach in src/test/org/vinwiki/RecommenderTest.java should still be useful to you.

Conclusion

So this concludes the four-part series on VinWiki. As you can see, integrating Mahout is easy, but it does require experimentation and tuning. You also have to be cognizant of scalability issues, such as how often to refresh your recommender. The framework I added to VinWiki should be useful for your application too. Please leave comments on my blog if you have questions or would like to suggest improvements to any of the features I discussed in any of the four posts.

VinWiki Part 3: Authentication with Facebook Connect and Sharing Content with Friends

2010-07-08T16:48:00.001-07:00

Introduction

This is the third post in a four part series about a wine rating and recommendation Web application built using open source Java technology. The purpose of this series is to document key design and implementation decisions that can be applied to other Web applications. Please read the first and second posts to get up-to-speed. You can download the project (with source) from here.

In this posting, I implement several common features that should help users find and contribute to your application. Specifically, I integrate with Facebook Connect (based on OAuth 2) to allow Facebook users to instantly register and authenticate using their Facebook profile. From there, I integrate the Facebook Like social plug-in which allows users to share content in your application with their friends on Facebook.

Authentication using Facebook Connect

The Facebook team has made integrating Facebook Connect into your application very easy. There are open source Java libraries available for integrating with Facebook, however I found it easier to just use the JavaScript SDK. Let's go through the process in five simple steps:

I. Register Your Application with Facebook

First, you need to register an application in Facebook to get a unique Application ID (please use something other than "VinWiki" for your application name since I'll be using that one in the near future). Update resources/WEB-INF/components.xml to set the facebookAppId property on the app component:

<component name="app" auto-create="true" scope="application" class="org.vinwiki.App">
    ...
    <property name="facebookAppId">ENTER_YOUR_FB_APPLICATION_ID_HERE</property>
  </component>

II. Initialize the Facebook JavaScript Library

Second, you need to initialize the Facebook JavaScript library when your page loads. For this, I created a new Facelets include file view/WEB-INF/facelets/facebook.xhtml and loaded it into the footer in my layout template view/layout/template.xhtml:

<h:panelGroup rendered="#{app.isFacebookEnabled()}">
    <ui:include src="/WEB-INF/facelets/facebook.xhtml"/>
  </h:panelGroup>

In facebook.xhtml, I have the following JavaScript:

(function() {
    var e = document.createElement('script'); e.async = true;
    e.src = document.location.protocol + '//connect.facebook.net/en_US/all.js';
    document.getElementById('fb-root').appendChild(e);
  }());

This function, borrowed from the Facebook developer documentation, asynchronously loads the Facebook JavaScript file into your page.

III. Register a JavaScript callback handler for Facebook session events

Once the JavaScript library is loaded, the window.fbAsyncInit function is called automatically.

window.fbAsyncInit = function() {    
    FB.init({appId:'#{app.facebookAppId}', status:true, cookie:true, xfbml:true});
    FB.Event.subscribe('auth.sessionChange', function(response) {
      if (response.session) {
        // Login successful
        var uid = response.session.uid;
        FB.api('/me', function(resp) {
          onFbLogin(uid, resp.email, resp.name, resp.first_name, resp.last_name, resp.gender);
        });
      } else {
        // The user has logged out, and the cookie has been cleared
        onFbLogout();
      }
    });
  };

After initializing the library (see FB.init), the application registers a listener for auth.sessionChange events (login or logout). On login, I use Facebook's Graph API to get some basic information about the current user (see FB.api). In the response callback handler for FB.api('/me'), I invoke a JavaScript function onFbLogin that executes the #{guest.onFbLogin()} action:

<a4j:form prependId="false">
  <s:token/>
  <a4j:jsFunction immediate="true" name="onFbLogin" ajaxSingle="true" action="#{guest.onFbLogin()}">
    ... params here ...
  </a4j:jsFunction>
</a4j:form>

Notice that I'm using Seam's <s:token/> tag to prevent cross-site request forgery since the #{guest.onFbLogin()} action simply trusts that the user was authenticated by Facebook on the client-side. For an overview of the thinking behind the <s:token/> tag, I refer you to Dan Allen's post at http://seamframework.org/Community/NewComponentTagStokenAimedToGuardAgainstCSRF. However, please realize that you must set javax.faces.STATE_SAVING_METHOD to "server" in web.xml for this method to secure your forms:

<context-param>
    <param-name>javax.faces.STATE_SAVING_METHOD</param-name>
    <param-value>server</param-value>
  </context-param>

The action handler on the server side is straight-forward because Seam supports alternative authentication mechanisms out-of-the-box. Specifically, all you need to do is invoke the acceptExternallyAuthenticatedPrincipal method of the org.jboss.seam.security.Identity object. I utilized the existing guest component because it handles other guest related actions such as register and login (see src/hot/org/vinwiki/user/GuestSupport.java)

Identity.instance().acceptExternallyAuthenticatedPrincipal(new FacebookPrincipal(user.getUserName()));
    Contexts.getSessionContext().set("currentUser", user);
    Events.instance().raiseEvent(Identity.EVENT_POST_AUTHENTICATE, Identity.instance());

I also raise the Identity.EVENT_POST_AUTHENTICATE event manually so that my nav component can re-configure the default for the authenticated user instead of showing the guest view.

IV. Show Facebook Connect Button on Login Panel

Lastly, we need to let users know that they login (or register) using their Facebook credentials. This is accomplished with the <fb:login-button> tag, see view/WEB-INF/facelets/guestSupport.xhtml.

<fb:login-button perms="email,publish_stream">
    <fb:intl>Login with Facebook</fb:intl>
  </fb:login-button>

The new RichFaces Login dialog looks like:

Notice that VinWiki requests access to the email and publish_stream extended permissions. The publish_stream permission allows users to share wines of interest found on VinWiki with their friends on Facebook. When accessing VinWiki for the first time, users will see a dialog that allows them to grant permissions to the application:

V. Logout

It's doubtful whether many users will ever explicitly logout of your site unless they are accessing it from a public computer. Consequently, you'll want to keep your session timeout value as low as possible. That said, you still need to offer the ability to logout. In view/WEB-INF/facelets/headerControls.xhtml, the logout action is implemented using a simple JSF commandLink:

<h:commandLink rendered="#{nav.isFbSession()}" onclick="FB.logout();" styleClass="hdrLink">
  <h:outputText value="Logout"/>
</h:commandLink>

The magic is in our sessionChange event listener discussed above. The Facebook JavaScript function FB.logout() triggers an auth.sessionChange event, which in turn calls onFbLogout() to execute the #{guest.onFbLogout()} action on the server.

So that covers authentication using Facebook Connect. There are many other ways to integrate your application with Facebook. In the next section, I'll implement a way for users to share content in your application with their friends on Facebook. For this feature, it is helpful to have bookmarkable URLs.

Sharing Content with Friends

In VinWiki, users may want to share specific wines of interest with their friends on Facebook. For this, I used Facebook's Like social plug-in. There are other options, including just posting a shared link to the user's activity stream. There's not much to integrating the Like social plug-in into your page once you've accounted for bookmarkable URLs (see posting 1). The <fb:like> tag will use the URL of the current page if you don't specify an href attribute. However, I want to make sure the URL that is shared with Facebook is as clean as possible. Thus, I introduced a new setting for the org.vinwiki.App component named baseUrl. You should change this to match your server in resources/WEB-INF/components.xml:

<component name="app" auto-create="true" scope="application" class="org.vinwiki.App">
    ...
    <property name="baseUrl">http://192.168.1.2:8080/vinwiki/</property>
  </component>

I also decided to add the Open Graph meta tags to the header of my wine details page, view/wine.xhtml:

<meta property="og:title" content="#{currentWine.fullName}"/>
  <meta property="og:type" content="drink"/>
  <meta property="og:url" content="#{viewWine.openGraphUrl}"/>
  <meta property="og:site_name" content="VinWiki"/>
  <meta property="og:description" content="#{jsf.truncate(currentWine.description,100,'...')}"/>

Presumably, when you share a page that supports the Open Group protocol, Facebook is able to present this additional metadata to your friends. Of course, the URL needs to be public before this will actually work ;-)

What's Next?

In the next and final post in this series, I'll integrate Mahout for making wine recommendations and discuss some considerations for scaling the application.

VinWiki Part 2: Full-text Search with Hibernate Search and Lucene

2010-06-14T13:34:00.000-07:00

Introduction

This is the second post in a four part series about a wine recommendation Web application built using open source Java technology. The purpose of this series is to document key design and implementation decisions that can be applied to other Web applications. Please read the first post for an overview of the project. You can download the project (with source) from here.

In this posting, I leverage Hibernate Search and Lucene to provide full-text search for wines of interest. This is post is merely an attempt to complement the already existing documentation on Lucene and Hibernate Search. If you are familiar with Hibernate and are new to Lucene, then I recommend starting out with the online manual for Hibernate Search 3.1.1 GA. In addition, I highly recommend reading Hibernate Search in Action and Lucene in Action (Second Edition); both are extremely well-written books by experts in each field.

Basic Requirements

Full-text search allows users to find wines using simple keywords, such as "zinfandel" or "dry creek valley". Users have come to expect certain features from a full-text search engine. The following table summarizes the key features our wine search engine will support:

Feature	Description
Paginated search results (see first post for solution)	Return the first 5-10 results, ordered by relevance on the first page of the search results. Allow users to retrieve more results by advancing the page navigator. Allows users to set a bounded preference for how many items to show on earch page.
Query term highlighting (view solution)	Highlight query terms in search results to allow users to quickly scan the results for the most relevant items.
Spell correction (view solution)	Ability to detect and correct for misspelled terms in the query.
Type-ahead suggestions (view solution)	Show a drop-down selection box containing suggested search terms after the user has typed at least 3 characters.
Advanced search form (view solution)	Allow users to fine-tune the search with an advanced search form.

Setup

From the previous post, you'll recall that I'm using Hibernate to manage my persistent objects. As such, I've chosen to leverage Hibernate Search (referred to as HS hereafter) to integrate my Hibernate-based object model with Lucene. For now, I'm using Hibernate Search v. 3.1.1 GA with Hibernate Core 3.3.1 GA. On the Lucene side, I'm using version 2.9.2 and a few classes from Solr 1.4. In the last posting in this series, we'll upgrade to the latest Hibernate Search and Core, which rely on JPA 2 and thus will require a move up to JBoss 6.x. As far as I know, the latest HS and Core classes have trouble on JBoss 4.2 and 5.1 because of their dependency on JPA 2 (let me know if you get it working).

Why not use the database to search?

Some may be wondering why I'm using Lucene instead of doing full-text search in my database? While possible, I feel the database is not the best tool for full-text searching. For an excellent treatment of the mis-match between relational databases and full-text search, I recommend reading Hibernate Search in Action. The first chapter provides a strong case for using Lucene instead of your database for full-text search. Just in terms of scalability, the database is the most expensive and complex layer to scale in most applications. Offloading searches to a local Lucene running on each node in your cluster reduces the amount of work your database is performing and ensures searches return quickly. Moreover, distributing a Lucene index across a cluster of app servers is almost trivial since Hibernate Search has built-in clustering support.

Project Configuration Changes

The following JAR files need to be included in your Web application's LIB directory (/WEB-INF/lib):

hibernate-search.jar
    lucene-core-2.9.2.jar
    lucene-highlighter-2.9.2.jar
    lucene-misc-2.9.2.jar
    lucene-snowball-2.9.2.jar
    lucene-spatial-2.9.2.jar
    lucene-spellchecker-2.9.2.jar
    lucene-memory-2.9.2.jar
    solr-core-1.4.0.jar
    solr-solrj-1.4.0.jar

How does Seam know to use Hibernate Search's FullTextEntityManager?

Seam's org.jboss.seam.persistence.HibernatePersistenceProvider automatically detects if Hibernate Search is available on the classpath. If HS is available, then Seam uses the org.jboss.seam.persistence.FullTextEntityManagerProxy instead of the default EntityManagerProxy, meaning that you will have access to a FullTextEntityManager wherever you have a Seam in-jected EntityManager. You also need to add a few more properties to your persistence deployment descriptor (resources/META-INF/persistence-*.xml):

<property name="hibernate.search.default.directory_provider" value="org.hibernate.search.store.FSDirectoryProvider"/>
  <property name="hibernate.search.default.indexBase" value="lucene_index"/>
  <property name="hibernate.search.reader.strategy" value="shared"/>
  <property name="hibernate.search.worker.execution" value="sync"/>

We'll make some adjustments to these settings as the project progresses, but these will suffice for now.

Indexing

The first step in providing full-text search capabilities is to index the content you want to search. In most cases, our users will want to find Wine objects and to a lesser extent Winery objects. Let's start by indexing Wine objects.

Indexing Wine Objects

I'll tackle indexing Wine objects in a few passes, progressively adding features, so let's start with the basics. First, we tell HS to index Wine objects using the @Indexed annotation on the class. As for what to search, I tend to favor using a single field to hold all searchable text for each object because it simplifies working with other Lucene extensions, such as the More Like This and term highlighting features.

@Field(name=DEFAULT_SEARCH_FIELD, index=Index.TOKENIZED)
    @Analyzer(definition="wine_stem_en")
    public String getContent() {
        // return one string containing all searchable text for this object
    }

Notice that the content field is Index.TOKENIZED during indexing. This means that the String value returned by the getContent method will be broken into a stream of tokens using a Lucene Analyzer; each token returned by the Analyzer will be searchable. Your choice of Analyzer depends on the type of text you are processing, which in our case is English text provided by our users when creating Wine objects. Hibernate Search leverages the text analysis framework provided by Solr. You can specify your Analyzer using the @Analyzer annotation. Behind the scenes, Hibernate Search creates the Analyzer using the Factory defined by an @AnalyzerDef class annotation. In our case

@AnalyzerDef(name = "wine_stem_en",
        tokenizer = @TokenizerDef(factory = org.vinwiki.search.Lucene29StandardTokenizerFactory.class),
        filters = {
          @TokenFilterDef(factory = StandardFilterFactory.class),
          @TokenFilterDef(factory = LowerCaseFilterFactory.class),
          @TokenFilterDef(factory = StopFilterFactory.class),
          @TokenFilterDef(factory = EnglishPorterFilterFactory.class)
    })

This looks more complicated than it is ... Let's take a closer look at the @AnalyzerDef annotation:

  name = "wine_stem_en"

This gives our analyzer definition a name so we can refer to it in the @Analyzer annotation. This will also come in handy when we start parsing user queries using Lucene's QueryParser, which also requires an Analyzer.

  tokenizer = 
  @TokenizerDef(factory = org.vinwiki.search.Lucene29StandardTokenizerFactory.class)

Specify a factory for Tokenizer instances. In this case, I'm supplying a custom factory class that creates a Lucene 2.9.x StandardTokenizer as the default factory provided by Solr creates a Lucene 2.4 StandardTokenizer (most likely for backwards compatibility).

  filters = {
    @TokenFilterDef(factory = StandardFilterFactory.class),
    @TokenFilterDef(factory = LowerCaseFilterFactory.class),
    @TokenFilterDef(factory = StopFilterFactory.class),
    @TokenFilterDef(factory = EnglishPorterFilterFactory.class)
  }

A tokenizer can pass tokens through a filter-chain to perform various transformations on each token. In this case, we're passing the tokens through four filters, in the order listed above.

StandardFilter: Removes dots from acronyms and 's from the end of tokens. Works only on typed tokens, i.e., those produced by StandardTokenizer or equivalent.
LowerCaseFilter: Lowercases letters in tokens
StopFilter: Discards common English words like "an" "the" and "of".
EnglishPorterFilter: Extracts the stem for each token using Porter Stemmer implemented by the Lucene Snowball extension.

You can index other fields, such as the wine type and style using the @Field annotation.

@Field(name="type",index=Index.UN_TOKENIZED,store=Store.YES)    
    public WineType getType() {
        return this.type;
    }

Building the Index

If you worked through the first post, then you should already have a database containing 2,025 wines. Hibernate Search will automatically index new Wine objects for us when the insert transaction commits, but we need to manually index the existing wines. During startup, the org.vinwiki.search.IndexHelper component counts the number of documents in the index and re-builds the index from the objects in the database if needed. During startup, you should see log output similar to:

[IndexHelper] Observed event org.vinwiki.event.REBUILD_INDEX from Thread QuartzScheduler1_Worker-8
[IndexHelper] Re-building Wine index for 2025 objects.
[IndexHelper] Flushed index update 300 from Thread Quartz ...
[IndexHelper] Flushed index update 600 from Thread Quartz ...
[IndexHelper] Flushed index update 900 from Thread Quartz ...
[IndexHelper] Flushed index update 1200 from Thread Quartz ...
[IndexHelper] Flushed index update 1500 from Thread Quartz ...
[IndexHelper] Flushed index update 1800 from Thread Quartz ...
[IndexHelper] Took 4094 (ms) to re-build the index containing 2025 documents.

Alright, with a few lines of code and some clever annotations, we now have a full-text search index for Wine objects. Let's do some searching!

Basic Search Box

From the first post, we already have a server-side pagination framework in place for displaying Wine objects. To integrate full-text search capabilities, we simply need to implement the org.vinwiki.action.PagedDataFetcher interface in terms of Hibernate Search. Here is a simple implementation (we'll add more features later):

This is sufficient to get us searching quickly.

Next, we need to pass the user's query from the search box to our nav component. In view/WEB-INF/facelets/searchControls.xhtml, update the JSF inputText tag to bind the user's query to an Event-scoped component named searchCriteria:

<h:inputText id="basicSearchQuery" styleClass="searchField" value="#{searchCriteria.basicSearchQuery}"/>

The searchCriteria component is in-jected into nav, which passes it on to an instance of org.vinwiki.search.Search which implements the PagedDataFetcher interface. You may be thinking that having a separate component for a single input field is over-kill, but the searchCriteria component will also come in very handy once we add an advanced search form. Here is what happens in nav:

@In(required=false) private SearchCriteria searchCriteria;
    ...
    public void doSearch() {
        cleanup();
        dataFetcher = new FeatureRichSearch(log, searchCriteria);
        initFirstPageOfItems(); // fetches the first page of Items
    }

Execute Search on Enter

There is the obligatory Search button next to my search input field, but most user's will just hit enter to execute the search. RichFaces makes this trivial to support using the <rich:hotKey> tag:

<rich:hotKey key="return" selector="#basicSearchQuery" 
          handler="#{rich:element('searchBtn')}.onclick();return false;"/>

The <rich:hotKey> tag binds a JavaScript event listener for the search box to submit the form when the user hits "enter". Be sure to use the selector attribute to limit the listener to the search box and not all input text boxes on the page! Any valid jQuery selector will do ...

At this point, re-compile and re-deploy these changes (at the command-line: seam clean reexplode). A search for "zinfandel" results in:

Hopefully, you get something similar in your environment ;-) Yes, those images are terribly ugly right now ... They were pulled dynamically from Freebase; if this were a real production application, then I'd work on getting better images. Next, we'll add some more features to the search engine, such as query term highlighting, spell correction, more like this, and filters. These additional features are implemented in the org.vinwiki.search.FeatureRichSearch class.

Highlighting Query Terms in Results

Highlighting query terms in results is a common and very useful feature of most search engines. The contrib/highlighter module in Lucene makes this feature easy to implement. There are two aspects to consider when displaying results to the user. First, we want to display the best "fragment" from the document text for the specific query. In other words, rather than just showing the first X characters of the description, we should show the best section of the description matching the user's query. Second, highlight the query terms in the chosen fragment. However, most of the description text for our Wine objects is short enough to display the full description for each result. Based on my current UI layout, 390 is a safe maximum length for fragments as it is long enough to provide useful information about each wine, yet short enough to keep from cluttering up the screen with text. This is something you'll have to work out for your application.

Highlighting Fragments with Hibernate Search

Recall that we stuff all the searchable text information about a Wine into a single searchable field "content". While good for searching, it's probably not something you want to display to your users directly. Instead, I've chosen to highlight terms in the wine description only. Here is how to construct a Highlighter (from the Lucene org.apache.lucene.search.highlight package):

// Pass the Lucene Query to a Scorer that extracts matching spans for the query
    // and then uses these spans to score each fragment    
    QueryScorer scorer = new QueryScorer(luceneQuery, Wine.DEFAULT_SEARCH_FIELD);
    // Highlight using a CSS style    
    SimpleHTMLFormatter formatter = new SimpleHTMLFormatter("<span class='termHL'>", "</span>");
    Highlighter highlighter = new Highlighter(formatter, scorer);
    highlighter.setTextFragmenter(new SimpleSpanFragmenter(scorer, Nav.MAX_FRAGMENT_LEN));

Also notice that I get a reference to the "wine_stem_en" Analzyer using:

Analyzer analyzer = searchFactory.getAnalyzer("wine_stem_en");

While iterating over the results, we pass the Analyzer and the actual description text to the Highlighter. The observant reader will notice that I didn't "stem" the description text during indexing, but now I am stemming the description text for highlighting. You'll see why I'm taking this approach in the next section when I add spell correction.

highlightedText = highlighter.getBestFragment(analyzer, Wine.SPELL_CHECK_SEARCH_FIELD, description);

FeatureRichSearch saves the dynamically highlighted fragments in a Map because fragments are specific to each query. If the query changes, then so must the fragments for the results. The getResultDisplayText method on the nav component interfaces with the FeatureRichSearch dataFetcher to get fragments for search results.

Spell Correction

For spell correction, I'm using the contrib/spellchecker module in Lucene (see http://wiki.apache.org/lucene-java/SpellChecker), which is a good place to start, but you should realize that handling spelling mistakes in search is a complex problem so this is by no means the "best" solution.

Spell Correction Dictionary

To begin, you need a dictionary of correctly spelled terms from which to base spell corrections on. The spellchecker module builds a supplementary index from the terms in your main index using fields based on n-grams of each term. Internally, SpellChecker creates several n-gram fields for each word depending on word length. Here is a screenshot of the word "zinfandel" in the SpellChecker index courtesy of Luke:

Recall that we're stemming terms in our default search field "content". If you build the spell checker index from this field, the dictionary will only contain stemmed terms. This has un-desired side-effect that the corrected term will look misspelled to the user. For example, if you search for "strawbarry", the spell checker will probably recommend "strawberri", which is good, but we really want to show the user "Did you mean 'strawberry'?". Thus, we need to base our spell checker index off non-stemmed text. When we ask the spell checker to suggest a term for "strawbarry", then it will return "strawberry". When we query the search index, we need to query for "strawberri", which is why I pass the "wine_stem_en" Analyzer to the QueryParser after applying the spell correction process.

Hibernate Search Automatic Indexing and Spell Correction

Lucene's SpellChecker builds a supplementary index from the terms in your main index. If your main index changes, e.g. after adding a new entity, then you need to update the spell correction index. From discussing this within the HS community, there's nothing built into HS to help you determine when to update the spell checker index, especially if you're using hibernate.search.worker.execution=async. In other words, you don't know when Hibernate Search is finished updating the Lucene index. You have a couple of options to consider here: 1) update the spell index incrementally as new content is added (or updated), or 2) update the index periodically in a background job. This depends on your requirements and how much you trust the source of the changes to the main index. For now, I'm using Seam Events to incrementally update the spell index after a new Wine is added or an existing Wine is updated (see src/hot/org/vinwiki/search/IndexHelper.java for details).

Edit Distance as Measure of Similarity Between Terms

Lucene's SpellChecker uses the notion of "edit distance" to measure the similarity between terms. When you call spellChecker.suggestSimilar(wordToRespell, ...), the checker consults an instance of org.apache.lucene.search.spell.StringDistance to score hits from the spell checker index. Here's how a SpellChecker is constructed:

Directory dir = FSDirectory.open(new java.io.File("lucene_index/spellcheck"));
    SpellChecker spell = new SpellChecker(dir);
    spell.setStringDistance(new LevensteinDistance());
    spell.setAccuracy(0.8f);
    String[] suggestions = spell.suggestSimilar(wordToRespell, 10,
                             indexReader, Wine.SPELL_CHECK_SEARCH_FIELD, true);

The LevensteinDistance class implements StringDistance by calculating the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character.

Spell Correction Heuristics

The spellchecker module does a fine job of suggesting possible terms, but we still need to decide how to handle these suggestions depending on the structure of the user's query. To keep things simple, our first heuristic applies to single term queries where the mis-spelled term does not exist in our spell checker index. In this case, we simply re-issue the search using the best suggestion from the spell checker. Here is a screen shot of how we present the results to the user:

However, we can't be sure that a term is mis-spelled if it exists in the spell checker index. Thus, if a single mis-spelled term exists in the spell correction index, then you need to decide if you are going to just "OR" the term mis-spelled and suggested terms together or let the user decide (it seems like Google doesn't always do one or the other, so their implementation is a bit more advanced than the one I'll provide here).

In the screenshot above, the user queried for "velvety tanins"; both terms exist in the spell correction index, but "tanins" is mis-spelled. During query processing, the SpellChecker suggested "tannins" for the mis-spelled word "tanins", which is correct. Thus, we search for "velvety tanins", but also suggest a spell-corrected query "velvety tannins", allowing the user to click on the suggested correction to see better results (hopefully). Please refer to the checkSpelling method in the src/hot/org/vinwiki/search/FeatureRichSearch.java for details on how these simple heuristics are implemented.

Type-ahead Search Suggestions

Most users appreciate when an application spares them un-necessary effort, such as typing common phrases into the search box. Thus, we'll use RichFaces <rich:suggestionbox> tag to provide a drop-down suggestion list of known search terms after the user types 3 or more characters. While trivial, this feature can help users be more productive with your search interface and helps minimize spelling mistakes.

In /WEB-INF/facelets/searchControls.xhtml, we attach the <rich:suggestionbox> to the search input text field using:

<rich:suggestionbox id="suggestionBox" for="basicSearchQuery" ignoreDupResponses="true"
                    immediate="true" limitToList="true" height="100" width="250" minChars="3"
                    usingSuggestObjects="true" suggestionAction="#{nav.suggestSearchTerms}"
                    selfRendered="true" first="0" var="suggestion">
  <h:column><h:outputText value="#{suggestion}"/></h:column>
</rich:suggestionbox>

In the first posting, I discussed queue and traffic flood protection using RichFace's Global Default Queue. You should definitely active an AJAX request queue if you are using type-ahead suggestions. I've also added an animated GIF and loading message using the <a4j:status> tag.

On the Java side, I've resorted to a simple LIKE query to the database, but you can also query a field analyzed with Lucene's org.apache.lucene.analysis.ngram.EdgeNGramTokenFilter in your full-text index.

Advanced Search

Sometimes a single search field isn't enough to pin-point the information you're looking for; advanced search addresses this, albeit uncommon, need for users to formalize complex queries. The advanced search form is application specific but most allow the user construct a query composed of AND, OR, exact phrase, and NOT clauses, as shown in the following screen shot:

As mentioned above, the org.vinwiki.search.SearchCriteria class manages the advanced search form data. I'll refer you to the source code for further details about advanced search. Notice that I'm using a RangeQuery to implement the Date field on the search form.

Testing Search

Be careful when building tests involving Lucene because lib/test/thirdparty-all.jar contains an older version of Lucene. To remedy this, I added the following list of JARs to the top of the test.path path in build.xml:

<fileset dir="${lib.dir}">
    <include name="lucene-core-2.9.2.jar"/>
    <include name="lucene-highlighter-2.9.2.jar"/>
    <include name="lucene-misc-2.9.2.jar"/>
    <include name="lucene-snowball-2.9.2.jar"/>
    <include name="lucene-spatial-2.9.2.jar"/>
    <include name="lucene-spellchecker-2.9.2.jar"/>
    <include name="lucene-memory-2.9.2.jar"/>
    <include name="solr-core-1.4.0.jar"/>
    <include name="solr-solrj-1.4.0.jar"/>
  </fileset>

A new test case was added for testing search, see src/test/org/vinwiki/test/SearchTest.java. While fairly trivial, this class helped me root out a few issues (like updating the spell correction index after an update) while developing the code for this post. So test-driven development shows its worth once again!

Future Enhancements to the Search Engine

There are still a few search-related features I think this application needs, including tagging, phrase handling, and synonyms. Specifically, user's should be able to add tags like "jammy" or "flabby" when rating wines. The application should be able to render a tag cloud from these user-supplied tags as another form of navigation. User tags should also be fed into the search index (using caution of course). Phrase detection complements user-supplied tagging by recognizing multi-term tags during text analysis. How you handle stop words also affects exact phrase matching. The contrib/shingles module helps speed up phrase searches involving common terms, so I'd definitely like to investigate its applicability for this application. Lastly, synonyms help supply the search index (and/or search queries) with additional terms that mean the same thing as other words in your documents. If time allows, I'll try to add a fifth post dealing with tags, phrases, and synonyms. I'd also like to hear from the community on other features that might be helpful.

TODO: Evaluating Search Quality with contrib/benchmark

The contrib/benchmark module is one extension all Lucene developers should be familiar with; benchmark allows you to run repeatable performance tests against your index. I'll refer you to the JavaDoc for using benchmark for performance testing. In the future, I'd like to use benchmark's quality package to measure relevance of search results. The classes in the quality package allow us to measure precision and recall for our search engine. Precision is a measure of "exactness" that tells us the proportion of the top N results that are relevant to the search. Recall is a measure of "completeness" that tells us the proportion of all relevant documents included in the top results. In other words, if there are 100 relevant documents for a query and the results return 80, then recall is 0.8. Sometimes, the two metrics conflict with each other in that as you return more results (to increase recall), you can introduce irrelevant documents, which decreases precision. The quality package will give us a way to benchmark precision and recall and then tune Lucene to improve these as much as possible. If I get time, then I'll post my results (and source).

A last words about scalability

While Lucene is very fast, you can run into search performance issues if you are constantly updating the index, which requires Hibernate Search to close and re-open its shared IndexReader. Hibernate Search has built-in support for a Master / Slave architecture where updates occur on a single master server and searches are executed on replicated, read-only slave indexes in your cluster. I use this configuration in my day job and it scales well. However, you'll need a JMS queue and Message-driven EJB to allow slave nodes to send updates to the master node, so you'll have to deploy your application as an EAR instead of a WAR (to deploy the MDB). Please refer to the online documentation provided with Hibernate Search for a good discussion on how to setup a master / slave cluster.

What's Next?

In the next post, I'll add authentication and registration using Facebook Connect.

VinWiki Part 1: Building an intelligent Web app using Seam, Hibernate, RichFaces, Lucene and Mahout

2010-05-24T15:17:00.000-07:00

Introduction

This is the first post in a four part series about a wine rating and recommendation Web application, named VinWiki, built using open source technology. The purpose of this series is to document key design and implementation decisions, which may be of interest to anyone wanting to build an intelligent Web application using Java technologies. The end result will not be a 100% functioning Web application, but will have enough functionality to prove the concepts. Specifically, here is a basic roadmap of the concepts covered in each post:

Introduction (May 24, 2010): Covers project setup, primary domain objects, and basic UI constructs such as server-side pagination, dynamic menus, and bookmarkable URLs with JSF / RichFaces / Facelets.
Full-text Search (June 14, 2010): Implements full-text search using Hibernate Search with Lucene 2.9.2.
Integrating with the Web (July 8, 2010): Authentication and registration using Facebook Connect and sharing/liking bookmarkable URLs on Facebook.
Recommendations (Sept 23, 2010): Provide wine recommendations using Apache Mahout.

The idea for VinWiki was born out of two interests in my life: wine and collective intelligence. I'm a wine connoisseur and love red wines from Piedmont (Italy) and the Dry Creek and Alexander Valley's in Sonoma. I don't, however, drink expensive wine. Rather, I'm always on the lookout for that $10-15/bottle wine that just tastes good (who isn't right?) My strategy is to know a lot about a little. For example, I know the major varietals in Piedmont, their characteristics, which years were good and which were not, etc. I know these things because I want to be able to drink great inexpensive wines and help my friends do the same. Wouldn't it be nice if you had access to all this knowledge in my head the next time you go out for Italian food in North Beach?

One of my other interests is figuring out how to harvest intelligence from the interactions and contributions of users on a Web site and then apply that intelligence to improve the personal experience with the application as well as the application as a whole. The science behind this is called Collective Intelligence. To keep things simple, I consider a Web application to be "intelligent" if it has the following key ingredients:

automatically improves its capabilities as more users contribute to it,
scalable (machine learning algorithms do better with large amounts of data),
mashable (simple Web services that expose data and services to other applications), and
doesn't pretend to be more than it is!

There's a wealth of knowledge available about how to implement the first ingredient using machine learning. And, as we'll see in the 4th post in this series, there is a wonderful open source project named Mahout that provides scalable implementations of many popular machine learning algorithms, such as clustering, classification, and recommendations. However, please keep in mind that it takes rigorous testing and experimentation to ensure these algorithms produce good results for your data. Don't assume that because an algorithm sounds sophisticated that it will produce good results in your application. Hence, the last key ingredient! In this application, intelligence comes in the form of making recommendations for wines from previous ratings using collaborative filtering. Presumably, as more users contribute wines and ratings, the application will improve for all users.

Toolset

The solution is built using the following open-source technologies:

JBoss AS 4.2.3
Seam 2.2.0
Hibernate (Core, EntityManager, Annotations, Validator & Search)
JSF 1.2 / RichFaces 3.3.2 SR1 / Facelets
Lucene 2.9.2
Mahout 0.3 / Hadoop 0.20.2

Minor updates to the Seam-based user registration example ...

2009-05-26T20:37:00.000-07:00

Based on feedback from the community, I made the following updates to the Seam example (nothing too major except verifying that the solution works with the latest RichFaces release)

Tested with RichFaces 3.3.1 GA -- works great!
Updated the GuestSupport class to use Seam's StatusMessages instead of working directly with FacesMessage(s).
Updated the registration form to use Seam's <s:decorate> tag and a Facelets template, which really simplified the form.
Moved the solution-specific text messages from i18n.properties into the Seam default messages.properties ... got tired of having two separate message bundles at work ...
Cleaned up a few JPA annotations to be more inline with the spec.
Localized the last few Hibernate Validator error messages that had hard-coded English text in them.
Created the Gender and ProfilePolicy enums used by the Preferences entity.

The updated source code is available here.

NOTE: If you want to upgrade to RichFaces 3.3.1, then you need to remove the version information from the JAR filenames:

richfaces-api-3.3.1.GA.jar rename to richfaces-api.jar
richfaces-impl-3.3.1.GA.jar rename to richfaces-impl.jar
richfaces-ui-3.3.1.GA.jar rename to richfaces-ui.jar

User registration solution using JBoss Seam with JSF / RichFaces and Hibernate

2009-05-13T19:39:00.001-07:00

In this article, I show you how to implement a Web 2.0 style user registration solution using JBoss Seam with JSF/Facelets/RichFaces and Hibernate. This article evolved from a real Web 2.0 site I'm building. You can learn more about me on my LinkedIn profile. As for what's in it for you, I've provided the source code and a working solution using:

JSF 1.2 (Mojarra) with Facelets and RichFaces 3.2.2 SR1
Hibernate (EntityManager, Annotations, and Validator)
Seam 2.1.1.GA on JBoss 4.2.2.GA

In future articles, I plan to incorporate Hibernate Search and some collective intelligence techniques such as tag clouds, clustering, collaborative filtering, and recommendations. While writing this article, I assumed the reader would have a basic understanding of JSF and Hibernate. However, I do spend extra time explaining some of the key features of Seam and RichFaces.

I'd like to give credit to Dan Allen as I've used many of the ideas he teaches in Seam In Action in my own Seam development; his book is one that all Seam developers should own!

Basic Requirements

Here are the high-level requirements this solution satisfies. Notice that most these requirements apply to many Web 2.0 sites seen on the Web today.

1. Public-facing Web site with read-only access to Guests
The site should be useful without logging in and the content should be accessible to search engines. Since casual surfers will be visiting the site, it must load fast and be easy to use on all major browsers (IE 6 / 7, FireFox 3.x, and Safari 3.x).

Jump to solution ...

2. Open registration process with email verification
Guests should be able sign-up for a new account using a simple form that collects minimal information from the user; CAPTCHAs should be used to prevent bots from creating accounts. Each user must agree to the site's terms of use before registering. After registering, the user will also need to click on an activation link sent to their email, which ensures they have access to the email account provided on the form.

Jump to solution ...

3. Remember Me
Registered users can request the site to remember their credentials for automatic login for future visits.

Jump to solution ...

4. Ability to recover forgotten passwords
Since we have a valid email for every user, users can request a new temporary password to be emailed to them. Upon logging in with the temporary password, the user should be prompted to change their password to a permanent value.

Jump to solution ...

5. User Preferences
Registered users can share personal information, upload a photo, and setup site preferences, such as whether they are interested in receiving emails about special promotions from the site.

Jump to solution ...

6. Strong Server-side Validation
Since we're exposing the site to the Wild Wild Web, server-side validation is essential; client-side validation using JavaScript is a nice to have, but is not sufficient. It's trivial for a hacker to by-pass client-side validation so you want to make sure your server code is protected as well. Where appropriate, use AJAX to give immediate feedback to the user when they've entered invalid data.

Jump to solution ...

Seam-based Architecture

In a previous blog posting, I used Spring with JSF / RichFaces to satisfy these requirements and advocated a layered architecture. While that is still a valid approach, I take a different approach in this article to highlight key features of Seam. Specifically, instead of covering each layer, I describe how I implemented each requirement. The layers are there if you need them, but with Seam you're not forced to think in terms of stateless layers. What initially attracted me to Seam was its get down to business approach. First off, it comes with a tool seam-gen which creates the skeleton project for your application, including generating JPA-based Entity code by reverse engineering a database schema. With Seam, you can get right to work building your UI that directly interacts with your entities. In contrast, with my Spring-based solution, I had to flush out the persistence and business-service layers consisting of the following interfaces and classes:

example.user.UserService (interface)
example.user.impl.UserServiceImpl (class)
example.user.dao.UserDao (interface)
example.user.dao.RoleDao (interface)
example.user.dao.impl.HibernateUserDao (class)
example.user.dao.impl.HibernateRoleDao (class)

Seam, on the other hand, treats the persistence manager as the DAO and provides a framework for CRUD components (see Seam Application Framework). I'll address the Seam Application Framework in a future article, for now I'm going to stay focused on the requirements at hand. I just wanted to make sure you knew that I wasn't throwing the persistence and business-service layers out the window by using Seam. They are there when you need them but are not in the way otherwise.

Here are some of the key aspects of Seam that I'll be using to satisfy the requirements:

POJOs with bi-directional dependency injection known as bijection (note: I'm not using EJB3 for this project, but Seam makes it really easy to use EJBs if you need them)
Object-Relational Mapping (ORM) using the Java Persistence API (JPA) and Hibernate under the covers.
Declarative transaction management using Seam annotations
JSF with Facelets and RichFaces
Test-driven development based on TestNG

Seam Project Organization

Here is the link to the Seam project created by seam-gen. Before digging too deeply into the details of the configuration, take moment to familiarize yourself with the organization of the Seam project:

NOTE: You'll need to copy the lib directory from your Seam 2.1.1.GA directory to my project directory as I excluded the lib directory to minimize the size of the download.

Object Persistence via JPA

Domain Entities

There are three primary domain entities: User, Preferences, and Role, which should be self-explanatory given the requirements outlined above. The entities are POJOs with the exception of using Seam, JPA, and Hibernate annotations in a few key places. For example, in the following code snippet, we annotate the example.user.User class as a JPA Entity and map it to the ex_user table in the database:

@Entity
@Table(name = "ex_user", uniqueConstraints = @UniqueConstraint(columnNames = "user_name"))
public class User implements Serializable ...

Notice that I'm using surrogate keys for my domain object identifiers, which are auto-generated by Hibernate using the best approach for the specific database-type:

@Id @GeneratedValue(generator="native")
@GenericGenerator(name="native", strategy = "native")
@Column(name = "user_id", unique = true, nullable = false)
public Long getUserId() {
    return userId;
}

I'm also using the Hibernate Validator annotations to restrict the values for some of the members, such as the user's email address:

@Column(name = "email", nullable = false, length = 60)
@NotNull
@Length(max = 60)
@Email
public String getEmail() {
    return this.email;
}

Domain Object Associations

There is a one-to-one relationship between the Preferences and User objects. The Preferences object is loaded with the User object (fetch=FetchType.EAGER) because it contains information that is needed to render the UI for the user.

@OneToOne(cascade = CascadeType.ALL,fetch = FetchType.EAGER)
@PrimaryKeyJoinColumn
public Preferences getPreferences() {
    return this.preferences;
}

There is a bidirectional many-to-many relationship between the User and Role objects using a join table ex_user_role.

@UserRoles
@ManyToMany(targetEntity = Role.class,cascade = {CascadeType.PERSIST,CascadeType.MERGE})
@JoinTable(name = "ex_user_role", joinColumns = @JoinColumn(name = "user_id"), inverseJoinColumns = @JoinColumn(name = "role_id"))
public Set getUserRoles() {
    if (userRoles == null) {
        userRoles = new HashSet ();
    }
    return userRoles;
}

On the Role object, we have:

@ManyToMany(cascade={CascadeType.PERSIST,CascadeType.MERGE},mappedBy="userRoles",targetEntity=User.class)
public Set getRoleUsers() {
    if (roleUsers == null) {
        roleUsers=new HashSet ();
    }
    return roleUsers;
}

This solution comes directly from the Hibernate documentation, section 7.5.3.

Notice that my entity classes are in the src/main directory as these classes cannot be incrementally hot-deployed by Seam. This means that if you change the code in these classes, then you need to recycle the Web application before the change is activated. On the other hand, classes in the src/hot directory are updated by Seam on-the-fly without recycling the Web application. Consequently, you want to put as many of your classes in the src/hot directory as possible (see: 2.8. Seam and incremental hot deployment).

Now that you understand the key domain entities involved in this solution, you're ready to see how I implemented each requirement.

Solution for Requirement #1 (Public-facing Web site with read-only access to Guests)

Much like many Web 2.0 sites today, my site will be useful to casual guests of the site. So it needs to load quickly and look attractive on all major browsers (IE 6-7, FireFox 3.x, and Safari 3.x). Thankfully, JSF and RichFaces satisfy these requirements out-of-the-box. Here are some of the reasons why I like JSF with RichFaces:

Modern, professional looking skin out-of-the-box (I don't like to fool with CSS).
Extensive set of easy-to-use, yet flexible UI components, such as trees, data grids, tabbed panes, etc.
Easy to create reusable custom components.
Large community of users and developers building real apps.
Performance: loads and runs fast on all modern browsers.
Minimizes hand-coded JavaScript.
Templates using Facelets

Key design decision: I decided to use the RichFaces ModalPanel component for my forms because it helps the user keep their current orientation with the site. Instead of navigating to a different page, I simply open a dialog over the current view. Consequently, I didn't have any need for JSF navigation rules in this example because there is only one page.

Solution for Requirement #2 (Open registration process with email verification)

For this requirement, it's useful to break it down into several detailed steps:

Guests are presented with a link to register.
Simple form to collect minimal user information.
Require the user to solve a CAPTCHA to prevent bots from creating accounts.
Require the user to agree to the site's terms of use.
Validate the user's email address before activating the account.
Activate the account when the user clicks on a link in the activation email.

Solution for Requirement #2a (Register Link)

In the view/WEB-INF/facelets/headerControls.xhtml file, I open the registerPanel dialog using a rich:componentControl which calls the show method of the rich:modalPanel component when the link is clicked:

<h:outputLink value="#" id="registerLink">
  <h:outputText value="#{i18n.register}"/>
  <rich:componentControl for="registerPanel" attachTo="registerLink" operation="show" event="onclick"/>
</h:outputLink>

Solution for Requirement #2b (Simple form to collect minimal user information)

In the view/WEB-INF/facelets/guestSupport.xhtml file, I use a rich:modalPanel to hold the registration form. My registration form only requires the user to supply a screen name, password, email, first, and last name. Notice that the form fields are bound to properties of a guest component, for example:

<h:inputText label="#{i18n.login_username}" id="userName" required="true" 
value="#{guest.registrationScreenName}" size="12" maxlength="12" redisplay="true">
  ...
</h:inputText>

The guest object is a Seam component example.action.GuestSupport that handles the registration process, see: src/hot/example/action/GuestSupport.java. Notice the annotations on the GuestSupport class:

@Name("guest")
@Scope(ScopeType.EVENT)
public class GuestSupport ...

The @Name annotation means that this is a Seam component that can be referenced in the EL using the identifier guest, such as: #{guest.registrationScreenName}. The @Scope(ScopeType.EVENT) annotation means that the component is managed in Seam's EVENT scope, which means the component exists from the Restore View phase until the end of the Render Response phase in the JSF lifecyle. Essentially, the guest component exists only to handle a guest action and then is released by the Seam container. If no guest actions are performed, then this component is never created.

The form is bound to the guest.doRegister method:

<a:commandButton type="button" id="registerButton" value="#{i18n.register}" action="#{guest.doRegister}"/>

Notice that I'm using a RichFaces AJAX form <a:form> which means the register action will be submitted as an AJAX request without reloading the entire page.

Solution for Requirement #2c (CAPTCHA)

In my Spring solution, I integrated with reCAPTCHA. However, Seam comes with a CAPTCHA solution built-in so I decided to give it a try. Here is how to embed the Seam CAPTCHA solution into your form:

<s:decorate id="verifyCaptchaField" template="captcha.xhtml">
  <h:graphicImage id="captchaChallenge" value="/seam/resource/captcha" styleClass="captchaChallenge"/>
  <h:inputText id="verifyCaptcha" value="#{captcha.response}" required="true" size="3"/>
</s:decorate>

The default solution is to show an image of a simple addition problem, which unlike reCAPTCHA is much easier to read and is probably pretty safe unless your site is for preschoolers who might not be able to solve the addition problem.

From what I've seen, it's also pretty easy to extend the built-in solution to provide your own image creation solution. You should note that if you use Seam's CAPTCHA solution, then you must make sure the Seam Resource Servlet is activated in your web.xml file (done automatically by seam-gen).

Solution for Requirement #2d (Terms of Use)

I built a placeholder HTML file to hold the site's legal text, see: view/WEB-INF/facelets/legal.html. If the user submits the form without agreeing to the terms of use, then a nice error message is returned. This is handled by the following code in my doRegister method:

if (!agreedToTermsOfUse) {
    String msg = facesSupport.getMessage("please_agree_to_terms", extCtxt.getRequestLocale());
    jsf.addMessage("registerForm:registerButton", new FacesMessage(FacesMessage.SEVERITY_ERROR, msg, null));
    return;
}

Solution for Requirement #2e (Email Verification)

The doRegister method creates the user but does not activate the account. Instead, it creates an activation key which is an MD5 hash of the user's information and the current timestamp. The timestamp is needed because there is a very slight chance of getting the same hash code for two different users. The activation key is sent to the user as a link back to the site that will activate their account.

Seam uses Facelet templates as email templates. For account verification, I created the view/WEB-INF/facelets/email/activation.xhtml template. Notice that I can use the EL to reference my Seam components just as I would for an HTML template:

<m:from name="JSF / RF Seam Example" address="support@saastk.com"/>
<m:to name="#{inactiveNewUser.email}">#{inactiveNewUser.email}</m:to>
<m:subject>JSF / RF Seam Example Account Activation</m:subject>
<m:body>
  <html>
    <body>
      <p>Dear #{inactiveNewUser.displayName},</p>
      <p>Please click on the following link to activate your account:</p>
      <p><a href="#{inactiveNewUser.activationLink}">#{inactiveNewUser.activationLink}</a></p>
    </body>
  </html>
</m:body>

The registrationMailer component processes the email templates and sends the email, see: src/hot/example/action/RegistrationMailer.java. Notice the annotations on the RegistrationMailer class:

@Name("registrationMailer")
@AutoCreate
@Scope(ScopeType.APPLICATION)
public class RegistrationMailer ...

The @Name annotation means that this is a Seam component that can be referenced in the EL using the identifier registrationMailer. The @Scope(ScopeType.APPLICATION) annotation means that the component is managed in Seam's APPLICATION scope, which means the component exists while the Web application is active. Essentially, there will be only one registrationMailer used by the Seam container. @AutoCreate means that Seam will create this component for us automatically.

I decided to send the email as part of the transaction meaning that if the email fails, then no account is created. The following code puts an instance of the aptly named InactiveNewUser bean into EVENT scope and the raises a synchronous application event.

Contexts.getEventContext().set("inactiveNewUser",
  new InactiveNewUser(newUser.toString(), newUser.getUserName(), newUser.getEmail(), newUserLink));
Events.instance().raiseEvent("userRegistered");

The RegistrationMailer is an observer of this event:

@Observer("userRegistered")
public void sendActivationEmail() {
    renderer.render("/WEB-INF/facelets/email/activation.xhtml");
}

Supposedly, you can send the email asynchronously, but I ran into trouble so I stayed with the synchronous approach for now. I'll post to the Seam forum to see what I can find out ...

NOTE: Be sure to configure a <mail:mail-session host="???" port="25"/> in the Seam configuration file: resources/WEB-INF/components.xml. See Chapter 21. Email

Solution for Requirement #2f (Account Activation)

Assuming all goes well and the email provided by the user is valid, then they will receive an email with a link back to the site, such as:

http://localhost:8989/jsf_rf_seam_user_reg/home.seam?act=19eb2e7567913c47a931e305cdeb6d311242247928343

When the user clicks on the link, we need to process the act request parameter. Fortunately, Seam makes this really easy to do using a page action. In resources/WEB-INF/pages.xml I declare an action for the home.xhtml page:

<action execute="#{guest.doActivate(request.getParameter('act'))}" 
  if="#{request.getParameter('act') != null}"/>

In plain English, Seam invokes the guest.doActivate method if the request contains the act parameter. If activation is successful, then I trigger the Login form to open up with the newly activated username pre-populated.

@Transactional
public void doActivate(String activationKey) {
    Query q = entityManager.createQuery("from User u where u.active=0 AND u.activationKey=:activationKey");
    q.setParameter("activationKey", activationKey);
    User activatedUser = (User)q.getSingleResult();
    activatedUser.setActive(true);
    activatedUser.setActivationKey(null);
    loginUserId = activatedUser.getUserName();
    credentials.setUsername(loginUserId);
    log.info("User {0} activated successfully.", loginUserId);
}

NOTE: You may want to have a background process that cleans up accounts that haven't been activated after so many days.

Solution for Requirement #3 (Remember Me)

Before I can tackle the Remember Me requirement, I need to make sure you know how login works with Seam. If you search my code, you'll see that there is no login method. Instead, I rely on Seam's Identity Management framework to do the login work for me. The Identity Management framework requires an identity store in resources/WEB-INF/components.xml:

<security:jpa-identity-store user-class="example.user.User" role-class="example.user.Role"/>

Notice that I'm reusing my User and Role Entities that I've already developed. My login form is bound to Seam's identity.login method:

<h:commandButton type="submit" id="loginBtn" action="#{identity.login}" value="#{i18n.login}"/>

See loginPanel in view/WEB-INF/facelets/guestSupport.xhtml

If login is successful, then Seam raises the Identity.EVENT_LOGIN_SUCCESSFUL event; if login fails, then Seam raises the Identity.EVENT_LOGIN_FAILED event. Thus, my GuestSupport component is configured as an @Observer for these events:

@Observer(Identity.EVENT_LOGIN_SUCCESSFUL)
public void onLogin() {
    this.currentUser = byUserName(credentials.getUsername());
    log.info("User {0} logged in successfully.", this.currentUser.getUserName());
}

@Observer(Identity.EVENT_LOGIN_FAILED)
public void onLoginFailed() {
    this.currentUser = null;
    FacesContext jsf = FacesContext.getCurrentInstance();
    String msg = facesSupport.getMessage("login_error", jsf.getExternalContext().getRequestLocale());
    jsf.addMessage("loginForm:loginBtn", new FacesMessage(FacesMessage.SEVERITY_ERROR, msg, null));
}

In the onLogin method, we see an application of Seam's outjection @Out in action. If login succeeds, then we outject the currentUser object into Seam's SESSION Scope as seen by:

@Out(required=false, scope=ScopeType.SESSION)
private User currentUser;

So, unlike Spring, Seam can easily export runtime objects into the container using simple annotations! Once outjected, the currentUser component can be used seamlessly in our JSF UI.

At this point, you may be wondering how Seam verifies the credentials supplied by the user against the database. There is a little annotation magic going on here that is not immediately apparent. If you study the example.user.User class, you'll see that I'm using a few annotations from the org.jboss.seam.annotations.security.management package:

@Column(name = "user_name",unique = true,nullable = false,length = 16)
@NotNull
@Length(min = 4,max = 16)
@Pattern(regex = "^[a-zA-Z\\d_]{4,12}$",message = "Invalid screen name.")
@UserPrincipal
public String getUserName() {
    return this.userName;
}

...

@Column(name = "password",nullable = false,length = 128)
@NotNull
@Length(max = 128)
@UserPassword(hash = "SHA")
public String getPasswordHash() {
    return this.passwordHash;
}

The @UserPrincipal and @UserPassword annotations are used by the Seam Identity Management framework to verify the credentials during login. With these annotations, Seam has all the information it needs to validate credentials against a one-way SHA-1 hash stored in the database.

Remember Me

Seam supports two approaches to remembering users when they return to your site, aptly named Remember Me: 1) only the username is remembered, or 2) a persistent access token is stored in a cookie to allow full automatic login. If you read the Seam documentation, you are strongly advised against the latter approach due to cross-site scripting vulnerabilities. While I understand those concerns, it is still a really nice feature to offer for users. Consequently, I'm going to show you how I implemented the persistent token approach and if you don't decide to use it, then it is very easy to rollback to the username only solution. I actually think the username-only solution is pretty slick if implemented correctly. LinkedIn is a good example of the username only solution where it recognizes you and shows your profile in read-only mode. LinkedIn only requires you to login if you try to make a change to your profile.

There are four steps you need to do to enable automatic login:

First, you need to activate Seam's rememberMe component in the resources/WEB-INF/components.xml file and set the mode to autoLogin:

<security:jpa-token-store token-class="example.security.AutoLoginToken"/>
  <security:remember-me mode="autoLogin"/>

The rememberMe component needs a token store. Seam provides a JPA-based implementation org.jboss.seam.security.JpaTokenStore that stores the token in your database, but you need to provide the Entity class, see src/main/example/security/AutoLoginToken.java. The AutoLoginToken implementation came directly from 15.3.5.1. Token-based Remember-me Authentication in the Seam documentation.

Second, you need to add a checkbox on the login form bound to the rememberMe component's enabled property:

<h:selectBooleanCheckbox id="rememberMe" value="#{rememberMe.enabled}"/>

Third, you need to invoke the identity.tryLogin method when the home page is requested. As with the activation link processing, I configured a simple page action for home.xhtml in pages.xml:

<action execute="#{identity.tryLogin}" if="#{not identity.loggedIn}"/>

The Seam documentation mentioned using the following event declarations in components.xml, but that did not work for me:

<event type="org.jboss.seam.security.notLoggedIn">
  <action execute="#{redirect.captureCurrentView}"/>
  <action execute="#{identity.tryLogin()}"/>
</event>
<event type="org.jboss.seam.security.loginSuccessful">
  <action execute="#{redirect.returnToCapturedView}"/>
</event>

Lastly, you need to observe the Identity.EVENT_POST_AUTHENTICATE event and out-ject the currentUser component into Session scope if auto-login was successful:

@Observer(Identity.EVENT_POST_AUTHENTICATE)
public void postAuthenticate(Identity identity) {
    if (identity != null && identity.isLoggedIn() && this.currentUser == null) {
        String userName = identity.getUsername();
        if (userName == null) userName = identity.getPrincipal().getName();
        if (userName != null) {
            this.currentUser = byUserName(userName);
            log.info("User {0} logged in successfully.", this.currentUser.getUserName());
        }
    }
}

The Identity.EVENT_LOGIN_SUCCESSFUL event was not raised by the rememberMe component upon successful auto-login, which may be a bug? I'll ask the Seam folks ... for now, you have to have this additional @Observer in place :-(

Solution for Requirement #4 (Ability to recover forgotten passwords)

If a user forgets their password, then they can request the site to email a temporary password by submitting their screen name and email.

Since this is an action performed by a guest of the site, the form is processed by the guest component:

@Transactional
public void doRecoverLostPassword() throws Exception {
    ...
}

The @Transactional annotation tells Seam that this method must be invoked within the context of a transaction. If the method completes successfully, then Seam will commit the work for you regardless of the underlying transaction manager your application is using.

When the user logs into the site with their temporary password (sent in an email), the site needs to prompt them to change the password to a permanent value. As before, to keep the user's orientation with the site, I pop-up a modal dialog to require the user to change their password, see chngPswdPanel in view/WEB-INF/facelets/userPreferences.xhtml.

The modal panel will only show itself if the user's password is temporary:

<rich:modalPanel id="chngPswdPanel" width="320" height="240" rendered="#{identity.loggedIn}" showWhenRendered="#{currentUser.temporaryPassword}">

The change password form is handled by the changePassword component, see: src/hot/example/action/ChangePasswordAction.java

Solution for Requirement #5 (User Preferences)

Once logged in, the user can click on the Preferences link to edit their profile and control how they want the site to interact with them. The prefPanel dialog in userPreferences.xhtml is activated using the following AJAX Link a:commandLink in headerControls.xhtml:

<a:commandLink id="prefLink" action="#{updatePreferences.beginEditPreferences(currentUser.userId)}" oncomplete="#{rich:component('prefPanel')}.show()" value="#{i18n.preferences}"/>

Look closely at the EL for the action attribute; I'm using a new component updatePreferences, see: src/hot/example/action/UpdatePreferencesAction.java.

@Name("updatePreferences")
@Scope(ScopeType.CONVERSATION)
@Restrict("#{identity.loggedIn}")
@Transactional
public class UpdatePreferencesAction implements java.io.Serializable ...

The Scope of this component is ScopeType.CONVERSATION, which means this component will exist for the duration of a conversation, possibly spanning multiple requests from the user. The Seam documentation equates a conversation with a single unit-of-work from the perspective of the user (not the database). In this case, the unit-of-work is to edit preferences, which may include uploading a photo as well as submitting an AJAX form one or more times. So the conversation begins when the user opens the preferences dialog and ends when they close it. Here is the beginEditPreferences method:

@Begin(flushMode = FlushModeType.MANUAL, join = true)
public void beginEditPreferences(Long userId) {
    currentUser = entityManager.find(User.class, userId); // attach to this EntityManager
    log.info("User {0} has started editing preferences.", currentUser.getUserName());
}

When the conversation begins, I use the entityManager to get the User Entity for the current user, identified by the userId parameter. Of course, there is a currentUser component in Session Scope (out-jected during the login process). However, it is not attached to the updatePreferences component's entityManager (it is a detached Entity that is only useful for reading data about the current user). When the beginEditPreferences method completes, the currentUser stays attached to the entityManager for the duration of the conversation.

While the preferences dialog is open, the user can submit several requests to the server, such as uploading an image. For image upload, I employ the RichFaces rich:fileUpload component:

<rich:fileUpload fileUploadListener="#{updatePreferences.fileUploadListener}" maxFilesQuantity="1" id="uploadPhoto" immediateUpload="true" acceptedTypes="jpg,gif,png" allowFlash="false" listHeight="110">
  <a:support event="onuploadcomplete" reRender="prefPanel,uploadPhotoPanel"/>
</rich:fileUpload>

The fileUploadListener method participates in the conversation accessing the currentUser that was attached to the entityManager when the conversation began.

When the user closes the preferences dialog, the conversation ends by invoking an AJAX call to updatePreferences.endEditPreferences which is annotated with @End:

@End
public void endEditPreferences() {
    entityManager.flush(); // flush the changes to the database
    log.info("User {0} finished editing preferences.", currentUser.getUserName());
}

Notice that the updatePreferences component out-jects the currentUser component back into Session scope (as the guest component did after a successful login). This is needed so that changes are propagated back to the UI, such as the user's display name shown in the top right corner of the application.

It may be hard to appreciate Seam conversations with such simplistic objects at play. However, as your object associations become more complex, conversations are immensely powerful in helping you manage updates to your entities.

Solution for Requirement #6 (Strong Server-side Validation using AJAX)

With Seam, you don't need any JavaScript validation--all validation can be done efficiently on the server. For example, notice how I ensure the user's password is strong:

<h:inputSecret label="#{i18n.login_password}" id="password" required="true" value="#{passwordSupport.password}" size="12" maxlength="12">
  <f:validateLength minimum="8" maximum="12"/>
  <rich:beanValidator/>
</h:inputSecret>

On the server, the password is validated against a regular expression using Hibernate Validator:

@NotEmpty
@Length(min=8,max=12)
@Pattern(regex="(?=^.{8,}$)((?=.*\\d)|(?=.*\\W+))(?![.\\n])(?=.*[A-Z])(?=.*[a-z]).*$", message="Password is not secure enough!")
private String password = null;

See: src/hot/example/action/PasswordSupport.java

Most of the forms used in this solution are submitted using AJAX requests. The login form and the image upload form are the only two forms that redisplay the entire page when submitted. Here are some other areas where I've used the AJAX features of RichFaces:

Country / Region

One of the nice things about RichFaces is that you can attach AJAX-driven logic to any JSF component without writing any JavaScript. For example, suppose we want to limit the list of regions based on the selected country on the Preferences form. This is trivial using the <a4j:support> tag provided by RichFaces:

<a4j:support event="onchange" rerender="region" ajaxsingle="true">

This fires an AJAX request when the country is changed and upon success, the region component is re-rendered with a country-specific list of regions.

User name check with AJAX

During registration, if the user's desired screen name is already taken by another user, we should let them know immediately. To do this, we use the <a4j:support> tag provided by RichFaces on the userName <h:inputText> field:

<h:inputText label="#{i18n.login_username}" id="userName" required="true" value="#{guest.registrationScreenName}" size="12" maxlength="12" redisplay="true">
  <f:validateLength minimum="4" maximum="12"/>
  <rich:beanValidator summary="#{i18n.invalid_screen_name}"/>
  <a:support event="onblur" reRender="userName" ajaxSingle="true"/>
</h:inputText>

When the onblur event occurs, RichFaces sends an AJAX request to update the guest.registrationScreenName property. If the desired screen name is already taken, a FacesMessage is created to be displayed by the <rich:message> tag associated with the username field.

<rich:message for="userName">
  <f:facet name="errorMarker"><h:graphicimage value="img/error.gif"></f:facet>
</rich:message>

Test Driven Development with Seam

One of the major attractions of POJO development and frameworks like Spring is the ease in which you can test your code as you develop it outside the container. Seam offers a similar solution, but does require the JBoss micro-container to execute tests. Other than the tests running much slower than similar Spring-based tests, I found testing with Seam to be very easy and effective. On the other hand, I wasn't able to find a simple JSF testing solution for Spring anyway, so just having the SeamTest framework in place is a benefit over Spring, even if it is slower. Seam tests are executed by TestNG. I've implemented a test that performs all the actions needed to support the requirements described above, see: src/test/example/test/UserRegistrationTest.java. Notice that you use the same EL as your JSF Facelets do!

Lastly, I want to mention one thing you should to do to fix an issue with incremental hot deploy. The seam explode command is supposed to hot-deploy your Facelets and Java classes from the hot directory. However, it also updates the JCA ConnectionFactory in the JBoss deploy directory. This can cause issues (see: https://jira.jboss.org/jira/browse/JBSEAM-3844?focusedCommentId=12442804). My solution was to add a new target to my project's build.xml file hotdeploy, which is the same as the explode target except datasource is no longer a dependency:

<target name="hotdeploy" depends="stage" description="Deploy the exploded archive but not the datasource">
  <fail unless="jboss.home">
    jboss.home not set
  </fail>
  <mkdir dir="${war.deploy.dir}"/>
  <copy todir="${war.deploy.dir}">
    <fileset dir="${war.dir}"/>
  </copy>
</target>

You also have to update the build.xml file in the seam-gen directory of your Seam distribution:

<target name="hotdeploy" depends="validate-project" description="Deploy the project as an exploded directory">
  <echo message="Deploying project '${project.name}' to JBoss AS as an exploded directory"/>
  <ant antfile="${project.home}/build.xml" target="hotdeploy" inheritall="false"/>
</target>

IMPORTANT: I excluded the lib directory from the download zip to reduce the file size. You simply need to copy the lib directory from your seam installation to my project folder to get the build working.

User authentication / registration with JSF / RichFaces, Spring, and Hibernate

2009-04-16T15:13:00.001-07:00

In this article, I show you how to implement a Web 2.0-style user registration solution using JSF/RichFaces, Spring, and Hibernate. This article evolved from a real Web 2.0 site I'm building; I hope to document more lessons learned as I have time. You can learn more about me on my LinkedIn profile. As for what's in it for you, I've provided the source code and a working solution using:

JSF 1.2 (Mojarra) with RichFaces 3.3.1 (using JSP not Facelets)
Hibernate Core 3.3.1, Hibernate Annotations 3.4.0, Hibernate Validator 3.1.0
Spring Framework 2.5.6
Spring Security 2.0.4

In future articles, I also plan to incorporate Hibernate Search and implementations of collective intelligence techniques such as tag clouds, clustering, collaborative filtering, and recommendations.

While writing this article, I assumed the reader would have a basic understanding of the technologies listed above and focused more on how to integrate them to solve the requirements discussed in the next section.

Basic Requirements

Here are the high-level requirements this solution satisfies. Notice that most these requirements apply to many Web 2.0 sites seen on the Web today.

3. Remember Me
Registered users can request the site to remember their credentials for automatic login for future visits.

Layered Architecture

To satisfy these requirements, I've implemented a layered architecture depicted in the diagram below:

For simple authentication, Spring Security may seem like overkill. However, as the application evolves, I'm also going to need authorization. In addition, Spring Security supports OpenID, which is also of interest for this site.

Web Application Organization

Here is the link to the source code and WAR file. A simple Ant build script is provided so you can build the source. Before digging too deeply into the details of the configuration, take moment to familiarize yourself with the organization of the Web application:

Persistence Layer

Domain Objects

There are three primary domain objects: User, Preferences, and Role, which should be self-explanatory given the requirements outlined above. The domain objects are POJOs with the exception of also using Hibernate annotations in a few key places. For example, in the following code snippet, we annotate the User class as a Hibernate Entity and map it to the ex_user table in the database:

@Entity
@Table(name="ex_user")
public class User implements Serializable ...

Notice that we are using surrogate keys for our domain objects that are auto-generated by Hibernate using the best approach for the specific database-type:

@Id
@GeneratedValue(strategy = GenerationType.AUTO)
public Long getUserId() {
    return userId;
}

We also define the userName property to be the natural ID for the User entity. While rare, you can allow the userName to change, such as if a user gets married and wants to change her name.

@NaturalId(mutable=true)
public String getUserName() {
    return userName;
}

I'm also using the Hibernate Validator annotations to restrict the values for some of the members, such as the userName:

@NotEmpty
@Length(min=4,max=16)
@Pattern(regex="^[a-zA-Z\\d_]{4,12}$", message="Invalid screen name.")
private String userName;

Unfortunately, the default localized message for the @Pattern annotation isn't very user-friendly so I've had to resort to a hard-coded English message until I figure out a cleaner solution.

Domain Object Associations

@OneToOne(cascade=CascadeType.ALL, fetch=FetchType.EAGER)
@PrimaryKeyJoinColumn
public Preferences getPreferences() {
    return preferences;
}

There is a bidirectional many-to-many relationship between the User and Role objects using a join table ex_user_role.

@ManyToMany(targetEntity=example.user.Role.class, cascade={CascadeType.PERSIST, CascadeType.MERGE})
@JoinTable(name="ex_user_role",joinColumns=@JoinColumn(name="userId"), inverseJoinColumns=@JoinColumn(name="roleId"))
protected Set<Role> getUserRoles() {
    if (userRoles == null) userRoles = new HashSet<Role>();
    return userRoles;
}

On the Role object, we have:

@ManyToMany(cascade = {CascadeType.PERSIST, CascadeType.MERGE}, mappedBy="userRoles", targetEntity=User.class)
public Set<User> getRoleUsers() {
    if (roleUsers == null) roleUsers = new HashSet<User>();
    return roleUsers;
}

This solution comes directly from the Hibernate documentation, section 7.5.3.

Data Access Object (DAO)

The persistence layer adopts the Data Access Object (DAO) design pattern to hide the details of the underlying persistence mechanism from the business logic of the application. For this solution, there are two DAO interfaces defined: UserDAO and RoleDAO.

The HibernateUserDao implements the UserDAO interface and extends Spring's HibernateDaoSupport base class. HibernateDaoSupport uses the Template method pattern to greatly simplify the process of using Hibernate correctly. Additionally, all checked Hibernate exceptions are translated into unchecked exceptions that extend from org.springframework.dao.DataAccessException. This allows developers to handle most persistence exceptions, which are non-recoverable, only in the appropriate layers, without having annoying boilerplate catch-and-throw blocks and exception declarations in the DAOs. For example, an attempt to insert a new record that violates a primary key constraint can only be resolved by changing the values the end-user is providing, such as a User ID during registration. With the Spring DataAccessException strategy, the DAO will throw a DataIntegrityViolationException and allow the presentation layer to catch it and prompt the end-user to change their input accordingly. Moreover, the Spring DataAccessException strategy insulates our business-logic layer from Hibernate specific classes. This is helpful if the underlying persistence mechanism needs to change for your application.

Hibernate

Each DAO implementation bean is injected with an instance of AnnotationSessionFactoryBean. This Spring FactoryBean provides a shared Hibernate SessionFactory and supports JDK 1.5+ annotation metadata for mappings, which alleviates the need to have hbm.xml files for our persistent objects.

<bean id="hbSessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
  <property name="dataSource" ref="jdbcDs">
  <property name="annotatedClasses">
    <list>
      <value>example.user.User</value>
      <value>example.user.Role</value>
      <value>example.user.Preferences</value>
    </list>
  </property>
  <property name="hibernateProperties">
    <value>hibernate.dialect=example.user.dao.hibernate.HSQLDialect_HHH_1598</value>
  </property>
  <property name="schemaUpdate" value="true">
</bean>

The hbSessionFactory bean is configured to auto-generate the database schema and is injected with our Spring DataSource. The custom hibernate.dialect property is provided to fix an issue with HSQLDB (see: http://forum.hibernate.org/viewtopic.php?p=2324183).

Under the covers, we use an Apache Commons DBCP connection pool that is exposed as a javax.sql.DataSource in the JNDI tree in Tomcat. A javax.sql.DataSource is defined by the JDBC specification to hide connection pooling and transaction management issues from application code. The web-application-config.xml file defines a reference to the javax.sql.DataSource using:

<bean class="org.springframework.jndi.JndiObjectFactoryBean" id="jdbcDs">
  <property name="jndiName" value="java:comp/env/jdbc/exampleDs">
</bean>

The database specific properties for the pool are configured in the META-INF/context.xml file:

<Resource name="jdbc/exampleDs"
          auth="Container"
          type="javax.sql.DataSource"
          username="sa"
          password=""
          driverClassName="org.hsqldb.jdbcDriver"
          url="jdbc:hsqldb:file:example"
          initialSize="0"
          maxActive="10"
          maxIdle="10"
          minIdle="0"
          maxWait="30000"
          poolPreparedStatements="true"
          />

Be sure to put the hsqldb.jar file into the tomcat/lib directory if you use HSQL.

Service Layer

One could allow the presentation layer beans to interact directly with the DAOs. However, this approach leads to your business logic being infected with presentation logic, which makes it difficult to support other interfaces, such as Web services. The Service Layer sits between your presentation layer and persistence layer. The Service Layer is where you implement the business-logic of your application.

For this solution, we have only one Spring bean in the Service Layer: UserService. The UserService provides a high-level interface to the operations provided by the UserDAO and RoleDAO. UserService is an application of the Session Facade design pattern in that it aggregates fine-grained data access operations into a single high-level transaction-aware service. Moreover, the UserService keeps business-logic out of the DAO implementations, which are only concerned with efficient object storage and retrieval.

Transactions

As mentioned above, the UserService is transaction-aware. This is accomplished using a Spring TransactionProxyFactoryBean which works with the HibernateTransactionManager. However, since we're using Spring 2.5.x and Java 5, we can use a more compact, annotation driven approach for making our UserServiceImpl transactional. First, you need to include the following element in your Spring configuration:

<tx:annotation-driven/>

Second, you need to mark your service's concrete implementation class with the @Transactional annotation:

@Transactional
public class UserServiceImpl implements UserService ...

For more information about declarative transactions with Spring, see: Declarative transaction management.

Mail Service

As described in the requirements, we'll need to send emails from time to time. For this, we'll be using Spring's JavaMailSender. The environment specific settings for your SMTP server are pulled from the example-env.properties file.

Presentation Layer

I chose JSF with RichFaces for this project based on the following criteria:

Modern, professional looking skin out-of-the-box (I don't want to fool with CSS).
Extensive set of easy-to-use, yet flexible UI components, such as trees, data grids, tabbed panes, etc.
Easy to create reusable custom components.
Large community of users and developers building real apps.
Performance: loads and runs fast on all modern browsers.
Minimizes hand-coded JavaScript.

RichFaces ModalPanel

I decided to use the RichFaces ModalPanel component <rich:modalPanel> for my dialogs because it helps the user keep their current orientation with the site. Instead of navigating to a different page, I simply open a dialog over the current view. Consequently, I didn't have any need for Spring Web Flow or JSF navigation rules in this example.

Spring-managed JSF Controller Beans

For this solution, there are two Spring-managed JSF controller beans of interest: example.jsf.GuestSupport and example.jsf.UserSession. Here is the definition from the example-jsf.xml file:

<bean id="guest" class="example.jsf.GuestSupport" scope="request">
<property name="userService" ref="userService"/>
<property name="facesSupport" ref="facesSupport"/>
</bean>

The GuestSupport bean (request-scoped) is the controller for actions performed by guests, such as login and register. Spring creates a new instance per request (if needed).

<bean id="userSession" class="example.jsf.UserSession" scope="session">
<property name="userService" ref="userService"/>
<property name="facesSupport" ref="facesSupport"/>
</bean>

The UserSession bean (session-scoped) is the controller for actions performed by authenticated users, such as managing preferences. There will be one instance of UserSession per HttpSession; Spring makes sure a new UserSession bean is created whenever a new HttpSession is created, which happens before the user is logged in. Thus, there are two possible states for the UserSession bean:

User has not been authenticated ( remoteUser == null )
User has logged in ( remoteUser != null )

Thus, you need to make sure not to use this bean in your JSF EL unless the user is logged in.

Registration

When a user clicks on the Register link on the home page, a RichFaces <rich:componentcontrol> is used to open the RichFaces ModalPanel with id registerPanel.

<rich:componentControl for="registerPanel" attachTo="registerLink" operation="show" event="onclick"/>

The form is bound to the doRegister method on our GuestSupport bean using a standard JSF commandButton. Under the covers, JSF uses the SpringBeanELResolver to get an instance of the GuestSupport bean from Spring. This is configured in faces-config.xml:

<application>
  <el-resolver>org.springframework.web.jsf.el.SpringBeanFacesELResolver</el-resolver>
</application>

Registration requires the user to solve a CAPTCHA. The easiest way to add this to your site is to use reCAPTCHA. The best way to get started with reCAPTCHA is to read their developer's guide at http://recaptcha.net/whyrecaptcha.html. I had to use a custom JSF tag to embed reCAPTCHA JavaScript into the registration form because the standard JSF script tag was being written to the page header instead of inline. This is due to the following setting in the web.xml:

<context-param>
  <param-name>com.sun.faces.externalizeJavaScript</param-name>
  <param-value>true</param-value>
</context-param>

This allows browsers to cache our JSF JavaScript instead of having to write it each time the page is accessed, which presumably will help our pages load faster. However, we need the reCAPTCHA script to be inline and not in the header. Fortunately, JSF makes it very easy to create custom tags.

<example:script src="#{uiSupport.recaptchaScriptUrl}">

Strong Password Validation

Notice that the registration form binds the password to the guest.passwordSupport.password.

<h:inputSecret label="#{i18n.login_password}" id="password" required="true" value="#{guest.passwordSupport.password}" size="12" maxlength="12">
  <rich:beanvalidator/>
</h:inputSecret>

This is because the User password is expected to be a one-way MD5 hash instead of plain text. The example.user.PasswordSupport utility bean validates the password using Hibernate Validator:

@NotEmpty
@Length(min=8,max=12)
@Pattern(regex="(?=^.{8,}$)((?=.*\\d)|(?=.*\\W+))(?![.\\n])(?=.*[A-Z])(?=.*[a-z]).*$", message="Password is not secure enough!")
private String password = null;

Upon successful registration, the guest.newUserName and guest.newUserEmail properties are updated on the guest bean. This activates the regPending modal panel to notify the user that their registration must be activated by clicking on an activation link sent to their email. However, if registration fails, then the registerPanel is displayed with the appropriate error message(s).

<a:commandButton id="registerButton" value="#{i18n.register}"
    action="#{guest.doRegister}" styleClass="rich-fileupload-button" reRender="regPending"
    oncomplete="if (document.getElementById('jsfMsgMaxSev').value != '2') { #{rich:component('registerPanel')}.hide(); #{rich:component('regPending')}.show(); }"/>

The oncomplete attribute will close the registerPanel and open the regPending panel if no errors are present in the AJAX response. This relies on the use of an A4J <a:outputPanel>, see index.jsp.

Account Activation

The activation link includes the act parameter containing a hex-encoded MD5 hash code linked to the pending registration. The guest.getUser() method checks for the act parameter and if present tries to activate the user. If successful, the login panel is displayed with the new user name pre-popluated.

For authentication, we delegate to Spring Security using an ExternalContext dispatch to /j_spring_security_check. This handler expects two parameters provided by our Login form: j_username and j_password. Notice that the login form is configured with prependId="false" so that JSF does not add loginForm: to the parameter names. The request to /j_spring_security_check is handled by the springSecurityFilterChain configured in web.xml:

<filter>
  <filter-name>springSecurityFilterChain</filter-name>
  <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>

Spring Security, configured in example-security.xml, uses a userDetailsService based authentication provider. In this case, we configure two simple SQL queries based on the Hibernate schema for the org.springframework.security.userdetails.jdbc.JdbcDaoImpl.

Authenticate by Username and Password:

SELECT userName, password, active from ex_user where userName=?

Get Roles for User:

SELECT ij.userId, r.name FROM ex_role r INNER JOIN (SELECT u.userId, ur.roleId FROM ex_user_role ur, ex_user u WHERE ur.userId=u.userId AND u.userName=?) ij ON r.roleId=ij.roleId

If authentication fails, Spring Security puts the exception into the session keyed by: AbstractProcessingFilter.SPRING_SECURITY_LAST_EXCEPTION_KEY. In the JSF layer, we need to check for this exception and if present display a generic JSF error message to the user. It's a good idea to show a generic message because the actual exception may contain information a hacker can use to break into the site. Unfortunately, I had to resort to invoking a static function GuestSupport.setLoginPanelVisibility from a JSP Scriptlet in auth.jspf to check for the Spring Security exception.

Remember Me

Recall that one of our requirements was to support a Remember Me feature so that returning users are automatically logged into the site. With Spring Security, this is trivial; the JSF login form simply needs to pass the _spring_security_remember_me parameter. Under the covers, Spring Security is configured to use the simple hash-based token approach. However, you can also use a more secure solution described here.

JavaScript and AJAX Tricks

In this section, I demonstrate a few of the JavaScript and AJAX tricks used by the solution.

Country / Region

<a4j:support event="onchange" rerender="region" ajaxsingle="true">

This fires an AJAX request when the country is changed and upon success, the region component is re-rendered with a country-specific list of regions.

User name check with AJAX

<a4j:support event="onblur" rerender="userName" ajaxsingle="true">

When the onblur event occurs, RichFaces sends an AJAX request to update the guest.registrationScreenName property. If the desired screen name is already taken, a FacesMessage is created to be displayed by the <rich:message> tag associated with the userName field.

<rich:message for="userName">
  <f:facet name="errorMarker"><h:graphicimage value="img/error.gif"></f:facet>
</rich:message>

JPA Entity	Description
Region	Encapsulates information about a geographical area that produces wine, such as Bordeaux. Implements the org.vinwiki.model.Composite interface to represent a hierarchical tree of regions and sub-regions.
Varietal	Encapsulates information about grape variety used to make wine, such as Chardonnay.
VarietalPct	Represents the percentage of a varietal in a specific wine.
Winery	Encapsulates information about a wine producer, such as Robert Mondavi. A Winery may have a latitude and longitude specified, which would allow us to show the winery on map of wineries in a region.
Wine	Encapsulates information about a specific wine, uniquely identified by name, winery, and vintage (typically the year the grapes were harvested).
User	Encapsulates information about a registered user in the application.
Preferences	Encapsulates user-supplied settings used to personalize the application.
Tag	Holds information for a keyword created by a user for rating a wine.
UserTag	Associates a tag with a user's rating.
Rating	Represents a specific user's rating (score and comments) of a specific wine.