VinWiki Part 2: Full-text Search with Hibernate Search and Lucene

Introduction

This is the second post in a four part series about a wine recommendation Web application built using open source Java technology. The purpose of this series is to document key design and implementation decisions that can be applied to other Web applications. Please read the first post for an overview of the project. You can download the project (with source) from here.

In this posting, I leverage Hibernate Search and Lucene to provide full-text search for wines of interest. This is post is merely an attempt to complement the already existing documentation on Lucene and Hibernate Search. If you are familiar with Hibernate and are new to Lucene, then I recommend starting out with the online manual for Hibernate Search 3.1.1 GA. In addition, I highly recommend reading Hibernate Search in Action and Lucene in Action (Second Edition); both are extremely well-written books by experts in each field.

Basic Requirements

Full-text search allows users to find wines using simple keywords, such as "zinfandel" or "dry creek valley". Users have come to expect certain features from a full-text search engine. The following table summarizes the key features our wine search engine will support:

Feature	Description
Paginated search results (see first post for solution)	Return the first 5-10 results, ordered by relevance on the first page of the search results. Allow users to retrieve more results by advancing the page navigator. Allows users to set a bounded preference for how many items to show on earch page.
Query term highlighting (view solution)	Highlight query terms in search results to allow users to quickly scan the results for the most relevant items.
Spell correction (view solution)	Ability to detect and correct for misspelled terms in the query.
Type-ahead suggestions (view solution)	Show a drop-down selection box containing suggested search terms after the user has typed at least 3 characters.
Advanced search form (view solution)	Allow users to fine-tune the search with an advanced search form.

Setup

From the previous post, you'll recall that I'm using Hibernate to manage my persistent objects. As such, I've chosen to leverage Hibernate Search (referred to as HS hereafter) to integrate my Hibernate-based object model with Lucene. For now, I'm using Hibernate Search v. 3.1.1 GA with Hibernate Core 3.3.1 GA. On the Lucene side, I'm using version 2.9.2 and a few classes from Solr 1.4. In the last posting in this series, we'll upgrade to the latest Hibernate Search and Core, which rely on JPA 2 and thus will require a move up to JBoss 6.x. As far as I know, the latest HS and Core classes have trouble on JBoss 4.2 and 5.1 because of their dependency on JPA 2 (let me know if you get it working).

Why not use the database to search?

Some may be wondering why I'm using Lucene instead of doing full-text search in my database? While possible, I feel the database is not the best tool for full-text searching. For an excellent treatment of the mis-match between relational databases and full-text search, I recommend reading Hibernate Search in Action. The first chapter provides a strong case for using Lucene instead of your database for full-text search. Just in terms of scalability, the database is the most expensive and complex layer to scale in most applications. Offloading searches to a local Lucene running on each node in your cluster reduces the amount of work your database is performing and ensures searches return quickly. Moreover, distributing a Lucene index across a cluster of app servers is almost trivial since Hibernate Search has built-in clustering support.

Project Configuration Changes

The following JAR files need to be included in your Web application's LIB directory (/WEB-INF/lib):

hibernate-search.jar
    lucene-core-2.9.2.jar
    lucene-highlighter-2.9.2.jar
    lucene-misc-2.9.2.jar
    lucene-snowball-2.9.2.jar
    lucene-spatial-2.9.2.jar
    lucene-spellchecker-2.9.2.jar
    lucene-memory-2.9.2.jar
    solr-core-1.4.0.jar
    solr-solrj-1.4.0.jar

How does Seam know to use Hibernate Search's FullTextEntityManager?

Seam's org.jboss.seam.persistence.HibernatePersistenceProvider automatically detects if Hibernate Search is available on the classpath. If HS is available, then Seam uses the org.jboss.seam.persistence.FullTextEntityManagerProxy instead of the default EntityManagerProxy, meaning that you will have access to a FullTextEntityManager wherever you have a Seam in-jected EntityManager. You also need to add a few more properties to your persistence deployment descriptor (resources/META-INF/persistence-*.xml):

<property name="hibernate.search.default.directory_provider" value="org.hibernate.search.store.FSDirectoryProvider"/>
  <property name="hibernate.search.default.indexBase" value="lucene_index"/>
  <property name="hibernate.search.reader.strategy" value="shared"/>
  <property name="hibernate.search.worker.execution" value="sync"/>

We'll make some adjustments to these settings as the project progresses, but these will suffice for now.

Indexing

The first step in providing full-text search capabilities is to index the content you want to search. In most cases, our users will want to find Wine objects and to a lesser extent Winery objects. Let's start by indexing Wine objects.

Indexing Wine Objects

I'll tackle indexing Wine objects in a few passes, progressively adding features, so let's start with the basics. First, we tell HS to index Wine objects using the @Indexed annotation on the class. As for what to search, I tend to favor using a single field to hold all searchable text for each object because it simplifies working with other Lucene extensions, such as the More Like This and term highlighting features.

@Field(name=DEFAULT_SEARCH_FIELD, index=Index.TOKENIZED)
    @Analyzer(definition="wine_stem_en")
    public String getContent() {
        // return one string containing all searchable text for this object
    }

Notice that the content field is Index.TOKENIZED during indexing. This means that the String value returned by the getContent method will be broken into a stream of tokens using a Lucene Analyzer; each token returned by the Analyzer will be searchable. Your choice of Analyzer depends on the type of text you are processing, which in our case is English text provided by our users when creating Wine objects. Hibernate Search leverages the text analysis framework provided by Solr. You can specify your Analyzer using the @Analyzer annotation. Behind the scenes, Hibernate Search creates the Analyzer using the Factory defined by an @AnalyzerDef class annotation. In our case

@AnalyzerDef(name = "wine_stem_en",
        tokenizer = @TokenizerDef(factory = org.vinwiki.search.Lucene29StandardTokenizerFactory.class),
        filters = {
          @TokenFilterDef(factory = StandardFilterFactory.class),
          @TokenFilterDef(factory = LowerCaseFilterFactory.class),
          @TokenFilterDef(factory = StopFilterFactory.class),
          @TokenFilterDef(factory = EnglishPorterFilterFactory.class)
    })

This looks more complicated than it is ... Let's take a closer look at the @AnalyzerDef annotation:

name = "wine_stem_en" This gives our analyzer definition a name so we can refer to it in the @Analyzer annotation. This will also come in handy when we start parsing user queries using Lucene's QueryParser, which also requires an Analyzer.

tokenizer = @TokenizerDef(factory = org.vinwiki.search.Lucene29StandardTokenizerFactory.class) Specify a factory for Tokenizer instances. In this case, I'm supplying a custom factory class that creates a Lucene 2.9.x StandardTokenizer as the default factory provided by Solr creates a Lucene 2.4 StandardTokenizer (most likely for backwards compatibility).

filters = { @TokenFilterDef(factory = StandardFilterFactory.class), @TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = StopFilterFactory.class), @TokenFilterDef(factory = EnglishPorterFilterFactory.class) } A tokenizer can pass tokens through a filter-chain to perform various transformations on each token. In this case, we're passing the tokens through four filters, in the order listed above. StandardFilter: Removes dots from acronyms and 's from the end of tokens. Works only on typed tokens, i.e., those produced by StandardTokenizer or equivalent. LowerCaseFilter: Lowercases letters in tokens StopFilter: Discards common English words like "an" "the" and "of". EnglishPorterFilter: Extracts the stem for each token using Porter Stemmer implemented by the Lucene Snowball extension.

You can index other fields, such as the wine type and style using the @Field annotation.

@Field(name="type",index=Index.UN_TOKENIZED,store=Store.YES)    
    public WineType getType() {
        return this.type;
    }

Building the Index

If you worked through the first post, then you should already have a database containing 2,025 wines. Hibernate Search will automatically index new Wine objects for us when the insert transaction commits, but we need to manually index the existing wines. During startup, the org.vinwiki.search.IndexHelper component counts the number of documents in the index and re-builds the index from the objects in the database if needed. During startup, you should see log output similar to:

[IndexHelper] Observed event org.vinwiki.event.REBUILD_INDEX from Thread QuartzScheduler1_Worker-8
[IndexHelper] Re-building Wine index for 2025 objects.
[IndexHelper] Flushed index update 300 from Thread Quartz ...
[IndexHelper] Flushed index update 600 from Thread Quartz ...
[IndexHelper] Flushed index update 900 from Thread Quartz ...
[IndexHelper] Flushed index update 1200 from Thread Quartz ...
[IndexHelper] Flushed index update 1500 from Thread Quartz ...
[IndexHelper] Flushed index update 1800 from Thread Quartz ...
[IndexHelper] Took 4094 (ms) to re-build the index containing 2025 documents.

Alright, with a few lines of code and some clever annotations, we now have a full-text search index for Wine objects. Let's do some searching!

Basic Search Box

From the first post, we already have a server-side pagination framework in place for displaying Wine objects. To integrate full-text search capabilities, we simply need to implement the org.vinwiki.action.PagedDataFetcher interface in terms of Hibernate Search. Here is a simple implementation (we'll add more features later):

This is sufficient to get us searching quickly.

Next, we need to pass the user's query from the search box to our nav component. In view/WEB-INF/facelets/searchControls.xhtml, update the JSF inputText tag to bind the user's query to an Event-scoped component named searchCriteria:

<h:inputText id="basicSearchQuery" styleClass="searchField" value="#{searchCriteria.basicSearchQuery}"/>

The searchCriteria component is in-jected into nav, which passes it on to an instance of org.vinwiki.search.Search which implements the PagedDataFetcher interface. You may be thinking that having a separate component for a single input field is over-kill, but the searchCriteria component will also come in very handy once we add an advanced search form. Here is what happens in nav:

@In(required=false) private SearchCriteria searchCriteria;
    ...
    public void doSearch() {
        cleanup();
        dataFetcher = new FeatureRichSearch(log, searchCriteria);
        initFirstPageOfItems(); // fetches the first page of Items
    }

Execute Search on Enter

There is the obligatory Search button next to my search input field, but most user's will just hit enter to execute the search. RichFaces makes this trivial to support using the <rich:hotKey> tag:

<rich:hotKey key="return" selector="#basicSearchQuery" 
          handler="#{rich:element('searchBtn')}.onclick();return false;"/>

The <rich:hotKey> tag binds a JavaScript event listener for the search box to submit the form when the user hits "enter". Be sure to use the selector attribute to limit the listener to the search box and not all input text boxes on the page! Any valid jQuery selector will do ...

At this point, re-compile and re-deploy these changes (at the command-line: seam clean reexplode). A search for "zinfandel" results in:

Hopefully, you get something similar in your environment ;-) Yes, those images are terribly ugly right now ... They were pulled dynamically from Freebase; if this were a real production application, then I'd work on getting better images. Next, we'll add some more features to the search engine, such as query term highlighting, spell correction, more like this, and filters. These additional features are implemented in the org.vinwiki.search.FeatureRichSearch class.

 

Highlighting Query Terms in Results

Highlighting query terms in results is a common and very useful feature of most search engines. The contrib/highlighter module in Lucene makes this feature easy to implement. There are two aspects to consider when displaying results to the user. First, we want to display the best "fragment" from the document text for the specific query. In other words, rather than just showing the first X characters of the description, we should show the best section of the description matching the user's query. Second, highlight the query terms in the chosen fragment. However, most of the description text for our Wine objects is short enough to display the full description for each result. Based on my current UI layout, 390 is a safe maximum length for fragments as it is long enough to provide useful information about each wine, yet short enough to keep from cluttering up the screen with text. This is something you'll have to work out for your application.

Highlighting Fragments with Hibernate Search

Recall that we stuff all the searchable text information about a Wine into a single searchable field "content". While good for searching, it's probably not something you want to display to your users directly. Instead, I've chosen to highlight terms in the wine description only. Here is how to construct a Highlighter (from the Lucene org.apache.lucene.search.highlight package):

// Pass the Lucene Query to a Scorer that extracts matching spans for the query
    // and then uses these spans to score each fragment    
    QueryScorer scorer = new QueryScorer(luceneQuery, Wine.DEFAULT_SEARCH_FIELD);
    // Highlight using a CSS style    
    SimpleHTMLFormatter formatter = new SimpleHTMLFormatter("<span class='termHL'>", "</span>");
    Highlighter highlighter = new Highlighter(formatter, scorer);
    highlighter.setTextFragmenter(new SimpleSpanFragmenter(scorer, Nav.MAX_FRAGMENT_LEN));

Also notice that I get a reference to the "wine_stem_en" Analzyer using:

Analyzer analyzer = searchFactory.getAnalyzer("wine_stem_en");

While iterating over the results, we pass the Analyzer and the actual description text to the Highlighter. The observant reader will notice that I didn't "stem" the description text during indexing, but now I am stemming the description text for highlighting. You'll see why I'm taking this approach in the next section when I add spell correction.

highlightedText = highlighter.getBestFragment(analyzer, Wine.SPELL_CHECK_SEARCH_FIELD, description);

FeatureRichSearch saves the dynamically highlighted fragments in a Map because fragments are specific to each query. If the query changes, then so must the fragments for the results. The getResultDisplayText method on the nav component interfaces with the FeatureRichSearch dataFetcher to get fragments for search results.

 

Spell Correction

For spell correction, I'm using the contrib/spellchecker module in Lucene (see http://wiki.apache.org/lucene-java/SpellChecker), which is a good place to start, but you should realize that handling spelling mistakes in search is a complex problem so this is by no means the "best" solution.

Spell Correction Dictionary

To begin, you need a dictionary of correctly spelled terms from which to base spell corrections on. The spellchecker module builds a supplementary index from the terms in your main index using fields based on n-grams of each term. Internally, SpellChecker creates several n-gram fields for each word depending on word length. Here is a screenshot of the word "zinfandel" in the SpellChecker index courtesy of Luke:

Recall that we're stemming terms in our default search field "content". If you build the spell checker index from this field, the dictionary will only contain stemmed terms. This has un-desired side-effect that the corrected term will look misspelled to the user. For example, if you search for "strawbarry", the spell checker will probably recommend "strawberri", which is good, but we really want to show the user "Did you mean 'strawberry'?". Thus, we need to base our spell checker index off non-stemmed text. When we ask the spell checker to suggest a term for "strawbarry", then it will return "strawberry". When we query the search index, we need to query for "strawberri", which is why I pass the "wine_stem_en" Analyzer to the QueryParser after applying the spell correction process.

Hibernate Search Automatic Indexing and Spell Correction

Lucene's SpellChecker builds a supplementary index from the terms in your main index. If your main index changes, e.g. after adding a new entity, then you need to update the spell correction index. From discussing this within the HS community, there's nothing built into HS to help you determine when to update the spell checker index, especially if you're using hibernate.search.worker.execution=async. In other words, you don't know when Hibernate Search is finished updating the Lucene index. You have a couple of options to consider here: 1) update the spell index incrementally as new content is added (or updated), or 2) update the index periodically in a background job. This depends on your requirements and how much you trust the source of the changes to the main index. For now, I'm using Seam Events to incrementally update the spell index after a new Wine is added or an existing Wine is updated (see src/hot/org/vinwiki/search/IndexHelper.java for details).

Edit Distance as Measure of Similarity Between Terms

Lucene's SpellChecker uses the notion of "edit distance" to measure the similarity between terms. When you call spellChecker.suggestSimilar(wordToRespell, ...), the checker consults an instance of org.apache.lucene.search.spell.StringDistance to score hits from the spell checker index. Here's how a SpellChecker is constructed:

Directory dir = FSDirectory.open(new java.io.File("lucene_index/spellcheck"));
    SpellChecker spell = new SpellChecker(dir);
    spell.setStringDistance(new LevensteinDistance());
    spell.setAccuracy(0.8f);
    String[] suggestions = spell.suggestSimilar(wordToRespell, 10,
                             indexReader, Wine.SPELL_CHECK_SEARCH_FIELD, true);

The LevensteinDistance class implements StringDistance by calculating the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character.

Spell Correction Heuristics

The spellchecker module does a fine job of suggesting possible terms, but we still need to decide how to handle these suggestions depending on the structure of the user's query. To keep things simple, our first heuristic applies to single term queries where the mis-spelled term does not exist in our spell checker index. In this case, we simply re-issue the search using the best suggestion from the spell checker. Here is a screen shot of how we present the results to the user:

However, we can't be sure that a term is mis-spelled if it exists in the spell checker index. Thus, if a single mis-spelled term exists in the spell correction index, then you need to decide if you are going to just "OR" the term mis-spelled and suggested terms together or let the user decide (it seems like Google doesn't always do one or the other, so their implementation is a bit more advanced than the one I'll provide here).

In the screenshot above, the user queried for "velvety tanins"; both terms exist in the spell correction index, but "tanins" is mis-spelled. During query processing, the SpellChecker suggested "tannins" for the mis-spelled word "tanins", which is correct. Thus, we search for "velvety tanins", but also suggest a spell-corrected query "velvety tannins", allowing the user to click on the suggested correction to see better results (hopefully). Please refer to the checkSpelling method in the src/hot/org/vinwiki/search/FeatureRichSearch.java for details on how these simple heuristics are implemented.

 

Type-ahead Search Suggestions

Most users appreciate when an application spares them un-necessary effort, such as typing common phrases into the search box. Thus, we'll use RichFaces <rich:suggestionbox> tag to provide a drop-down suggestion list of known search terms after the user types 3 or more characters. While trivial, this feature can help users be more productive with your search interface and helps minimize spelling mistakes.

In /WEB-INF/facelets/searchControls.xhtml, we attach the <rich:suggestionbox> to the search input text field using:

<rich:suggestionbox id="suggestionBox" for="basicSearchQuery" ignoreDupResponses="true"
                    immediate="true" limitToList="true" height="100" width="250" minChars="3"
                    usingSuggestObjects="true" suggestionAction="#{nav.suggestSearchTerms}"
                    selfRendered="true" first="0" var="suggestion">
  <h:column><h:outputText value="#{suggestion}"/></h:column>
</rich:suggestionbox>

In the first posting, I discussed queue and traffic flood protection using RichFace's Global Default Queue. You should definitely active an AJAX request queue if you are using type-ahead suggestions. I've also added an animated GIF and loading message using the <a4j:status> tag.

On the Java side, I've resorted to a simple LIKE query to the database, but you can also query a field analyzed with Lucene's org.apache.lucene.analysis.ngram.EdgeNGramTokenFilter in your full-text index.

 

Advanced Search

Sometimes a single search field isn't enough to pin-point the information you're looking for; advanced search addresses this, albeit uncommon, need for users to formalize complex queries. The advanced search form is application specific but most allow the user construct a query composed of AND, OR, exact phrase, and NOT clauses, as shown in the following screen shot:

As mentioned above, the org.vinwiki.search.SearchCriteria class manages the advanced search form data. I'll refer you to the source code for further details about advanced search. Notice that I'm using a RangeQuery to implement the Date field on the search form.

Testing Search

Be careful when building tests involving Lucene because lib/test/thirdparty-all.jar contains an older version of Lucene. To remedy this, I added the following list of JARs to the top of the test.path path in build.xml:

<fileset dir="${lib.dir}">
    <include name="lucene-core-2.9.2.jar"/>
    <include name="lucene-highlighter-2.9.2.jar"/>
    <include name="lucene-misc-2.9.2.jar"/>
    <include name="lucene-snowball-2.9.2.jar"/>
    <include name="lucene-spatial-2.9.2.jar"/>
    <include name="lucene-spellchecker-2.9.2.jar"/>
    <include name="lucene-memory-2.9.2.jar"/>
    <include name="solr-core-1.4.0.jar"/>
    <include name="solr-solrj-1.4.0.jar"/>
  </fileset>

A new test case was added for testing search, see src/test/org/vinwiki/test/SearchTest.java. While fairly trivial, this class helped me root out a few issues (like updating the spell correction index after an update) while developing the code for this post. So test-driven development shows its worth once again!

Future Enhancements to the Search Engine

There are still a few search-related features I think this application needs, including tagging, phrase handling, and synonyms. Specifically, user's should be able to add tags like "jammy" or "flabby" when rating wines. The application should be able to render a tag cloud from these user-supplied tags as another form of navigation. User tags should also be fed into the search index (using caution of course). Phrase detection complements user-supplied tagging by recognizing multi-term tags during text analysis. How you handle stop words also affects exact phrase matching. The contrib/shingles module helps speed up phrase searches involving common terms, so I'd definitely like to investigate its applicability for this application. Lastly, synonyms help supply the search index (and/or search queries) with additional terms that mean the same thing as other words in your documents. If time allows, I'll try to add a fifth post dealing with tags, phrases, and synonyms. I'd also like to hear from the community on other features that might be helpful.

TODO: Evaluating Search Quality with contrib/benchmark

The contrib/benchmark module is one extension all Lucene developers should be familiar with; benchmark allows you to run repeatable performance tests against your index. I'll refer you to the JavaDoc for using benchmark for performance testing. In the future, I'd like to use benchmark's quality package to measure relevance of search results. The classes in the quality package allow us to measure precision and recall for our search engine. Precision is a measure of "exactness" that tells us the proportion of the top N results that are relevant to the search. Recall is a measure of "completeness" that tells us the proportion of all relevant documents included in the top results. In other words, if there are 100 relevant documents for a query and the results return 80, then recall is 0.8. Sometimes, the two metrics conflict with each other in that as you return more results (to increase recall), you can introduce irrelevant documents, which decreases precision. The quality package will give us a way to benchmark precision and recall and then tune Lucene to improve these as much as possible. If I get time, then I'll post my results (and source).

A last words about scalability

While Lucene is very fast, you can run into search performance issues if you are constantly updating the index, which requires Hibernate Search to close and re-open its shared IndexReader. Hibernate Search has built-in support for a Master / Slave architecture where updates occur on a single master server and searches are executed on replicated, read-only slave indexes in your cluster. I use this configuration in my day job and it scales well. However, you'll need a JMS queue and Message-driven EJB to allow slave nodes to send updates to the master node, so you'll have to deploy your application as an EAR instead of a WAR (to deploy the MDB). Please refer to the online documentation provided with Hibernate Search for a good discussion on how to setup a master / slave cluster.

What's Next?

In the next post, I'll add authentication and registration using Facebook Connect.

VinWiki Part 1: Building an intelligent Web app using Seam, Hibernate, RichFaces, Lucene and Mahout

Introduction

This is the first post in a four part series about a wine rating and recommendation Web application, named VinWiki, built using open source technology. The purpose of this series is to document key design and implementation decisions, which may be of interest to anyone wanting to build an intelligent Web application using Java technologies. The end result will not be a 100% functioning Web application, but will have enough functionality to prove the concepts. Specifically, here is a basic roadmap of the concepts covered in each post:

1. Introduction (May 24, 2010): Covers project setup, primary domain objects, and basic UI constructs such as server-side pagination, dynamic menus, and bookmarkable URLs with JSF / RichFaces / Facelets.
2. Full-text Search (June 14, 2010): Implements full-text search using Hibernate Search with Lucene 2.9.2.
3. Integrating with the Web (July 8, 2010): Authentication and registration using Facebook Connect and sharing/liking bookmarkable URLs on Facebook.
4. Recommendations (Sept 23, 2010): Provide wine recommendations using Apache Mahout.

The idea for VinWiki was born out of two interests in my life: wine and collective intelligence. I'm a wine connoisseur and love red wines from Piedmont (Italy) and the Dry Creek and Alexander Valley's in Sonoma. I don't, however, drink expensive wine. Rather, I'm always on the lookout for that $10-15/bottle wine that just tastes good (who isn't right?) My strategy is to know a lot about a little. For example, I know the major varietals in Piedmont, their characteristics, which years were good and which were not, etc. I know these things because I want to be able to drink great inexpensive wines and help my friends do the same. Wouldn't it be nice if you had access to all this knowledge in my head the next time you go out for Italian food in North Beach?

One of my other interests is figuring out how to harvest intelligence from the interactions and contributions of users on a Web site and then apply that intelligence to improve the personal experience with the application as well as the application as a whole. The science behind this is called Collective Intelligence. To keep things simple, I consider a Web application to be "intelligent" if it has the following key ingredients:

1. automatically improves its capabilities as more users contribute to it,
2. scalable (machine learning algorithms do better with large amounts of data),
3. mashable (simple Web services that expose data and services to other applications), and
4. doesn't pretend to be more than it is!

There's a wealth of knowledge available about how to implement the first ingredient using machine learning. And, as we'll see in the 4th post in this series, there is a wonderful open source project named Mahout that provides scalable implementations of many popular machine learning algorithms, such as clustering, classification, and recommendations. However, please keep in mind that it takes rigorous testing and experimentation to ensure these algorithms produce good results for your data. Don't assume that because an algorithm sounds sophisticated that it will produce good results in your application. Hence, the last key ingredient! In this application, intelligence comes in the form of making recommendations for wines from previous ratings using collaborative filtering. Presumably, as more users contribute wines and ratings, the application will improve for all users.

Toolset

The solution is built using the following open-source technologies:

• JBoss AS 4.2.3
• Seam 2.2.0
• Hibernate (Core, EntityManager, Annotations, Validator & Search)
• JSF 1.2 / RichFaces 3.3.2 SR1 / Facelets
• Lucene 2.9.2
• Mahout 0.3 / Hadoop 0.20.2

Recommended Reading List

Here are a few excellent books that introduce you to the subject of intelligent Web applications:

• Algorithms of the Intelligent Web
• Collective Intelligence in Action
• Mahout in Action
• Programming Collective Intelligence (code samples in Python, but still a great read and Python is a fun scripting language to know anyway ;-)

If you are interested in full-text search, the following two books are invaluable assets:

• Hibernate Search in Action
• Lucene in Action (Second Edition)

Basic Requirements

There are three primary activities you can do with this application:

1. Rate wines you've already tried
2. Find and read ratings for wines you may like to try
3. Receive recommendations of new wines you should to try based on your previous ratings

These activities equate to the following use cases:

• Register New User Account   view solution
• Basic Navigation: Browse Wine by Region, Date, Tag, or Popularity   view solution
• View detailed information for a specific wine   view solution
• Add Rating for an Existing Wine   view solution
• Add New Wine   view solution
• Edit Wine   view solution
• Search for Wine   view solution
• Share wines and ratings with Facebook friends   view solution
• View Recommendations   view solution

Here is a screen shot of the home page (note: best I could do with the image quality since blogger limits the width to 800).

Domain Objects

The following UML Class Diagram depicts the primary domain objects in our model.

JPA Entity	Description
Region	Encapsulates information about a geographical area that produces wine, such as Bordeaux. Implements the org.vinwiki.model.Composite interface to represent a hierarchical tree of regions and sub-regions.
Varietal	Encapsulates information about grape variety used to make wine, such as Chardonnay.
VarietalPct	Represents the percentage of a varietal in a specific wine.
Winery	Encapsulates information about a wine producer, such as Robert Mondavi. A Winery may have a latitude and longitude specified, which would allow us to show the winery on map of wineries in a region.
Wine	Encapsulates information about a specific wine, uniquely identified by name, winery, and vintage (typically the year the grapes were harvested).
User	Encapsulates information about a registered user in the application.
Preferences	Encapsulates user-supplied settings used to personalize the application.
Tag	Holds information for a keyword created by a user for rating a wine.
UserTag	Associates a tag with a user's rating.
Rating	Represents a specific user's rating (score and comments) of a specific wine.

Relational Database Model

Here is the Hibernate model translated into a MySQL database model:

Seed Data

Since an application like this isn't very interesting without some real data, I extracted some wine-related entities from Freebase using their RESTful Web Services API, along with some manual tweaking to better fit the desired data model. Specifically, the seed data contains:

• 104 wine regions / sub-regions
• 487 grape varietals
• 772 wineries
• 2,025 wines

So this should get us started with some real data, but as we'll see below, users will also be able to add new wines as needed.

Persistence Annotations

Most of the JPA annotations in the domain model are straight-forward, so I'll refer you to the source code for further study. However, I'm using a few annotations that warrant further discussion, including:

Hierarchical Regions (composite pattern)

A wine Region can have sub-regions, such as France has Bordeaux and Alsace. JPA makes modeling composites really easy using @ManyToOne with @JoinColumn:

@ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "parent_region", nullable = true)
    public Region getParentRegion() {
        return this.parentRegion;
    }

Yep! It really is that easy ...

@javax.persistence.MappedSuperclass

Entities typically share a common set of fields, such as ID, name, description so it is common to encapsulate these common fields into a base class. The javax.persistence.MappedSuperclass annotation designates a class whose mapping information is applied to the entities that inherit from it. As you can see from the class diagram, the org.vinwiki.model.AbstractItemBase class serves as the @MappedSuperclass in our model.

@org.hibernate.annotations.Cache

If you think about it, most of the wine-related entities in this model are primarily read-only in nature. The main user activity in this application is to add ratings to existing wine objects. Thus, the Wine, Winery, Varietal, and Region entities are good candidates for caching in what is called the second-level cache. We use the @org.hibernate.annotations.Cache annotation to define the cache concurrency strategy for each entity we want to cache. I'm using org.hibernate.annotations.CacheConcurrencyStrategy.NONSTRICT_READ_WRITE because the application may need to update these entities occasionally, but in most cases they are used as read-only objects. For more information about cache concurrency strategy, please refer to The Second Level Cache section in the Hibernate Core documentation.

Of course, we also need to enable the second-level cache in persistence.xml. I found it easiest to use Ehcache initially but you should research the other providers, e.g. JBoss Cache, to determine the most appropriate solution for your application. I'll revisit this decision when I tackle clustering this application in the cloud. For now, here are the salient properties specified in persistence.xml:

<property name="hibernate.cache.provider_class" value="net.sf.ehcache.hibernate.EhCacheProvider"/>
  <property name="hibernate.cache.use_query_cache" value="true"/>
  <property name="hibernate.cache.use_second_level_cache" value="true"/>
  <property name="hibernate.cache.region_prefix" value=""/>
  <property name="hibernate.generate_statistics" value="true"/>
  <property name="hibernate.session_factory_name" value="SessionFactories/vinwikiSF"/>

Notice that I'm enabling Hibernate Statistics using the hibernate.generate_statistics property. During startup, I deploy the Hibernate StatisticsService MBean using:

hibernateMBeanName = new ObjectName("Hibernate:type=statistics,application=vinwiki");
    StatisticsService mBean = new StatisticsService();
    mBean.setSessionFactoryJNDIName("SessionFactories/vinwikiSF");
    ManagementFactory.getPlatformMBeanServer().registerMBean(mBean, hibernateMBeanName);

I also use Ehcache as the Seam cache provider, in resources/WEB-INF/components.xml:

<cache:eh-cache-provider/>

This will come in handy when we start displaying tag clouds and other expensive data structures to our users.

Be sure to include ehcache.jar in your deployed-jars.list file!

Solutions

In this section, I walk you through the solutions to each use case described above.

Register New User Account

For this requirement, I leveraged the solution from my previous blog post. We'll see in post #3 in this series how to allow Facebook users to automatically register and authenticate using their Facebook account. However, I did make a few minor improvements to the handling of user preferences so be sure to review the code in the org.vinwiki.user package after reading the aforementioned blog post.

Basic Navigation

There are a number of ways a user can browse for wines in the application, including by region, most recent, best rated, as well as search. Moreover, a user can switch between these mechanisms at any time. The nav component (org.vinwiki.action.Nav) deployed in session scope supports user navigation.

The nav component can be in one of two states during its lifecycle: A) guest mode when #{identity.loggedIn} is false or B) authenticated user mode when #{identity.loggedIn} is true. This is because Seam does not create a new session component after a user is authenticated.

When a new session is created, we need to generate a default view of the wine data; a list of the Most Recent Wines added to the application seems like a good choice. I use Seam's @Factory annotation to provide the default view (the Seam documentation calls this pull-style MVC):

@Factory("mainMenu")
    public void initMainMenu() {
        if (mainMenu == null) {
            mainMenu = buildMainMenu();
        }
        // Ensure the current PagedDataFetcher is in-sync with the current request
        syncDataFetcherWithRequest();
    }

To handle the switch from state A to B, nav observes the Identity.EVENT_POST_AUTHENTICATE event:

@Observer(Identity.EVENT_POST_AUTHENTICATE)
    public void postAuthenticate(Identity identity) {
        cleanupDataProvider();
        // after login, display the user menu instead of the guest menu
        mainMenu = getMainMenu();
        syncDataFetcherWithRequest();
    }

Dynamic RichFaces PanelMenu

One method of navigating is to browse wines by region. Recall that Region implements the org.vinwiki.model.Composite interface so we can compose a hierarchical tree structure of regions. I chose the RichFaces PanelMenu for this example, but you could also adapt the code to build another type of menu. Menu items are dynamic so we need to programmatically build a org.richfaces.component.html.HtmlPanelMenu using the RichFaces API. Because the menu is a form of navigation, the nav component builds the menu. On the home page, we have a <rich:panelMenu> tag with the binding set to "mainMenu".

<a4j:form prependId="false">
    <rich:panelMenu id="mainMenu" binding="#{mainMenu}"/>
  </a4j:form>

Seam invokes the initMainMenu method of our nav component to resolve the binding. All guests share the same menu, but each user can have their own menu. You could imagine allowing users to hide specific regions they are not interested in, which will impact the rendering of the main menu. One caveat is that you should manually set the ID for all the UI components your binding creates or you will most likely encounter duplicate ID exceptions, especially after hot deployments.

Bookmarkable Navigation URLs

It would be nice if the application allowed users to bookmark a specific filter like "Most Recent" or region like "California". For example, in the screen shot below, notice that the URL in the browser address bar reflects the fact that the user clicked on California in the main menu. The user can bookmark this page in their browser to instantly view the list of Californian wines available in VinWiki.

Seam makes this possible using URL re-writing and page parameters. My solution is largely based on the Blog example provided in the Seam documentation. When the menu was constructed, I set the action for the California region menu item to be /region.xhtml?r=California. In pages.xml, I link the "r" request parameter to the region property of an event-scoped component bookmarkable, see org.vinwiki.action.BookmarkableRequestInfo. In addition, I use a rewrite rule to make the URL more intuitive (/region/California instead of /region.seam?r=California):

<page view-id="/region.xhtml">
    <rewrite pattern="/region/{r}"/>
    <param name="r" value="#{bookmarkable.region}"/>
  </page>

This is where the aforementioned syncDataFetcherWithRequest method comes into play; this method uses the injected bookmarkable component to determine which data to show to the user, which in this case is the list of Californian wines.

One drawback to allowing navigation requests to be bookmarked in this manner is that I had to duplicate the contents of view/home.xhtml into filter.xhtml because it seems that adding multiple rewrite patterns on a single page leads to some weird URLs. I'm sure this could be overcome with some work, but since I'm using Facelets, the amount of duplication is minimal (and of course the UI is not final so the filter page may end up being different anyway). In the next posting, I'll show you how to make search results bookmarkable as well.

Server-side Pagination

Server-side pagination is essential for gracefully handling a large number of objects. The basic idea is to only display a small subset of the total results to the user at one time. The key, of course, is to extend the sub-set concept to the server and only load small sub-set of data from the database at a time. Only doing client-side pagination will become a major performance issue as the number of objects in your database increases. Thankfully, RichFaces does most of the work for us! Here is a screenshot of VinWiki's scrollable data table backed by server-side pagination:

And, the corresponding JSF syntax (from view/WEB-INF/facelets/itemTable.xhtml:

1. My solution is based largely on the sample code provided with RichFaces (see org.richfaces.demo.extendeddatamodel.AuctionDataModel). Essentially, my <rich:dataTable> interacts with an event-scoped component navDataModel that extends org.ajax4jsf.model.SerializableDataModel. Under the covers, the DataModel (see org.vinwiki.action.PagedDataModelBase) component does everything needed to support AJAX-driven pagination except the actual loading of data. To load data, the DataModel delegates to a org.vinwiki.action.PagedDataProvider, whose implementation is a session-scoped component that loads and caches items from the database.
2. Under the covers, the number of rows displayed to the user per page comes from the user's preferences.
3. The RichFaces <rich:datascroller> provides the paging mechanism for our paged data table.

Here is a UML class diagram showing the relationship between the DataModel and DataProvider (most of these classes can be used in your project as they have nothing to do with Wine):

As you might expect, the concrete implementation of org.vinwiki.action.PagedDataProvider is our handy nav component which extends org.vinwiki.action.PagedDataProviderBase. PagedDataProviderBase does most of the work except the actual fetching of a page of data from the database, which is provided by a org.vinwiki.action.PagedDataFetcher. There is a concrete implementation of org.vinwiki.action.PagedDataFetcher for each type of navigation. For example, the org.vinwiki.action.FetchRegion class provides Wine objects for a specific region to the data provider. Notice that my current PagedDataFetcher implementations are *not* Seam components and expect the EntityManager and User ID to be passed to them. I chose this approach to make it really easy to build new types of navigation queries; all you have to do is provide a query to return a page of data and a query to return the total number of items available to the current user.

The PagedDataProviderBase maps each starting index to a List of Item objects. A Map is a good solution if you are using the <rich:datascroller> because the user can jump from page 1 to 10 without going through pages 2-9 first. If you're using a sequential paging mechanism then a simple List should suffice. Of course the Map could grow very big if the user visits many pages in the scroller. I'll leave it as an exercise for the reader to solve using SoftReferences.

View Wine Details

Assuming the user finds a wine of interest while browsing the application, they may want to view more information for the wine as well as scroll through other users' ratings.
You didn't know the Rat Pack liked wine and wrote Latin did you ;-)

There are a number of possible activities the wine details view can offer the user, including:

• Add / edit rating
• View a list of similar wines (MoreLikeThis)
• Contextual display Ads
• Navigation controls to view the next wine in the results
• Edit information about the wine itself

Consequently, it makes sense to implement a new page for viewing wine details ( view/wine.xhtml ). From the home page, we take the user to the wine details page using a Seam <s:link> tag, which allows us to pass the Wine ID as a request parameter:

<s:link view="/wine.xhtml" value="#{item.fullName}" style="font-size:14px;">
    <f:param name="wid" value="#{item.id}"/>
  </s:link>

In pages.xml, I map a request parameter to the wineId property of the viewWine component:

<page view-id="/wine.xhtml">
    <rewrite pattern="/wine/{wid}"/>
    <param name="wid" value="#{viewWine.wineId}"/>
  </page>

Notice that I'm also re-writing the URL. Link the "wid" parameter to #{viewWine.wineId} begins a new conversation for viewing the wine:

@Begin(flushMode = FlushModeType.MANUAL, join = true)
    public void setWineId(Long wineId) {
       ...
    }

There are a number of ways to begin a conversation so be sure to research the best approach for your application in the Seam documentation.

The wine details page can be bookmarked as well. In this case, however, I'm using the push-style MVC approach discussed in the Seam documentation. I had to get clever with the Home link on the details page because if the user comes through a bookmark, history.go(-1) won't work for returning the user to the VinWiki home page. Thus, I rely on an AJAX call to end the conversation and then invoke the backToHome JavaScript function:

<a4j:commandLink value="#{messages.backToHome}" action="#{viewWine.endViewWine()}" oncomplete="backToHome()" style="font-size:14px;"/>

From view/layout/template.xhtml:

function backToHome() {
    var currentHost = document.location.protocol+"//"+document.location.host;
    if (document.referrer && document.referrer.startsWith(currentHost)) {
        history.back();
    } else {
        window.location.replace("/vinwiki/");
    }
}

Add Rating for Existing Wine

There are two ways a user can add a rating:

1. lookup a wine on-the-fly from the home page, or
2. view details for a specific wine and then add the rating from the wine details view.

Rating with on-the-fly Lookup

For authenticated users, the header menu on the home page includes a link to "Add Rating". Clicking on the Add Rating link, triggers the #{ratingHome.beginRatingWine()} action using AJAX. When the action completes, the addRatingPanel is shown to the user.

From view/WEB-INF/facelets/headerControls.xhtml

<a4j:commandLink value="Add Rating" action="#{ratingHome.beginRatingWine()}" reRender="addRatingPanel"
            oncomplete="#{rich:component('addRatingPanel')}.show()" styleClass="hdrLink"/>

The conversational ratingHome component (org.vinwiki.action.RatingHome based on Seam's Application Framework) manages adding and editing Rating objects by users.

So now let's see how on-the-fly lookup works ... from view/WEB-INF/facelets/addRating.xhtml

1. Decorate the form field as a required field using Seam's <s:decorate> tag and a Facelets template. If a validation error occurs, Seam will decorate the field with error information.
2. When the "onblur" event fires on the input field, send an AJAX request to the server to determine if the user selected a known wine.
3. Attach a RichFaces suggestion box <rich:suggestionbox> to the input field to auto-complete the wine name as the user types. On the server, the RichFaces suggestion box invokes the #{ratingHome.autoCompleteWine} action. I'll discuss the RichFaces request queue in more detail below, but for now, you should realize that it is dangerous to have an auto-complete component without some sort of request flood control in place.

So assuming the user has successfully filled-in the rating form, let's see what happens when the form is submitted:

There's a fair amount of complex machinery going on in this one tag! Let's analyze it step-by-step:

1. A RichFaces <a4j:commandButton> submits the form using AJAX to invoke the #{ratingHome.saveRating} action within the same conversation started when the user clicked on Add Rating.
2. The reRender attribute tells RichFaces to re-render the addRatingPanel and itemTable components in the component tree and updated in the browser DOM after the AJAX response is completed. This ensures that the panel will display any errors that occur on the server.
3. If there are no errors, close the modal panel. Notice that I'm calling the hasJsfErrMsg JavaScript function, which returns true if there are any error messages queued in the FacesContext. The hasJsfErrMsg function is defined in my main Facelets layout template (view/layout/template.xhtml) and relies on an <a4j:outputPanel> to update the value of a hidden field with ID 'jsfMsgMaxSev' on every AJAX request. The value for this field is pulled from the FacesContext.getMaximumSeverity() method.
   

   <a4j:outputPanel ajaxRendered="true">
       <a4j:form prependId="false">
         <h:inputHidden id="jsfMsgMaxSev" value="#{jsf.maxMsgSev}"/>
       </a4j:form>
     </a4j:outputPanel>
   

Rating from Details

When a user views a wine, a new conversation is started with the viewWine component.

Recall that we are already in a conversation when viewing the wine. Thus, the Add Rating operation will occur in a nested conversation using the ratingHome component.

@Begin(nested=true, flushMode=FlushModeType.MANUAL)
    public void beginRatingWine(Wine currentWine) {
        // attach the objects needed to create a rating to the
        // extended PersistenceContext for this conversation
        user = getEntityManager().merge(currentUser);
        wine = getEntityManager().find(Wine.class, currentWine.getId());
        updateStateForWine();
        info("User {0} has starting rating wine {1} in NESTED conversation {2}.", user.getUserName(), wine.getFullName(), Conversation.instance().getId());
    }

Upon starting the nested conversation, you need to "attach" the User and Wine objects to the extended PersistenceContext (entityManager).

Add New Wine

What type of wiki would this be if user's could not add new content on-the-fly? Of course, this application is not a full-featured wiki (see the wiki project in the Seam examples), but any authenticated user can add a new Wine (and dependent objects like Winery and Region) to the application. The wineHome component does most of the heavy lifting for adding a new Wine to the system, see org.vinwiki.action.WineHome. WineHome provides persistence operations for Wine entities by extending org.jboss.seam.framework.EntityHome from the Seam Application Framework. The implementation is mostly uninteresting from a re-use perspective, with the exception of being a @Factory for a component named wine.

@Factory("wine")
    public Wine initWine() {
        return getInstance();
    }

This allows EL expressions in /admin.xhtml to refer to the new Wine object simply as wine, such as #{wine.type}. The wine is managed in conversation scope along with wineHome.

We'll need to address the threat of spam and cross-site scripting (XSS) hacks before we can release this application on the Web, which I'll address in post #3.

Edit Wine

Edit wine also uses the wineHome component and /admin.xhtml.

So that about covers the specific use cases for this posting. Now, let's look at some specific UI implemenation details that will help you build better apps with Seam and RichFaces.

Other Useful UI Implementation Details

Facelets

I'm using Facelets for templating and UI component re-use. The main layout template is view/layout/template.xhtml. Each view references this template in the root Facelets <ui:composition ... template="layout/template.xhtml"> element.

RichFaces Semantic Layouts

I've started experimenting with the semantic layouts support provided by RichFaces (see view/layout/template.xhtml). I think the layouts help reduce CSS you need to manage for application so their definitely worth checking out.

RichFaces ModalPanel, Forms, and Seam Conversations

The RichFaces ModalPanel component is an excellent tool for allowing the user to perform a quick operation without losing their current context with your application. For example, the Add Rating ModalPanel allows the user to add a rating to the wine they are currently viewing without going to a separate page. Unfortunately, ModalPanel can also be major source of frustration if not handled correctly, especially when working with an AJAX-driven form on the panel. In this section, I hope to spare you some of that frustration by tackling some of the troublesome areas you may encounter when working with ModalPanels.

Opening the ModalPanel (and beginning a Conversation)

For the complete code for this section, please see view/WEB-INF/facelets/userPreferences.xhtml.


Here are some tips to keep in mind when opening a ModalPanel within a conversation:

1. The reRender attribute should include the ID of the panel you are opening. This ensures the UI components in the panel reflect the state of the conversation.
2. When the action completes, use JavaScript to show the panel via oncomplete="#{rich:component('prefPanel')}.show()"
   
3. In the action handler on the server, be sure to load any entities needed by the ModalPanel into the extended PersistenceContext for the conversation:

   // Out-ject the User object that we're updating, vs. the one that came in ...
       @Out protected User updatingUser = null;
   
       // Just a handy reference to the preferences object we're updating ...
       @Out protected Preferences updatingPrefs = null;
   
       @Begin(flushMode = FlushModeType.MANUAL, join = true)
       public void beginEditPreferences() {
           // load the User entity to edit preferences for into the extended PC for this conversation
           updatingUser = getEntityManager().find(User.class, currentUser.getId());
           updatingPrefs = updatingUser.getPreferences();
           initDob();
       }
   

Processing the Form on the ModalPanel (and ending the Conversation)

So now you have a visible ModalPanel with a form displayed, within a Seam conversation. When the user submits the form, we need to validate the input and display any errors that occur. If no errors occur, then close the panel when the action completes.

1. The action handler should end the conversation only if it completes successfully. So be careful with using @End with AJAX action handlers of type void. I prefer to use Conversation.instance().end() when dealing with AJAX actions as it makes it more explicit when the conversation ends.
2. Be sure to re-render the form after submit so validation errors are displayed correctly.
3. As mentioned previously, use a simple JavaScript function to determine if there are FacesMessages with severity ERROR or greater queued in the FacesContext; close the panel if there are no errors.

Queue and Traffic Flood Protection using the Global Default Queue

I've enabled the global queue for this application in web.xml:

<context-param>
    <param-name>org.richfaces.queue.global.enabled</param-name>
    <param-value>true</param-value>
  </context-param>

Project Setup

Building

You can download the project from here.
The project should build with minimal effort using Seam 2.2.0 with Java 6 from the command-line or Eclipse (I used v. 3.4.2). Once deployed to your JBoss 4.2.3 server, you can register or login as "vinwiki" with password "S00ner$1". However, you should also register your own account to see how the registration system works.

Database Setup

I've included a mysqldump file (vinwiki.sql) in the root directory which contains some wine related objects pulled from Freebase. To install this database in your MySQL 5.1.x database, do the following:

mysql> create database vinwiki;
mysql> grant all privileges on vinwiki.* to 'vinwiki'@'localhost' identified by 'vinwiki';
mysql> use vinwiki;
mysql> source vinwiki.sql;

Testing

The tests are configured to run against a MySQL database named vinwiki_test so that tests do not affect the development database. The JDBC connection parameters are specified in: bootstrap/deploy/hsqldb-ds.xml. From the MySQL command-line, do the following:

mysql> create database vinwiki_test;
mysql> grant all privileges on vinwiki_test.* to 'vinwiki'@'localhost' identified by 'vinwiki';

What's Next?

So by now you should have a good understanding of the basic structure of this application and be able to build it and run the unit tests. Please continue on to the second post which covers Hibernate Search and Lucene to help users find wines of interest.