Serving jokes locally with Ratpack and MongoDB

In two previous posts, I discussed applications I created that were simple client-side front ends for the Internet Chuck Norris Database (ICNDB), located at http://icndb.com. This post gives the details of the local server I created, using Groovy, MongoDB, and the cool Ratpack project (note new URL). The earlier posts contained parts of that app, but since then I’ve updated it to the latest version of Ratpack, revised the gradle build file accordingly, added a couple of integration tests, and checked the whole thing into GitHub.

I often use ICNDB in my Groovy presentations, because it’s easy to access, returns a simple JSON object, and is rather amusing.

(This, by the way, is in direct contrast to Mr. Norris himself, whose politics are somewhat to the right of Attila the Hun. Still, I only care about the jokes themselves, which are pretty funny.)

Accessing the site is trivial:

import groovy.json.JsonSlurper

def url = 'http://api.icndb.com/jokes/random'
def json = new JsonSlurper().parseText(url.toURL().text)
def joke = json?.value?.joke
println joke

The Groovy JDK adds a toURL method to java.lang.String, which returns an instance of java.net.URL. It also adds a getText method to java.net.URL that returns the entire response as a string. I use the parseText method in JsonSlurper to parse it and just dig through the properties to get the contained joke.

Of course, my demo relies on that site being available, and that’s far from a sure thing, especially if Carlos Ray’s lawyers ever get to it. It also has a tendency to go down at inconvenient intervals, like when I’m trying to present the above script.

(Quick aside: this blog is hosted on WordPress, which has a much better uptime record than my actual home page. My site is hosted on a very old Windows laptop in my office. You’ve heard of “five nines” uptime, where a site guarantees it’ll be available 99.999% of the time? My site only promises “nine fives” uptime. It’s online a bit more than half the time.)

It behooves me, therefore, to keep a local copy if possible. That turns out to be pretty easy. First, there aren’t a whole lot of jokes. The ICNDB API offers the link http://api.icndb.com/jokes/count to return the total number of jokes. That request returns a JSON object:

{
    "type": "success",
    "value": 546
}

I can certainly handle a database of that size locally. In fact, in my app I load the whole thing into memory because why not?

Since the format of the jokes is JSON, I decided to use a MongoDB database to store them. The native format for MongoDB is BSON, or binary JSON, so all I have to do is grab the jokes and I can just append them to a local database. To make the code easier, I use the GMongo project, which is just a Groovy wrapper around Mongo’s Java client library.

The other useful method in the ICNDB API is http://api.icndb.com/jokes/random/num, where num represents the number of jokes you want returned. I want all of them, so I replace num with the total.

For example, if I access http://api.icndb.com/jokes/random/5, I get something like:

{
    "type": "success",
    "value": [
        {
            "id": 204,
            "joke": "Nagasaki never had a bomb dropped on it. Chuck Norris jumped out of a plane and punched the ground",
            "categories": []
        },
        {
            "id": 329,
            "joke": "There are only two things that can cut diamonds: other diamonds, and Chuck Norris.",
            "categories": []
        },
        {
            "id": 348,
            "joke": "There?s an order to the universe: space, time, Chuck Norris.... Just kidding, Chuck Norris is first.",
            "categories": []
        },
        {
            "id": 360,
            "joke": "Two wrongs don't make a right. Unless you're Chuck Norris. Then two wrongs make a roundhouse kick to the face.",
            "categories": []
        },
        {
            "id": 406,
            "joke": "Chuck Norris doesn't say "who's your daddy", because he knows the answer.",
            "categories": []
        }
    ]
}

The only difference from the original URL is that now the value property returns all of the contained jokes, but that’s not a problem. The overall script is therefore:

import groovy.json.JsonSlurper
import com.gmongo.GMongo

// Drop the current icndb database, if it exists
GMongo mongo = new GMongo()
def db = mongo.getDB('icndb')
db.cnjokes.drop()

// Get the total number of available jokes
JsonSlurper slurper = new JsonSlurper()
String jsonTxt = 'http://api.icndb.com/jokes/count'.toURL().text
def json = slurper.parseText(jsonTxt)
int total = json.value.toInteger()

// Grab all of them at once
jsonTxt = "http://api.icndb.com/jokes/random/${total}".toURL().text
json = slurper.parseText(jsonTxt)

// Save them all locally
def jokes = json.value
jokes.each {
    db.cnjokes << it
}
assert total == jokes*.id.size()
assert total == db.cnjokes.find().count()

How cool is it that all I have to do is grab the jokes and append them to the collection to save them in the database? Truly, we live in magical times. 🙂

I can browse the database in Eclipse if I use the MonjaDB plugin. Here’s a screenshot showing it:
MonjaDB

Now that the database is populated, I can build the Ratpack app. I started off using the lazybones builder inside of gvm, the Groovy enVironment Manager, which I discussed in the earlier post. Ratpack keeps evolving, though, and lazybones hasn’t kept up, so the changes I’ve made to the resulting app are a bit more substantial than I originally intended.

Here’s the JokeServer class. In the constructor, I load all the jokes into a Groovy map, where the keys are the id’s and the values are the joke strings themselves.

@Singleton
class JokeServer {
    GMongo mongo = new GMongo()
    Map jokes = [:]
    List ids = []
    
    JokeServer() {
        DB db = mongo.getDB('icndb')
        def jokesInDB = db.cnjokes.find([categories: 'nerdy'])
        jokesInDB.each { j ->
            jokes[j.id] = j.joke
        }
        ids = jokes.keySet() as List
    }
    ...

The other method in the class returns a random joke by shuffling the id’s and returning the associated joke. (The id’s themselves aren’t consecutive, partly because I’m only using the “nerdy” ones and partly because the original site skipped ID values.)

    ...   
    String getJoke(String firstName = 'Chuck', 
                   String lastName = 'Norris') {
        Collections.shuffle(ids)
        String joke = jokes[ids[0]]
        if (!joke) println "Null joke at id=$id"
        if (firstName != 'Chuck')
            joke = joke.replaceAll(/Chuck/, firstName)
        if (lastName != 'Norris')
            joke = joke.replaceAll(/Norris/, lastName)
        return joke
    }
}

I’m using Groovy’s nice optional arguments mechanism here, so if I invoke getJoke without arguments I get the original joke, but if I supply a first name or a last name they’re used in the joke itself.

Here’s my test case to make sure at least this much is working. It’s a regular JUnit test, implemented in Groovy.

import static org.junit.Assert.*;

import org.junit.Before;
import org.junit.Test;

class JokeServerTest {
    JokeServer server = JokeServer.instance  // @Singleton on server

    @Test
    public void testGetJokeFirstNameLastName() {
        String joke = server.getJoke('Patton', 'Boggs')
        assert !joke.contains('Chuck')
        assert !joke.contains('Norris')
        assert joke.contains('Patton')
        assert joke.contains('Boggs')
    }

    @Test
    public void testGetJoke() {
        String joke = server.joke
        assert joke
    }

}

Did you notice that I added @Singleton on the JokeServer, just for fun? That’s why the test grabs the server using the instance property. My first test then uses the strings “Patton” and “Boggs”, the name of the law firm that sent me the takedown notice. The second test accesses the joke property, which calls the getJoke method by the normal Groovy idiom.

Ratpack applications use a script called ratpack.groovy to set the various handlers. Since my script is so simple, I just added everything to that handler:

import static ratpack.groovy.Groovy.*

import com.kousenit.JokeServer

JokeServer server = JokeServer.instance

ratpack {
    handlers {
        get {
            String message
            if(request.queryParams.firstName ||
               request.queryParams.lastName) {
                message = server.getJoke(
                    request.queryParams.firstName, 
                    request.queryParams.lastName)
            } else {
                message = server.joke
            }
            response.headers.set 'Content-Type', 'application/json'
            response.send message
        }
        
        assets "public"
    }
}

The ratpack method takes a closure containing the various handlers. I only have a single handler, which is accessed using a get request. Then I get the relevant joke, set the Content-Type header to the MIME type for JSON, and return it. Actually, since my server returns the actual string, I probably shouldn’t set the header at all, but it hasn’t hurt anything so far.

The only difference between this script and the one I showed previously is the import statement at the top. Now the ratpack method is a static method in the ratpack.groovy.Groovy class, which Eclipse (actually, Groovy / Grails Tool Suite) can’t find even though it’s in the dependencies.

The next piece of the puzzle is the Gradle build script itself. Ratpack recently changed its package structure and moved its deployed versions to JFrog. Here’s the updated build script:

apply plugin: "ratpack-groovy"

buildscript {
  repositories {
    maven { url "http://oss.jfrog.org/repo" }
    mavenCentral()
  }
  dependencies {
    classpath 'io.ratpack:ratpack-gradle:0.9.0-SNAPSHOT'
  }
}

repositories {
  maven { url "http://oss.jfrog.org/repo" }
  mavenCentral()
  maven { url "http://repo.springsource.org/repo" } // for springloaded
}

dependencies {
    compile 'com.gmongo:gmongo:1.0'
    testCompile "org.spockframework:spock-core:0.7-groovy-2.0", {
        exclude module: "groovy-all"
    }

    // SpringLoaded enables runtime hot reloading.
    springloaded "org.springsource.springloaded:springloaded-core:1.1.4"
}

task wrapper(type: Wrapper) {
    gradleVersion = "1.8"
}

The buildscript block includes the information for downloading the ratpack-groovy plugin, which has changed the group id of the ratpack dependency to io.ratpack. I excluded the groovy-all module from the Spock dependency because it’s already part of the ratpack-groovy plugin, and I updated the Gradle version property in the wrapper to 1.8, but otherwise this is the same as the one generated by lazybones.

Before I run, though, I still want some sort of integration test. Most of the available examples online don’t have any tests (sigh), and I spent far too many hours figuring out how to get one to work, so I’m including it here.

package com.kousenit

import ratpack.groovy.test.LocalScriptApplicationUnderTest
import ratpack.groovy.test.TestHttpClient
import spock.lang.Specification

class ServerIntegrationSpec extends Specification {

    def aut = new LocalScriptApplicationUnderTest()
    @Delegate TestHttpClient client = aut.httpClient()

    def setup() {
        resetRequest()
    }

    def "regular get request returns Chuck Norris string"() {
        when:
        String result = get('/').asString()

        then:
        println result
        result.contains('Chuck Norris')
    }

    def "firstName and lastName parameters work"() {
        when:
        def response = get('?firstName=Carlos&lastName=Ray')?.asString()
        
        then:
        println response
        response.contains('Carlos Ray')
    }
}

As with most Spock tests, the tests themselves are pretty self-explanatory. The hard part was figuring out the proper URLs to invoke and knowing to use the asString method, which wasn’t obvious at all. I’m also not clear on the mechanism used to get the TestHttpClient instance, but I’m sure Luke Daley will explain it to me when I see him next. 🙂

One last quirk should be noted for anyone trying to duplicate this. To run the server, I type:

> gradle run

which starts up the server on port 5050. Access the server using http://localhost:5050 or http://localhost:5050/firstName=Carlos&lastName=Ray and you’re all set.

For some reason, the script that runs the app requires the ratpack.groovy script to be located in the src/ratpack folder. If I want to run the tests, however,

> gradle test

then I have to have the ratpack.groovy file in the root of the project. I have no idea why, nor do I know how to configure them so I only need one and not the other, so I just copied the file into both locations.

(I know — that’s ugly and awkward. I promise to fix it when find out why it’s necessary in the first place.)

So, at long last, if you want the code, it’s all stored in my GitHub repository at https://github.com/kousen/cnjokeserver. Feel free to clone it and wreak whatever havoc upon it you desire.

——-

For those who might be interested, a few notes about my book, Making Java Groovy:

  • The book page at Amazon now has 24 reviews. All are four or five stars but one, which is extremely gratifying. More importantly, the text of each makes it clear that the book is finding its intended audience: Java developers who want to make their jobs easier by adding Groovy.
  • The lone three-star review doesn’t say anything bad about the book anyway, so at least I’ve got that going for me, which is nice.
  • If you get the book at Amazon rather than at Manning, you can still get the ebook versions (pdf, mobi, and epub). The mechanism to do so is included in an insert in the book.
  • The response to my silly marketing ideas in my last couple of posts has been silence punctuated by the occasional cricket, so I’m going to stop doing that unless something really good occurs to me. Oh well. They made me laugh during the writing process, and anything that keeps you writing is a Good Thing(TM).

Making Java Groovy: ratpack, MongoDB, and Chuck Norris

Before I get to the good parts of this post (the technical content), let me take care of a few marketing issues.

First, as I mentioned in my last post, I gave my “Making Java Groovy” presentation at JavaOne last week. If you were there and you haven’t completed your Session Surveys, please do so. I really want to Party Like a (JavaOne) Rock Star, Party Like a (JavaOne) Rock Star, etc., though I expect in reality that would involve sharing a quiet dinner at home with my wife. She probably wouldn’t appreciate it if I trashed the room, too, at least not more than it’s already trashed*.

*Yeah, my 21 year old son still does live at home, why do you ask?

That did give me the opportunity to see my book on the shelf of a book store for the first time:

book_shelf_photo

I should also mention that as of 9/30, if you buy the book at Manning, you can now get all the electronic formats (pdf, mobi, and epub).

Finally, I got my first Amazon review today, and was so good I walked around with a smile all day.

Now on to the real content. In my Groovy presentations, I often like to access a RESTful web service and show (1) how easy it is to make a GET request using Groovy, and (2) how to use a JsonSlurper to parse a JSON response to get the information inside it. For this purpose my default site is ICNDB, the Internet Chuck Norris Database.

That site has caused me problems, though. First, it resulted in my first ever take down notice from a lawyer, which I’ll show in a moment. Seriously. As part of my Android presentations at No Fluff, Just Stuff conferences I build a simple app to access that site, parse the results (using a GSON parser, since I don’t have Groovy available) and update the display. When running, the app looks like:

icndb_emulator_image

To make it easy for the attendees of my Android talks to play with, I uploaded the app to the Google Play store. That was fun until I received the following email, from an address I’ll leave out:

Dear Sir/Madam:

Patton Boggs LLP represents Carlos Ray Norris, aka Chuck Norris, the famous actor and celebrity.

We are contacting you because we recently learned that you have developed and are distributing a software application that uses Mr. Norris’s name and/or image without authorization on Google Play.

Mr. Norris appreciates all of his fans. However, the unauthorized use of his name and/or image severely harms my client and jeopardizes his existing business relationships.

Mr. Norris owns legal rights in his name and image which includes copyright, trademark, and publicity rights (the “Norris Properties”). He uses the Norris Properties regularly in his own business endeavors. Therefore we have asked Google to remove your application from Google Play because it violates Mr. Norris’s intellectual property rights.

We request that you (1) immediately stop developing and distributing “Chuck Norris” applications; (2) remove all “Chuck Norris” applications that you have developed or control from all websites under your control; and (3) do not use Mr. Norris’s name or image, or any cartoon or caricature version of Mr. Norris’s name or image for any endeavor, including in connection with software applications, without Mr. Norris’s permission.

Thank you for honoring Mr. Norris’s legal rights. Please contact me if you have questions.

Sincerely, …

A few points immediately come to mind:

  • Carlos Ray Norris? Seriously? I had no idea.
  • Does it matter that my app is free and only consumes data from a publicly available web site? Apparently not.
  • If I changed the icon and replaced the words “Chuck” and “Norris” everywhere with the words “Patton” and “Boggs” (the name of the law firm :)), could I upload it again?

I’m still not exactly sure what I was doing wrong, but of course I did not want to cross Carlos Ray “Chuck” Norris, to say nothing of his lawyers. But after talking to a few people, I decided to ignore the letter, at least until any money was involved.

I haven’t heard anything since, but about a week later Google Play took down my app.

So be it. If you want the source code, though, check out the ICNDB project in my GitHub repository. It’s mostly just a demonstration of how to Spring’s RestTemplate, Google’s GSON parser, and an Android AsyncTask together. In the repo is another branch that uses a Java Timer to refresh the joke every 10 seconds.

The real question is, why haven’t the barracudas lawyers gone after the original ICNDB site? Worse, what happens to my poor app (and, more importantly, my presentation) when they do?

I decided my best course of action was to download as many of the jokes as possible and be ready to serve them up locally in case I need them. That, at long last, brings me to the technical part of this blog post.

The original web site returns jokes in JSON format. That means storing them in a MongoDB database is trivial, because Mongo’s native format is BSON (binary JSON) and I can even query on the joke properties later.

How do I grab all the jokes? There’s no obvious query in the API for that, but there is a work around. If I first access http://api.icndb.com/jokes/count, I can get the total number of jokes. Then there is a URL called http://api.icndb.com/jokes/random/:num, which fetches num random jokes. According to the web site, it returns them in the form:

{ "type": "success", "value": [ { "id": 1, "joke": "Joke 1" }, { "id": 5, "joke": "Joke 5" }, { "id": 9, "joke": "Joke 9" } ] }

The value property is a list containing all the individual jokes.

To work with MongoDB, I’ll use the driver from the GMongo project. It follows the typical Groovy idiom, in that it takes an existing Java API (in this case, the ugly and awkward Java driver for Mongo) and wraps it in a much simpler Groovy API. As a beautiful illustration of the process, here’s an excerpt from the com.gmongo.GMongo class:

class GMongo {

  @Delegate
  Mongo mongo

  // ... lots of overloaded constructors ...

  DB getDB(String name) {
    patchAndReturn mongo.getDB(name)
  }

  static private patchAndReturn(db) {
    DBPatcher.patch(db); return db
  }
}

Note the use of the @Delegate annotation, which exposes all the methods on the existing Java-based Mongo class through the GMongo wrapper.

Based on that, here’s my script to download all the jokes and store them in a local MongoDB database:

import groovy.json.*
import com.gmongo.GMongo

GMongo mongo = new GMongo()
def db = mongo.getDB('icndb')
db.cnjokes.drop()

String jsonTxt = 'http://api.icndb.com/jokes/count'.toURL().text
def json = new JsonSlurper().parseText(jsonTxt)
int total = json.value.toInteger()

jsonTxt = "http://api.icndb.com/jokes/random/${total}".toURL().text
json = new JsonSlurper().parseText(jsonTxt)
def jokes = json.value
jokes.each {
    db.cnjokes << it
}
println db.cnjokes.find().count()

Note the nice overloaded left-shift operator to add each joke to the collection.

I’ve run this script several times and I consistently get 546 total jokes. That means the entire collection easily fits in memory, a fact I take advantage of when serving them up myself.

My client needs to request a random joke from the server. I want to do this in a public forum, so I’m not interested in any off-color jokes. Also, the ICNDB server itself offers a nice option that I want to duplicate. If you specify a firstName or lastName property on the URL, the joke will replace the words “Chuck” and “Norris” with what you specify. I like this because, well, the more I find out about Carlos Ray the more I wish I didn’t know.

Here’s my resulting JokeServer class:

package com.kousenit

import java.security.SecureRandom
import com.gmongo.GMongo
import com.mongodb.DB

@Singleton
class JokeServer {
    GMongo mongo = new GMongo()
    Map jokes = [:]
    List ids = []
    
    JokeServer() {
        DB db = mongo.getDB('icndb')
        def jokesInDB = db.cnjokes.find([categories: [$ne : 'explicit']])
        jokesInDB.each { j ->
            jokes[j.id] = j.joke
        }
        ids = jokes.keySet() as List
    }
    
    String getJoke(String firstName = 'Chuck', String lastName = 'Norris') {
        Collections.shuffle(ids)
        String joke = jokes[ids[0]]
        if (!joke) println "Null joke at id=$id"
        if (firstName != 'Chuck')
            joke = joke.replaceAll(/Chuck/, firstName)
        if (lastName != 'Norris')
            joke = joke.replaceAll(/Norris/, lastName)
        return joke
    }
}

MongoDB has a JavaScript API which uses qualifiers like $ne for “not equals”. The Groovy API wrapper lets me add those as keys in a map supplied to the find method.

I added the @Singleton annotation on the class, though that may be neither necessary or appropriate. I may want multiple instances of this class for scaling purposes. I’ll have to think about that. Let me know if you have an opinion.

I’m sure there’s an easier way to get a random joke out of the collection, but I kept running into issues when I tried using random integers. This way works, though it’s kind of ugly.

The getJoke method uses Groovy’s cool optional arguments capability. If getJoke is invoked with no arguments, I’ll use the original version. If firstName and/or lastName are specified, I use the replaceAll method from the Groovy JDK to change the value in the joke. Strings are still immutable, though, so the replaceAll method returns a new object and I have to reuse my joke reference to point to it. I actually missed that the first time (sigh), but that’s what test cases are for.

Speaking of which, here’s the JUnit test (written in Groovy) to verify the JokeServer is working properly:

package com.kousenit

import static org.junit.Assert.*

import org.junit.Before
import org.junit.Test

class JokeServerTest {
    JokeServer server = JokeServer.instance

    @Test
    public void testGetJokeFirstNameLastName() {
        String joke = server.getJoke('Patton', 'Boggs')
        assert !joke.contains('Chuck')
        assert !joke.contains('Norris')
        assert joke.contains('Patton')
        assert joke.contains('Boggs')
    }

    @Test
    public void testGetJoke() {
        assert server.joke
    }
}

I can invoke the getJoke method with zero, one, or two arguments. The basic testGetJoke method uses zero arguments. Or, rather, it accesses the joke property, which then calls getJoke() using the usual Groovy idiom.

Now I need to use this class inside a web server, and that gave me a chance to dig into the ratpack project. Ratpack is a Groovy project and I’ve checked out the source code from GitHub, but there’s a simpler way to create a new ratpack application based on the Groovy enVironment Manager (gvm) tool.

First I used gvm to install lazybones:

> gvm install lazybones

Then I created my ratpack project:

> lazybones create ratpack icndb

That creates a simple structure (shown in the ratpack manual) with a Gradle build file. The only modification I made to build.gradle was to add “compile 'com.gmongo:gmongo:1.0'” to the dependencies block.

Here’s the file ratpack.groovy, the script that configures the server:

import static org.ratpackframework.groovy.RatpackScript.ratpack

import com.kousenit.JokeServer

JokeServer server = JokeServer.instance

ratpack {
    handlers {
        get {
            String message
            if(request.queryParams.firstName || request.queryParams.lastName) {
                message = server.getJoke(
                    request.queryParams.firstName, 
                    request.queryParams.lastName)
            } else {
                message = server.joke
            }
            response.headers.set 'Content-Type', 'application/json'
            response.send message
        }
    }
}

I configured only a single “get” handler, which is invoked on an HTTP GET request. I check to see if either a firstName or lastName query parameter is supplied, and, if so, I invoke the full getJoke method. Otherwise I just access the joke property. I made sure to set the Content-Type header in the response to indicate I was returning JSON data, and sent the message.

If I type ./gradlew run from the command line, ratpack starts up its embedded Netty server on port 5050 and serves jokes on demand. Here’s a picture of the default request and response (using the Postman plugin in Chrome):

postman_icndb_default

Here is the response if I specify a first and last name:

postman_patton_boggs

There you have it. I wanted to write a test inside ratpack to check my hander, but the infrastructure for that appears to still be under development. I tried to imitate one of the samples and extend the ScriptAppSpec class, but that class isn’t in my dependent jars. I also tried to follow the Rob Fletcher’s mid-century ipsum example, but while I was able to create the test, it failed when running because it couldn’t find the ratpack.groovy file inside the ratpack folder, since it expected it to be in the root. The right approach would probably be to turn the contents of my get block into a separate Handler, because the manual talks about how to test them, but I haven’t done that yet.

As they say, if you live on the cutting edge, sometimes you get cut. I expect that will all settle down by 1.0.

For my presentations, I now need to update my Android client so that it accesses http://localhost:5050 and understands that the JSON coming back is a bit simpler than that served up by the original server. That’s beyond the scope of this (way too long) blog post, however.

I haven’t yet committed the application to GitHub, but I will eventually. In the meantime I just have to hope that my new app also doesn’t run afoul of Carlos, Ray, Patton, and Boggs, Esq.

%d bloggers like this: