Concurrent Kitties Using GPars

On today’s Groovy Podcast, I mentioned that I was teaching a Groovy training class this week at Research Now in Plano, TX. That’s not how I said it, though. I said that I was broadcasting live in front of a studio audience and that they were the most amazingly awesome group I’ve ever encountered.

(Yeah, okay, sometimes I shamelessly pander for good evals. I’ll let you know if it worked after the class ends. Unless it doesn’t, in which case I probably won’t.)

During the podcast, I told my inestimable co-host, Peter Ledbrook, that we got a chance to use GPars in class. The app we used it on was all about the primary goal of the internet, which is to display cat pictures.

Peter then shamed me into writing a blog post about it, which you’re reading now.

I’ve actually written about this app before, for another client. My post there was originally entitled, “The Reason The Internet Was Invented, Or Cat Pictures FTW”, but the host changed it to the far more mundane Calling RESTful Services in Groovy.

The basic idea is that Flickr (remember them? Me neither) has a RESTful API that lets you search for photos. The “flickr.photos.search” request doesn’t require authentication, but does require a whole set of query parameters, including an API key.

Funny story: in order to get a Flickr API key, you actually have to register at Yahoo! Remember them, too? Yeah, neither did I.

At any rate, I registered and got my key, so I can now do the searches. Here’s the start of my Groovy script to do it:

import groovy.json.*

String key = new File('flickr_key.txt').text
String endPoint = 'https://api.flickr.com/services/rest?'
def params = [method        : 'flickr.photos.search',
              api_key       : key,
              format        : 'json',
              tags          : 'kitty',
              nojsoncallback: 1,
              media         : 'photos',
              per_page      : 6]

// Build URL and download JSON data
String qs = params.collect { it }.join('&')
String jsonTxt = "$endPoint$qs".toURL().text

The query string is constructed from the map of params by running a collect on each element (which returns key=value for each Map.Entry) and then joining the resulting list with an ampersand. Notice the tags key was assigned to the word “kitty”.

The next part of my script writes out the results and appends them to a file.

// write formatted JSON data to file
File f = new File('cats.json')
if (f) f.delete()
f << JsonOutput.prettyPrint(jsonTxt)
println JsonOutput.prettyPrint(jsonTxt)

Here’s a sample formatted JSON response:

{
    "photos": {
        "page": 1,
        "pages": 127979,
        "perpage": 6,
        "total": "767873",
        "photo": [
            {
                "id": "17418175405",
                "owner": "31469819@N02",
                "secret": "9055856685",
                "server": "5453",
                "farm": 6,
                "title": "A Ghostly Cat",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            },
            {
                "id": "16795470464",
                "owner": "95966544@N07",
                "secret": "cc4af0d44f",
                "server": "8799",
                "farm": 9,
                "title": "Looking for a home",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            },
            {
                "id": "17228164988",
                "owner": "92936362@N06",
                "secret": "d42c68bbf3",
                "server": "8734",
                "farm": 9,
                "title": "peaches the cat",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            },
            {
                "id": "17208304157",
                "owner": "102705402@N02",
                "secret": "582fff8f44",
                "server": "8688",
                "farm": 9,
                "title": "This is the sweetest cat in the world!",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            },
            {
                "id": "17228717179",
                "owner": "37561081@N07",
                "secret": "eb8d0119fe",
                "server": "7722",
                "farm": 8,
                "title": "\u65e9\u5b89",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            },
            {
                "id": "17388635206",
                "owner": "127041099@N08",
                "secret": "6310c6012a",
                "server": "7745",
                "farm": 8,
                "title": "Tsim Tung Brother Cream (\u5c16\u6771\u5fcc\u5ec9\u54e5)",
                "ispublic": 1,
                "isfriend": 0,
                "isfamily": 0
            }
        ]
    },
    "stat": "ok"
}

Note that nowhere in the various photo elements do you find a URL for the actual image. It turns out that to assemble the image you have to plug various pieces of the photo elements into a string, which is something Groovy is good at. First, however, I have to parse this and grab the photo elements:

// parse JSON data and build URL for pictures
def json = new JsonSlurper().parseText(jsonTxt)
def photos = json.photos.photo

The photos variable is now a list of maps for each photo, which I can transform into URLs using a collect:

def images = photos.collect { p ->
    String url =
        "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg"
    url.toURL().bytes
}

The Groovy string uses the farm, server, id, and secret elements of the response in each photo and builds a complete URL for the JPG image. Then I convert that to an actual URL and call getBytes() to return byte arrays.

I can then use a SwingBuilder to assemble a trivial GUI showing all the images:

// build UI using Swing
new SwingBuilder().edt {
    frame(title: 'Cat pictures', visible: true, pack: true,
            defaultCloseOperation: WC.EXIT_ON_CLOSE,
            layout: new GridLayout(0, 2, 2, 2)) {
        images.each {
            label(icon: new ImageIcon(it))
        }
    }
}

That requires some additional imports:

import groovy.swing.SwingBuilder

import java.awt.GridLayout
import javax.swing.ImageIcon
import javax.swing.WindowConstants as WC  // Ooh, aliased imports

Here’s where we improved the system using GPars. The download of the images can be done in a multithreaded fashion by adding a GParsPool:

import static groovyx.gpars.GParsPool.*

// ...

def images = []
withPool {
    images = photos.collectParallel { p ->
        String url =
                "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg"
        url.toURL().bytes
    }
}

// ...

That uses the default pool size, which is the number of processors you have plus one. The images are now downloaded concurrently as part of transforming the photo elements into byte arrays using collectParallel.

Here’s the whole script together:

import static groovyx.gpars.GParsPool.*

import groovy.json.*
import groovy.swing.SwingBuilder

import java.awt.GridLayout
import javax.swing.ImageIcon
import javax.swing.WindowConstants as WC

String key = new File('flickr_key.txt').text
String endPoint = 'https://api.flickr.com/services/rest?'
def params = [method        : 'flickr.photos.search',
              api_key       : key,
              format        : 'json',
              tags          : 'kitty',
              nojsoncallback: 1,
              media         : 'photos',
              per_page      : 6]

// Build URL and download JSON data
String qs = params.collect { it }.join('&')
String jsonTxt = "$endPoint$qs".toURL().text

// write formatted JSON data to file
File f = new File('cats.json')
if (f) f.delete()
f << JsonOutput.prettyPrint(jsonTxt)
println JsonOutput.prettyPrint(jsonTxt)

// parse JSON data and build URL for pictures
def json = new JsonSlurper().parseText(jsonTxt)
def photos = json.photos.photo

def images = []
withPool {
    images = photos.collectParallel { p ->
        String url =
                "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg"
        url.toURL().bytes
    }
}

// build UI using Swing
new SwingBuilder().edt {
    frame(title: 'Cat pictures', visible: true, pack: true,
            defaultCloseOperation: WC.EXIT_ON_CLOSE,
            layout: new GridLayout(0, 2, 2, 2)) {
        images.each {
            label(icon: new ImageIcon(it))
        }
    }
}

Here is the result of a sample run:

So there you have it, except for the stupid Flickr key, which I decided to let you register for on your own. Hey, I had to go through that pain, so everybody else does, too.

Well, not everybody. As part of my pandering for evals technique, I did give my key to the students in my class, who no doubt will reward me with stellar evals once we’re done. Probably. It could happen. Either way, at least there were cat pictures, and that’s a Good Thing.