Concurrent Kitties Using GPars
May 8, 2015 3 Comments
On today’s Groovy Podcast, I mentioned that I was teaching a Groovy training class this week at Research Now in Plano, TX. That’s not how I said it, though. I said that I was broadcasting live in front of a studio audience and that they were the most amazingly awesome group I’ve ever encountered.
(Yeah, okay, sometimes I shamelessly pander for good evals. I’ll let you know if it worked after the class ends. Unless it doesn’t, in which case I probably won’t.)
During the podcast, I told my inestimable co-host, Peter Ledbrook, that we got a chance to use GPars in class. The app we used it on was all about the primary goal of the internet, which is to display cat pictures.
Peter then shamed me into writing a blog post about it, which you’re reading now.
I’ve actually written about this app before, for another client. My post there was originally entitled, “The Reason The Internet Was Invented, Or Cat Pictures FTW”, but the host changed it to the far more mundane Calling RESTful Services in Groovy.
The basic idea is that Flickr (remember them? Me neither) has a RESTful API that lets you search for photos. The “flickr.photos.search” request doesn’t require authentication, but does require a whole set of query parameters, including an API key.
Funny story: in order to get a Flickr API key, you actually have to register at Yahoo! Remember them, too? Yeah, neither did I.
At any rate, I registered and got my key, so I can now do the searches. Here’s the start of my Groovy script to do it:
import groovy.json.* String key = new File('flickr_key.txt').text String endPoint = 'https://api.flickr.com/services/rest?' def params = [method : 'flickr.photos.search', api_key : key, format : 'json', tags : 'kitty', nojsoncallback: 1, media : 'photos', per_page : 6] // Build URL and download JSON data String qs = params.collect { it }.join('&') String jsonTxt = "$endPoint$qs".toURL().text
The query string is constructed from the map of params
by running a collect
on each element (which returns key=value
for each Map.Entry
) and then joining the resulting list with an ampersand. Notice the tags
key was assigned to the word “kitty”.
The next part of my script writes out the results and appends them to a file.
// write formatted JSON data to file File f = new File('cats.json') if (f) f.delete() f << JsonOutput.prettyPrint(jsonTxt) println JsonOutput.prettyPrint(jsonTxt)
Here’s a sample formatted JSON response:
{ "photos": { "page": 1, "pages": 127979, "perpage": 6, "total": "767873", "photo": [ { "id": "17418175405", "owner": "31469819@N02", "secret": "9055856685", "server": "5453", "farm": 6, "title": "A Ghostly Cat", "ispublic": 1, "isfriend": 0, "isfamily": 0 }, { "id": "16795470464", "owner": "95966544@N07", "secret": "cc4af0d44f", "server": "8799", "farm": 9, "title": "Looking for a home", "ispublic": 1, "isfriend": 0, "isfamily": 0 }, { "id": "17228164988", "owner": "92936362@N06", "secret": "d42c68bbf3", "server": "8734", "farm": 9, "title": "peaches the cat", "ispublic": 1, "isfriend": 0, "isfamily": 0 }, { "id": "17208304157", "owner": "102705402@N02", "secret": "582fff8f44", "server": "8688", "farm": 9, "title": "This is the sweetest cat in the world!", "ispublic": 1, "isfriend": 0, "isfamily": 0 }, { "id": "17228717179", "owner": "37561081@N07", "secret": "eb8d0119fe", "server": "7722", "farm": 8, "title": "\u65e9\u5b89", "ispublic": 1, "isfriend": 0, "isfamily": 0 }, { "id": "17388635206", "owner": "127041099@N08", "secret": "6310c6012a", "server": "7745", "farm": 8, "title": "Tsim Tung Brother Cream (\u5c16\u6771\u5fcc\u5ec9\u54e5)", "ispublic": 1, "isfriend": 0, "isfamily": 0 } ] }, "stat": "ok" }
Note that nowhere in the various photo
elements do you find a URL for the actual image. It turns out that to assemble the image you have to plug various pieces of the photo elements into a string, which is something Groovy is good at. First, however, I have to parse this and grab the photo elements:
// parse JSON data and build URL for pictures def json = new JsonSlurper().parseText(jsonTxt) def photos = json.photos.photo
The photos
variable is now a list of maps for each photo
, which I can transform into URLs using a collect
:
def images = photos.collect { p -> String url = "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg" url.toURL().bytes }
The Groovy string uses the farm
, server
, id
, and secret
elements of the response in each photo and builds a complete URL for the JPG image. Then I convert that to an actual URL and call getBytes()
to return byte arrays.
I can then use a SwingBuilder to assemble a trivial GUI showing all the images:
// build UI using Swing new SwingBuilder().edt { frame(title: 'Cat pictures', visible: true, pack: true, defaultCloseOperation: WC.EXIT_ON_CLOSE, layout: new GridLayout(0, 2, 2, 2)) { images.each { label(icon: new ImageIcon(it)) } } }
That requires some additional imports:
import groovy.swing.SwingBuilder import java.awt.GridLayout import javax.swing.ImageIcon import javax.swing.WindowConstants as WC // Ooh, aliased imports
Here’s where we improved the system using GPars. The download of the images can be done in a multithreaded fashion by adding a GParsPool:
import static groovyx.gpars.GParsPool.* // ... def images = [] withPool { images = photos.collectParallel { p -> String url = "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg" url.toURL().bytes } } // ...
That uses the default pool size, which is the number of processors you have plus one. The images are now downloaded concurrently as part of transforming the photo
elements into byte arrays using collectParallel
.
Here’s the whole script together:
import static groovyx.gpars.GParsPool.* import groovy.json.* import groovy.swing.SwingBuilder import java.awt.GridLayout import javax.swing.ImageIcon import javax.swing.WindowConstants as WC String key = new File('flickr_key.txt').text String endPoint = 'https://api.flickr.com/services/rest?' def params = [method : 'flickr.photos.search', api_key : key, format : 'json', tags : 'kitty', nojsoncallback: 1, media : 'photos', per_page : 6] // Build URL and download JSON data String qs = params.collect { it }.join('&') String jsonTxt = "$endPoint$qs".toURL().text // write formatted JSON data to file File f = new File('cats.json') if (f) f.delete() f << JsonOutput.prettyPrint(jsonTxt) println JsonOutput.prettyPrint(jsonTxt) // parse JSON data and build URL for pictures def json = new JsonSlurper().parseText(jsonTxt) def photos = json.photos.photo def images = [] withPool { images = photos.collectParallel { p -> String url = "http://farm${p.farm}.staticflickr.com/${p.server}/${p.id}_${p.secret}.jpg" url.toURL().bytes } } // build UI using Swing new SwingBuilder().edt { frame(title: 'Cat pictures', visible: true, pack: true, defaultCloseOperation: WC.EXIT_ON_CLOSE, layout: new GridLayout(0, 2, 2, 2)) { images.each { label(icon: new ImageIcon(it)) } } }
Here is the result of a sample run:
So there you have it, except for the stupid Flickr key, which I decided to let you register for on your own. Hey, I had to go through that pain, so everybody else does, too.
Well, not everybody. As part of my pandering for evals technique, I did give my key to the students in my class, who no doubt will reward me with stellar evals once we’re done. Probably. It could happen. Either way, at least there were cat pictures, and that’s a Good Thing.
Pingback: Settimanale Groovy #69 | BME
Pingback: Diario di Grails (settimana 19 del 2015) | BME
Reblogged this on Andrey Hihlovskiy.