{"id":5971,"date":"2019-05-17T11:27:27","date_gmt":"2019-05-17T10:27:27","guid":{"rendered":"https:\/\/irsg.bcs.org\/informer\/?p=5971"},"modified":"2019-05-17T11:27:27","modified_gmt":"2019-05-17T10:27:27","slug":"conference-review-haystack-us-2019-relevance-avengers-assemble","status":"publish","type":"post","link":"https:\/\/archive-irsg.bcs.org\/informer\/?p=5971","title":{"rendered":"Conference Review – Haystack US 2019 – Relevance Avengers Assemble!"},"content":{"rendered":"\n
\n

Last year I <\/em>attended the Haystack search relevance conference<\/em><\/a>\u00a0in Charlottesville, USA as a guest of our partners OpenSource Connections (OSC). In 2019 we\u00a0<\/em>merged my old business Flax with OSC<\/em><\/a>\u00a0so I returned as one of the conference organisers.<\/em><\/p>\n

Haystack is a conference all about search relevance – making sure that the results your users see fit their requirements and your business needs. Unlike some events Haystack has no sponsors and no vendor pitches and we try hard to keep the price low to promote accessibility. It’s a great chance to network with other search people – and no-one will ask you if what you do for a living ‘is a bit like Google’!<\/p>\n<\/div>\n

<\/p>\n

\n

This year the venue was a cinema in downtown Charlottesville, which gave us much needed extra space and easier access to the Downtown Mall and its array of restaurants, snack shops and bars. Plus points included reclining seats, an onsite cafe and some very big screens, although we did discover some issues with WiFi coverage (perhaps an aid to concentration however?) and the movie projector didn\u2019t always play nice with presenters\u2019 laptops. We\u2019ll sort this out for next time I\u2019m sure – of course the affected presenters were professionals and coped admirably with the glitches. Also, I\u2019m hoping none of the conference attendees felt they missed out on seeing Avengers Endgame, on show in one of the other theatres\u2026but just in case they did I\u2019ll introduce some of the marvel-lous characters we saw onstage at Haystack.<\/p>\n

The first day was introduced by Max \u201cIronman\u201d Irwin of OSC who gave us a keynote on\u00a0What is Search Relevance?<\/em>. Max showed us the three aspects of search quality: performance, experience and of course relevance, and went on to discuss how we can score judgements, cope with disagreements between human raters and fold in user engagement data. He also showed us a list of the speakers to come and welcomed over 140 attendees from the USA and Europe to Haystack.<\/p>\n

The next talk I saw was by Alessandro \u201cDr Strange\u201d Benedetti of Sease Ltd. (OK, I\u2019ll stop the Avengers references now before I infer one of our speakers was green and angry) on the\u00a0Rated Ranking Evaluator<\/a>\u00a0relevance testing tool. He showed us the heirarchical model for test queries they have developed and how the open source RRE can be used to run a huge amount of tests on a Solr or Elasticsearch instance as part of the Maven build process, producing a set of relevance metrics. These metrics in turn can be emitted to a spreadsheet, RRE\u2019s own server dashboard or as JSON (RRE also uses JSON for the relevance judgements that must be provided to it).<\/p>\n

Tara Diedrichsen & Tito Sierra of\u00a0LexisNexis<\/a>\u00a0followed with a fascinating talk on best practices for gathering human judgements for relevance testing. It\u2019s clear that LexisNexis have put huge amounts of work into this area to help them identify problem areas to focus on and to evaluate new algorithms. I\u2019m pleased they stressed that it\u2019s important to record\u00a0why<\/strong>\u00a0a search result is good or bad – this is essential information for relevance engineers who may be unfamiliar with the subject area.<\/p>\n

Lunch followed, and conference attendees scattered to the various restaurants on the Downtown Mall – luckily as far as I can tell they all came back afterwards. The next talk I saw came from Ren\u00e9 Kriegler on\u00a0Query Relaxation<\/a>\u00a0which was fascinating – Ren\u00e9 showed us various ways to remove terms from a query to increase the number of results and eventually suggested using a neural network to work out the best term to lose.<\/p>\n

Unfortunately I missed the next session as I was preparing to run the Lightning Talks, our last session of the day. The Lightning Talks started with a moving tribute to Ted Sullivan by his friend and colleague Eric Hatcher – sadly we lost Ted this year, I was very privileged to be able to meet him at last year\u2019s Haystack.<\/p>\n

The talks featured speakers on subjects including Zookeeper on AWS, the new\u00a0Quaerite<\/a>\u00a0relevance test tool, Solr on Kubernetes and the challenges of full text search at the Hathi Trust over 17 million documents. Thanks to everyone who volunteered to speak at such short notice!<\/p>\n

The first day concluded with dinner at Kardinal Hall nearby which was great fun and a chance to network and chat with other attendees.<\/p>\n

Jeremiah Via of the New York Times was the first presentation I attended on Day 2. Jeremiah described how Elasticsearch is used to index 18 million items at the Times and how they developed both online and offline metrics to improve relevance. The Times\u2019 index contains over 22 million unique tokens and nearly 2 million tags. He stressed the importance of being able to easily iterate through configuration changes – as he said \u201cimproving search is about making lots of little improvements\u201d.<\/p>\n

Next up was Tom Burgmans, describing how his team established a relevance focused culture at Wolters Kluwer. I particularly enjoyed seeing a screenshot of their advanced relevance testing tool which showed relevance judgements and also broke down the various contributions to relevance scores – I hope as he did that this tool eventually becomes open source. Wolters Kluwer have also developed a set of loosely coupled reusable search components which help to share knowledge and experience across the organisation. His last point was \u2018don\u2019t stop\u2019 – relevance improvement is never finished!<\/p>\n

My colleague Bertrand Rigaldies of OSC then talked about Solr query parsers (he noted that there are no less than 29 different query parsers supplied with Solr, including a good few I\u2019d never heard of). He showed how to build a simple proximity query parser (to handle queries like \u201c\u2018fish\u2019 within 3 words of \u2018chips\u2019\u201d) and stressed that although custom parsers can be very powerful, they are complex to write and one should try to use an out-of-the-box parser where possible.<\/p>\n

Lunch followed, attendees again taking advantage of the various outlets in Charlottesville\u2019s Downtown Mall.<\/p>\n

John Berryman, one half of the team behind the\u00a0Relevant Search<\/a>\u00a0book and now at Eventbrite, gave an engaging talk on automatic tagging using search logs and machine learning. His system creates a training set from user interactions (the events that users clicked after a particular query) then attempts to predict what tags to apply to other events – the tags being the search queries themselves.<\/p>\nThe next session was a panel discussion on Does Learning to Rank Actually Work (my alternative title \u2018Learning to Rank – or learning to tank?\u2019 was sadly discarded \ud83d\ude42 with Ren\u00e9 Kriegler, Doug Turnbull, Xun Wang (Snag) and Erik Bernhardson (Wikimedia). The audience provided some great questions for the panel.\n

I sadly missed most of Simon Hughes of DHI\u2019s talk on Search with Vectors but what I did see was very interesting, including how he had built a special query parser for Lucene that stored vectors as payloads. Luckily there\u2019s lots of detail in this\u00a0Github repository<\/a>.<\/p>\n

The conference ended with thanks to all the speakers, organisers and most importantly the attendees – without whom Haystack would of course not be possible! Thanks to everyone who came and made it such a great event. Haystack will return!<\/p>\n

If you\u2019d like a richer description of the conference including some of the talks I missed please do read\u00a0Jettro Coenradie\u2019s blog<\/a>. Alessandro Benedetti of Sease has also\u00a0written about his experience<\/a>\u00a0of the event. You can also join many of the conference attendees in\u00a0Relevance Slack<\/a>\u00a0– there\u2019s a\u00a0#haystack-conference<\/strong>\u00a0channel.<\/p>\n

You\u2019ll be glad to know we will be releasing the slides for all the main talks\u00a0and<\/strong>\u00a0the Lightning Talks very soon, and unlike last year we managed to video all the sessions – so anything you (or I) missed (or simply didn\u2019t understand well enough at the time) will be available to peruse at your leisure. Keep watching the\u00a0conference website<\/a>\u00a0for updates.<\/p>\n

You might want to know that we’ll be running a Haystack EU conference in Berlin on October 28th 2019 – do keep an eye on the\u00a0Haystack website<\/a>\u00a0and\u00a0follow me on Twitter<\/a>\u00a0for more updates.<\/p>\n<\/div>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Last year I attended the Haystack search relevance conference\u00a0in Charlottesville, USA as a guest of our partners OpenSource Connections (OSC). In 2019 we\u00a0merged my old business Flax with OSC\u00a0so I returned as one of the conference organisers. Haystack is a conference all about search relevance – making sure that the results your users see fit… Continue reading Conference Review – Haystack US 2019 – Relevance Avengers Assemble!<\/span><\/a><\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[195,213],"tags":[],"class_list":["post-5971","post","type-post","status-publish","format-standard","hentry","category-conference-review","category-spring-2019","entry"],"_links":{"self":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts\/5971","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5971"}],"version-history":[{"count":0,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts\/5971\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}