Over the last two decades most of my search-related projects have been with large multi-national companies so an invitation to participate in a NATO working on enterprise and federated search in March 2019 was irresistible. The fact that it was to be held in a 5 star hotel on the Adriatic Sea at Split, Croatia had nothing to do with my acceptance!. The event was an annual European Think-Tank for Information, Decision and Execution Superiority (TIDE) Sprint. There is a related event in the USA later in the year. Given that it was a NATO event and not open to the public I hope you will appreciate that I am not going to get into too much detail about what was disclosed and discussed.
TIDE had multiple technical streams, all focused on developing some specific deliverables around standards and good practice. One of these streams was on enterprise and federated search, which was on the agenda for the first time. The main reason for this two-day stream was that NATO staff had been working on a standard for federated search in a mission (think Afghanistan) situation. In a mission member states of NATO come together to work towards a resolution of the conflict. The emerging requirement was for member states not only to share information but to enable staff to search across the information repositories of those states on the mission using a federated search application.
A considerable amount of work had been undertaken on a draft standard and the TIDE event was an opportunity to present it to NATO staff and member state representatives, and also to gain the perspective of vendors and service providers. In all there were around 50 people in the room. On the first day speakers from Findwise, Datafari, Semantic Web Company, Amazon AWS and Microsoft provided an introduction to the current state of enterprise search. My own contributions were to walk people through the modules of a typical search application and also provide a vendor-neutral view of the future of enterprise search. In the afternoon I facilitated a discussion between NATO staff and some of the vendors and consultants to define mission requirements. What was missing from the draft standard was any context for the requirement. The second day was then a deep dive into federated search architecture options.
Among the issues that emerged were the ownership of the search index (if a member state left what happened to their information – did they take it with them?), security management, and whether the query time architecture proposed in the standard was flexible enough to meet mission requirements. This discussion introduced me to the Open Search standard. I was aware of the standard but certainly not at a level needed to make a contribution. What was notable was the very open discussion environment. No one tried to push a particular agenda and as a result the discussions were lively and very productive.
The outcome was an agreement that the standard should be less prescriptive in terms of the architecture. As the discussions moved into the third (unplanned!) day it became clear to all concerned that index time, query time and hybrid approaches all had merits and challenges. This change of direction is now going to be discussed internally, from which a revised version of the standard will emerge. This change of approach was led by Fredric Landqvist (Findwise), Tomasz Sobczak (Findwise) and Cedric Ulmer (Datafari). All of us were working on a pro bono basis but the learning we gained from our participation in the event was well worth the investment.
An important outcome of the event was the compilation of a Record of Exploration of the presentation and discussions. This ended up as a 70pp report providing NATO staff across the entire organisation with a very comprehensive account of the current status and future directions for enterprise search technology. The report was compiled by Tomasz Rakowski, who set up the event and acted as the overall facilitator. Sadly it is not a public document.
This was without doubt the best enterprise search event I have participated in both in terms of the quality of the presentations and the strong focus on a specific deliverable instead of a collection of case studies. Somewhat to the surprise of some of the NATO staff the consultants present worked together as a very harmonious team. To be sure we had different perspectives but the commonality was a grounding in understanding user requirements, something that was novel to the largely NATO IT community participants. I’m seeing a lot of interest in federated search now that the vision of one-search-to-search-them-all is recognised as not being achievable. Looking through my collection of over 1000 research papers broadly related to enterprise search there are only a few that consider federated search. The conference was especially timely as I have been running a project on the development of a federated search strategy for a large chemical company. Indirectly the client has benefited considerably from this NATO initiative though I can’t tell them how!
I will finish by thanking Tomasz Rakowski for inviting me to the event and entrusting me with leading the requirements-gathering session without ever having met me or seen me in action.