MICES 2024

Registration

Welcome & Introduction

René Kriegler / Sebastian Russ
Organisers

Planning of e-commerce search relevance work

Doug Turnbull
Reddit

Is your quarterly search planning a headache? Maybe you're a PM that wants to try the next big AI thingy. Or a techie and you want to explore a cool conference talk. But herein lies a paradox - how can you try these new ideas, without prematurely making a deep investment in time and people? In this talk, I will propose a methodology for rapid prototyping of dozens of search relevance ideas at once. By using the magic of SmallData™ - quick, timeboxed ideas can test whether promise exists before going deeper into building. I’ll talk through how I’ve evolved my thinking of realistic relevance planning within an organization. How to we empower partners / our own curiosity without constantly saying no… Yet at the same time minimizing time spent on work that doesn’t lead to success. I’ll walk through how I think about this with a real e-commerce dataset, thinking through how to maximize the number of ideas we try, not just getting obsessed during the project.

Doug Turnbull has been enthusiastic about search relevance since 2013. He co-authored Relevant Search and AI Powered Search. He created Quepid and Splainer for search relevance testing. He co-created the Elasticsearch Learning to Rank plugin with Wikimedia Foundation and Snagajob. Doug loves learning from other search practitioners, and hopes you'll bring inquisitive curiosity and experiences to this talk.
Doug currently works at Reddit where he's helping bring Machine Learning to search. Recently Doug worked at Shopify to help improve merchant search attributed revenue by 19% year over year. Doug spent 8 years consulting at dozens of organizations during his time as CTO at OpenSource Connections. Doug blogs about search and other topics at http://softwaredoug.com.

Break

Vectorizing consumer electronic goods

Johannes Peter (Principal Search Consultant) & Ruchi Juneja (MediaMarktSaturn)

Comprehensive quality analyses of our keyword-based search revealed that it works very well for the short-head, but has room for improvement in the long-tail area. As a consequence, we decided to enhance our keyword-based search by the vector search approach since it appears to be promising to address the identified long-tail related issues. The goal of our first experimental iteration was to evaluate, whether publicly available models are able to help us when they are fine-tuned with our own data. The outcome of this experiment was that all fine-tuned public models showed an extremely poor search performance, irrespective of size of training data or number of epochs. The reasons were quickly identified: 1) the data and especially the vocabulary of these models are only minimally representative for descriptions of consumer electronic goods, 2) the product descriptions used for the training contained a lot of irrelevant information and 3) the way vector search works out-of-the-box is only partially suitable for e-commerce purposes and requires customization. Consequently, the goals of our second experimental iteration were 1) to train our own model from scratch, 2) to build our own use-case-specific product descriptions and 3) to customize the model layers in order to eliminate undesirable factors like length of description or term frequency. The results of this experimental iteration will be discussed.

Johannes is a passionate leader, consultant and engineer in the areas of search, data and cloud. He is experienced in driving and implementing solutions from initial ideas towards systems running in productive environments and measurably contributing to business KPIs. Throughout his entire career, Johannes has been passionate about open source. In his early days, he contributed several features to Apache NiFi to improve its integration with Apache Solr. Later he became a main contributor and committer for Querqy, a query rewriting solution for Apache Solr and Elasticsearch used by various big retail companies worldwide.

Ruchi works as a Sr. Data Scientist at MediaMarktSaturn in the Search ML team. For over a decade, she has immersed herself in the world of data science working on various business problems. The last five years have truly defined her path to the fascinating realm of search. Her passion for search technologies has led her on a journey across various industries, from education to music and e-commerce, delving deep into the intricacies of search algorithms, specialising in search ranking and query understanding.

How semantic search projects fail

Roman Grebennikov
Delivery Hero

Build embeddings with your preferred vendor and put them into the vector search database. What can go wrong? However, in practice, this is the moment where real problems arise:

- Different embeddings, but the search results are very similar and still bad. Do embeddings matter? Should we opt for OpenAI or open-source?
- Customers signal that “ketchup” is irrelevant to the “tomato” query, but the ML model thinks otherwise. How can we control a black-box embedding? Is there value in fine-tuning?
- Search results are relevant for English queries but not as relevant for Spanish-Arabic-Chinese. How should we handle multilingual searches? What about typos?
- A semantic search always returns results, even for the most wrong and bizarre queries such as ‘how to bake a car’ or ‘why is the sky made of cheese’. When should you stop the search? How can you balance precision and recall?

At Delivery Hero, we made three attempts to implement a semantic search for all 40 countries where we deliver food — only the last one **partially** surpassed the strong lexical search baseline we spent years hand-tuning. In this talk, we’ll discuss challenges and failures we encountered throughout this journey.

Roman is Principal ML engineer at Delivery Hero and an ex startup CTO working on modern search and recommendations problems. A pragmatic fan of open-source software, functional programming, LLMs and performance engineering.

Break

Offline evaluation of product search with model-based judgments at Shopify

Alberto Castelo Becerra
Shopify

At Shopify the efficacy of search systems significantly influences user experience and business performance. Offline evaluation of search is crucial for enabling fast and efficient iteration cycles necessary for continuous improvement. However, the reliance on raw behavioral data such as clicks and orders presents challenges due to its inherent sparsity and multiple biases. In this talk, I will discuss how cutting-edge machine learning models, specifically cross-encoder models, can be utilized to overcome these challenges. We will explore how these models can robustly assess and enhance the output of product search pipelines, ensuring more accurate evaluations and scalable testing before deployment.

Alberto is a Senior Machine Learning engineer at Search Foundations team at Shopify where he has been for the last 3 years. He primarily focuses on leveraging machine learning for search and recommendations problems.

Visual Recommendations: the Good, the Bad, and the Ugly Use-cases

Paul-Louis Nech & Raed Chammam
Algolia

What makes a recommendation stand out? Is it just the quality of the product itself that you see? The relevance of this item in the context of what you’re looking at? Maybe it’s simply that some use-cases shine while others are, hum, interesting? In this talk, Paul-Louis will walk you through several real-life examples using image recommendation systems to provide visual recommendations to various kinds of users. You’ll see use-cases that are really good, some initiatives that are rather bad if you don't take careful steps… and a few experiments which can only be described as ugly. The audience will get tips to find what kind of use-cases would benefit most from Image Recommendations, and how to adapt their data to make the most of it.

Paul-Louis Nech is a Machine Learning engineer with 10 years of experience crafting software for a global audience. Throughout his career, he has leveraged advanced algorithms to empower developers and users alike. From SwiftKey to Algolia, Paul-Louis has built tools that bridge the gap between state-of-the-art algorithms (which work in the lab) and end products (which succeed in the wild). This requires not only great technology, but more importantly great user-experience, to make sure you build something people want.

Raed works as Senior Frontend Engineer at Algolia.

Lunch

User behavior insights

Stavros Macrakis (OpenSearch at AWS) & Charlie Hull (OpenSource Connections)

Every e-commerce search professional needs data about users' behavior. Data is fundamental for analyzing user behavior and improving search relevance, both with manual tuning and with machine learning. The User Behavior Insights system provides a standard way to do that. Until now, collecting fine-grained user behavior data has been haphazard. Many search developers have developed their own ad hoc data collection systems and analysis tools. Proprietary systems collect data but mostly track page-to-page flow, and not granular data on the flow of search results through the system, mostly sharing aggregations with their customers. And open-source systems generally don't offer data collection mechanisms. Our open-source User Behavior Insights (UBI) system provides a client-side library for instrumenting web pages, server-side tools for collecting data, and analytical tools for understanding it. Critically, it defines a standard schema for behavior data so that the community can contribute additional analytical tools. We will share a call to action to the e-commerce search community on the need to make it simpler to seamlessly track, the steps of a user’s search journey in an ethical and safe manner in order to build the experiences of the future.

Stavros is passionate about search relevance. At AWS, Google, GLG, FAST, and Lycos, he has worked with a wide variety of organizations and applications, and is frequently astounded at how even large, sophisticated organizations fail to collect the data needed to evaluate and tune their search systems.

Charlie Hull is the Marketing Director at OSC and also leads client projects. He keeps a strategic view on developments in the search industry and is in demand as a speaker at conferences across the world.

Search & Privacy as One

Ángel Maldonado (Motive.co, Empathy.co, VisiblePrivacy.com) & Alex Barrett (Spaceheater.ai)

AI enhanced Search & Privacy can co-exist when the wealth of possible customer data lives in a neutral space, one that is intrinsically trusted. In this talk, Alex and Angel will share three stories that demonstrate that there are alternatives to centralised AI Search when user rights are considered a priori.

Through these three different retrieval experiences (Customers, Merchants and CPGs), the talk exemplifies a realisation of retrieval that encapsulates privacy, control and agency within the user's computational capsule.

Angel is a computer scientist with a passion for philosophy and design. Founded the co-authoring platform Open Innovation (oi.empathy.co) to finance and bring to market new ventures such as PrivacyCloud.com, Empathy.co, Motive.co, VisiblePrivacy.com or the EthicalAlliance.co. Lives in London and enjoys music and meditation :)

With almost a decade of experience in early-stage consumer-facing Me2B / SaaS / CX related startups, Alex Barrett has put forth a great deal of thought and energy, passionately focused on customer-centric experience, relationships and behaviors. Alex’s latest venture Spaceheater.ai, aims to unlock previously unimaginable digital experiences through the prioritization and empowerment of user privacy and control.

Break

An AI Assistant in the life of a Search Engine Administrator

Lucian Precup & Aline Paponaud
all.site / Adelean

The integration of AI assistants into administrative tools like Kibana and OpenSearch Dashboards has opened new possibilities beyond traditional observability and monitoring functions. In this abstract, we propose an innovative AI assistant tailored specifically for e-commerce administration consoles, focusing on optimizing search engine relevance and performance. Unlike existing assistants, our solution not only responds to natural language commands but also proactively suggests actionable insights to improve user experience. For instance, it can recommend creating filtered redirections based on user behavior patterns or propose synonym enhancements to minimize zero results queries. Moreover, administrators can leverage the assistant to execute complex tasks such as adjusting product visibility or fine-tuning search parameters through intuitive dialogue. Our presentation will delve into the architectural requirements for implementing such an assistant effectively, as well as outline a comprehensive set of capabilities and recommendations for maximizing its utility in real-world scenarios. Join us to explore the future of AI-driven e-commerce administration and discover how our innovative approach can revolutionize search engine management.

Lucian is the CTO of all.site - the collaborative search engine developed at Station F in Paris. With his colleagues at Adelean, Lucian develops solutions for indexing, searching and analyzing data. Lucian regularly shares his knowledge in specialized conferences and organizes the Search & Data Meetup.

Aline works with many customers on implementing search engines, helping them improve efficiency and use it for various use cases from search to observability. She is CTO of Adelean.

Barcamp / self-organising sessions