Ranking Companion: A Visual Analytics Approach to Item-Based Ranking with Hybrid Item Selection

Personalizing item rankings is difficult because users often struggle to articulate their preferences through abstract attribute scoring. While item-based ranking—where users provide feedback on concrete items—is more intuitive, existing systems typically rely on a single, fixed method for selecting which items a user should evaluate. This research asks how a hybrid approach, combining multiple complementary item-selection strategies, can improve user control and the quality of personalized rankings.

The authors developed Ranking Companion, a visual analytics system that integrates a closed-loop machine learning workflow. The system supports three core tasks: selecting candidate items, externalizing listwise preferences, and inspecting the resulting ranking. To avoid the limitations of single-method selection, the system offers six distinct item-selection methods (ISM):

• Baseline Search: User-driven retrieval of known items.
• Similarity: Suggests items similar to those already highly ranked.
• Trend: Suggests items based on external dataset popularity.
• Uncertainty: An active learning approach that targets items in regions where the model lacks evidence.
• UMAP Projection: A spatial map for visual discovery.
• Ranking Panel: Allows users to select items directly from the current ranking output.

Users provide feedback by ordering selected items via drag-and-drop, which the system uses to retrain a LightGBM ranking model. The system also provides global explanations using SHAP values to help users understand which attributes are driving the model's decisions.

A formative user study with 10 participants demonstrated that providing multiple, complementary item-selection methods allows users to navigate the trade-offs between different ranking goals. Participants reported that different methods were better suited for different stages of the ranking process—for example, using search for bootstrapping and uncertainty-based selection for model refinement. The study suggests that a unified interactive space for item selection increases user agency and helps mitigate the "filter bubble" effects common in systems that rely on a single selection strategy.