ShapeTalk: Combining Natural Language and Sketch for Time-Series Pattern Querying

ShapeTalk addresses the challenge of searching for specific patterns in univariate time-series data, a task common in fields like finance and climate science. Traditional tools often force users to choose between rigid structured filters or precise sketches, both of which struggle with vague or composite search intents. ShapeTalk introduces a coordinated, multimodal approach that treats natural language and sketching as complementary tools rather than a single fused input stream.

The system centers on an LLM-based semantic parsing pipeline that translates free-form natural language into interpretable, editable shape-feature constraints. Users can describe patterns—such as a "sharp drop followed by a plateau"—and the system converts these into quantifiable local and global features. Because the system makes these feature constraints explicit, users can inspect and manually adjust them if the initial interpretation is incorrect.

ShapeTalk allows for fluid, iterative refinement. A user might start with a natural language query to get a broad set of results, then switch to a sketch to refine the geometric details of the pattern. Conversely, a user can sketch a shape and ask the system to generate a natural language description of it, bridging the gap between visual intuition and formal query parameters. This design ensures that the system remains adaptable to different datasets without requiring a fixed vocabulary of pre-defined patterns.

By providing a transparent and editable interface, ShapeTalk lowers the barrier for exploratory data analysis. It allows analysts to express complex, partially formed ideas in their own words while retaining the precision of geometric sketching, ultimately making time-series pattern discovery more accessible and robust to user error.