Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization

Identifying problem and method sentences is critical for tasks like knowledge graph construction and automated summarization. However, manual annotation is labor-intensive, resulting in small-scale datasets. This scarcity leads models to overfit by memorizing high-frequency 'formulaic expressions' (FEs)—common n-gram phrases like 'was used for'—rather than learning the underlying semantic structure of the research. This paper aims to improve extraction accuracy by expanding training data and better incorporating surrounding context.

The authors introduce two primary innovations to address these challenges:

1. FE Desensitization: To reduce model reliance on rigid patterns, the authors developed data augmenters that identify and replace FEs with placeholders. They tested three selection strategies: dependency-tree-based, LLM-based, and using existing open-source FE lists.
2. Context-Enhanced Transformer: Recognizing that adjacent sentences often contain clarifying information, the authors designed a transformer architecture that uses context as a query to weight the importance of words in the target sentence, effectively enriching semantics while filtering out noise from unrelated adjacent text.

The researchers manually constructed two datasets (abstract-based and full-text) and defined three categories: problem sentences, method sentences, and co-occurrence sentences. Quantitative experiments demonstrated that the proposed model outperformed baselines, achieving macro F1 score improvements of 3.71% and 2.67% on the respective datasets. Additionally, the study evaluated LLM-based in-context learning (ICL) for this task, concluding that current ICL methods are not suitable for high-accuracy problem and method extraction.