Mahyar Ghazanfari, Amin Tabrizian, Armin Mehrabian, Peng Wei
4 min
Earth observation (EO) research is inherently combinatorial, requiring researchers to fuse diverse datasets—such as atmospheric chemistry profiles with cryospheric ice-velocity measurements—to test scientific hypotheses. With over 8,000 NASA datasets available, the space of potential pairings is vast and largely unexplored. This paper asks: can we automate the discovery of scientifically coherent, novel dataset pairings and articulate them into structured, testable research hypotheses?
The authors propose a three-agent pipeline grounded in the NASA Earth Observation Knowledge Graph (EO-KG). First, a heterogeneous graph neural network (GNN) trained on historical dataset co-usage relations ranks candidate pairs. Second, a three-agent LLM cascade (Filter, Generator, Judge) processes these candidates. The Filter agent assesses plausibility and novelty, the Generator agent articulates a structured hypothesis (question, testable claim, method, expected finding), and the Judge agent evaluates the output. To ensure robustness, the authors conduct a 2x2x2 factorial experiment across two LLM backbones (GPT-5.2 and Claude Sonnet 4.6) to decompose how much score variance is attributable to the model identity versus the hypothesis quality.
The pipeline successfully generated 160 structured hypotheses across diverse domains, including glaciology and ecohydrology. A key finding is that the GNN-predicted novel pairings are rated by LLM judges as nearly as plausible as actual co-usages observed in 2024 publications. However, the factorial experiment reveals a critical limitation: while hypothesis rankings are stable across different LLM backbones, absolute scores are highly sensitive to the judge's identity. Specifically, judge identity accounts for 25% of the variance in importance scores, while the choice of dataset context (blind vs. contextual) is the primary driver of variance in tractability scores.
This work shifts the paradigm of AI-driven discovery in observational science from unstructured literature retrieval to structured, data-grounded ideation. By pinning every hypothesis to specific, actionable NASA datasets, the pipeline produces outputs that are ready for immediate investigation. The study also provides a necessary cautionary tale for the scientific community regarding the use of LLMs as evaluators, demonstrating that absolute scores from a single LLM judge are not portable and should be interpreted with caution.
Large language models have recently been explored for scientific hypothesis generation, but most prior work relies on unstructured literature and free-form textual claims. We present a pipeline for Earth observation that grounds hypothesis generation directly in the NASA Earth Observation Knowledge Graph. A heterogeneous graph neural network trained on historical co-usage relations ranks candidate dataset pairings, and a three-agent LLM pipeline filters, generates, and evaluates structured research hypotheses. Applied to 1,475 NASA datasets, the system produces 160 hypotheses spanning multiple Earth-science domains, including ecohydrology, glaciology, aerosol--cloud interactions, vegetation phenology, and stratospheric chemistry. Model-predicted novel dataset pairings are rated nearly as plausible as held-out real co-usages from the literature, indicating that the pipeline surfaces scientifically coherent yet unexplored combinations. A 2*2*2 factorial experiment across GPT-5.2 and Claude Sonnet 4.6 shows that hypothesis rankings remain stable, while absolute scores depend strongly on judge identity, highlighting limitations of single-judge LLM evaluation.
Sam: [realizing] So the absolute scores are essentially not portable across judge configurations. A hypothesis that scores highly under one model might look mediocre under another, and you can't straightforwardly compare runs that used different judges.
Alex: [calm, precise] Correct. But the more practically useful finding is that the *rankings* are robust. The relative ordering of hypotheses stays consistent even when absolute scores shift across judge configurations. So a researcher using this system shouldn't be reading the scores as cardinal values — they should be treating the output as a ranked shortlist. The top-tier recommendations are stable; the numerical magnitudes are not. [[RP_SECTION:practical-deployment-considerations|Practical Deployment Considerations]]
Sam: [analytical] That's an important distinction for anyone thinking about deploying this. It means the system is defensible as a prioritization tool, but you'd want to be cautious about any downstream process that treats the scores as ground truth — like an automated funding filter or a threshold-based triage system.
Alex: [concluding] That's the right framing. The pipeline transforms what would otherwise be an intractable survey problem into a structured retrieval task. It doesn't replace the scientist's judgment — it compresses the search space so that judgment can be applied where it actually matters. The limitation to keep in mind is that the evaluation framework itself has a meaningful judge-dependence, and the paper is transparent about that. For a careful referee, the next question would be whether the ranking stability holds under distribution shift — that is, when the judge models are updated or replaced with architectures the system wasn't calibrated against. That's left open.
Sam: [nodding] So the honest summary is: a well-motivated graph-based grounding mechanism, a plausibility result that holds up under scrutiny, and an evaluation design that's candid about its own variance structure. The tool is real, but the scoring layer needs to be handled carefully.
Alex: [settled] That's it. And for anyone working in Earth observation or adjacent fields with large heterogeneous data catalogs, the architectural pattern here — using co-usage graphs to constrain LLM ideation — is probably the transferable contribution, independent of the specific NASA context. Thanks for listening to ResearchPod.