Junli Liu, Zhongrui You, Chenhui Li, Zhen Li, Yuyang Tian, Qizhen Chen, DeLin Qin, Haoming Song, Dong Wang, Xuelong Li, Zhigang Wang, Bin Zhao
8 min
Abstract
Human cognition flexibly draws on a shared perceptual foundation to understand environments, attend to referenced entities, reason about spatial relationships, and act. Endowing aerial robots with an analogous functional organization remains challenging, as existing systems are often specialized for individual capabilities, dependent on offboard computation, or unable to sustain closed-loop operation on resource-constrained platforms. We present OpenFlyV2, a unified onboard vision-language-action architecture that uses a shared vision-language representation to support semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control within a single model. A compact action expert converts the shared multimodal context into short-horizon flight waypoints, while motion-aware temporal memory and action priors reduce the computational cost of closed-loop execution. We construct \dataset{}, a simulation-based dataset comprising approximately 70,000 navigation trajectories and 60,000 vision-language instruction--response pairs. OpenFlyV2 generates action chunks, each comprising 12 waypoints, entirely onboard at 10 Hz using an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to physical quadrotor flight, achieving a 48\% navigation success rate across four real-world environments without real-world aerial training data. Extensive results demonstrate that broad vision-language competence and continuous closed-loop flight can coexist within a single model on a resource-constrained aerial platform. Code will be made publicly available at https://github.com/Ideal-ljl/AeroBrain.
Alex: Take the document’s example: fly toward the red clock tower and stop nearby. As the view changes, the model needs to connect the landmark instruction with current and earlier observations. The memory supplies motion-linked visual context, while the current image remains directly available.
Sam: Compression can save computation while throwing away useful information. Did they test whether this memory actually helps navigation?
Alex: In supervised comparisons, replacing historical images with event tokens cut the input token count by at least half. Tokens are the units the model processes. The memory also improved every reported navigation metric at both tested action-refinement budgets, while representing a longer flight history.
Sam: That addresses looking backward. What makes generating the next movement cheaper?
Alex: Their Motion-Structured Prior supplies a learned starting pattern for the proposed movement. The authors found that short flight segments concentrate around recurring shapes. They cluster training trajectories into prototypes, then select and adjust one using the current context.
Sam: Does choosing a prototype restrict the drone to a small menu of maneuvers? That could make computation cheaper at the expense of flexibility.
Alex: The prototype initializes the prediction; it does not fix the final path. An action expert refines that starting proposal using the full visual and language context. The proposal includes both movement and stopping decisions, and the expert can move away from the selected pattern.
Sam: So instead of starting from an unstructured proposal, it starts nearer a plausible flight segment. Does that advantage survive when you allow more refinement?
Alex: Not uniformly. The prior’s advantage was concentrated at the single-refinement budget used for deployment. With additional refinement, memory alone had the strongest supervised success rate. That makes this a result about the available computation, not a claim that adding the prior always improves navigation.
Sam: Before we get to physical flight, where did the training experience come from? “Simulation-trained” can cover quite different kinds of data.
Alex: They combined engine-authored environments with scenes reconstructed from real imagery using a rendering method called Gaussian splatting. All task-specific training observations were rendered, and navigation actions came from simulation. Language annotations were generated automatically and then checked by human reviewers; only approved examples entered the final dataset.
Sam: And did learning to answer questions actually coexist with learning to fly, rather than one capability replacing the other?
Alex: The model trained jointly on navigation and aerial question-answering tasks, including distance estimation and locating described objects. Later, navigation received reinforcement post-training: learning from simulated rollout outcomes. Only the action expert changed during that stage; the shared visual and language components stayed fixed.
Sam: What does the navigation comparison show beyond an attractive demonstration?
Alex: In unseen simulated environments, OpenFlyV2 succeeded in about 53 percent of episodes. DualVLN, one of the comparison models, reached about 37 percent on the same episodes. OpenFlyV2 also led the tested methods in seen environments and on measures incorporating path efficiency.
Sam: What counted as success there? Passing close to the destination could hide a problem with knowing when to stop.
Alex: Simulation success required an explicit stop within the specified goal region. The authors also measured whether the drone entered that region at any point. In the supervised component tests, reaching the region was more common than successful termination, exposing exactly that stopping gap.
Sam: Now the central claim: did this run entirely aboard an aircraft, or was some of the decision-making still remote?
Alex: Navigation inference ran aboard a DJI aircraft equipped with an NVIDIA Jetson Orin NX. The model generated chunks of twelve waypoints, meaning successive flight targets. It produced those chunks at ten hertz, with new observations feeding subsequent decisions.
Sam: Generating another chunk while flying the previous one sounds like a coordination problem of its own.
Alex: They handled that asynchronously: the next chunk was generated while the current chunk was executing. Training taught the model to continue from commands already committed during that inference interval. A low-level controller then tracked the resulting waypoints.
Sam: How convincing were the actual flights, especially beyond places represented during training?
Alex: Across one hundred physical trials, the pooled success rate was 48 percent. Those trials mixed single-landmark and sequential-landmark instructions, across locations both represented and unrepresented in training. After the initial instruction, flights proceeded without manual intervention.
Sam: Then “no real flight training” does not mean every test location was completely unfamiliar. Some locations had contributed reconstructed scenes.
Alex: Correct. There was no additional training on real flight trajectories or test-site fine-tuning, but some locations were represented through rendered reconstructions. The pooled result does not isolate performance at unseen locations. Physical success was manually assessed, and its criterion differed from the simulation distance threshold.
Sam: And the competing models were tested only in simulation, because they could not run on this onboard hardware. So physical feasibility is demonstrated more directly than comparative superiority outdoors.
Alex: That is my reading. The paper supports onboard integration, with useful simulation comparisons behind it. It does not establish reliable field navigation; the authors identify detecting deviations, recovering from mistakes, and resuming navigation as unresolved directions.
Sam: Who should read the full document, and where should they start?
Alex: Researchers in aerial robotics and resource-constrained robot learning should start with “History compression and single-step action generation.” Then read the physical evaluation protocol and onboard deployment methods. Those sections separate the component evidence from the field claim. For everyone else: language-guided flight can run onboard, but running onboard is not the same as flying reliably.
Sam: Keep that distinction in mind.