Kotaro Oishi, Kazuma Ikeda, Ryosei Hara, Ryo Yoshida, Mariko Isogawa, Kentaro Yoshioka
8 min
Abstract
LiDAR is a critical sensor for autonomous driving and robotics, yet it remains vulnerable to adverse conditions such as fog, rain, transparent objects, and low-reflectance surfaces. Full-waveform LiDAR (FWL) addresses these limitations by capturing the complete return waveform as a histogram, preserving rich reflection characteristics including multi-path echoes and weak signals. While recent Transformer-based approaches have demonstrated promise on FWL data, two fundamental bottlenecks remain: high computational cost from redundant tokens in low-intensity background regions, and limited generalization due to scarce annotated training data. We address both bottlenecks by exploiting the physical structure inherent to FWL data. First, we propose FWL-ToPM, a token pruning and merging mechanism that leverages intensity distributions to selectively reduce tokens while retaining critical waveform content. Second, we introduce FWLAug, a data augmentation framework designed as a regularization method that preserves waveform structure, retaining the Time-of-Flight waveform of each ray and the frustum geometry, to improve generalization to unseen scenes. Evaluated on the Ghost-FWL benchmark, FWL-ToPM achieves over 8 times speedup at real-time throughput alongside a nearly 9-point F1 gain over the prior state of the art, establishing the first real-time FWL Transformer that simultaneously improves accuracy. FWLAug further improves performance on both the standard benchmark and an out-of-distribution test set of unseen scenes. Together, these methods push FWL-based perception toward practical deployment. The code and the OOD dataset are publicly available and can be accessed via the project page: https://keio-csg.github.io/FWL-ToPM
Alex: What is the actual decision rule? Could another researcher implement the basic idea without a learned selector?
Sam: Each patch gets a score: its highest signal intensity. The method ranks patches by that score and removes the lowest-ranked fraction before the encoder, which extracts their features. In the main configuration, it prunes seventy percent of patches.
Alex: That sounds close to keeping only the strongest returns. Wouldn’t that lose the richer information that justified recording the waveform?
Sam: It selects patches, not individual peaks. One example in the document contains a glass return and strong ghost returns along the same ray. Keeping only the strongest peaks drops an object return. Their patch-level pruning retains that object return, together with the waveform inside its patch.
Alex: So the distinction is a fixed peak budget along each ray versus selecting regions across the frame. What happens to regions that are removed?
Sam: During training, their classification losses are excluded. During inference, every location in a removed patch is labeled noise. That makes the method efficient, but it also makes the intensity threshold a consequential decision.
Alex: And after pruning, there is still information to combine. How do they avoid treating the surviving patches as a disconnected pile?
Sam: They order the surviving patches along a path through the original three-dimensional grid. Nearby patches tend to remain nearby in that ordering. Then they select representative tokens at regular intervals and combine sufficiently similar tokens with those representatives for the attention calculation.
Alex: That keeps representatives distributed through the surviving data, rather than concentrated in one region. Does the evidence show both stages are necessary?
Sam: The component comparison is useful here. Pruning alone delivers almost all the accuracy improvement. Merging alone helps accuracy less, but also speeds processing. Together, they reach about fifty-three full frames per second, compared with roughly six and a half for the previous leading method.
Alex: What exactly does “better recognition” mean in that comparison? We should distinguish a classification gain from improved downstream driving.
Sam: They measure F1, which balances precision and recall, averaged across the non-noise classes. The combined method improves that score by about nine points over the prior leader. These are classification results, not tests of driving performance or robotic safety.
Alex: Does the comparison isolate token reduction, or could the new backbone account for the whole improvement?
Sam: Their own model without reduction already outperforms that prior leader in accuracy, though it is slower. Adding the proposed components improves its accuracy further while greatly increasing throughput. They also compare against adapted image-based pruning and merging methods, which perform worse in these experiments.
Alex: That supports the computational argument. But scarce training data was the other problem. How do they add variety without corrupting the measurements?
Sam: Their augmentation framework, FWLAug, preserves each ray’s temporal waveform and respects the sensor’s viewing geometry. One operation pastes regions from another scene, preferentially selecting glass, an underrepresented class. Crucially, it copies the entire waveform, not an arbitrary slice through return time.
Alex: And the other operation changes distance. How do they do that without just sliding an object around inside a data array?
Sam: They use the return timing to locate points in three-dimensional space, move them along the sensor rays, and shift their waveforms accordingly. Regions left blank are filled with waveform data from another scene. The authors argue that leaving zeros would introduce unrealistic training artifacts.
Alex: Pasting glass into a scene could change the reflections themselves. Are these synthetic examples physically accurate simulations?
Sam: No, and the authors explicitly say they are not. These are regularization methods intended to improve learning. They don’t recompute all light paths or distance-dependent intensity attenuation, and pasted boundaries can introduce discontinuities.
Alex: Then the strongest test is whether the trained model handles genuinely unfamiliar scenes. What did they hold out?
Sam: They created a separate-location test set containing five hundred frames, with dense glass configurations absent from the original benchmark. The sensor and annotation procedure stayed the same. Using both augmentations improved average F1 there by about five points over training without augmentation.
Alex: Does combining them win every comparison, or is the benefit more specific?
Sam: It wins the average score on that unfamiliar-scene test. The pasting operation alone does better on glass specifically, while the distance-shifting operation alone does better on the original benchmark’s average. So the combined method is not uniformly best across every reported outcome.
Alex: What should make us most cautious about transferring this to an actual robot?
Sam: Weak returns can matter. The authors warn that pruning may remove distant objects, dark surfaces, or weak weather-affected reflections. Those patches become noise predictions, potentially creating false negatives. And their speed measurement covers the model, not the complete sensor-to-decision pipeline.
Alex: Given those boundaries, who should read the full document, and where should they start?
Sam: Researchers in full-waveform perception and efficient Transformers should start with the pruning-and-merging ablation, then the augmentation comparisons and limitations. The evidence supports the approach on this benchmark and unfamiliar scenes from the same sensor platform. It does not establish cross-sensor robustness or embedded deployment. For everyone else: removing background can improve both speed and accuracy, but weak signals are not automatically unimportant.
Alex: That distinction is worth carrying into the next model you build.