TCG-AR: Real-Time Multi-View Augmented Reality for Trading Card Game Streaming

Trading card game (TCG) streams are typically limited to flat, top-down camera views that offer little visual engagement for spectators. While some high-end productions use specialized hardware—such as embedded chips or instrumented tables—to overlay virtual creatures onto cards, these solutions are costly and inaccessible to casual players. This paper asks: can we achieve real-time, markerless augmented reality (AR) for TCG streaming using only standard, off-the-shelf RGB cameras?

The authors introduce TCG-AR, a pipeline that treats TCG streaming as a game-state recognition problem. The system operates on two concurrent paths: a recognition path that updates the game state (card identity, position, and orientation) from a zenithal view, and a rendering path that overlays virtual models onto cards across all camera feeds.

To avoid the high cost of manual data labeling, the authors developed an automatic, game-agnostic procedure that generates synthetic training data from a reference set of card images. This allows the system to be adapted to any TCG by simply providing a new reference set. The detection model uses an oriented R-CNN to handle cards at any angle, while the identification model employs an ArcFace-based metric learning approach to retrieve card identities from a large reference database in real time.

TCG-AR successfully demonstrates that high-quality AR overlays are possible without specialized hardware. The system maintains a fluid 30 FPS rendering rate while refreshing the game state at approximately 10 FPS on consumer-grade hardware (NVIDIA RTX 2080 Ti). The use of synthetic data for training proved highly effective, with the detection model achieving a 0.90 mAP on real-world evaluation data. Furthermore, the identification stage is robust enough to correctly recognize cards even when they are partially occluded by damage counters or other game markers.