ResearchPod Summary
Traditional input devices like physical keyboards and mice can be cumbersome for mobile users, while existing virtual keyboard alternatives often require expensive hardware, specialized projectors, or complex AR/VR setups. This paper investigates whether a practical, budget-friendly virtual keyboard can be constructed using only a standard camera and a printed paper keyboard layout.
The proposed system consists of two primary modules: a keyboard processing module and a touch processing module. The keyboard module uses YOLOv8n-seg to segment the keyboard region from a video feed, followed by a homography transformation to map the paper into an orthogonal view. This allows the system to identify individual keys using a fine-tuned YOLOv8n model. The touch processing module detects keystrokes by analyzing the color of the user's fingernail, which changes due to blood displacement when pressure is applied to a surface. The system also uses Google Mediapipe to track fingertip movement, triggering the color analysis only when a potential touch event is detected.
Experiments showed that the system is highly effective at identifying the keyboard region (92% average precision) and individual keys (70% average precision). However, the touch detection accuracy remains limited at 36%, primarily due to environmental factors such as lighting intensity and the difficulty of maintaining consistent camera angles for all fingers. Despite these technical challenges, a user study with 20 participants indicated that users found the system interesting and functional for basic text entry tasks.
This research demonstrates a path toward highly accessible, zero-cost human-computer interaction. By relying on ubiquitous hardware—a standard webcam and a piece of paper—the system eliminates the barrier to entry posed by expensive AR headsets or specialized depth-sensing cameras, making virtual input more viable for everyday mobile computing.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.