Fabriccio: Touchless Gestural Input on Interactive Fabrics

Motivation

As computing becomes increasingly ubiquitous and blends into our everyday devices (e.g. thermostats or speakers), the need to bring interactivity to everyday objects, including those made of soft and lightweight fabrics (e.g., garments, toys, and furniture) has grown significantly. This need has led to advances in sensing techniques that enable input to be carried out on interactive fabrics, such as touching or deforming the fabric.

However, a challenge with existing input modalities is that physical contact with fabric must occur during the interaction. Thus, opportunities are missed for users to utilize other methods, such as touchless (or mid-air) hand gestures, commonly seen on smartphones, smart watches, car infotainment systems, and smart IoT devices. The touchless, mid-air gestures performed by a hand or fingers near a sensor, significantly extends the input vocabularies of interactive fabrics including those carrying special meaning that can’t be replaced by touch (e.g., waving the hand for a greeting). Touchless gestures are also useful in common scenarios where physical contact with a fabric is undesirable by a user (e.g., the hands are unclean when eating or exercising).

In this paper, we bring near-field touchless gestural input to interactive fabric using doppler motion sensing. With our technique, soft objects augmented with a textile motion sensor can detect nearby finger gestures (e.g. in ~10 cm distance) to trigger a desired application. This enables new types of interactions in a variety of contexts. For example, a plush dog toy can make a greeting sound to respond to a child’s wave in front of it. When standing or walking, a user can perform micro finger gestures (e.g., sliding on the index finger using the thumb) with the hand hanging naturally alongside the body to discretely interact with a screen. This type of gesture is subtle, easy to perform, and can now be sensed through trousers, instead of needing heavy, leg mounted depth cameras which are used in current methods for such scenarios.

Applications

Interactive Furniture. The first application we implemented is an integrated media controller for a sofa, where a user controls the media playing on a TV, with gestures performed above an armrest. A swipe gesture can navigate the program, while a push gesture can pause or play media currently playing on the TV. In the scenarios where a user does not want to touch the sofa because their hands are unclean (e.g., when eating), the touchless hand gestures are useful additions to touch input on fabric.

Interactive Clothing. Our second scenario involves turning everyday clothing into interactive wearables. For example, we augmented the logo of a sports shirt with Fabriccio to allow a user engaged in a fitness activity to receive different types of audio information using gestures. For example, a user performing a check-mark gesture above a logo can be used for checking the percentage of their fitness goal, and similarly, a thumb circle gesture can be used for listening to their fitness schedule through the headphone. In another example, we instrumented a pair of pants with Fabriccio on the side. This allows a user to perform subtle arms-down gestures alongside the body to interact with a smartphone.

Selected Press Coverage

Hackster.io: Fabriccio Doppler-Based Touchless Sensor Brings Flexible Gesture Sensing to Interactive Fabrics