Apple patent filing involves using hand gestures to interact with computer-generated environments

Apple has filed for a patent (number 20210286502) for “devices, methods, and graphics user interfaces for providing computer-generated experiences.” It involves interaction with virtual reality (VR) and augmented reality (AR) on devices such as the Mac with, for instance, hand gestures. It also involves the rumored “Apple Glasses,” a VR/AR head-mounted display (HMD).

About the patent filing

In the patent filing, Apple notes that the development of computing systems for AR has increased significantly in recent years. Examples of AR environments include at least some virtual elements that replace or augment representations of the physical world. 

Input devices, such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch-screen displays for computing systems and other electronic computing devices are used to interact with virtual/augmented reality environments. Examples of virtual elements include virtual three-dimensional objects, digital images, video, text, icons, and control elements such as buttons and other graphics. 

However, Apple says that methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, AR environments, mixed reality environments, and VR environments) are “cumbersome, inefficient, and limited.” 

The tech giant adds that, for example, “systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment, and systems in which manipulation of virtual objects are complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment.”

In addition, Apple says that these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. 

In addition, many systems that provide VR and/or mixed reality experiences use HMDs that physically shield the users’ faces from their surroundings, and hinder social interaction and information exchange with the outside world…