Today was a great day for patents granted related to Apple’s future headphone device(s). The US Patent and Trademark Office has officially published a series of five newly granted patents covering various attributes that can be incorporated into a mixed reality headset. For those of you who follow the news of future Apple headphone(s), there are plenty of things to dive into for rich details such as the use of head motion sensors referred to as “Head Odometers”; trace the look with the help of gloss; Camera-based gaze tracking using neural networks and more.
Eye tracking using head motion sensors or ‘head distance meters’
Today, the United States Patent and Trademark Office officially granted Apple a patent titled “Eye Tracking Using Low Resolution Images” relating to various methods and devices for eye tracking in virtual and mixed or augmented reality (VR/AR) applications using Head motion sensors are referred to as “vertical odometers”.
The key to providing accurate eye tracking is knowing the location of the user’s eyes in relation to eye tracking cameras. In some embodiments of the eye-tracking system, to accurately determine the location of the user’s eyes in relation to the eye-tracking cameras, the controller may implement an algorithm that performs a three-dimensional (3D) reconstruction using images captured by the eye-tracking cameras to create 3D models of the user’s eyes.
3D models of the eyes indicate the 3D position of the eye in relation to eye-tracking cameras, allowing eye-tracking algorithms implemented by the controller to accurately track eye movement.
Methods and devices for tracking the relative motion of the device with respect to the user’s head are described where sensors (hereinafter referred to as head motion sensors or head odometers) are placed in one or more positions in or on the device, for example in or near the user’s ears to track The pitch is primarily at or near the bridge of the nose to primarily track the y movement.
Signals from vertical odometers can be used to detect the movement of the device in relation to the user’s eyes. This may allow 3D reconstruction to be performed only when the device motion is detected in relation to the user’s eyes, which greatly reduces the power consumption by the eye tracking system.
Apple FIG patent. Figure 2 below shows an example VR/AR HMD implementing an eye tracking system that includes sensors to detect the movement of the HMD in relation to the user’s eyes.
For more information on this topic, see the patent granted by Apple 11,442,540.
Glint-Assisted Gaze Tracker. Glitter-assisted gaze tracker
The granted Apple patent covers methods and devices for luminescence-assisted gaze tracking in head-mounted VR/AR displays (HMDs). Images captured by gaze-tracking cameras may be entered into the flicker detection and pupil detection process, for example performed by one or more processors from an HMD controller.
The process of detecting flicker may in pictures and pass the scintillation information to the pupil detection process, where the flashes detected can be used to detect the location and surroundings of the pupil.
Luminosity information can also be passed through the luminescence detection process to an LED flicker matching process that matches the detected luminosity with certain light-emitting elements of the gaze tracking system. The results of the glint-LED matching process (detected luminosity and LED correspondence) and the pupil detection process (pupil ellipse detection) are passed to the gaze estimation process, eg performed by one or more processors from the console, to estimate the user’s point of staring.
Fig Apple patented. 9A through 9C graphically shows the matching of glint-LED in the image space; Figs. 11A and 11B show an example of luminosity matching results in the image space method compared to the results when applying the luminosity matching method in 3D space to detect and correct potential mismatches using luminosity matching in the image space method.
Three additional patents for MR stethoscopes granted today include:
11442271Display lighting systems
11,442,539Camera-based gaze tracking using neural networks
11,442,543: Electronic devices capable of estimating monocular