Image stabilization for color-sequential displays

• US 8,970,495 B1
• Filed: 08/21/2012
• Issued: 03/03/2015
• Est. Priority Date: 03/09/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Biffle, Clifford L., Weaver, Josh
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): ONYEKABA, AMY
• Application Number:
  US13/590,378
• Time in Patent Office: 924 Days
• Field of Search: 345 7- 8, 345690-699, 348 42- 60
• US Class Current: 345/158
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G09G 2310/0235 Field-sequential colour display
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/08 Cursor circuits

Abstract

A computing system that includes a color sequential display (CSD) and an eye-tracking device may determine motion of an eye with respect to the CSD, while displaying temporally sequential color sub-frames of a video display image on CSD. Based on the determined motion, the computing system may determine an expected change in position of the eye with respect to the CSD over a time interval from a first temporally sequential color sub-frame to a second temporally sequential color sub-frame that follows the first temporally sequential color sub-frame. Then, based on the determined expected change in position, the computing system may compensate for the determined motion of the eye by adjusting a display position on the CSD of the second temporally sequential color sub-frame with respect to a display position on the CSD of the first temporally sequential color sub-frame. A head-mounted display is an example of such a computing system.

Claims

1. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

while displaying temporally sequential color sub-frames of a video display image on a color-sequential display (CSD) of the HMD, determining motion of an eye of a user of the HMD with respect to the CSD;

based on the determined motion, computing an expected change in position of the eye with respect to the CSD over a time interval from a first temporally sequential color sub-frame to a second temporally sequential color sub-frame that follows the first temporally sequential color sub-frame; and

based on the computed expected change in position, compensating for the determined motion of the eye by adjusting a display position on the CSD of the second temporally sequential color sub-frame with respect to a display position on the CSD of the first temporally sequential color sub-frame,

wherein computing the expected change in position of the eye with respect to the CSD over the time interval from the first temporally sequential color sub-frame to the second temporally sequential color sub-frame based on the determined motion comprises;

determining a predicted change in angular orientation of an eye-gaze direction of the eye with respect to a plane of the CSD as a function of the determined motion over the time interval from the first temporally sequential color sub-frame to the second temporally sequential color sub-frame.

2. The method of claim 1, wherein the second temporally sequential color sub-frame is followed by a third temporally sequential color sub-frame, the method further comprising:

based on the determined motion, computing a subsequent expected change in position of the eye with respect to the CSD over a time interval from the second temporally sequential color sub-frame to the third temporally sequential color sub-frame; and

based on the computed subsequent expected change in position, adjusting a display position on the CSD of the third temporally sequential color sub-frame with respect to the adjusted display position on the CSD of the second temporally sequential color sub-frame.

3. The method of claim 1, wherein the video display image comprises temporally sequential frames, each of the temporally sequential frames comprising a temporal sequence of N color sub-frames corresponding to N display colors,

and wherein displaying the temporally sequential color sub-frames of the video display image on the CSD comprises displaying the temporal sequence of the N color sub-frames of each of the temporally sequential frames.

4. The method of claim 3, wherein N=3, and the N color sub-frames of each of the temporally sequential frames comprise a red (R), a green (G), and a blue (B) color sub-frame of each of the temporally sequential frames.

5. The method of claim 3, wherein the CSD comprises a display region bounded by a display border,

wherein the second temporally sequential color sub-frame is one of the N color sub-frames of a given temporally sequential frame,

and wherein the method further comprises:

determining that adjusting the display position on the CSD of the second temporally sequential color sub-frame will cause at least part of the second temporally sequential sub-frame to be positioned outside of the display border, and thereby cause a portion of the display region to display fewer than the N display colors for the given temporally sequential frame; and

in response, cropping the portion of the display region in which the fewer than the N display colors for the given temporally sequential frame will be displayed.

6. The method of claim 1, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises tracking motion of the eye with an eye-tracking camera of the HMD.

7. The method of claim 1, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises determining angular motion of an eye-gaze direction of the eye of the user with respect to the CSD.

8. The method of claim 1, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises:

measuring motion of the HMD with a motion detector of the HMD; and

analytically converting the measured motion of the HMD into computed motion of the CSD with respect to the eye of the user of the HMD.

9. The method of claim 8, wherein the motion detector comprises a three-axis accelerometer and a three-axis gyroscope,

and wherein measuring motion of the HMD comprises measuring both angular movement and linear movement.

10. The method of claim 1, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises:

tracking motion of the eye with an eye-tracking camera of the HMD;

measuring motion of the HMD with a motion detector of the HMD; and

analytically combining the tracked motion of the eye with the measured motion of the HMD into computed motion of the CSD with respect to the eye of the user of the HMD.

11. The method of claim 1, wherein the CSD is configured to display the temporally sequential color sub-frames at a sub-frame rate,

and wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises measuring motion of the eye of the user at a measuring rate that exceeds the sub-frame rate.

13. The method of claim 1, wherein the expected change in position corresponds to a change from (i) a first eye-gaze direction of the eye in a plane of the CSD at a time of the first temporally sequential color sub-frame to (ii) a second eye-gaze direction of the eye in the plane of the CSD at a time of the second temporally sequential color sub-frame,

and wherein adjusting the display position on the CSD of the second temporally sequential color sub-frame based on the computed expected change in position comprises displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction.

14. The method of claim 13, wherein the plane of the CSD comprises a two-dimensional pixel array,

wherein the first eye-gaze direction corresponds to a first pixel location in the pixel array, and the second eye-gaze direction corresponds to a second pixel location in the pixel array,

and wherein displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction comprises offsetting a display position of the second temporally sequential sub-frame relative to a display position of the first temporally sequential sub-frame by an amount equal to an offset from the first pixel location to the second pixel location.

12. In a wearable head-mounted display (HMD), a computer-implemented method comprising:

while displaying temporally sequential color sub-frames of a video display image on a color-sequential display (CSD) of the HMD, determining motion of an eye of a user of the HMD with respect to the CSD;

based on the determined motion, computing an expected change in position of the eye with respect to the CSD over a time interval from a first temporally sequential color sub-frame to a second temporally sequential color sub-frame that follows the first temporally sequential color sub-frame; and

based on the computed expected change in position, compensating for the determined motion of the eye by adjusting a display position on the CSD of the second temporally sequential color sub-frame with respect to a display position on the CSD of the first temporally sequential color sub-frame,

wherein computing the expected change in position of the eye with respect to the CSD over the time interval from the first temporally sequential color sub-frame to the second temporally sequential color sub-frame based on the determined motion comprises;

determining a location of a first point in a plane of the CSD corresponding to a first eye-gaze direction of the eye at a time of display of the first temporally sequential color sub-frame; and

based on the determined location of the first point and the determined motion, making a prediction of a location of a second point in the plane of the CSD corresponding to a predicted eye-gaze of the eye at a time of display of the second temporally sequential color sub-frame, the prediction being made prior to the time of display of the second temporally sequential color sub-frame.

15. In a system including a color-sequential display (CSD), a computer-implemented method comprising:

while displaying temporally sequential color sub-frames of a display image on the CSD, determining motion of an eye with respect to the CSD;

based on the determined motion, computing an expected change in position of the eye with respect to the CSD from one temporally sequential color sub-frame to a next temporally sequential color sub-frame; and

based on the computed expected change in position, adjusting a display position on the CSD of the next temporally sequential color sub-frame with respect to a display position on the CSD of the one temporally sequential color sub-frame,

wherein computing the expected change in position of the eye with respect to the CSD from the one temporally sequential color sub-frame to the next temporally sequential color sub-frame comprises;

determining a predicted change in angular orientation of an eye-gaze direction of the eye with respect to a plane of the CSD as a function of the determined motion over the time interval from the one temporally sequential color sub-frame to the next temporally sequential color sub-frame.

16. A wearable head-mounted display (HMD) comprising:

means for determining motion of an eye of a user of the HMD with respect to a color-sequential display (CSD) of the HMD, while displaying temporally sequential color sub-frames of a video display image on the CSD;

means for computing an expected change in position of the eye with respect to the CSD over a time interval from a first temporally sequential color sub-frame to a second temporally sequential color sub-frame that follows the first temporally sequential color sub-frame, wherein the computing is based on the determined motion; and

means for using the computed expected change in position to compensate for the determined motion of the eye by adjusting a display position on the CSD of the second temporally sequential color sub-frame with respect to a display position on the CSD of the first temporally sequential color sub-frame,

wherein the expected change in position corresponds to a change from (i) a first eye-gaze direction of the eye in a plane of the CSD at a time of the first temporally sequential color sub-frame to (ii) a second eye-gaze direction of the eye in the plane of the CSD at a time of the second temporally sequential color sub-frame,

and wherein adjusting the display position on the CSD of the second temporally sequential color sub-frame based on the computed expected change in position comprises displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction.

17. The wearable HMD of claim 16, wherein the second temporally sequential color sub-frame is followed by a third temporally sequential color sub-frame, the wearable HMD further comprising:

means for computing a subsequent expected change in position of the eye with respect to the CSD over a time interval from the second temporally sequential color sub-frame to the third temporally sequential color sub-frame; and

means for adjusting a display position on the CSD of the third temporally sequential color sub-frame with respect to the adjusted display position on the CSD of the second temporally sequential color sub-frame.

18. The wearable HMD of claim 16, wherein the video display image comprises temporally sequential frames, each of the temporally sequential frames comprising a temporal sequence of a red (R) color sub-frame, a green (G) color sub-frame, and a blue (B) color sub-frame, the temporal sequence corresponding to N=3 display colors,

and wherein displaying the temporally sequential color sub-frames of the video display image on the CSD comprises displaying the temporal sequence of the R color sub-frame, the G color sub-frame, and the B color sub-frame of each of the temporally sequential frames.

19. The wearable HMD of claim 18, wherein the CSD comprises a display region bounded by a display border,

wherein the second temporally sequential color sub-frame is one of the R color sub-frame, the G color sub-frame, and the B color sub-frame of a given temporally sequential frame,

and wherein the wearable HMD further comprises:

means for making a determination that adjusting the display position on the CSD of the second temporally sequential color sub-frame will cause at least part of the second temporally sequential sub-frame to be positioned outside of the display border, and thereby cause a portion of the display region to display fewer than the N=3 display colors for the given temporally sequential frame; and

means for responding to the determination by cropping the portion of the display region in which the fewer than the N=3 display colors for the given temporally sequential frame will be displayed.

20. The wearable HMD of claim 16, wherein means for determining motion of the eye of the user of the HMD with respect to the CSD comprise means for analyzing eye-tracking data from an eye-tracking camera of the HMD to determine angular motion of an eye-gaze direction of the eye of the user with respect to the CSD.

21. The wearable HMD of claim 16, wherein means for determining motion of the eye of the user of the HMD with respect to the CSD comprises:

means for analyzing eye-tracking data from an eye-tracking camera of the HMD to determine angular motion of an eye-gaze direction of the eye of the user with respect to the CSD;

means for analyzing motion data from a motion detector of the HMD to determine both angular movement and linear movement the HMD;

means for analytically converting the angular movement and linear movement the HMD into computed motion of the CSD with respect to the eye of the user of the HMD; and

means for analytically combining the eye-tracking data with the motion of the HMD into an eye-gaze direction of the eye of the user with respect to the CSD.

22. The wearable HMD of claim 16, wherein the CSD is configured to display the temporally sequential color sub-frames at a sub-frame rate,

and wherein means for determining the motion of the eye of the user of the HMD with respect to the CSD comprise means for measuring motion of the eye of the user at a measuring rate that exceeds the sub-frame rate.

23. The wearable HMD of claim 16, wherein the plane of the CSD comprises a two-dimensional pixel array,

wherein the first eye-gaze direction corresponds to a first pixel location in the pixel array, and the second eye-gaze direction corresponds to a second pixel location in the pixel array,

and wherein displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction comprises offsetting a display position of the second temporally sequential sub-frame relative to a display position of the first temporally sequential sub-frame by an amount equal to an offset from the first pixel location to the second pixel location.

24. A non-transitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a wearable head-mounted display (HMD), cause the wearable HMD to carry out operations comprising:

while displaying temporally sequential color sub-frames of a video display image on a color-sequential display (CSD) of the HMD, determining motion of an eye of a user of the HMD with respect to the CSD;

based on the determined motion, computing an expected change in position of the eye with respect to the CSD over a time interval from a first temporally sequential color sub-frame to a second temporally sequential color sub-frame that follows the first temporally sequential color sub-frame; and

based on the computed expected change in position, compensating for the determined motion of the eye by adjusting a display position on the CSD of the second temporally sequential color sub-frame with respect to a display position on the CSD of the first temporally sequential color sub-frame,

wherein computing the expected change in position of the eye with respect to the CSD over the time interval from the first temporally sequential color sub-frame to the second temporally sequential color sub-frame based on the determined motion comprises;

determining a location of a first point in a plane of the CSD corresponding to a first eye-gaze direction of the eye at a time of display of the first temporally sequential color sub-frame; and

based on the determined location of the first point and the determined motion, making a prediction of a location of a second point in the plane of the CSD corresponding to a predicted eye-gaze of the eye at a time of display of the second temporally sequential color sub-frame, the prediction being made prior to the time of display of the second temporally sequential color sub-frame.

25. The non-transitory computer-readable medium of claim 24, wherein the second temporally sequential color sub-frame is followed by a third temporally sequential color sub-frame,

and wherein the operations further comprise:

based on the determined motion, computing a subsequent expected change in position of the eye with respect to the CSD over a time interval from the second temporally sequential color sub-frame to the third temporally sequential color sub-frame; and

based on the computed subsequent expected change in position, adjusting a display position on the CSD of the third temporally sequential color sub-frame with respect to the adjusted display position on the CSD of the second temporally sequential color sub-frame.

26. The non-transitory computer-readable medium of claim 24, wherein the video display image comprises temporally sequential frames, each of the temporally sequential frames comprising a temporal sequence of N color sub-frames corresponding to N display colors,

and wherein displaying the temporally sequential color sub-frames of the video display image on the CSD comprises displaying the temporal sequence of the N color sub-frames of each of the temporally sequential frames.

27. The non-transitory computer-readable medium of claim 26, wherein N=3, and the N color sub-frames of each of the temporally sequential frames comprise a red (R), a green (G), and a blue (B) color sub-frame of each of the temporally sequential frames.

28. The non-transitory computer-readable medium of claim 26, wherein the CSD comprises a display region bounded by a display border,

wherein the second temporally sequential color sub-frame is one of the N color sub-frames of a given temporally sequential frame,

and wherein the method further comprises:

determining that adjusting the display position on the CSD of the second temporally sequential color sub-frame will cause at least part of the second temporally sequential sub-frame to be positioned outside of the display border, and thereby cause a portion of the display region to display fewer than the N display colors for the given temporally sequential frame; and

in response, cropping the portion of the display region in which the fewer than the N display colors for the given temporally sequential frame will be displayed.

29. The non-transitory computer-readable medium of claim 24, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises analyzing eye-tracking data from an eye-tracking camera of the HMD to determine angular motion of an eye-gaze direction of the eye of the user with respect to the CSD.

30. The non-transitory computer-readable medium of claim 24, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises:

analyzing motion data from a motion detector of the HMD to determine both angular movement and linear movement the HMD; and

analytically converting the angular movement and linear movement the HMD into computed motion of the CSD with respect to the eye of the user of the HMD.

31. The non-transitory computer-readable medium of claim 24, wherein determining motion of the eye of the user of the HMD with respect to the CSD comprises:

analyzing eye-tracking data from an eye-tracking camera of the HMD to determine angular motion of an eye-gaze direction of the eye of the user with respect to the CSD;

analyzing motion data from a motion detector of the HMD to determine both angular movement and linear movement the HMD;

analytically converting the angular movement and linear movement the HMD into computed motion of the CSD with respect to the eye of the user of the HMD;

analytically combining the eye-tracking data with the motion of the HMD into an eye-gaze direction of the eye of the user with respect to the CSD.

32. The non-transitory computer-readable medium of claim 24, wherein the expected change in position corresponds to a change from (i) a first eye-gaze direction of the eye in a plane of the CSD at a time of the first temporally sequential color sub-frame to (ii) a second eye-gaze direction of the eye in the plane of the CSD at a time of the second temporally sequential color sub-frame,

and wherein adjusting the display position on the CSD of the second temporally sequential color sub-frame based on the computed expected change in position comprises displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction.

33. The non-transitory computer-readable medium of claim 32, wherein the plane of the CSD comprises a two-dimensional pixel array,

wherein the first eye-gaze direction corresponds to a first pixel location in the pixel array, and the second eye-gaze direction corresponds to a second pixel location in the pixel array,

and wherein displaying the second temporally sequential sub-frame at a relative position with respect to the second eye-gaze direction that is substantially the same as a relative position of the first temporally sequential sub-frame with respect to the first eye-gaze direction comprises offsetting a display position of the second temporally sequential sub-frame relative to a display position of the first temporally sequential sub-frame by an amount equal to an offset from the first pixel location to the second pixel location.

34. A non-transitory computer-readable medium having instructions stored thereon that, upon execution by one or more processors of a system, cause the system to carry out operations comprising:

while displaying temporally sequential color sub-frames of a display image on a color-sequential display (CSD) of the system, determining motion of an eye with respect to the CSD;

based on the determined motion, computing an expected change in position of the eye with respect to the CSD from one temporally sequential color sub-frame to a next temporally sequential color sub-frame; and

based on the computed expected change in position, adjusting a display position on the CSD of the next temporally sequential color sub-frame with respect to a display position on the CSD of the one temporally sequential color sub-frame,

wherein computing the expected change in position of the eye with respect to the CSD from the one temporally sequential color sub-frame to the next temporally sequential color sub-frame comprises;

determining a predicted change in angular orientation of an eye-gaze direction of the eye with respect to a plane of the CSD as a function of the determined motion over the time interval from the one temporally sequential color sub-frame to the next temporally sequential color sub-frame.