Video noise reduction for video augmented reality system

• US 10,354,140 B2
• Filed: 01/31/2017
• Issued: 07/16/2019
• Est. Priority Date: 01/31/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Guenter, Brian K., Gal, Ran
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US15/421,150
• Publication Number:
  US 20180218217A1
• Time in Patent Office: 896 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Video noise reduction for a video augmented reality system is provided. A head mounted display includes a display unit; a camera for generating frames of display data. A frame store is provided for storing previous frames of displayed information that was sent to the display unit; and a motion processor is provided in communication with the camera, display unit, and the frame store. The motion processor is operable to: identify an area of interest in a current frame of display data; match the area of interest to similar areas in previous frames stored in the frame store; rotate and translate the matched areas of interest from the one or more previous frames stored in the frame store to match the area of interest in the current frame; and average the prior matched areas of interest with the current area of interest to generate a displayed area of interest.

Claims

1. A head mounted display device, comprising:

a display unit;

a camera, having a fast shutter speed, for generating frames of display data during movement of a user'"'"'s head;

a memory comprising a frame store for storing a previous frame of displayed information that was sent to the display unit; and

a motion processor in communication with the camera, display unit, and the frame store, wherein the motion processor is operable to;

identify a first area of interest in a current frame of display data, wherein the first area of interest is a first portion of the current frame, wherein the first area of interest is identified by eye tracking the first portion of the current frame at which a user of the head mounted display is looking, wherein a location of the first area of interest moves based on movement of a user'"'"'s eyes;

identify a second area of interest, similar to the first area of interest, in the previous frame stored in the frame store, wherein the second area of interest is identified by eye tracking a second portion of the previous frame at which the user of the head mounted display was looking;

rotate and translate the second area of interest, from the previous frame stored in the frame store, to match the first area of interest in the current frame;

average the second area of interest with the first area of interest to generate a displayed area of interest to increase light in the displayed area to compensate for the fast shutter speed; and

display the displayed area of interest.

2. The head mounted display of claim 1, further comprising an eye tracker coupled to the motion processor.

3. The head mounted display of claim 2, wherein the first area of interest is identified based on information received from the eye tracker denoting the first area of interest on the display at which the user of the head mounted display is looking.

4. The head mounted display of claim 1, further comprising a second display coupled to the motion processor.

5. The head mounted display of claim 4, further comprising a second camera coupled to the motion processor.

6. The head mounted display of claim 1, wherein the average of the first matched area of interest with the second area of interest is a weighted average.

7. The head mounted display of claim 6, wherein more recent frames are more heavily weighted than older frames.

8. The head mounted display of claim 6, wherein the weighting of a frame is based on the quality of the match between the second area of interest of the previous frame and the first area of interest of the current frame.

9. A system comprising a computing device, the computing device comprising:

at least one processing device in communication with at least one camera, wherein the camera has a fast shutter speed, which causes frames of display data generated from the camera to have low light; and

at least one computer readable data storage device storing instructions that, when executed by the at least one processing device are operative to;

receive an image from the camera;

determine a first area of interest in the image, wherein the first area of interest is a first portion of the current frame, wherein the first area of interest is identified by eye tracking the first portion of the current frame at which a user of the head mounted display is looking, wherein a location of the first area of interest moves based on movement of a user'"'"'s eyes;

find a comparable second area of interest in a previous frame, wherein the second area of interest is identified by eye tracking a second portion of the previous frame at which the user of the head mounted display was looking;

average the first area of interest in the image with the comparable second area of interest in the previous frame to increase light in the displayed area to compensate for the fast shutter speed; and

display the averaged area of interest embedded within the image from the camera.

10. The system of claim 9 further operative to:

receive eye tracking information, from an eye tracker, containing information on where a user of a head mounted display is looking.

11. The system of claim 10, further operative to:

use the eye tracking information, from the eye tracker, to determine the first area of interest.

12. The system of claim 9, wherein to average the first area of interest in the image with the second area of interest in the previous frame, the system is further operative to:

rotate and translate the second area of interest based on movement of the first area of interest compared to the second area of interest based on movement of the user'"'"'s eyes; and

average the rotated and translated second area of interest with the first area of interest in the image.

13. The system of claim 9, wherein to average the first area of interest in the image with the second area of interest in the previous frame, the system is further operative to perform a weighted average of the areas of interest.

14. The system of claim 13, wherein the first area of interest in the image is more heavily weighted than the second area of interest.

15. The system of claim 13 wherein the comparable area of interest is weighted based on its similarity to the first area of interest in the image.

16. A method comprising:

receiving a current image from a camera wherein the camera has a fast shutter speed, which causes frames of display data generated from the camera to have low light;

retrieving from a frame store a previous image that was captured by the camera;

determining a first area of interest, wherein the first area of interest is identified by eye tracking a first portion of the current frame at which a user of the head mounted display is looking, wherein a location of the first area of interest moves based on movement of a user'"'"'s eyes;

finding a comparable second area of interest in the previous image, wherein the second area of interest is identified by eye tracking a second portion of the previous frame at which the user of the head mounted display was looking;

averaging the first area of interest in the current image with the second area of interest in the previous image to create an averaged area of interest, wherein the averaging increases light in the averaged area to compensate for the fast shutter speed; and

display the averaged area of interest.

17. The method of claim 16, further comprising displaying the averaged area of interest within the current image from the camera.

18. The method of claim 17 wherein averaging is performed on a weighted average basis.

19. The method of claim 18 wherein the areas of interest are weighted based on the age of the previous images.

20. The method of claim 18, wherein the weighting is based on the quality of match between the first area of interest with similar areas in the previous images.