Foveated camera for video augmented reality and head mounted display

• US 10,298,840 B2
• Filed: 01/31/2017
• Issued: 05/21/2019
• Est. Priority Date: 01/31/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Guenter, Brian K.
• Primary Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US15/421,185
• Publication Number:
  US 20180220068A1
• Time in Patent Office: 840 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Using a foveated camera for video augmented reality via a head mounted display is provided. A camera may be placed in a binning mode; a full frame binned image may be captured using the camera and the binning mode of the camera may be turned off. A region of interest may be selected within the full frame; the region of interest may be captured at a higher resolution than the resolution of the camera when in binning mode; the region of interest may be composited with the full frame binned image; and the composited image may be displayed.

Claims

1. A method, comprising:

providing a head-mounted display comprising a right lens, a left lens, and a camera;

placing the camera in a binning mode;

capturing a full frame binned image using the camera and turning off the binning mode of the camera;

selecting a region of interest within the full frame;

capturing the region of interest at a higher resolution than the resolution of the camera when in the binning mode;

compositing the region of interest with the full frame binned image; and

displaying the composited image, wherein the left lens is refreshed with the composited image at a first time, wherein the right lens is refreshed with a next composited image at a second time, and wherein the second time is half a refresh period from the first time.

2. The method of claim 1, further comprising performing late stage re-projection on the full framed binned image prior to compositing the region of interest with the full frame binned image.

3. The method of claim 2, wherein performing late stage re-proj ection of the full framed binned image comprises rotating the full framed binned image to match any rotation of the region of interest.

4. The method of claim 2, wherein performing late stage re-projection of the full framed binned image comprises translating the full framed binned image to match any translation of the region of interest.

5. The method of claim 1, further comprising capturing a partial frame binned image immediately around the region of interest, where the binning value is less than a binning value of the captured full frame image.

6. The method of claim 5, wherein compositing further comprises compositing the region of interest with the partial frame binned image with the full frame binned image.

7. The method of claim 1, wherein compositing further comprises feathering the area around the edge of the region of interest.

8. A head mounted display, comprising:

a camera capable of capturing a region of interest and of binning;

a display comprising;

a right lens; and

a left lens;

a processor coupled to the camera and the display, the processor operable to;

place the camera in a binning mode;

capture a full frame binned image;

designate a region of interest within the camera frame;

capture the region of interest using the camera;

composite the region of interest with the full frame binned image;

refresh the left lens with the region of interest with the full frame binned image at a first time; and

refresh the right lens with a next region of interest with a next full frame binned image at a second time, and wherein the second time is half a refresh period from the first time.

9. The head mounted display of claim 8, further comprising an eye tracking unit coupled to the processor.

10. The head mounted display of claim 9, wherein the processor is further operable to designate a region of interest within the camera frame based on information received from the eye tracking unit.

11. The head mounted display of claim 8, wherein the processor is operable to, prior to compositing, perform late stage re-projection.

12. The head mounted display of claim 11, wherein performing late stage re-projection comprises rotating the full frame binned image to match any rotation of the region of interest.

13. The head mounted display of claim 11, wherein performing late stage re-projection comprises translating the full frame binned image to match any translation of the region of interest.

14. The head mounted display of claim 8, wherein the processor is further operable to take a binned image of the area immediately surrounding the region of interest and wherein the binning value of this intermediate area is less than the binning value of the full framed binned image.

15. The head mounted display of claim 8, wherein the composite action of the processor further includes the capability of compositing the area of interest with the intermediate area and with the full framed binned image.

16. A computer storage medium associated with a head-mounted display comprising a right lens, a left lens, and a camera, the computer storage medium containing computer executable instructions which, when executed by a computer, perform a method comprising the acts of:

receiving a lower resolution binned full frame image from the camera;

receiving a higher resolution region of interest image from within a frame of the camera;

compositing the received lower resolution image with the higher resolution image; and

displaying the composited image, wherein the left lens is refreshed with the composited image at a first time, wherein the right lens is refreshed with a next composited image at a second time, and wherein the second time is half a refresh period from the first time.

17. The computer storage medium of claim 16 having further instructions for:

feathering the pixels around the higher resolution image where it meets the lower resolution area.

18. The computer storage medium of claim 17 having further instructions for placing the camera in a binning mode prior to receiving the lower resolution binned full frame image from the camera.

19. The computer storage medium of claim 16 having further instructions for placing the camera in a binning mode and capturing an intermediate region around the region of interest at a binning value less than a binning value of the full frame image.

20. The computer storage medium of claim 19 having further instructions for compositing the received lower resolution image with the intermediate region with the higher resolution region of interest.