Techniques for setting focus in mixed reality applications

• US 10,044,925 B2
• Filed: 08/18/2016
• Issued: 08/07/2018
• Est. Priority Date: 08/18/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, Hang, Jain, Vishal, Lei, Maria Cheng, Guyman, William Douglas
• Primary Examiner(s): Nazrul, Shahbaz
• Application Number:
  US15/240,850
• Publication Number:
  US 20180054562A1
• Time in Patent Office: 719 Days
• CPC Class Codes: G03B 13/32 Means for focusing
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

Described are examples for capturing one or more real world images for display with one or more computer-generated images (e.g., holograms). One or more computer-generated images for overlaying over one or more real world images can be received, and a depth for overlaying at least one of the computer-generated images over the one or more real world images can be determined. A lens of a camera can be focused based on this depth, and the one or more real world images can be captured via the camera with the lens focused based on the depth. The one or more real world images can be provided as one or more mixed reality images with the one or more computer-generated images overlaid on the one or more real world images, such that the image is focused on objects near the one or more computer-generated images.

Claims

1. A computing device, comprising:

a camera comprising a lens configured to capture and provide one or more real world images; and

at least one processor configured to determine whether to transform the one or more real world images to a mixed reality image;

wherein when the at least one processor determines not to transform the one or more real world images, the at least one processor is configured to perform one or more auto-focusing processes of the camera to focus the one or more real world images; and

wherein when the at least one processor determines to transform the one or more real world images, the at least one processor is configured to;

overlay one or more computer-generated images over the one or more real world images; and

override, based on determining to transform the one or more real world images, the one or more auto-focusing processes of the camera by causing the camera to focus the lens for capturing the one or more real world images based at least in part on a depth at which the at least one processor overlays at least one of the one or more computer-generated images.

2. The computing device of claim 1, wherein the at least one processor is further configured to cause the camera to focus the lens at the depth of the at least one of the one or more computer-generated images.

3. The computing device of claim 1, wherein the at least one processor is configured to determine a selection of the at least one computer-generated image based on a size of the at least one computer-generated image, and wherein the at least one processor is configured to cause the camera to focus the lens at least in part in response to the selection.

4. The computing device of claim 1, wherein the at least one processor is configured to cause the camera to focus the lens at the depth corresponding to a point of the at least one of the one or more computer-generated images having a level of detail that achieves a threshold.

5. The computing device of claim 1, wherein the at least one processor is configured to cause the camera to focus the lens at the depth corresponding to a size or distance from the computing device of the at least one of the one or more computer-generated images.

6. The computing device of claim 1, wherein the at least one processor is configured to receive a selection of the at least one of the one or more computer-generated images, and wherein the at least one processor is configured to cause the camera to focus the lens at least in part in response to the selection.

7. The computing device of claim 6, wherein the selection corresponds to a detected gesture to select the at least one of the one or more computer-generated images presented on an interface on a display.

8. The computing device of claim 6, further comprising a gaze tracking sensor configured to track a position of one or more eyes of a user, wherein the selection corresponds to a determination that the one or more eyes of the user are positioned to view the at least one of the one or more computer-generated images.

9. The computing device of claim 6, wherein the selection corresponds to determining one or more heuristics of the at least one of the one or more computer-generated images.

10. The computing device of claim 1, further comprising a display for displaying the one or more real world images overlaid with the one or more computer-generated images, wherein the display is at least one of positioned on the computing device or remotely located from the computing device.

11. A method for capturing one or more real world images for providing with one or more computer-generated images, comprising:

receiving one or more computer-generated images for overlaying over one or more real world images;

determining a depth for overlaying at least one of the one or more computer-generated images over the one or more real world images;

overriding, based on receiving the one or more computer-generated images, an auto-focusing process of a camera at least in part by instead focusing a lens of a camera based on the depth;

capturing the one or more real world images via the camera with the lens focused based on the depth; and

providing the one or more real world images as one or more mixed reality images with the one or more computer-generated images overlaid on the one or more real world images.

12. The method of claim 11, wherein focusing the lens further comprises focusing the lens at the depth for overlaying the at least one of the one or more computer-generated images.

13. The method of claim 11, further comprising determining a selection of the at least one computer-generate image based on a size of the at least one computer-generated image, and wherein focusing the lens is based at least in part on the selection.

14. The method of claim 11, wherein focusing the lens further comprises focusing the lens at the depth corresponding to a point of the at least one of the one or more computer-generated images having a level of detail that achieves a threshold.

15. The method of claim 11, further comprising receiving a selection of the at least one of the one or more computer-generated images, and wherein focusing the lens is based at least in part on the selection.

16. The method of claim 15, further comprising detecting a gesture to select the at least one of the one or more computer-generated images presented on an interface on the display, wherein the selection corresponds to the gesture.

17. The method of claim 15, further comprising:

tracking position of one or more eyes of a user; and

determining that the one or more eyes of the user are positioned to view the at least one of the one or more computer-generated images, wherein the selection corresponds to the determination.

18. A non-transitory computer-readable medium comprising code for capturing one or more real world images for providing with one or more computer-generated images, the code comprising:

code for receiving one or more computer-generated images for overlaying on one or more real world images;

code for determining a depth for overlaying at least one of the one or more computer-generated images on the one or more real world images;

code for overriding, based on receiving the one or more computer-generate images, an auto-focusing process of a camera at least in part by instead focusing a lens of a camera based on the depth;

code for capturing the one or more real world images via the camera with the lens focused based on the depth; and

code for providing the one or more real world images as one or more mixed reality images with the one or more computer-generated images overlaid on the one or more real world images.

19. The non-transitory computer-readable medium of claim 18, wherein the code for focusing the lens focuses the lens at the depth for overlaying the at least one of the one or more computer-generated images.

20. The non-transitory computer-readable medium of claim 18, wherein the code further comprises code for determining a selection of the at least one computer-generate image based on a size of the at least one computer-generated image, and wherein focusing the lens is based at least in part on the selection.