Super-resolution apparatus and method for virtual and mixed reality

• US 10,482,575 B2
• Filed: 09/28/2017
• Issued: 11/19/2019
• Est. Priority Date: 09/28/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Li, Zhengmin, Kuwahara, Atsuo, Vembar, Deepak
• Primary Examiner(s): Hon, Ming Y
• Application Number:
  US15/719,458
• Publication Number:
  US 20190096035A1
• Time in Patent Office: 782 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

An apparatus and method for efficiently improving virtual/real interactions in augmented reality. For example, one embodiment of a method comprises: capturing a raw image including depth data; identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects; generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; detecting interactions between the virtual objects and the real objects using the super-resolution map; and performing one or more graphics processing or general purpose processing operations based on the detected interactions.

Claims

1. A method comprising:

capturing a raw image including depth data;

identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;

generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;

detecting interactions between the virtual objects and the real objects using the super-resolution map; and

performing one or more graphics processing or general-purpose processing operations based on the detected interactions.

2. The method of claim 1 wherein the machine-learning techniques comprise:

performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and

evaluating results of the comparisons.

3. The method of claim 2 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.

4. The method of claim 3 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.

5. The method of claim 4 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.

6. The method of claim 1 wherein identifying the one or more regions of interest further comprises:

if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.

7. The method of claim 1 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.

8. The method of claim 1 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.

18. The non-transitory machine-readable medium of claim 1 wherein the machine-learning techniques comprise:

performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and

evaluating results of the comparisons.

19. The non-transitory machine-readable medium of claim 18 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.

20. The non-transitory machine-readable medium of claim 19 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.

21. The non-transitory machine-readable medium of claim 20 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.

9. An apparatus comprising:

a camera to capture a raw image including depth data;

region detection circuitry to identify one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;

depth super-resolution circuitry to generate a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;

interaction detection circuitry to detect interactions between the virtual objects and the real objects using the super-resolution map; and

wherein one or more graphics processing or general-purpose processing operations are to be performed based on the detected interactions.

10. The apparatus of claim 9 wherein the machine-learning techniques comprise:

performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and

evaluating results of the comparisons.

11. The apparatus of claim 10 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.

12. The apparatus of claim 11 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.

13. The apparatus of claim 12 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.

14. The apparatus of claim 9 wherein identifying the one or more regions of interest further comprises:

if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.

15. The apparatus of claim 9 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.

16. The apparatus of claim 9 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.

17. A non-transitory machine-readable medium having program code stored therein which, when executed by a machine, causes the machine to perform the operations of:

capturing a raw image including depth data;

identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;

generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;

detecting interactions between the virtual objects and the real objects using the super-resolution map; and

performing one or more graphics processing or general-purpose processing operations based on the detected interactions.

22. The non-transitory machine-readable medium of claim 17 wherein identifying the one or more regions of interest further comprises:

if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.

23. The non-transitory machine-readable medium of claim 17 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.

24. The non-transitory machine-readable medium of claim 17 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.