Localized depth map generation

• US 10,362,296 B2
• Filed: 08/17/2017
• Issued: 07/23/2019
• Est. Priority Date: 08/17/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Price, Raymond Kirk, Bleyer, Michael, Demandolx, Denis Claude Pierre
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/680,188
• Publication Number:
  US 20190058859A1
• Time in Patent Office: 705 Days
• Field of Search: 348 43
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Techniques involving localized depth map generation, the techniques including receiving pixel data for a frame captured by an image sensor, the pixel data including at least one light intensity value, corresponding to an amount of light received by the image sensor during a frame period, for each of a plurality of pixels; identifying a subset of the pixels as being associated with a physical object detected based on at least the pixel data; selecting a region of the frame, the region corresponding to at least the subset of the pixels; and selectively generating a localized depth map for the frame period corresponding to the selected region. A portion of the frame outside of the selected region is not associated with a depth map generated for the frame period.

Claims

1. A device employing depth sensing, the device comprising:

a first image sensor configured to measure received amounts of light;

an object detector configured to;

receive first pixel data for a first frame captured by the first image sensor, the first pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during a first frame period, for each of a plurality of first pixels, and

identify a first subset of the first pixels as being associated with a first physical object detected based on at least the first pixel data;

a region of interest selection module configured to select a first region of the first frame, the first region corresponding to at least the first subset of the first pixels identified by the object detector; and

a depth map calculation module configured to selectively generate a localized depth map for the first frame period corresponding to the first region selected by the region of interest selection module, wherein;

a portion of the first frame outside of the first region is not associated with a depth map generated for the first frame period;

the device further comprises a localized depth map generator;

including the depth map calculation module,

including a resizing module configured to determine a target size for the localized depth map and determine an initial size for the first region, and

configured to upsample or downsample image data for the first frame period based on a ratio between the initial size and the target size; and

the depth map calculation module is configured to generate the localized depth map based on at least the upsampled or downsampled image data.

2. The device according to claim 1, wherein the depth map calculation module is configured to generate the localized depth map based on at least second pixel data for the first frame captured by the first image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during the first frame period, for each of a plurality of second pixels.

3. The device according to claim 1, further comprising a second image sensor configured to measure received amounts of light, wherein

the second image sensor is included in a different camera than the first image sensor, and

the depth map calculation module is configured to:

receive second pixel data for a second frame captured by the second image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the second image sensor during a second frame period overlapping the first frame period, for each of a plurality of second pixels, and

generate the localized depth map based on at least the received second pixel data.

4. The device according to claim 1, wherein:

the first image sensor is a time-of-flight imaging sensor;

the device is configured to determine at least one active brightness value for each of the plurality of first pixels;

the object detector is configured to identify the first subset of the first pixels based on the determined active brightness values; and

the device is configured to perform phase calculation or dealiasing depth calculation for the first subset of the first pixels after the identification of the first subset of the first pixels.

5. The device according to claim 1, wherein the image data is second pixel data for the first frame captured by the first image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during the first frame period, for each of a plurality of second pixels.

6. The device according to claim 1, further comprising a depth map processing module configured to receive the localized depth generated by the depth map calculation module configured to process the localized depth map to generate events and/or data,

wherein the resizing module is configured to receive the target size from the depth map processing module.

7. The device according to claim 1, wherein:

the region of interest selection module is configured to identify a second subset of the first pixels, which includes the first subset of the first pixels and a plurality of the first pixels immediately surrounding the first subset of the first pixels; and

the first region corresponds to the second subset of the first pixels.

8. The device according to claim 1, further comprising:

a head-mounted display arranged to be worn and viewed by an end user of the device;

a controller including one or more processors;

one or more machine readable media storing instructions which, when executed by the one or more processors, cause the controller to;

process the localized depth map to generate events and/or data, and

display and/or change one or more user interface elements via the head-mounted display in response to the events and/or data.

9. The device according to claim 1, wherein:

the localized depth map generator is configured to downsample image data for the first frame period based on the ratio between the initial size and the target size; and

the depth map calculation module is configured to generate the localized depth map based on at least the downsampled image data.

10. A device employing depth sensing, the device comprising:

a first image sensor and a second image sensor arranged for stereoscopic depth estimation and each configured to measure received amounts of light;

an object detector configured to;

receive first pixel data for a first frame captured by the first image sensor, the first pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during a first frame period, for each of a plurality of first pixels,

identify a first subset of the first pixels as being associated with a first physical object detected based on at least the first pixel data,

receive second pixel data for a second frame captured by the second image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the second image sensor during a second frame period overlapping the first frame period, for each of a plurality of second pixels, and

identify a second subset of the second pixels as being associated with the first physical object detected based on at least the second pixel data;

a region of interest selection module configured to;

select a first region of the first frame, the first region corresponding to at least the first subset of the first pixels identified by the object detector, and

select a second region of the second frame, the second region corresponding to at least the second subset of the second pixels identified by the object detector; and

a depth map calculation module configured to perform stereo matching of pixels selected as corresponding to the first region of the first frame and pixels selected as corresponding to the second region of the second frame to generate a localized depth map corresponding to the first and second regions selected by the region of interest selection module, wherein;

a portion of the first frame outside of the first region is not associated with a depth map generated for the first frame period, and a portion of the second frame outside of the second region is not associated with a depth map generated for the first or second frame periods;

the device further comprises a localized depth map generator;

including the depth map calculation module,

including a resizing module configured to determine a target size for the localized depth map and determine an initial size for the first region, and

configured to upsample or downsample image data for the first frame period based on a ratio between the initial size and the target size; and

the depth map calculation module is configured to generate the localized depth map based on at least the upsampled or downsampled image data.

11. A method of localized depth map generation, the method comprising:

receiving first pixel data for a first frame captured by a first image sensor, the first pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during a first frame period, for each of a plurality of first pixels;

identifying a first subset of the first pixels as being associated with a first physical object detected based on at least the first pixel data;

selecting a first region of the first frame, the first region corresponding to at least the first subset of the first pixels identified as being associated with the first physical object;

selectively generating a localized depth map for the first frame period corresponding to the selected first region;

determining a target size for the localized depth map; and

determining an initial size for the first region, wherein;

a portion of the first frame outside of the first region is not associated with a depth map generated for the first frame period, and

the generating the localized depth map includes upsampling or downsampling image data for the first frame period based on a ratio between the initial size and the target size.

12. The method of claim 11, wherein the localized depth map is generated based on at least second pixel data for the first frame captured by the first image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during the first frame period, for each of a plurality of second pixels.

13. The method of claim 11, further comprising:

receiving second pixel data for a second frame captured by a second image sensor included in a different camera than the first image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the second image sensor during a second frame period overlapping the first frame period, for each of a plurality of second pixels,

wherein the localized depth map is generated based on at least the second pixel data.

14. The method of claim 13, wherein:

the first image sensor is a color image sensor; and

the first pixel data includes multiple light intensity values, corresponding to amounts of light received for respective color components by the first image sensor during the first frame period, for each of the plurality of first pixels.

15. The method of claim 11, wherein:

the first image sensor is a time-of-flight imaging sensor;

the method further comprises determining at least one active brightness value for each of the plurality of first pixels;

the identifying the first subset of the first pixels is based on the determined active brightness values; and

the identifying the first subset of the first pixels is performed prior to phase calculation or dealiasing depth calculation for the first subset of the first pixels.

16. The method of claim 11, wherein the image data is second pixel data for the first frame captured by the first image sensor, the second pixel data including at least one light intensity value, corresponding to an amount of light received by the first image sensor during the first frame period, for each of a plurality of second pixels.

17. The method of claim 11, further comprising:

providing the localized depth map to a depth map processing module; and

receiving the target size from the depth map processing module.

18. The method of claim 11, wherein the first physical object is a human hand.

19. The method of claim 11, further comprising:

identifying a second subset of the first pixels, which includes the first subset of the first pixels and a plurality of the first pixels immediately surrounding the first subset of the first pixels, wherein

the first region corresponds to the second subset of the first pixels.

20. The method of claim 11, wherein the generating the localized depth map includes downsampling image data for the first frame period based on the ratio between the initial size and the target size.