Foreground analysis on parametric background surfaces

• US 9,551,922 B1
• Filed: 07/06/2012
• Issued: 01/24/2017
• Est. Priority Date: 07/06/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Liu, Qiang, Yao, Ning, Gopalan, Sowmya
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Khoo, Stacy
• Application Number:
  US13/543,522
• Time in Patent Office: 1,663 Days
• Field of Search: 345173-178, 345/419, 382/103, 178 1801- 1809
• US Class Current: 1/1
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 35/00 Stereoscopic photography
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 21/80 Generation or processing of content or additional data by content creator independently of the distribution process; Content per se arrangements for generating broadcast information H04H60/02

Abstract

Techniques are described for projecting content onto a display surface and for analyzing the hand of a user that is placed over the display surface, such as when the user is making a command gesture. Based on the location and known shape of the display surface, a surface depth map is calculated corresponding to the display surface. Points of the calculated surface depth map are then subtracted from respective points an observed depth map to produce a foreground depth map corresponding to the user hand. The hand may be further characterized and tracked based on the foreground depth map.

Claims

1. A system comprising:

one or more processors;

an image sensor;

a projector;

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

capturing one or more images of an environment using the image sensor;

determining a location of a display medium within the environment based at least in part on the one or more images, wherein the display medium has a known parametric description;

projecting content onto the display medium using the projector;

determining an observed depth map of at least a portion of the environment that contains the display medium, wherein the observed depth map includes first depth information associated with the display medium and an object within the environment that at least partially occludes the display medium;

computing a display depth map of the display medium based at least in part on the location and the known parametric description of the display medium, wherein the display depth map includes second depth information associated with the display medium, the second depth information including a first depth for a first point of the display medium that is unobscured in the one or more images and a second depth for a second point of the display medium that is obscured by the object and that is not depicted in the one or more images; and

producing a foreground depth map based at least in part on differences between the second depth information of the display depth map and the first depth information of the observed depth map, the foreground depth map indicating a difference in depth between a third point associated with the observed depth map and a corresponding third point associated with the display depth map.

2. The system of claim 1, wherein the display medium is a handheld medium.

3. The system of claim 1, wherein the parametric description of the display medium defines a shape of a surface of the display medium.

4. The system of claim 1, wherein determining the observed depth map comprises time-of-flight observations.

5. The system of claim 1, wherein the display medium is partially occluded by a hand of a user.

6. The system of claim 1, wherein the foreground depth map corresponds to a hand of a user that partially occludes the display medium.

7. The system of claim 1, wherein:

the observed depth map comprises depths of observed surface points of the at least the portion of the environment;

the display depth map comprises calculated depths of surface points of the display medium; and

producing the foreground depth map comprises subtracting respective depths of the display depth map from corresponding depths of the observed depth map.

8. A method, comprising:

capturing one or more images of an environment that contain a first representation of a target object;

determining an observed depth map of at least a portion of the environment, wherein the observed depth map includes first depth information associated with the target object;

computing a target depth map based on a known location of the target object and a known parametric description associated with the target object, wherein the target depth map includes second depth information associated with the target object, the second depth information including a first depth for a first point of the target object that is unobscured in the one or more images and a second depth for a second point of the target object that is obscured by a foreground object in the one or more images; and

determining a foreground depth map based at least in part on differences between the second depth information of the target depth map and the first depth information of the observed depth map, the foreground depth map indicating a difference in depth between a third point associated with the observed depth map and a corresponding third point associated with the target depth map.

9. The method of claim 8, wherein the foreground depth map includes third depth information associated with the foreground object.

10. The method of claim 8, wherein determining the observed depth map comprises time-of-flight observations.

11. The method of claim 8, wherein the target object is moveable within the environment.

12. The method of claim 8, further comprising:

determining the known location of the target object based at least in part on the one or more images.

13. The method of claim 8, further comprising determining the known location of the target object based at least in part on the observed depth map.

14. The method of claim 8, further comprising projecting content onto the target object.

15. The method of claim 8, wherein the parametric description associated with the target object defines a shape of a surface of the target object.

16. The method of claim 8, wherein the target object is partially occluded by a hand of a user.

17. The method of claim 8, wherein the foreground depth map corresponds to a hand of a user that partially occludes the target object.

18. The method of claim 8, wherein:

the observed depth map comprises depths of observed surface points of the environment;

the target depth map comprises calculated depths of surface points of the target object; and

producing the foreground depth map comprises subtracting respective depths of the target depth map from corresponding depths of the observed depth map.

19. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

determining, from one or more images of an environment, a location of a target object within the environment, wherein the target object is partially occluded by a hand of a user;

determining an observed depth map of at least a portion of the environment that contains the target object, wherein the observed depth map includes first depth information associated with the at least the portion of the environment;

computing a target depth map of the target object based on the location and a known parametric description of the target object, wherein the target depth map includes second depth information associated with the target object, the second depth information including a first depth for a first point of the target object that is unobscured in the one or more images and a second depth for a second point of the target object that is obscured by the hand of the user in the one or more images; and

producing a foreground depth map corresponding to the hand of the user based at least in part on differences between the second depth information associated with the target depth map and the first depth information associated with the observed depth map, the foreground depth map indicating a difference in depth between a third point associated with the observed depth map and a corresponding third point associated with the target depth map.

20. The one or more non-transitory computer-readable media of claim 19, wherein the known parametric description defines a shape of a surface of the target object.

21. The one or more non-transitory computer-readable media of claim 19, wherein determining the observed depth map comprises time-of-flight observations.

22. The one or more non-transitory computer-readable media of claim 19, wherein the target object comprises a handheld display.

23. The one or more non-transitory computer-readable media of claim 19, the acts further comprising projecting content onto the target object at the location.

24. The one or more non-transitory computer-readable media of claim 19, wherein determining the location of the target object is based at least on part on the observed depth map.

25. The one or more non-transitory computer-readable media of claim 19, wherein:

the observed depth map comprises depths of observed surface points of the at least the portion of the environment;

the target depth map comprises calculated depths of surface points of the target object; and

producing the foreground depth map comprises subtracting respective depths of the target depth map from corresponding depths of the observed depth map.