3D silhouette sensing system

• US 9,720,506 B2
• Filed: 01/14/2014
• Issued: 08/01/2017
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Powell, Karlton David, Kim, David, Izadi, Shahram, Bathiche, Steven, Pradeep, Vivek, Large, Timothy Andrew
• Primary Examiner(s): MARINELLI, PATRICK
• Application Number:
  US14/154,571
• Publication Number:
  US 20150199018A1
• Time in Patent Office: 1,295 Days
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A 3D silhouette sensing system is described which comprises a stereo camera and a light source. In an embodiment, a 3D sensing module triggers the capture of pairs of images by the stereo camera at the same time that the light source illuminates the scene. A series of pairs of images may be captured at a predefined frame rate. Each pair of images is then analyzed to track both a retroreflector in the scene, which can be moved relative to the stereo camera, and an object which is between the retroreflector and the stereo camera and therefore partially occludes the retroreflector. In processing the image pairs, silhouettes are extracted for each of the retroreflector and the object and these are used to generate a 3D contour for each of the retroreflector and object.

Claims

1. A 3D sensing system comprising:

a stereo camera, the stereo camera including an infra-red camera;

a light source; and

a 3D sensing module arranged to track a retroreflector during a time that an object partially occluding the retroreflector is tracked by the 3D sensing module using image pairs captured by the stereo camera, and to detect a touch event or determine a pressure value based at least in part on a depression of the retroreflector;

the 3D sensing module further comprising a synchronization module arranged to generate a trigger pulse to trigger capture of a pair of images by the stereo camera and simultaneous illumination by the light source and an analysis module arranged to analyze the captured images to simultaneously track both the retroreflector and the object partially occluding the retroreflector, the analysis module further arranged to analyze the captured images to generate a 3D contour of each of the retroreflector and the object by extracting a 2D contour of the retroreflector and a 2D contour of the object from each image in an image pair and performing stereo matching between the 2D contours from each image in the image pair; and

the 3D sensing module further arranged to remove distortion from each image in the image pair, rectify each image in the image pair, and identify parts of each image to not consider parts of the scene not contained in the other image in the image pair in the analysis.

2. The system according to claim 1,

wherein the retroreflector is movable relative to the stereo camera.

3. The system of claim 2, wherein the retroreflector is transparent at visible wavelengths of light and retroreflective at a wavelength range captured by the stereo camera.

4. The system of claim 2, wherein the retroreflector comprises a user interface layout formed on an upper surface of the retroreflector, wherein the user interface layout is transparent at a wavelength range over which the retroreflector retroreflects.

5. The system of claim 1, the 3D sensing module being at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device.

6. The system according to claim 1, wherein the stereo camera comprises a pair of infra-red (IR) cameras at a fixed spacing.

7. The system according to claim 1, wherein the light source comprises a plurality of diffuse IR LEDs arranged around a lens of the infrared camera.

8. The system according to claim 1 further comprising a printed display content placard arranged to be removably placed on a back side of the retroreflector.

9. A method of 3D sensing comprising:

generating a trigger pulse to cause a stereo camera to capture a pair of images of a scene during a time that the scene is illuminated by a light source;

analyzing the pair of images to track both a retroreflector in the scene and an object partially occluding the retroreflector and to detect a touch event or determine a pressure value based at least in part on a depression of the retroreflector;

analyzing the pair of images further comprising computing a 3D depth contour for the retroreflector in a first pass and computing a 3D depth contour for the object in a second pass;

computing the 3D depth contour for the retroreflector comprising extracting a 2D contour for each bright silhouette in each image in the pair and performing stereo matching between the 2D contours extracted from a first image in the pair and the 2D contours extracted from a second image in the pair; and

removing distortion from each image in the image pair, rectifying each image in the image pair, and identifying parts of each image to not consider parts of the scene not contained in the other image in the image pair in the analysis.

10. A method according to claim 9, further comprising repeating the method to capture pairs of images at a pre-defined frame rate and wherein the analysis is performed on a frame by frame basis in real time.

11. The method according to claim 9, wherein computing a 3D depth contour for the object comprises:

for each image in the pair of images;

generating a convex hull of each extracted 2D contour in the image;

generating a modified image by inverting pixels within the image which are located within a convex hull; and

extracting a 2D contour for each bright silhouette in the modified image; and

performing stereo matching between the 2D contours extracted from a first modified image in the pair and the 2D contours extracted from a second modified image in the pair.

12. The method according to claim 9, wherein analyzing the pair of images further comprises:

classifying at least one of the retroreflector and the object by applying machine learning techniques to the 3D depth contour.

13. The method according to claim 12, further comprising:

generating an input signal for a software program based on a classification of at least the retroreflector.

14. The method according to claim 13, wherein the input signal is generated based on a classification of at least the retroreflector and a relative position of the retroreflector and the object.

15. The method according to claim 9, further comprising, prior to analyzing the pair of images:

removing distortion from each image in the image pair;

rectifying each image in the image pair; and

identifying parts of each image to not consider parts of the scene not contained in the other image in the image pair in the analysis.

16. The method according to claim 9 further comprising analyzing the pair of images further comprising classifying at least one of the retroreflector and the object by applying machine learning techniques to the 3D depth contour, generating an input signal for a software program based on a classification of at least the retroreflector and a relative position of the retroreflector and the object.

17. A 3D sensing system comprising:

a stereo camera;

a light source;

a 3D sensing module arranged to track both a retroreflector and an object partially occluding the retroreflector using image pairs captured by the stereo camera and to generate a closed 3D contour of each of the retroreflector and the object; and

the 3D sensing module further arranged to remove distortion from each image in an image pair, rectify each image in the image pair, and identify parts of each image to not consider parts of the scene not contained in the other image in the image pair in the analysis.

18. The system according to claim 17 further comprising a printed display content placard arranged to be removably placed on a back side of the retroreflector.

19. The system of claim 17 wherein the retroreflector is transparent at visible wavelengths of light and retroreflective at a wavelength range captured by the stereo camera.

20. The system of claim 17, wherein the retroreflector comprises a user interface layout formed on an upper surface of the retroreflector, wherein the user interface layout is transparent at a wavelength range over which the retroreflector retroreflects.