Determining a segmentation boundary based on images representing an object

• US 10,156,937 B2
• Filed: 09/24/2013
• Issued: 12/18/2018
• Est. Priority Date: 09/24/2013
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Tretter, Daniel R, Kang, Jinman, Tan, Kar Han, Hong, Wei, Short, David Bradley, Sievert, Otto
• Primary Examiner(s): Amadiz, Rodney
• Application Number:
  US15/024,107
• Publication Number:
  US 20160231866A1
• Time in Patent Office: 1,911 Days
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/174 involving the use of two or more images

Abstract

Examples disclosed herein relate to determining a segmentation boundary based on images representing an object. Examples include an IR image based on IR light reflected by an object disposed between an IR camera and an IR-absorbing surface, a color image representing the object disposed between the color camera and the IR-absorbing surface, and determining a segmentation boundary for the object.

Claims

1. A computing system comprising:

an infrared (IR)-absorbing surface that is disposed approximately horizontally;

an IR camera disposed above and pointed at the IR-absorbing surface to capture an IR image representing an object disposed between the IR camera and the IR-absorbing surface based on IR light reflected by the object;

an RGB camera to capture an RGB image representing the object disposed between the RGB camera and the IR-absorbing surface;

a depth camera to capture a depth image representing the object disposed between the depth camera and the IR-absorbing surface;

a segmentation engine to determine a segmentation boundary representing at least one outer edge of the object based on the IR image, the depth image, and the RGB image, wherein the segmentation boundary is determined independent of any prior image of the object captured by any of the IR, depth, and RGB cameras;

a projection assembly to project visible images on the IR-absorbing surface and the object;

wherein the segmentation engine is to determine the segmentation boundary representing the at least one outer edge of the object based on the IR, depth, and RGB images captured during the projection of the visible images; and

wherein the IR-absorbing surface further comprises a touch-sensitive region to detect physical contact with the touch-sensitive region.

2. The computing system of claim 1, wherein:

the IR camera is a low-resolution IR camera and the IR image is a low-resolution IR image;

the depth camera is a low-resolution depth camera and the depth image is a low-resolution depth image; and

the RGB camera is a high-resolution RGB camera and the RGB image is a high-resolution RGB image.

3. The computing system of claim 2, wherein the segmentation engine comprises:

a combine engine to combine the IR and depth images into a single vector image;

a boundary engine to determine a preliminary segmentation boundary for the object based on the vector image;

an upsample engine to upsample the preliminary segmentation boundary to the resolution of the RGB image; and

refine engine to refine the upsampled preliminary segmentation boundary based on the RGB image to obtain the segmentation boundary.

4. The computing system of claim 2, wherein the segmentation engine comprises:

an upsample engine to upsample the IR and depth images to the resolution of the RGB image;

a combine engine to combine the RGB image and the upsampled IR and depth images into a single vector image; and

a boundary engine to determine the segmentation boundary based on the vector image.

5. The computing system of claim 2, wherein:

the IR camera and the IR image have a quarter video graphics array (QVGA) resolution;

the depth camera and the depth image have the QVGA resolution; and

the RGB camera and the RGB image have a resolution of multiple megapixels (MPs).

6. The computing system of claim 1, wherein:

the IR camera is to detect lesser intensity of IR light from the IR-absorbing surface than from the object such that portions of the IR image representing the IR-absorbing surface include lesser IR light intensity values than higher IR light intensity values included in portions of the IR image representing the object; and

the segmentation engine is to determine the segmentation boundary based at least in part on at least one boundary between the lesser IR light intensity values and the greater IR light intensity values.

7. A non-transitory machine-readable storage medium comprising instructions executable by a processing resource of a computing system comprising a horizontal infrared (IR)-absorbing surface and IR camera, a depth camera, and a color camera, each disposed above and pointed at the IR-absorbing surface, the instructions executable to:

acquire, from the IR camera, an IR image representing an object disposed between the IR camera and the IR-absorbing surface based on IR light reflected by the object;

acquire, from the depth camera, a depth image representing respective distances of portions of the object disposed between the depth camera and the IR-absorbing surface;

acquire, from the color camera, a color image having a higher resolution than each of the IR image and the depth image and representing the object disposed between the color camera and the IR-absorbing surface;

determine a preliminary segmentation boundary for the object based on the IR image data and the depth image data;

upsample the preliminary segmentation boundary to the resolution of the color image; and

refine the upsampled preliminary segmentation boundary based on the color image to determine a segmentation boundary for the object.

8. The storage medium of claim 7, wherein the instructions to determine comprise instructions to:

combine the IR and depth images into a single vector image comprising, for each pixel of the vector image, an IR intensity value from a corresponding pixel of the IR image and a depth value from a corresponding pixel of the depth image; and

detect edges in the vector image based on both the IR intensity value and depth values at each pixel of the vector image.

9. The storage medium of claim 7, wherein:

the color camera is an RGB camera and the color image is an RGB image;

the resolution of each of the IR, depth, and RGB images is a two-dimensional pixel resolution; and

the two-dimensional pixel resolution of the RGB image is at least ten times greater than the two-dimensional pixel resolution of the IR image in each dimension, and at least ten times greater than the two-dimensional pixel resolution of the depth image in each dimension.

10. A method comprising:

capturing a low-resolution infrared (IR) image with an IR camera disposed above and pointing at an IR-absorbing surface;

capturing a low-resolution depth image with a depth camera disposed above and pointing at the IR-absorbing surface;

capturing a high-resolution color image with a color camera disposed above and pointing at the IR-absorbing surface, wherein each of the IR image, the depth image, and the color image represents an object disposed between the IR-absorbing surface and the respective camera used to capture the image;

combining the IR image and the depth image into a single vector image comprising data from the IR image and from the depth image at each pixel;

determining a preliminary segmentation boundary for the object based on the vector image;

upsampling the preliminary segmentation boundary to the resolution of the color image; and

refining the upsampled preliminary segmentation boundary based on the color image to determine a segmentation boundary for the object.

11. The method of claim 10, wherein determining the segmentation boundary comprises:

detecting edges in the vector image where the presence of an edge is indicated by the either of the data from the IR image and the data from the depth image.

12. The method of claim 11, wherein:

the IR image has quarter video graphics array (QVGA) resolution;

the depth image has QVGA resolution;

the resolution of the color image is multiple mexapixels (MPs);

the color image is an RGB image; and

the refining comprises filtering the upsampled segmentation boundary based on the RGB image to obtain a refined segmentation boundary.