Object instance identification using three-dimensional spatial configuration

• US 10,311,593 B2
• Filed: 11/16/2016
• Issued: 06/04/2019
• Est. Priority Date: 11/16/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cohen, Benjamin M, Tzadok, Asaf, Tzur, Yochay
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/352,586
• Publication Number:
  US 20180137386A1
• Time in Patent Office: 930 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A system for identifying specific instances of objects in a three-dimensional (3D) scene, comprising: a camera for capturing an image of multiple objects at a site; at least one processor executable to: use a location and orientation of the camera to create a 3D model of the site including multiple instances of objects expected to be in proximity to the camera, and generate multiple candidate clusters each representing a different projection of the 3D model, detect at least two objects in the image, and determine a spatial configuration for each detected object; and match the detected image objects to one of the multiple candidate cluster using the spatial configurations, associate the detected objects with the expected object instances of the matched cluster, and retrieve information of one of the detected objects that is stored with the associated expected object instance; and a head-wearable display configured to display the information.

Claims

1. A system for identifying object instances in a three-dimensional (3D) scene, comprising:

a camera configured to capture an image of multiple objects at a site;

at least one hardware processor; and

a non-transitory memory device having embodied thereon program code executable by said at least one hardware processor to;

receive, from said camera, a captured image that depicts multiple objects that are physically present at the site,

detect at least two objects in the image,

retrieve 3D information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site,

generate, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster,

determine a spatial configuration of the objects detected in the image, with respect to each other and to said camera,

match the objects detected in the image to one of the multiple candidate clusters, by;

(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects on the respective candidate cluster, and

(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster,

associate the objects detected in the image with the objects of the most probable cluster, and

retrieve information of at least one of the objects of the most probable cluster.

2. The system of claim 1, wherein the program code is further executable by said at least one hardware processor to display the retrieved information.

3. The system of claim 2, further comprising a head-wearable display selected from the group consisting of:

an augmented reality display, and a virtual reality display,

wherein the displaying of the retrieved information is on the head-wearable display.

4. The system of claim 3, wherein one or more of the at least one hardware processors are disposed with the head-wearable display.

5. The system of claim 3, wherein the camera is disposed with the head-wearable display.

6. The system of claim 2, further comprising a user interface configured to receive a selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

7. A method for identifying multiple object instances, comprising:

capturing, by a camera, an image depicting multiple objects that are physically present at a site;

detecting at least two objects in the image;

retrieving three-dimensional (3D) information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site;

generating, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster;

determining a spatial configuration of the objects detected in the image, with respect to each other and to said camera;

matching the objects detected in the image to one of the multiple candidate clusters, by;

(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects in the respective candidate cluster, and

(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster;

associating the objects detected in the image with the objects of the most probable cluster; and

retrieving information of at least one of the objects of the most probable cluster.

8. The method of claim 7, further comprising displaying the retrieved information.

9. The method of claim 8, further comprising receiving a user selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

13. The method of claim 8, wherein the displaying of the retrieved information is on a head-wearable display selected from the group consisting of:

an augmented reality display, and a virtual reality display.

14. The method of claim 13, wherein the camera is disposed with the head-wearable display.

10. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor to:

receive, from a camera, a captured image that depicts multiple objects that are physically present at a site;

detect at least two objects in the image;

retrieve 3D information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site;

generate, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster;

determine a spatial configuration of the objects detected in the image, with respect to each other and to the camera;

match the objects detected in the image to one of the multiple candidate clusters, by;

(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects in the respective candidate cluster, and

(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster;

associate the objects detected in the image with the objects of the most probable cluster; and

retrieve information of at least one of the objects of the most probable cluster.

11. The computer program product of claim 10, wherein the program code is further executable to display the retrieved information.

12. The computer program product of claim 11, the program code is further executable to receive a user selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

15. The computer program product of claim 11, wherein the displaying of the retrieved information is on a head-wearable display selected from the group consisting of:

an augmented reality display, and a virtual reality display.

16. The computer program product of claim 15, wherein the camera is disposed with the head-wearable display.