Object ingestion through canonical shapes, systems and methods

• US 9,501,498 B2
• Filed: 02/16/2015
• Issued: 11/22/2016
• Est. Priority Date: 02/14/2014
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Wnuk, Kamil, McKinnon, David, Sudol, Jeremi, Song, Bing, Siddiqui, Matheen
• Primary Examiner(s): Shah, Utpal
• Application Number:
  US14/623,435
• Publication Number:
  US 20150302027A1
• Time in Patent Office: 645 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

An object recognition ingestion system is presented. The object ingestion system captures image data of objects, possibly in an uncontrolled setting. The image data is analyzed to determine if one or more a priori know canonical shape objects match the object represented in the image data. The canonical shape object also includes one or more reference PoVs indicating perspectives from which to analyze objects having the corresponding shape. An object ingestion engine combines the canonical shape object along with the image data to create a model of the object. The engine generates a desirable set of model PoVs from the reference PoVs, and then generates recognition descriptors from each of the model PoVs. The descriptors, image data, model PoVs, or other contextually relevant information are combined into key frame bundles having sufficient information to allow other computing devices to recognize the object at a later time.

Claims

1. An object recognition ingestion system comprising:

canonical shape database storing shape objects having geometrical attributes of canonical shapes, shape attributes, and having reference key frame points-of-view (PoVs); and

an object ingestion engine coupled with the canonical shape database and programmed to perform the steps of;

obtaining image data of at least one object;

deriving a set of edges related to the at least one object from the image data;

obtaining a shape result set from the canonical shape database where the shape result set include shape objects having shape attributes satisfying shape selection criteria determined as a function of geometrical information from the set of edges;

selecting at least one target shape object from shape objects in the shape result set;

generating an object model from the at least one target shape object and portions of the image data associated with the set of edges;

deriving a set of model key frame PoVs from the object model and the reference key frame PoVs associated with the at least one target shape object;

instantiating a descriptor object model from the object model, the descriptor model comprising recognition algorithm descriptors having locations on the object model relative to the model key frame PoVs;

creating a set of key frames bundles from the descriptor object model as a function of the set of model key frame PoVs; and

storing the set of key frame bundles in object recognition database.

2. The system of claim 1, further comprising the object recognition database.

3. The system of claim 1, wherein the shape objects stored in the canonical shape database include geometrical primitives.

4. The system of claim 3, wherein at least one of the shape objects comprise a compound shape object comprising at least two geometric primitives.

5. The system of claim 3, wherein the geometrical primitives include at least one of the following:

a line, a square, a cube, a circle, a sphere, a cylinder, a cone, a box, a torus, a platonic solid, a triangle, a pyramid, and a box.

6. The system of claim 1, wherein at least some of the shape objects stored in the canonical shape database represent 3D objects.

7. The system of claim 1, wherein the shape objects stored in the canonical shape database comprises topological classifications.

8. The system of claim 1, wherein the shape objects stored in the canonical shape database comprise object templates representing object classes.

9. The system of claim 8, wherein the object templates include at least one of the following:

a vehicle, a building, an appliance, a plant, a toy, a face, a person, and an internal organ.

10. The system of claim 1, wherein the reference key frame PoVs comprise a normal vector.

11. The system of claim 1, wherein the reference key frame PoVs comprise key frame PoV generation rules.

12. The system of claim 1, wherein the image data comprises at least one of the following types of data:

visible data, video data, video frame data, still image data, acoustic imaging data, medical image data, and game imaging data.

13. The system of claim 1, wherein the geometrical attributes include at least one of the following:

a length, a width, a height, a thickness, a radius, a diameter, an angle, a hole, a center, a formula, a texture, a bounding box, a chirality, a periodicity, an orientation, a pitch, and a number of sides.

14. The system of claim 1, further comprising a mobile device that includes the object ingestion engine.

15. The system of claim 14, wherein the mobile device further includes the canonical shape database.

16. The system of claim 14, wherein the mobile device further includes the object recognition database.

17. The system of claim 1, wherein the recognition ingestion engine is further programmed to perform the step of obtaining the shape result set as a function of edge descriptors associated with the set of edges.

18. The system of claim 17, wherein the canonical shape database indexes the shape objects stored in the canonical shape database based on edge descriptors.

19. The system of claim 1, wherein the recognition ingestion engine is programmed to perform the step of selecting the at least one target shape object based on a user selection.

20. The system of claim 1, wherein the recognition ingestion engine is programmed to perform the step of selecting the at least one target shape object based on a score.

21. The system of claim 20, wherein the score is determined as a function of at least one of the following:

a location, a time, and a descriptor match.

22. The system of claim 1, wherein the recognition algorithm descriptors include at least one of the following type of descriptors:

SIFT, FREAK, FAST, DAISY, and BRISK.

23. The system of claim 1, wherein at least one key frame bundle within the set of key frame bundles includes the following:

a normal vector, an image, and a descriptor.

24. The system of claim 1, further comprising an imaging sensor programmed to perform the step of capturing the image data of the at least one object.