Object ingestion through canonical shapes, systems and methods

• US 10,095,945 B2
• Filed: 10/18/2016
• Issued: 10/09/2018
• Est. Priority Date: 02/14/2014
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Wnuk, Kamil, McKinnon, David, Sudol, Jeremi, Song, Bing, Siddiqui, Matheen
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US15/297,053
• Publication Number:
  US 20170039442A1
• Time in Patent Office: 721 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

An object recognition ingestion system is presented. The object ingestion system captures image data of objects, possibly in an uncontrolled setting. The image data is analyzed to determine if one or more a priori know canonical shape objects match the object represented in the image data. The canonical shape object also includes one or more reference PoVs indicating perspectives from which to analyze objects having the corresponding shape. An object ingestion engine combines the canonical shape object along with the image data to create a model of the object. The engine generates a desirable set of model PoVs from the reference PoVs, and then generates recognition descriptors from each of the model PoVs. The descriptors, image data, model PoVs, or other contextually relevant information are combined into key frame bundles having sufficient information to allow other computing devices to recognize the object at a later time.

Claims

1. An object ingestion device comprising:

An image sensor;

A non-transitory computer readable memory storing object ingestion software instructions; and

At least one processor coupled with the non-transitory computer readable memory and image sensor, and that is configurable upon execution of the object ingestion software instructions by the at least one processor to;

obtain a digital representation of a scene including an image of a target object captured by the image sensor and a location;

determine a context associated with the scene based, at least in part, on the digital representation and the location;

identify contextually relevant shape objects in a shape database based on the context;

derive a set of edges from the image of the target object;

select at least one target shape object from the contextually relevant shape objects based on the set of edges;

generate a target object model from the at least one target shape object and portions of the image data associated with the set of edges;

create a set of key frame bundles from the target object model as a function of recognition algorithm descriptors and points of view associated with the at least one target shape object; and

send the set of key frame bundles to an object recognition database for storage.

2. The device of claim 1, wherein the image comprises at least one of the following types of image data:

a still image, video data, a medical image, non-visible spectrum data, and an ultrasound image.

3. The device of claim 1, wherein the contextually relevant shape objects have shape attributes that relate to the context.

4. The device of claim 3, wherein the shape attributes comprise a symmetry attribute.

5. The device of claim 4, wherein the symmetry attribute represents at least one of the following types of symmetry:

rotational symmetry, scale symmetry, fractal symmetry, reflection symmetry, helical symmetry, and translation symmetry.

6. The device of claim 3, wherein the shape attributes comprise at least one of the following attributes:

a geometrical attribute, the location, a size, a distance, a width, a ratio, a thickness, a depth, a hole, a number of sides, a geometric center, a texture, a formula, a bounding box, a chirality, a periodicity, an orientation, an angular pitch, a scaling, a name, a key, a shape index, and a relevant descriptor.

7. The device of claim 1, wherein the context comprises a positive association context.

8. The device of claim 7, wherein the positive association context causes an increase in weight of the contextually relevant shape objects relative to other shape objects in the shape database in selection of the at least one target shape object.

9. The device of claim 1, wherein the context comprises a negative association context.

10. The device of claim 9, wherein the negative association context causes a decrease in weight of other shape objects in the shape database relative to the contextually relevant shape objects in selection of the at least one target shape object.

11. The device of claim 1, wherein the context includes at least one the following types of data:

a GPS location, a time, a weather, an intent, and a user identity.

12. The device of claim 1, wherein the location comprises at least one of the following types of locations:

a road, a freeway, and a parking lot.

13. The device of claim 1, wherein the at least one target shape object comprises an object template.

14. The device of claim 13, wherein the object template represents at least one of the following types of objects:

a tree, a car, a plane, a tire, a vehicle, a building, a human, a face, an appliance, a toy, a tissue, and an organ.

15. The device of claim 1, wherein the target object comprises at least one of the following:

a building, a sign, a product, an automobile, a food, a document, a person, a face, clothing, a device, an organ, an animal, a plant, a game player, an inventory item, a book, a piece of laboratory equipment, a weapon, a landmark, a flower, an insect, and a plane.

16. The device of claim 1, further comprising a mobile device operating as the object ingestion device and that includes the at least one processor and the non-transitory computer readable memory.

17. The device of claim 16, wherein the mobile device comprises at least one of the following:

a cell phone, a robot, a game console, a game interface, a digital camera, a medical device, a head-mount visor, a toy, and a vehicle.

18. The device of claim 1, further comprising the shape database.

19. The device of claim 1, further comprising the object recognition database.

20. The device of claim 1, wherein the recognition algorithm descriptors include at least one of the following types of descriptors:

a SIFT descriptor, a FREAK descriptor, a FAST descriptor, a SURF descriptor, a DAISY descriptor, and a BRISK descriptor.

21. An object ingestion method comprising the steps of:

obtaining a digital representation of a scene including an image of a target object captured by the image sensor and a location;

determining a context associated with the scene based, at least in part, on the digital representation and the location;

identifying contextually relevant shape objects in a shape database based on the context;

deriving a set of edges from the image of the target object;

selecting at least one target shape object from the contextually relevant shape objects based on the set of edges;

generating a target object model from the at least one target shape object and portions of the image data associated with the set of edges;

creating a set of key frame bundles from the target object model as a function of recognition algorithm descriptors and points of view associated with the at least one target shape object; and

sending the set of key frame bundles to an object recognition database for storage.

22. An object ingestion device comprising a mobile device that comprises at least one of a cell phone, a robot, a game console, a game interface, a digital camera, a medical device, a head-mount visor, a toy, and a vehicle, the mobile device comprising:

an image sensor;

a non-transitory computer readable memory storing object ingestion software instructions; and

at least one processor coupled with the non-transitory computer readable memory and image sensor, and that is configurable upon execution of the object ingestion software instructions by the at least one processor to;

obtain a digital representation of a scene including an image of a target object captured by the image sensor and a location;

determine a context associated with the scene based, at least in part, on the digital representation and the location;

identify contextually relevant shape objects in a shape database based on the context;

derive a set of edges from the image of the target object;

select at least one target shape object from the contextually relevant shape objects based on the set of edges;

generate a target object model from the at least one target shape object and portions of the image data associated with the set of edges;

create a set of key frame bundles from the target object model as a function of recognition algorithm descriptors and points of view associated with the at least one target shape object; and

send the set of key frame bundles to an object recognition database for storage.