Fast recognition algorithm processing, systems and methods

• US 9,690,991 B2
• Filed: 10/27/2016
• Issued: 06/27/2017
• Est. Priority Date: 07/19/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Soon-Shiong, Patrick, McKinnon, David, Wnuk, Kamil, Song, Bing, Siddiqui, Matheen, Sudol, Jeremi, Dialameh, Orang
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US15/336,754
• Publication Number:
  US 20170046571A1
• Time in Patent Office: 243 Days
• Field of Search: 382190
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Systems and methods of quickly recognizing or differentiating many objects are presented. Contemplated systems include an object model database storing recognition models associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from key frames of the modeled object. Such objects are recognized by a recognition engine at a later time. The recognition engine can construct a recognition strategy based on a current context where the recognition strategy includes rules for executing one or more recognition algorithms on a digital representation of a scene. The recognition engine can recognize an object from the object model database, and then attempt to identify key frame bundles that are contextually relevant, which can then be used to track the object or to query a content database for content information.

Claims

1. A mobile device comprising:

at least one sensor;

at least one display;

a tangible, non-transitory computer readable memory storing software instructions; and

at least one hardware processor coupled with the memory and the sensor, and configurable, upon execution of the software instructions, to;

capture, via the at least one sensor, a digital representation a scene;

obtain access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key

frame bundles include recognition features associated with modeled features of at least

one known object;

recognize at least one scene object in the scene as the at least one known object according to at least one recognition algorithm and as a function of the recognition features and the digital representation;

obtain access to augmented reality (AR) content associated with the at least one known object, wherein the AR content comprises an object mask generated from an object model of the at least one known object;

anchor the AR content to an anchor point on the at least one scene object as a function of the modeled features corresponding to the recognition features used to

recognize the at least one scene object; and

render the AR content on the display in relation to the anchor point on the at least one scene object.

2. The mobile device of claim 1, wherein the at least one hardware processor is further configurable to track the at least one scene object in the digital representation in real-time as a function of the recognition features.

3. The mobile device of claim 2, wherein the at least one hardware processor is further configurable to track at least ten recognized scene objects in the digital representation in real-time as a function of corresponding recognition features.

4. The mobile device of claim 2, wherein the at least one hardware processor is further configurable to track the at least one scene object at a frame rate of a video rendering of the AR content on the display.

5. The mobile device of claim 4, wherein the frame rate is at least 30 frames per second.

6. The mobile device of claim 1, wherein the digital representation comprises at least one of image data, video data, audio data, location data, biometric data, tactile data, time data, temperature data, and accelerometer data.

7. The mobile device of claim 1, wherein the AR content is rendered as an overlay over video data captured as part of the digital representation.

8. The mobile device of claim 1, wherein the AR content further comprises at least one of a link, a URL, image data, video data, audio data, augmented reality data, social media data, product data, text data, object data, object model data, game data, news data, and multimedia data.

9. The mobile device of claim 1, wherein the

object mask comprises an inverse object mask.

10. The mobile device of claim 1, wherein the context is based on at least one of a time, a location, an orientation, a personal preference, a weather condition, a proximity, a temperature, and an acceleration.

11. The mobile device of claim 1, wherein the context includes at least one of a contextual device state, a shopping context, a commute context, a birthday context, a gaming context, and a non-visual context.

12. The mobile device of claim 1, wherein the contextually relevant key frame bundles include non-imaging modeled object properties of the at least one known object.

13. The mobile device of claim 12, wherein the AR content is rendered as a function of the non-imaging modeled object properties.

14. The mobile device of claim 12, wherein the non-imaging modeled object properties include at least one of a material property, a mechanical property, a magnetic property, an optical property, a thermal property, an acoustic property, a chemical property, and an electrical property.

15. The mobile device of claim 1, wherein the anchor point corresponds to at least one of the modeled features when the at least one scene object is deformed relative to the at least one known object.

16. The mobile device of claim 1, wherein the scene comprises a 3D scene.

17. The mobile device of claim 1, wherein the scene comprises a video scene.

18. The mobile device of claim 1, wherein the scene comprises a real-world environment.

19. The mobile device of claim 1, wherein the scene comprises a virtual setting.

20. The mobile device of claim 1, wherein the scene comprises a non-visual environment.

21. The mobile device of claim 1, wherein the at least one known object comprises at least one of a building, an automobile, a person, a face, a game player, a game character, a lamp post, and a vehicle.

22. The mobile device of claim 1, wherein the mobile device is one of a cell phone, a tablet, a game console, a game, an appliance, a robot, a vehicle, a toy, a hand-held game system, a personal data assistant, a medical device, a UAV, and a phablet.