Fast recognition algorithm processing, systems and methods

• US 9,904,850 B2
• Filed: 05/22/2017
• Issued: 02/27/2018
• Est. Priority Date: 07/19/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Wnuk, Kamil, Song, Bing, Siddiqui, Matheen, McKinnon, David, Sudol, Jeremi, Soon-Shiong, Patrick, Dialameh, Orang
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US15/602,022
• Publication Number:
  US 20170262704A1
• Time in Patent Office: 281 Days
• Field of Search: 382190
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Systems and methods of quickly recognizing or differentiating many objects are presented. Contemplated systems include an object model database storing recognition models associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from key frames of the modeled object. Such objects are recognized by a recognition engine at a later time. The recognition engine can construct a recognition strategy based on a current context where the recognition strategy includes rules for executing one or more recognition algorithms on a digital representation of a scene. The recognition engine can recognize an object from the object model database, and then attempt to identify key frame bundles that are contextually relevant, which can then be used to track the object or to query a content database for content information.

Claims

1. A mobile device comprising:

at least one sensor;

at least one display;

a tangible, non-transitory computer readable memory storing software instructions; and

at least one hardware processor coupled with the memory and the sensor, and configurable, upon execution of the software instructions, to;

capture, via the at least one sensor, a digital representation a scene;

obtain access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles include;

recognition features associated with modeled features of at least one known object, one or more key frames, and one or more of a content link, metadata, a normal vector of a key frame, AR content, contextual information, context attributes, and context links;

recognize at least one scene object in the scene as the at least one known object according to at least one recognition algorithm and as a function of the recognition features and the digital representation;

obtain access to augmented reality (AR) content associated with the at least one known object; and

render the AR content on the display in relation to an anchor point on the at least one scene object such that the AR content overlays or masks at least a portion of the at least one scene object, wherein the contextually relevant key frame bundles include one or more key frames and one or more of a content link, metadata, a normal vector of a key frame, AR content, contextual information, context attributes, and context links.

2. The mobile device of claim 1, wherein the contextually relevant key frame bundles include non-imaging modeled object properties of the at least one known object.

3. The mobile device of claim 2, wherein the AR content is rendered as a function of the non-imaging modeled object properties.

4. The mobile device of claim 2, wherein the non-imaging modeled object properties include at least one of a material property, a mechanical property, a magnetic property, an optical property, a thermal property, an acoustic property, a chemical property, and an electrical property.

5. The mobile device of claim 1, wherein the AR content is indexed in a content database based on data bound to the contextually relevant key frame bundles.

6. The mobile device of claim 1, wherein the AR content comprises at least one of a link, a URL, image data, video data, audio data, augmented reality data, social media data, product data, text data, object data, object model data, game data, news data, multimedia data, and content information corresponding to the at least one known object.

7. The mobile device of claim 1, wherein the AR content comprises an AR model generated from an object model of the at least one known object.

8. The mobile device of claim 7, wherein the AR model is updated based on an interaction with the at least one scene object.

9. The mobile device of claim 7, wherein the AR model comprises an object mask.

10. The mobile device of claim 9, wherein the object mask includes chroma key or chroma key compositing data.

11. The mobile device of claim 9, wherein the object mask removes at least some background features in the digital representation.

12. The mobile device of claim 9, wherein the object mask removes the at least one scene object.

13. The mobile device of claim 9, wherein the object mask comprises an inverse object mask.

14. The mobile device of claim 1, wherein the at least one hardware processor is further configurable to anchor the AR content to the anchor point on the at least one scene object as a function of the modeled features corresponding to the recognition features used to recognize the at least one scene object.

15. The mobile device of claim 1, wherein the anchor point corresponds to at least one of the modeled features when the at least one scene object is deformed relative to the at least one known object.

16. The mobile device of claim 1, wherein the at least one hardware processor is further configurable to track the at least one scene object based on information in the contextually relevant key frame bundles.

17. The mobile device of claim 16, wherein the at least one hardware processor is further configurable to track the at least one scene object in real-time as a function of the recognition features.

18. The mobile device of claim 17, wherein the at least one hardware processor is further configurable to track the at least one scene object at a frame rate of a video rendering of the AR content on the display.

19. The mobile device of claim 17, wherein the hardware processor is further configurable to track recognized scene objects relative to each other.

20. The mobile device of claim 19, wherein the hardware processor is further configurable to recognize a plurality of scene objects in parallel.

21. The mobile device of claim 19, wherein the hardware processor is further configurable to differentiate the at least one scene object from at least one of another scene object and background of the scene.

22. The mobile device of claim 1, wherein the hardware processor is further configurable to analyze or interpret one or more scene objects.

23. The mobile device of claim 22, wherein a gesture of a scene object is interpreted as a command, a control, or a communication.

24. The mobile device of claim 1, wherein obtaining access to the contextually relevant key frame bundles is triggered by one or more of a sound, music, song, jingle, location information, time, user profile information, or other non-visual information.

25. The mobile device of claim 24, wherein the location information triggers a pre-caching of the contextually relevant key frame bundles.

26. The mobile device of claim 1, wherein the digital representation comprises at least one of image data, video data, audio data, location data, biometric data, tactile data, time data, temperature data, and accelerometer data.

27. The mobile device of claim 1, wherein the context is based on at least one of a time, a location, an orientation, a personal preference, a weather condition, a proximity, a temperature, and an acceleration.

28. The mobile device of claim 1, wherein the context includes at least one of a contextual device state, a shopping context, a commute context, a birthday context, a gaming context, and a non-visual context.

29. The mobile device of claim 1, wherein the scene comprises one or more of a 3D scene, a video scene, a real-world environment, a virtual setting, and a non-visual environment.

30. The mobile device of claim 1, wherein the at least one known object comprises at least one of a building, an automobile, a person, a face, a game player, a game character, a ball, a lamp post, and a vehicle.

31. The mobile device of claim 1, wherein the mobile device is one of a cell phone, a tablet, a game console, a game, an appliance, a robot, a vehicle, a toy, a hand-held game system, a personal data assistant, a medical device, a UAV, and a phablet.