Fast recognition algorithm processing, systems and methods

• US 9,355,123 B2
• Filed: 07/15/2014
• Issued: 05/31/2016
• Est. Priority Date: 07/19/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Wnuk, Kamil, Song, Bing, Siddiqui, Matheen, McKinnon, David, Sudol, Jeremi, Soon-Shiong, Patrick, Dialameh, Orang
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US14/332,371
• Publication Number:
  US 20150023602A1
• Time in Patent Office: 686 Days
• Field of Search: 382/190
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Systems and methods of quickly recognizing or differentiating many objects are presented. Contemplated systems include an object model database storing recognition models associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from key frames of the modeled object. Such objects are recognized by a recognition engine at a later time. The recognition engine can construct a recognition strategy based on a current context where the recognition strategy includes rules for executing one or more recognition algorithms on a digital representation of a scene. The recognition engine can recognize an object from the object model database, and then attempt to identify key frame bundles that are contextually relevant, which can then be used to track the object or to query a content database for content information.

Claims

1. An apparatus comprising:

at least one sensor configured to obtain a digital representation of a plurality of objects; and

a recognition engine coupled with the at least one sensor and configured to;

obtain access to an object model database storing recognition models of known object models, the recognition models having features derived from key frames of known object models;

determine a recognition strategy based on the digital representation, the recognition strategy comprising an algorithm application rules set governing application of a plurality of algorithms to the digital representation;

generate at least one recognition feature by executing the plurality of algorithms on the digital representation according to the algorithm application rules set;

identify a target modeled object in the object model database where the corresponding recognition model has features similar to the at least one recognition feature;

identify a target key frame bundle having a content link based on the target modeled object;

retrieve content information associated with a plurality of object models corresponding to at least some of the plurality objects based on the content link of the target key frame bundle; and

render at least some of the content information on a display screen.

2. The apparatus of claim 1, wherein the known object models comprise at least three dimensional models.

3. The apparatus of claim 1, wherein the content information comprises an object model mask.

4. The apparatus of claim 3, wherein the content information comprises a green screen mask.

5. The apparatus of claim 1, wherein the plurality of algorithms include at least one of the following types of algorithmic components:

a feature detection algorithm, an edge detection algorithm, a corner detection algorithm, depth estimation determination algorithm, focal length estimation algorithm, camera intrinsics estimation algorithm, a character recognition algorithm, an image pattern detection algorithm, a symbol recognition algorithm, a biometric detection algorithm, and an audio recognition algorithm.

6. The apparatus of claim 1, wherein the plurality of algorithms include at least one of the following algorithms:

SIFT, FAST, DAISY, FREAK, SURF, BRISK, ASR, and OCR.

7. The apparatus of claim 1, wherein the target key frame bundle comprises object tracking features.

8. The apparatus of claim 1, the recognition engine is further configured to generate the recognition features substantially in real-time with respect to receiving the digital representation.

9. The apparatus of claim 1, wherein the digital representation includes at least one of the following types of data:

image data, text data, audio data, video data, and biometric data.

10. The apparatus of claim 1, wherein the recognition engine is configured to identify at least X of the object models per Y unit of time, where X/Y is at least 1 object model per second.

11. The apparatus of claim 10, wherein the X/Y is at least 10 object models per second.

12. The apparatus of claim 11, wherein the X/Y is at least 300 objects per second.

13. The apparatus of claim 1, wherein the recognition engine is further configured to generate the recognition features and identify the at least one target modeled object within one sampling period of the digital representation.

14. The apparatus of claim 13, the sampling period comprises a time period no greater than a single image frame rendering time.

15. The apparatus of claim 14, wherein the single image frame rendering time is no greater than approximately 1/24^th of a second.

16. The apparatus of claim 1, further comprising a mobile device incorporating the recognition engine.

17. The apparatus of claim 16, wherein the mobile device further incorporates the sensor.

18. The apparatus of claim 17, wherein the sensor is selected from the group consisting of:

a GPS sensor, a hall probe, a camera, an RFID reader, a near field radio, a microphone, a biometric sensor, a touch screen, an accelerometer, a magnetometer, a gyroscope, a spectrometer, a strain gauge, a stress gauge, a pulse oximeter, a seisometer, a galvanometer, a Radar sensor, a LIDAR sensor, an infrared sensor, a flow meter, an anemometer, a Geiger counter, a scintillator, a barometer, and a piezoelectric sensor.

19. The apparatus of claim 16, wherein the mobile devices comprises at least one of the following:

a cell phone, a tablet, a kiosk, an appliance, a vehicle, and a game console.

20. The apparatus of claim 1, wherein the content information comprises multi-media data.

21. The apparatus of claim 1, wherein the content information comprise at least one of the following:

image data, video data, audio data, augmented reality data, mask data, social media data, product data, text data, object data, object model data, game data, and news data.

22. The apparatus of claim 1, wherein the recognition engine is further configured to obtain access to the object model database by construction the object model database from the plurality of key frames.

23. The apparatus of claim 1, wherein the recognition engine is further configured to obtain access to the object model database by receiving the object model database from a server.

24. The apparatus of claim 1, wherein the recognition engine is further configured to obtain access to the object model database over a network.

25. The apparatus of claim 1, wherein the recognition engine is further configured to update the object model database based on new key frames.