Fast recognition algorithm processing, systems and methods

• US 10,628,673 B2
• Filed: 01/26/2018
• Issued: 04/21/2020
• Est. Priority Date: 07/19/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Wnuk, Kamil, Song, Bing, Siddiqui, Matheen, McKinnon, David, Sudol, Jeremi, Soon-Shiong, Patrick, Dialameh, Orang
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US15/881,043
• Publication Number:
  US 20180165519A1
• Time in Patent Office: 816 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Systems and methods of quickly recognizing or differentiating many objects are presented. Contemplated systems include an object model database storing recognition models associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from key frames of the modeled object. Such objects are recognized by a recognition engine at a later time. The recognition engine can construct a recognition strategy based on a current context where the recognition strategy includes rules for executing one or more recognition algorithms on a digital representation of a scene. The recognition engine can recognize an object from the object model database, and then attempt to identify key frame bundles that are contextually relevant, which can then be used to track the object or to query a content database for content information.

Claims

1. An autonomous vehicle apparatus comprising:

at least one sensor;

at least one hardware processor;

a tangible, non-transitory computer readable memory storing an object model database and software instructions; and

a recognition engine, executable on the at least one hardware processor, coupled with the memory and the at least one sensor, and configurable, upon execution of the software instructions, to;

capture, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;

obtain access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;

track recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at least one of another scene object and background of the scene;

recognize a scene object as the at least one known object using at least one recognition algorithm, the recognition features, and the digital representation; and

initiate a vehicle action responsive to the scene object.

2. The apparatus of claim 1, wherein the vehicle action comprises interpreting a gesture of a scene object as at least one of a command, a control, or a communication.

3. The apparatus of claim 1, wherein the vehicle action comprises obtaining access to content associated with the at least one known object.

4. The apparatus of claim 3, wherein the content comprises at least one of a link, a URL, image data, video data, audio data, augmented reality data, social media data, product data, text data, object data, object model data, game data, news data, multimedia data, and content information corresponding to the at least one known object.

5. The apparatus of claim 1, wherein the recognition engine is further configurable to recognize a plurality of scene objects in parallel.

6. The apparatus of claim 1, wherein the recognition engine is further configurable to analyze or interpret one or more scene objects.

7. The apparatus of claim 1, wherein obtaining access to the contextually relevant key frame bundles is triggered by one or more of a sound, location information, time, user profile information, or other non-visual information.

8. The apparatus of claim 1, wherein the contextually relevant key frame bundles are pre-cached in the memory based on vehicle position or location data obtained from the at least one sensor device.

9. The apparatus of claim 1, wherein key frame bundles comprise information related to one or more magnetic, optical, thermal, acoustic, chemical, and electrical properties of a modeled object.

10. The apparatus of claim 1, wherein the key frame bundles are contextually relevant to a current context of the recognition engine based on one or more of a location or motion of a vehicle.

11. The apparatus of claim 1, wherein the digital representation comprises at least one of image data, video data, audio data, location data, biometric data, tactile data, time data, temperature data, weather data, and accelerometer data.

12. The apparatus of claim 1, wherein the digital representation comprises multi-dimensional data.

13. The apparatus of claim 1, wherein the context comprises at least one of time information, location information, orientation information, ambient audio information, user profile information, directional information, weather information, proximity information, temperature information, tracking information, and acceleration information.

14. The apparatus of claim 1, wherein the context includes at least one of a contextual device state, a shopping context, a commute context, and a non-visual context.

15. The apparatus of claim 1, wherein the at least one known object comprises at least one of a building, an automobile or vehicle, a person, a face, a moveable object, and a street fixture.

16. The apparatus of claim 1, wherein the recognition engine comprises a cloud-based recognition service.

17. The apparatus of claim 1, wherein the at least one sensor comprises at least one of a GPS device, hall probe, camera, RFID reader, near field radio, microphone, biometric sensor, touch screen, accelerometer, magnetometer, gyroscope, spectrometer, strain or stress gauge, pulse oximeter, seismometer, galvanometer, Radar, LIDAR, infrared sensor, flow sensor, anemometer, Geiger counter, scintillator, barometer, and piezoelectric sensor.

18. The apparatus of claim 1, wherein the at least one recognition algorithm comprises an algorithm prioritizing execution of a plurality of processing algorithms based on the context derived from the digital representation.

19. The apparatus of claim 18, wherein prioritizing execution is based on one or more of a time, a location, an orientation, a contextual parameter, a GPS coordinate or position, a user, a license agreement, a digital representation attribute, a frame rate, and a hierarchy.

20. The apparatus of claim 18, wherein one of a plurality of processing algorithms is selected to process data from the digital representation based on a processing result corresponding to another one of the plurality of processing algorithms.

21. The apparatus of claim 1, wherein object models in the object model database are constructed from scanning real-world objects using one or more of a laser beam, LIDAR, a time-of-flight system, and a structured light system.

22. An autonomous vehicle method comprising:

capturing, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;

obtaining access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;

tracking recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at least one of another scene object and background of the scene;

recognizing a scene object as the at least one known object using at least one recognition algorithm, the recognition features, and the digital representation; and

initiating a vehicle action responsive to the scene object.

23. A non-transitory computer readable medium or media containing instructions for executing a method comprising:

capturing, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;

obtaining access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;

tracking recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at least one of another scene object and background of the scene;

recognizing a scene object as the at least one known object using at least one recognition algorithm, the recognition features, and the digital representation; and

initiating a vehicle action responsive to the scene object.