Augmented reality vision system for tracking and geolocating objects of interest

• US 9,495,783 B1
• Filed: 06/13/2013
• Issued: 11/15/2016
• Est. Priority Date: 07/25/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Oskiper, Taragay, Kumar, Rakesh, Sizintsev, Mikhail, Branzoi, Vlad
• Primary Examiner(s): Crawford, Jacinta M
• Assistant Examiner(s): Wills, Diane
• Application Number:
  US13/916,702
• Time in Patent Office: 1,251 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

Methods and apparatuses for tracking objects comprise one or more optical sensors for capturing one or more images of a scene, wherein the one or more optical sensors capture a wide field of view and corresponding narrow field of view for the one or more images of a scene, a localization module, coupled to the one or more optical sensors for determining the location of the apparatus, and determining the location of one more objects in the one or more images based on the location of the apparatus and an augmented reality module, coupled to the localization module, for enhancing a view of the scene on a display based on the determined location of the one or more objects.

Claims

1. A user-carried or user-worn apparatus for tracking objects, the apparatus comprising:

one or more optical sensors for capturing images of a scene, wherein the images include a wide field of view image of the scene and a narrow field of view image of the scene;

a display for displaying a displayed view of the scene; and

a processor coupled to a storage medium, the storage medium storing processor-executable instructions, which when executed by the processor, performs a method comprising;

geolocating the one or more optical sensors and a location of one or more objects depicted in the captured images based on a location of the one or more optical sensors;

recognizing an object of interest in the wide field of view image;

recognizing the object of interest in the narrow field of view image;

tracking a location of the recognized object of interest during transition from the wide field of view image to the narrow field of view image so as to correlate the location of the recognized object of interest in the wide field of view image and the narrow field of view image; and

enhancing the displayed view of the scene on the display based on the geolocating of the one or more objects, wherein the enhanced, displayed view comprises overlay content scaled in accordance with the tracking of the location of the recognized object of interest from the wide field of view to the narrow field of view.

7. The apparatus of claim 1, wherein the method performed by the processor based on the processor-executable instructions further comprises:

tracking location of the one or more objects from wide field of view images to narrow field of view images via the geolocation of the one or more objects in the wide field of view images.

8. The apparatus of claim 1, wherein the method performed by the processor based on the processor-executable instructions further comprises:

maintaining the enhanced, displayed view consistent in real-time with determined location of the recognized object of interest as a user of the apparatus relocates the apparatus.

9. The apparatus of claim 1, wherein the method performed by the processor based on the processor-executable instructions further comprises:

inserting objects into the enhanced, displayed view based on geographic data that indicates that the inserted object is to be occluded by another object in the enhanced, displayed view.

10. The apparatus of claim 1, wherein enhancing the displayed view comprises overlaying geographically located information from external sources onto the displayed view.

11. The apparatus of claim 1 wherein the apparatus is in communication with one or more similar apparatuses such that users of the apparatuses utilize the apparatuses in a gaming activity in a physical area.

12. The apparatus of claim 1, wherein the method performed by the processor based on the processor-executable instructions further comprises controlling the one or more optical sensors to capture one or more steps of field of view within the wide field of view image and the narrow field of view image.

13. The apparatus of claim 1, wherein the method performed by the processor based on the processor-executable instructions further comprises:

broadcasting the geolocation and additional descriptive information of the one or more objects to other apparatuses; and

providing on-display navigation cues in the enhanced, displayed view, the cues suggesting movement of the apparatus to locate the one or more objects in the scene and objects located and described by the other tracking apparatuses within view of the one or more optical sensors of the apparatus.

2. The apparatus of claim further comprising a first lens of the one or more optical sensors that captures the wide field of view and a second lens of the one or more optical sensors that captures the narrow field of view.

3. The apparatus of claim 2, further comprising a binocular unit that houses the first lens and the second lens.

4. The apparatus of claim 3, wherein the method performed by the processor based on the processor-executable instructions further comprises:

broadcasting the tracked location of the one or more objects and receiving tracked location information of objects outside a field of view of the one or more optical sensors, the received, tracked location information being received from other apparatuses.

5. The apparatus of claim 4, wherein the enhancing the displayed view further comprises:

augmenting the displayed view to indicate location of the one or more objects outside the field of view of the one or more optical sensors.

6. The apparatus of claim 3, wherein the enhancing the displayed view further comprises enhancing the displayed view to include descriptive information created by nearby users of other apparatuses.

14. A method for tracking objects by a user-carried or user-worn apparatus, the method comprising:

capturing, using one or more optical sensors, images of a scene, wherein the images include a wide field of view image of the scene and a narrow field of view image of the scene;

geolocating the one or more optical sensors, and a location of one or more objects depicted in the captured images based on a location of the one or more optical sensors;

recognizing an object of interest in the wide field of view image;

recognizing the object of interest in the narrow field of view image;

tracking a location of the recognized object of interest during transition from the wide field of view image to the narrow field of view image; and

enhancing a displayed view of the scene on a display based on the geolocating of the one or more objects, wherein the enhanced, displayed view comprises overlay content scaled in accordance with the tracking of the location of the recognized object of interest from the wide field of view image to the narrow field of view image.

15. The method of claim 14, further comprising:

capturing, using a first lens of the one or more optical sensors, the wide field of view image; and

capturing, using a second lens the one or more optical sensors, the narrow field of view image.

16. The method of claim 15, wherein the first lens and a second lens are housed in a binocular unit.

17. The method of claim 16, further comprising broadcasting the tracked location information for the one or more objects and receiving tracked location information for objects outside a field of view of the one or more optical sensors, the received, tracked location information being received from other apparatuses.

18. The method of claim 17, wherein the enhancing of the displayed view further comprises

enhancing the displayed view to indicate location of the one or more objects outside the field of view of the one or more optical sensors based on the received, tracked information received from other apparatuses.

19. The method of claim 14 further comprising tracking the one or more objects during a transition from displaying the wide field of view to displaying the narrow field of view as the enhanced, displayed view.

20. The method of claim 14, further comprising updating the enhanced, displayed view to remain consistent in real-time to indicate the location of the recognized object of interest as the location of the one or more optical sensors is modified.

21. The method of claim 14, further comprising inserting objects into the enhanced, displayed view based on geographic data that the inserted object is to be occluded by another object in the enhanced, displayed view.

22. The method of claim 14, wherein the enhancing of the displayed view further comprises overlaying geographically located information from external sources onto the displayed view.

23. The method of claim 14, further comprising:

storing one or more objects of interest marked by a user, tags associated with objects of interest, guiding directions to particular objects of interest, or user generated simulated objects for other users to observe from a location and viewing unit distinct from a location and viewing unit of the user.

24. The method of claim 23, further comprising guiding the user to a tagged object of interest, based on geographic data corresponding to the scene.

25. The method of claim 23 further comprising:

scanning an observed scene that is observed by a user via the apparatus and capturing and storing one or more image mosaics and landmark images for the observed scene;

designating a particular viewing angle and a designated location in relation to the observed scene; and

outputting guiding information that guides a user to the designated location using the image mosaics and landmark images, in response to the user moving away from the location.