Augmented reality panorama supporting visually impaired individuals

• US 9,526,658 B2
• Filed: 11/20/2013
• Issued: 12/27/2016
• Est. Priority Date: 02/24/2010
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Dialameh, Orange, Miller, Douglas, Blanchard, Charles, Dorcey, Timothy C., Sudol, Jeremi M.
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): HAQUE, MD NAZMUL
• Application Number:
  US14/085,541
• Publication Number:
  US 20140218493A1
• Time in Patent Office: 1,133 Days
• US Class Current: 1/1
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61H 2201/0157 portable
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 21/005 Details of specially-adapted software to access information, e.g. to browse through hyperlinked information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 5/125 the stations being mobile
• CPC Class Codes: H04M 1/72481 for visually impaired users

Abstract

There is presented a system and method for providing real-time object recognition to a remote user. The system comprises a portable communication device including a camera, at least one client-server host device remote from and accessible by the portable communication device over a network, and a recognition database accessible by the client-server host device or devices. A recognition application residing on the client-server host device or devices is capable of utilizing the recognition database to provide real-time object recognition of visual imagery captured using the portable communication device to the remote user of the portable communication device. In one embodiment, a sighted assistant shares an augmented reality panorama with a visually impaired user of the portable communication device where the panorama is constructed from sensor data from the device.

Claims

1. An augmented reality engine comprising:

a device interface configured to receive ambient sensor data from a mobile sensing device, the ambient sensor data reflecting a environment of a person and including a current field of view and device position data;

a panoramic engine configured to construct an augmented reality panorama representing a 3D digital model of the environment and including some of its objects at least partially based on the ambient sensor data with the current field of view integrated into the augmented reality panorama 3D digital model as a function of the device position; and

an assistant interface configured to;

present visually the 3D digital model of the augmented reality panorama along with the integrated current field of view to an assistant,

generate device commands based on an interaction between the assistant with at least one object in the 3D digital model and outside the current field of view,

transmit, via the device interface, the device commands to the remote mobile sensing device, the device commands instructing the mobile sensing device to capture additional object data associated with the objects in the augmented reality panorama outside the integrated current field of view, and

update the augmented reality panorama by incorporating the additional object data into the 3D digital model.

2. The engine of claim 1, wherein the current field of view comprises image data captured by the sensing device.

3. The engine of claim 1, wherein the augmented reality panorama comprises multi-modal panoramic data.

4. The engine of claim 3, where the multi-modal panoramic data includes haptic data and where the device commands comprises haptic feedback instructions associated with real-world objects in the remote environment proximate to the person and that correspond to the objects in the augmented reality panorama.

5. The engine of claim 1, wherein the mobile sensing device comprises a smart phone.

6. The engine of claim 1, augmented reality panorama comprises a personalized panorama associated with the person.

7. The engine of claim 6, the personalized panorama comprises private ambient sensor data under management of the person.

8. The engine of claim 7, wherein the personalized panorama comprises filtered portions as a function of the private ambient sensor data.

9. The engine of claim 1, wherein the augmented reality panorama comprises historical portions representative of historical ambient sensor data.

10. The engine of claim 9, wherein the augmented reality panorama comprises highlighted objects indicating differences between the historical portions and recently updated portions of the augmented reality panorama based on the current field of view.

11. The engine of claim 1, wherein the augmented reality panorama comprises public portions representative of publicly gathered sensor data.

12. The engine of claim 1, wherein the assistant interface is further configured to present a latency indicating a time difference between acquisition of the ambient sensor data and presentation of the current field of view.

13. The engine of claim 1, wherein assistant interface is further configured to correct placement of the current field of view in the augmented reality panorama based on at least two of the following ambient sensor data:

GPS location data, image registration data, accelerometer data, magnetometer data, wireless signal data, odometry data, and audio data.

14. The engine of claim 1, wherein the augmented reality panorama comprises metatags assigned to the objects within the augmented reality panorama.

15. The engine of claim 14, where wherein the metatags comprise metadata visible to the assistant and superimposed on the augmented reality panorama.

16. The engine of claim 14, wherein the metatags comprises non-visual metadata.

17. The engine of claim 1, wherein the assistant interface is further configured to indicate distances to the objects relative to the current field of view.

18. The engine of claim 1, wherein the device commands comprise instructions to the mobile sensing device to activate an actuator indicating an object'"'"'s position relative to the mobile sensing device.

19. The engine of claim 1, wherein the device commands comprise instructions to the mobile sensing device to emit audible signals indicating an object'"'"'s position relative to the mobile sensing device.

20. The engine of claim 1, wherein the panoramic engine comprises an object feature registration module configured to convert the ambient sensor data into object landmarks within the augmented reality panorama.

21. The engine of claim 20, wherein the object landmarks within the augmented reality panorama include image registration landmarks.

22. The engine of claim 21, wherein the image registration landmarks comprise Scale-Invariant Feature Transformation (SIFT) landmarks.

23. The engine of claim 1, wherein the device position data comprises device orientation data.

24. The engine of claim 1, wherein the device interface represents a remote device interface.

25. The engine of claim 24, wherein the mobile sensing device comprises a remote device.

26. The engine of claim 1, wherein the assistant interface is configured to render the augmented reality panorama to a sighted assistant.

27. The engine of claim 1, wherein the ambient sensor data represents an environment of a visually impaired person.

28. The engine of claim 1, wherein the device interface comprises a peer-to-peer network interface.

29. The engine of claim 1, wherein the mobile sensing device comprises glasses.