Augmented reality panorama supporting visually impaired individuals

• US 10,535,279 B2
• Filed: 11/16/2016
• Issued: 01/14/2020
• Est. Priority Date: 02/24/2010
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Dialameh, Orang, Miller, Douglas, Blanchard, Charles, Dorcey, Timothy C., Sudol, Jeremi M.
• Primary Examiner(s): Haque, Md N
• Application Number:
  US15/353,016
• Publication Number:
  US 20170069227A1
• Time in Patent Office: 1,154 Days
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61H 2201/0157 portable
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 21/005 Details of specially-adapted software to access information, e.g. to browse through hyperlinked information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: G09B 5/125 the stations being mobile
• CPC Class Codes: H04M 1/72481 for visually impaired users

Abstract

There is presented a system and method for providing real-time object recognition to a remote user. The system comprises a portable communication device including a camera, at least one client-server host device remote from and accessible by the portable communication device over a network, and a recognition database accessible by the client-server host device or devices. A recognition application residing on the client-server host device or devices is capable of utilizing the recognition database to provide real-time object recognition of visual imagery captured using the portable communication device to the remote user of the portable communication device. In one embodiment, a sighted assistant shares an augmented reality panorama with a visually impaired user of the portable communication device where the panorama is constructed from sensor data from the device.

Claims

1. An augmented reality panorama server comprising:

A remote device interface configured to receive sensor data from a plurality of remote devices over a network wherein the sensor data includes at least location data;

A tangible, non-transitory computer readable memory storing software instructions; and

At least one processor coupled with the remote device interface and the memory, and configurable, upon execution of the software instructions, to operate as a panorama engine that;

stores the sensor data, including the location data, from the plurality of remote devices in a recognition database, the sensor data stored in the recognition database including real-time sensor data whose location data specifies a current location of a first of the plurality of remote devices and historical sensor data whose location data specifies a historical location of a second of the plurality of remote devices, the second remote device being a mobile device;

identifies a target location;

collects sensor data from the recognition database as a function of the target location and the location data in the sensor data;

constructs an augmented reality panorama associated with the target location based on the collected sensor data, the augmented reality panorama constructed based on the sensor data of at least two different remote devices including the real-time sensor data of the first remote device and the historical sensor data of the second remote device;

generates commands related to objects within the augmented reality panorama; and

causes a display to present the augmented reality panorama, via a display, and make the commands available to a user.

2. The server of claim 1, wherein the remote device interface is configured to couple with a peer-to-peer network operating as the network.

3. The server of claim 1, wherein the remote device interface is configured to couple with at least one of the following types of remote devices over the network:

a mobile device, a portable device, a smart phone, and glasses.

4. The server of claim 1, wherein the sensor data includes at least one of the following data modalities:

image data, video data, video stream data, audio data, accelerometer data, magnetic data, bio data, barcode data, RFID data, odometer data, and health metric data.

5. The server of claim 1, wherein the sensor data comprises ambient sensor data from the plurality of remote devices.

6. The server of claim 1, wherein the recognition database stores recognizable objects as a function of the sensor data.

7. The server of claim 1, wherein augmented reality panorama comprises at least a two dimensional augmented reality panorama.

8. The server of claim 7, wherein augmented reality panorama comprises at least a three dimensional augmented reality panorama.

9. The server of claim 1, wherein the augmented reality panorama comprises a field of view of at least one of the plurality of remote devices.

10. The server of claim 8, wherein the field of view comprises a real-time field of view.

11. The server of claim 1, wherein the augmented reality panorama comprises a multi-modal panorama.

12. The server of claim 11, wherein the multi-modal panorama comprises at least one of the following data modalities:

visual data, audio data, haptic data, kinesthetic data, and metadata data.

13. The server of claim 1, wherein the augmented reality panorama comprises a live panorama.

14. The server of claim 1, wherein the panorama engines constructs the augmented reality panorama as a function of stitching the sensor data according to registration features within the sensor data.

15. The server of claim 14, wherein the registration features including at least one of the following:

SIFT features, SURF features, environment features, landmark features, text features, audio features, and barcode features.

16. The server of claim 1, wherein the augmented reality panorama comprises network links to information related to the objects.

17. The server of claim 16, wherein network links initiate location based services based on the location data.

18. The server of claim 1, wherein the location data comprises global positioning system data.

19. The server of claim 1, wherein the location data comprises image features.

20. The server of claim 1, further including the recognition database.

21. The server of claim 20, wherein the recognition database comprises a private database.

22. The server of claim 21, wherein the augmented reality panorama comprises a personalized augmented reality panorama constructed from the private database.

23. The server of claim 22, wherein the recognition database comprises a shared community database.

24. The server of claim 22, wherein the augmented reality panorama comprises a shared augmented reality panorama.

25. The server of claim 1, wherein the commands include device commands.

26. The server of claim 25, wherein the device command include at least one of the following:

change data acquisition rate, change sampling rate, adjust resolution, adjust compression parameters, adjust encryption parameters, alternate communication path, trigger an actuator, emit audible signals, and manage an application.

27. The server of claim 1, wherein the commands include panorama commands.

28. The server of claim 27, wherein the panorama commands include at least one of the following:

pan, tilt, rotate, filter, and time shift.

29. The server of claim 1, wherein the commands include object management commands related to the objects.

30. The server of claim 29, wherein the object management commands include at least one of the following:

add, delete, assign metadata, tag, update, select, filter, and save.

31. The server of claim 1, wherein the at least two different remote devices have different location data.