Media streaming methods, apparatus and systems

• US 10,078,971 B2
• Filed: 09/14/2015
• Issued: 09/18/2018
• Est. Priority Date: 09/03/2014
• Status: Active Grant

• Current Assignee: Aira Tech Corporation
• Original Assignee: ARIA Tech Corporation
• Inventors: Kanuganti, Suman, Chang, Yuja, Bock, Lawrence
• Primary Examiner(s): Walker, Jared T
• Application Number:
  US14/853,548
• Publication Number:
  US 20160063893A1
• Time in Patent Office: 1,100 Days
• Field of Search: 348 62
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G09B 21/001 Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/003 using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: H04M 1/72475 specially adapted for disabled users
• CPC Class Codes: H04N 21/25841 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/8545 for generating interactive applications
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Techniques and systems are provided for assisting a user through a guidance mode activity. Such a user may be visually or otherwise impaired, or a user requiring help for other reasons. In some embodiments, a client device, held or worn by the user, may comprise a video capture device and several sensors, and may send video data and sensor data to a server. The server may comprise a processor and artificial intelligence. The server may send the video data and sensor data to an agent device. The agent device may provide content for display on an agent interface. An agent may view the agent interface, and assist the user in real time through audio instructions or other feedback.

Claims

1. A portable electronic apparatus comprising:

a power source;

a storage device storing profile data of a user of the portable electronic apparatus;

a video capture device configured to capture video data of an environment in which the portable electronic apparatus is located;

a video coding device configured to receive video data from the video capture device and compress the video data to generate compressed video data;

a location sensor configured to obtain location data that indicates a location of the portable electronic apparatus;

an input device;

wireless communication circuitry;

a processor operatively coupled to the power source, the video coding device, the location sensor, the input device and the wireless communication circuitry, the processor configured to;

receive input corresponding to activation of a guidance mode;

activate the guidance mode and generate and send a guidance request in response to receiving the input; and

while guidance mode is active, transmit compressed video data and location data using the wireless communication circuitry to a second device, and receive guidance feedback data from the second device;

detect a low network condition; and

implement, in response to detecting the low network condition, a local mode in which the processor is configured to analyze the guidance request, either the video data or the compressed video data, and the location data to provide one or more guidance feedback instructions; and

an output device configured to output information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

2. The portable electronic apparatus of claim 1 wherein the input corresponding to activation of a guidance mode comprises user input received from the input device.

3. The portable electronic apparatus of claim 1 wherein the profile data includes at least visual impairment data indicating a visual impairment of the user.

4. The portable electronic apparatus of claim 1 wherein the processor is configured to stream the compressed video data as a live video stream to at least one or more of a server or a live agent device as the video data is captured by the video capture device and compressed by the video coding device.

5. The portable electronic apparatus of claim 1 wherein the processor is further configured to implement a privacy mode that, when enabled, disables the video capture device from capturing video data.

6. The portable electronic apparatus of claim 1 wherein the processor is configured to detect a battery level of the portable electronic apparatus being below a predefined threshold level, and implement the local mode in response to detecting the battery level being below the predefined threshold level.

7. The portable electronic apparatus of claim 1 wherein the portable electronic apparatus comprises a wearable electronic device having a frame front, first and second lenses attached to the frame front, and first and second temples that extend from the frame front and are sized and shaped to extend over left and right ears of the user, respectively.

8. The portable electronic apparatus of claim 1 wherein the output device is further configured to provide haptic output.

9. A method of providing assistance to a visually-impaired user, the method comprising:

storing profile data of the visually-impaired user of a client device;

capturing video data of an environment in which the client device is located;

compressing the video data to generate compressed video data;

obtaining sensor data from one or more sensors, the one or more sensors including at least a location sensor;

receiving input corresponding to activation of a guidance mode;

generating a guidance request in response to receiving the input;

providing the guidance request, the compressed video data, and the sensor data to at least one or more of a server or a live agent device;

receiving guidance feedback data from at least one or more of the server or the live agent device;

detecting a low network condition;

implementing, in response to detecting the low network condition, a local mode in which the client device is configured to analyze the guidance request, either the video data or the compressed video data, and the sensor data to provide one or more guidance feedback instructions; and

outputting information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

10. The method of claim 9 further comprising streaming the compressed video data as a live video stream to at least one or more of the server or the live agent device as the video data is captured and compressed.

11. The method of claim 9 further comprising disabling the client device from capturing video data upon implementation of a privacy mode.

12. The method of claim 9 further comprising detecting a battery level of the client device as being below a predefined threshold level and implementing the local mode in response to detecting the battery level being below the predefined threshold level.

16. The method of claim 9 wherein the profile data includes at least visual impairment data indicating a visual impairment of the user.

13. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

instructions that cause the one or more processors to store profile data of a user of a client device;

instructions that cause the one or more processors to capture video data of an environment in which the client device is located;

instructions that cause the one or more processors to compress the video data to generate compressed video data;

instructions that cause the one or more processors to obtain sensor data from one or more sensors, the one or more sensors including at least a location sensor;

instructions that cause the one or more processors to receive input corresponding to activation of a guidance mode;

instructions that cause the one or more processors to generate a guidance request in response to receiving the input;

instructions that cause the one or more processors to provide the guidance request, the compressed video data, and the sensor data to at least one or more of a server or a live agent device;

instructions that cause the one or more processors to receive guidance feedback data from at least one or more of the server or the live agent device;

instructions that cause the one or more processors to detect a low network condition;

instructions that cause the one or more processors to implement, in response to detecting the low network condition, a local mode in which the client device is configured to analyze the guidance request, either the video data or the compressed video data, and the sensor data to provide one or more guidance feedback instructions; and

instructions that cause the one or more processors to output information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

14. The non-transitory computer-readable memory of claim 13 further comprising instructions that cause the one or more processors to stream the compressed video data as a live video stream to at least one or more of the server or the live agent device as the video data is captured and compressed.

15. The non-transitory computer-readable memory of claim 13 further comprising instructions that cause the one or more processors to disable the client device from capturing video data upon implementation of a privacy mode.

17. The non-transitory computer-readable memory of claim 13 wherein the profile data includes at least visual impairment data indicating a visual impairment of the user.

18. The non-transitory computer-readable memory of claim 13 further comprising instructions that cause the one or more processors to detect a battery level of the client device as being below a predefined threshold level and to implement the local mode in response to detecting the battery level being below the predefined threshold level.

19. A portable electronic apparatus comprising:

a power source;

a storage device storing profile data of a user of the portable electronic apparatus;

a video capture device configured to capture video data of an environment in which the portable electronic apparatus is located;

a video coding device configured to receive video data from the video capture device and compress the video data to generate compressed video data;

a location sensor configured to obtain location data that indicates a location of the portable electronic apparatus;

an input device;

wireless communication circuitry;

a processor operatively coupled to the power source, the video coding device, the location sensor, the input device and the wireless communication circuitry, the processor configured to;

receive input corresponding to activation of a guidance mode;

activate the guidance mode and generate and send a guidance request in response to receiving the input; and

while guidance mode is active, transmit compressed video data and location data using the wireless communication circuitry to a second device, and receive guidance feedback data from the second device;

detect a battery level of the portable electronic apparatus being below a predefined threshold level; and

implement, in response to detecting the battery level being below the predefined threshold level, a local mode in which the processor is configured to analyze the guidance request, either the video data or the compressed video data, and the location data to provide one or more guidance feedback instructions; and

an output device configured to output information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

20. The portable electronic apparatus of claim 19 wherein the input corresponding to activation of a guidance mode comprises user input received from the input device.

21. The portable electronic apparatus of claim 19 wherein the profile data includes at least visual impairment data indicating a visual impairment of the user.

22. The portable electronic apparatus of claim 19 wherein the processor is configured to stream the compressed video data as a live video stream to at least one or more of a server or a live agent device as the video data is captured and compressed by the video coding device.

23. The portable electronic apparatus of claim 19 wherein the processor is further configured to implement a privacy mode that, when enabled, disables the video capture device from capturing video data.

24. The portable electronic apparatus of claim 19 wherein the processor is configured to detect a low network condition, and implement the local mode in response to detecting the low network condition.

25. The portable electronic apparatus of claim 19 wherein the portable electronic apparatus comprises a wearable electronic device having a frame front, first and second lenses attached to the frame front, and first and second temples that extend from the frame front and are sized and shaped to extend over left and right ears of the user, respectively.

26. The portable electronic apparatus of claim 19 wherein the output device is further configured to provide haptic output.

27. A method of providing assistance to a visually-impaired user, the method comprising:

storing profile data of the visually-impaired user of a client device;

capturing video data of an environment in which the client device is located;

compressing the video data to generate compressed video data;

obtaining sensor data from one or more sensors, the one or more sensors including at least a location sensor;

receiving input corresponding to activation of a guidance mode;

generating a guidance request in response to receiving the input;

providing the guidance request, the compressed video data, and the sensor data to at least one or more of a server or a live agent device;

receiving guidance feedback data from at least one or more of the server or the live agent device;

detecting a battery level of the client device as being below a predefined threshold level;

implementing, in response to detecting the battery level as being below the predefined threshold level, a local mode in which the client device is configured to analyze the guidance request, either the video data or the compressed video data, and the sensor data to provide one or more guidance feedback instructions; and

outputting information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

28. The method of claim 27 further comprising streaming the compressed video data as a live video stream to at least one or more of the server or the live agent device as the video data is captured and compressed.

29. The method of claim 27 further comprising disabling the client device from capturing video data upon implementation of a privacy mode.

30. The method of claim 27 further comprising detecting a low network condition and implementing the local mode in response to detecting the low network condition.

31. The method of claim 27 wherein the profile data includes at least visual impairment data indicating a visual impairment of the visually-impaired user.

32. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

instructions that cause the one or more processors to store profile data of a user of a client device;

instructions that cause the one or more processors to capture video data of an environment in which the client device is located;

instructions that cause the one or more processors to compress the video data to generate compressed video data;

instructions that cause the one or more processors to obtain sensor data from one or more sensors, the one or more sensors including at least a location sensor;

instructions that cause the one or more processors to receive input corresponding to activation of a guidance mode;

instructions that cause the one or more processors to generate a guidance request in response to receiving the input;

instructions that cause the one or more processors to provide the guidance request, the compressed video data, and the sensor data to at least one or more of a server or a live agent device;

instructions that cause the one or more processors to receive guidance feedback data from at least one or more of the server or the live agent device;

instructions that cause the one or more processors to detect a battery level of the client device as being below a predefined threshold level;

instructions that cause the one or more processors to implement, in response to detecting the battery level as being below the predefined threshold level, a local mode in which the client device is configured to analyze the guidance request, either the video data or the compressed video data, and the sensor data to provide one or more guidance feedback instructions; and

instructions that cause the one or more processors to output information based on one or more of the guidance feedback data or the one or more guidance feedback instructions.

33. The non-transitory computer-readable memory of claim 32 further comprising instructions that cause the one or more processors to stream the compressed video data as a live video stream to at least one or more of the server or the live agent device as the video data is captured and compressed.

34. The non-transitory computer-readable memory of claim 32 further comprising instructions that cause the one or more processors to disable the client device from capturing video data upon implementation of a privacy mode.

35. The non-transitory computer-readable memory of claim 32 further comprising instructions that cause the one or more processors to detect a low network condition and to implement the local mode in response to detecting the low network condition.

36. The non-transitory computer-readable memory of claim 32 wherein the profile data includes at least visual impairment data indicating a visual impairment of the user.