VOICE-DRIVEN INTERFACE TO CONTROL MULTI-LAYERED CONTENT IN A HEAD MOUNTED DISPLAY

• US 20180067717A1
• Filed: 09/05/2017
• Published: 03/08/2018
• Est. Priority Date: 09/02/2016
• Status: Abandoned Application

• Current Assignee: Allomind, Inc. (Skylights, Inc.)
• Original Assignee: Allomind, Inc. (Skylights, Inc.)
• Inventors: LIMAYE, ABHIJIT, LEUNG, THEOBOLT, VASA, NIRAV
• Application Number:
  US15/696,046
• Publication Number:
  US 20180067717A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

The present disclosure relates to user interfaces for virtual reality and augmented reality head mounted displays. One embodiment of the present disclosure involves: (i) initializing an application instance for interacting with multi-layered media content output to a head-mounted display; (ii) receiving digitized speech and processing the digitized speech to extract text; (iii) comparing one or more words from the extracted text with a predefined keyword list to determine a response identifier; (iv) determining if the response identifier matches a response identifier stored in a database; and (v) if the response identifier matches a response identifier stored in the database, triggering an action in the application instance or retrieving from the database metadata associated with a media layer of the multi-layered media content.

Claims

1. A method, comprising:

initializing an application instance for interacting with multi-layered media content output to a head-mounted display, wherein the multi-layered media content comprises a plurality of media layers;

receiving digitized speech and processing the digitized speech to extract text;

comparing one or more words from the extracted text with a predefined keyword list to determine a response identifier;

determining if the response identifier matches a response identifier stored in a database; and

if the response identifier matches a response identifier stored in the database, triggering an action in the application instance or retrieving from the database metadata associated with a media layer of the multi-layered media content.

2. The method of claim 1, further comprising:

determining that the response identifier matches an application context response identifier stored in the database and triggering an action in the application instance.

3. The method of claim 1, further comprising:

determining that the response identifier matches an object context response identifier stored in the database and retrieving from the database metadata associated with a media layer of the multi-layered media content.

4. The method of claim 3, further comprising:

processing the metadata to generate a response to be applied to the multi-layered content for output to the head mounted display.

5. The method of claim 4, wherein the response creates a new information layer that is combined with other media layers of the multi-layered content.

6. The method of claim 1, wherein if the response identifier does not match a response identifier stored in the database, initiating a text query at the application instance to a third party domain service to perform response identifier matching.

7. The method of claim 1, further comprising:

enabling an awake mode in the application instance such that a salutation is not required each time a user initiates a vocal interaction.

8. The method of claim 1, wherein the application instance displays a motion user interface over the multi-layered content, the motion user interface comprising a textual keyword, an audio prompt or haptic feedback that cues a user to make voice initiated interactions.

9. The method of claim 1, wherein the digitized speech is received from a microphone communicatively coupled to the head mounted display.

16. The non-transitory computer-readable medium of claim 9, wherein the digitized speech is received from a microphone communicatively coupled to the head mounted display.

10. A non-transitory computer-readable medium having computer executable program code stored thereon, the computer executable program code configured to cause a system to:

initialize an application instance for interacting with multi-layered media content output to a head-mounted display;

receive digitized speech and processing the digitized speech to extract text;

compare one or more words from the extracted text with a predefined keyword list to determine a response identifier associated with a media layer of the multi-layered media content;

determine if the response identifier matches a response identifier stored in a database; and

if the response identifier matches a response identifier stored in the database, trigger an action in the application instance or retrieve from the database metadata associated with a media layer of the multi-layered media content.

11. The non-transitory computer-readable medium of claim 10, wherein the computer executable program code is configured to further cause the system to:

determine that the response identifier matches an application context response identifier stored in the database and triggering an action in the application instance.

12. The non-transitory computer-readable medium of claim 10, wherein the computer executable program code is configured to further cause the system to:

determine that the response identifier matches an object context response identifier stored in the database and retrieving from the database metadata associated with a media layer of the multi-layered media content.

13. The non-transitory computer-readable medium of claim 12, wherein the computer executable program code is configured to further cause the system to:

process the metadata to generate a response to be applied to the multi-layered content for output to the head mounted display.

14. The non-transitory computer-readable medium of claim 13, wherein the response creates a new information layer that is combined with other media layers of the multi-layered content.

15. The non-transitory computer-readable medium of claim 10, wherein if the response identifier does not match a response identifier stored in the database, the computer executable program code is configured to further cause the system to initiate a text query at the application instance to a third party domain service to perform response identifier matching.

17. A system, comprising:

a head-mounted display; and

a non-transitory computer-readable medium having computer executable program code stored thereon, the computer executable program code configured to cause the system to;

initialize an application instance for interacting with multi-layered media content output to the head-mounted display;

receive digitized speech and processing the digitized speech to extract text;

compare one or more words from the extracted text with a predefined keyword list to determine a response identifier associated with a media layer of the multi-layered media content;

determine if the response identifier matches a response identifier stored in a database; and

if the response identifier matches a response identifier stored in the database, trigger an action in the application instance or retrieve from the database metadata associated with a media layer of the multi-layered media content.

18. The system of claim 17, wherein the non-transitory computer-readable medium is a memory of the head-mounted display.

19. The system of claim 17, wherein the non-transitory computer-readable medium is a memory of a mobile device communicatively coupled to the head-mounted display.

20. The system of claim 17, wherein the non-transitory computer-readable medium is a memory of a server communicatively coupled to the head-mounted display.