Switching between a first operational mode and a second operational mode using a natural motion gesture

• US 8,922,487 B2
• Filed: 11/12/2013
• Issued: 12/30/2014
• Est. Priority Date: 06/23/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ho, Casey Kwok Ching, Nanvati, Sharvil
• Primary Examiner(s): MERCEDES, DISMERY E
• Application Number:
  US14/078,498
• Publication Number:
  US 20140071166A1
• Time in Patent Office: 413 Days
• Field of Search: None
• US Class Current: 345/158
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mobile device is operative to change from a first operational mode to a second or third operational mode based on a user'"'"'s natural motion gesture. The first operational mode may include a voice input mode in which a user provides a voice input to the mobile device. After providing the voice input to the mobile device, the user then makes a natural motion gesture and a determination is made as to whether the natural motion gesture places the mobile device in the second or third operational mode. The second operational mode includes an augmented reality display mode in which the mobile device displays images recorded from a camera overlaid with computer-generated images corresponding to results output in response to the voice input. The third operational mode includes a reading display mode in which the mobile device displays, without augmented reality, results output in response to the voice input.

Claims

1. A method comprising:

receiving, by a processor and from an orientation sensor, a first set of orientation data representative of a mobile computing device being in a first orientation;

receiving, by the processor and from the orientation sensor, a second set of orientation data representative of the mobile computing device being in a second orientation;

changing, by the processor, an operational mode of the mobile computing device from a first operational mode to a second operational mode, the changing responsive to;

determining that the first set of orientation data approximates a first set of baseline data representative of a first state of the mobile computing device, the first state of the mobile computing device comprising an input receiving mode operative to receive voice input from a user, and

determining that the second set of orientation data approximates a second set of baseline data representative of a second state of the mobile computing device, the second state of the mobile computing device comprising a visual display mode operative to display a selected portion of a real-world environment overlaid by a computer-generated graphic element;

receiving, by the processor and while the mobile computing device is in the first operational mode, an indication of voice input;

outputting, by the processor and for display while the mobile computing device is in the second operational mode, a selected portion of a real-world environment physically located around the mobile computing device and a computer-generated graphic element overlaid on the displayed portion of the real-world environment, the computer-generated graphic comprising a visual indication of a search result, wherein the search result is determined responsive to a query based on the indication of the voice input.

2. The method of claim 1, wherein the second state of the mobile computing device comprises the mobile computing device being in the second operational mode.

3. The method of claim 1, wherein the first set of orientation data indicates that a front surface of the mobile computing device is positioned approximately against a head of the user.

4. The method of claim 1, wherein the second set of orientation data indicates that a front surface of the mobile computing device is approximately perpendicular to a ground surface.

5. The method of claim 1, wherein the second set of orientation data indicates that a front surface of the mobile computing device forms an approximate acute angle to a ground surface.

6. The method of claim 1, further comprising:

receiving, by the processor and from the orientation sensor, a set of transitional orientation data representative of transitional orientations of the mobile computing device;

wherein the changing of the operational mode of the mobile computing device from the first operational mode to the second operational mode is further responsive to determining that the set of transitional orientation data approximates a third set of baseline data representative of a transition of the mobile computing device from the first state to the second state.

7. The method of claim 1, wherein the selected portion of the real-world environment surrounding the mobile computing device is displayed in real-time.

8. The method of claim 1, wherein the selected portion of the real-world environment surrounding the mobile computing device was previously recorded and stored in a storage device.

9. The method of claim 1, the indication of the voice input associated with an address, the search result comprising an indication of a building associated with the address, wherein the displayed portion of the real-world environment comprises at least a portion of the building and the computer-generated graphic element includes a visual indication of the building.

10. The method of claim 1, the indication of the voice input associated with a destination different than a current location of the mobile computing device, wherein the computer-generated graphic element includes a visual indication of a direction to travel in order to reach the destination.

11. A non-transitory computer-readable medium that stores instructions that, when executed by a processor of a mobile computing device, cause the mobile computing device to perform a method comprising:

receiving, by the processor and from an orientation sensor, a first set of orientation data representative of the mobile computing device being in a first orientation;

receiving, by the processor and from the orientation sensor, a second set of orientation data representative of the mobile computing device being in a second orientation;

changing, by the processor, an operational mode of the mobile computing device from a first operational mode to a second operational mode, the changing responsive to;

determining that the first set of orientation data approximates a first set of baseline data representative of a first state of the mobile computing device, the first state of the mobile computing device comprising an input receiving mode operative to receive voice input from a user, and

determining that the second set of orientation data approximates a second set of baseline data representative of a second state of the mobile computing device, the second state of the mobile computing device comprising a visual display mode operative to display a selected portion of a real-world environment overlaid by a computer-generated graphic element;

receiving, by the processor and while the mobile computing device is in the first operational mode, an indication of voice input;

outputting, by the processor and for display while the mobile computing device is in the second operational mode, a selected portion of a real-world environment physically located around the mobile computing device and a computer-generated graphic element overlaid on the displayed portion of the real-world environment, the computer-generated graphic comprising a visual indication of a search result, wherein the search result is determined responsive to a query based on the indication of the voice input.

12. The non-transitory computer-readable medium of claim 11, wherein the second state of the mobile computing device comprises the mobile computing device being in the second operational mode.

13. The non-transitory computer-readable medium of claim 11, wherein the first set of orientation data indicates that a front surface of the mobile computing device is positioned approximately against a head of the user.

14. The non-transitory computer-readable medium of claim 11, wherein the second set of orientation data indicates that a front surface of the mobile computing device is approximately perpendicular to a ground surface.

15. The non-transitory computer-readable medium of claim 11, wherein the second set of orientation data indicates that a front surface of the mobile computing device forms an approximate acute angle to a ground surface.

16. The non-transitory computer-readable medium of claim 11, the method further comprising:

receiving, by the processor and from the orientation sensor, a set of transitional orientation data representative of transitional orientations of the mobile computing device;

wherein the changing of the operational mode of the mobile computing device from the first operational mode to the second operational mode is further responsive to determining that the set of transitional orientation data approximates a third set of baseline data representative of a transition of the mobile computing device from the first state to the second state.

17. The non-transitory computer-readable medium of claim 11, wherein the selected portion of the real-world environment surrounding the mobile computing device is displayed in real-time.

18. The non-transitory computer-readable medium of claim 11, wherein the selected portion of the real-world environment surrounding the mobile computing device was previously recorded and stored in a storage device.

19. The non-transitory computer-readable medium of claim 11, the indication of the voice input associated with an address, the search result comprising an indication of a building associated with the address, wherein the displayed portion of the real-world environment comprises at least a portion of the building and the computer-generated graphic element includes a visual indication of the building.

20. The non-transitory computer-readable medium of claim 11, the indication of the voice input associated with a destination different than a current location of the mobile computing device, wherein the computer-generated graphic element includes a visual indication of a direction to travel in order to reach the destination.