Image processing device, image processing method, and program for display of a menu on a ground surface for selection with a user's foot

• US 9,367,964 B2
• Filed: 09/25/2015
• Issued: 06/14/2016
• Est. Priority Date: 03/24/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kashitani, Tatsuki
• Primary Examiner(s): GILES, NICHOLAS G
• Application Number:
  US14/865,756
• Publication Number:
  US 20160012645A1
• Time in Patent Office: 263 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in an input image based on an environment map, a superimposition display image generating unit which generates a superimposition display image by setting superimposition display data at the position of the object determined by the superimposition display position determining unit, an image superimposing unit which superimposes the superimposition display image on a visual field of a user, an operating object recognizing unit which recognizes an operating object imaged in the input image, and a process executing unit which executes a process corresponding to an item selected based on a position of the operating object recognized by the operating object recognizing unit.

Claims

1. A device connected to an imaging device, the device comprising:

circuitry configured to

detect an orientation of the imaging device exceeding a predetermined value, indicating that a user bearing the imaging device looks down;

control a display to display a menu on a ground surface when the orientation of the imaging device exceeds the predetermined value; and

reproduce, when the orientation of the imaging device no longer exceeds the predetermined value, an output associated with a selection from the menu made using the user'"'"'s foot.

2. The device according to claim 1, wherein the menu includes a first layer of items.

3. The device according to claim 2, wherein the menu includes a second layer of items, which is displayed upon selection of an item from the first layer of items, the second layer of items being displayed at a location different from a location of the first layer of items.

4. The device according to claim 1, wherein the menu includes navigation or game as an item.

5. The device according to claim 4, wherein the menu includes navigation as an item and, when the navigation item is selected, the output is a navigation image.

6. The device according to claim 1, wherein the circuitry is further configured to acquire an environment map and detect the ground surface to display the menu on.

7. The device according to claim 1, wherein the menu is displayed in a semi-circular shape.

8. A method comprising:

detecting an orientation of an imaging device exceeding a predetermined value, indicating that a user bearing the imaging device looks down;

controlling a display to display a menu on a ground surface when the orientation of the imaging device exceeds the predetermined value; and

reproducing, when the orientation of the imaging device no longer exceeds the predetermined value, an output associated with a selection from the menu made using the user'"'"'s foot.

9. The method according to claim 8, wherein the menu includes a first layer of items.

10. The method according to claim 9, wherein the menu includes a second layer of items, which is displayed upon selection of an item from the first layer of items, the second layer of items being displayed at a location different from a location of the first layer of items.

11. The method according to claim 8, wherein the menu includes navigation or game as an item.

12. The method according to claim 11, wherein the menu includes navigation as an item and, when the navigation item is selected, the output is a navigation image.

13. The method according to claim 8, wherein the method further comprises acquiring an environment map and detecting the ground surface to display the menu on.

14. The method according to claim 8, wherein the menu is displayed in a semi-circular shape.

15. A non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a computer, cause the computer to perform a method comprising:

detecting an orientation of an imaging device exceeding a predetermined value, indicating that a user bearing the imaging device looks down;

controlling a display to display a menu on a ground surface when the orientation of the imaging device exceeds the predetermined value; and

reproducing, when the orientation of the imaging device no longer exceeds the predetermined value, an output associated with a selection from the menu made using the user'"'"'s foot.

16. The non-transitory computer-readable storage medium according to claim 15, wherein the menu includes a first layer of items.

17. The non-transitory computer-readable storage medium according to claim 16, wherein the menu includes a second layer of items, which is displayed upon selection of an item from the first layer of items, the second layer of items being displayed at a location different from a location of the first layer of items.

18. The non-transitory computer-readable storage medium according to claim 15, wherein the menu includes navigation or game as an item.

19. The non-transitory computer-readable storage medium according to claim 18, wherein the menu includes navigation as an item and, when the navigation item is selected, the output is a navigation image.

20. The non-transitory computer-readable storage medium according to claim 15, wherein the method further comprises acquiring an environment map and detecting the ground surface to display the menu on.

21. The non-transitory computer-readable storage medium according to claim 15, wherein the menu is displayed in a semi-circular shape.