Image processing device, image processing method, and program for displaying an image in accordance with a selection from a displayed menu and based on a detection by a sensor

• US 10,521,085 B2
• Filed: 11/26/2018
• Issued: 12/31/2019
• Est. Priority Date: 03/24/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kashitani, Tatsuki
• Primary Examiner(s): Giles, Nicholas G
• Application Number:
  US16/199,524
• Publication Number:
  US 20190095060A1
• Time in Patent Office: 400 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in an input image based on an environment map, a superimposition display image generating unit which generates a superimposition display image by setting superimposition display data at the position of the object determined by the superimposition display position determining unit, an image superimposing unit which superimposes the superimposition display image on a visual field of a user, an operating object recognizing unit which recognizes an operating object imaged in the input image, and a process executing unit which executes a process corresponding to an item selected based on a position of the operating object recognized by the operating object recognizing unit.

Claims

1. An apparatus including an image device, the apparatus comprising:

circuitry configured to;

detect a tilt of the apparatus with a sensor;

acquire an environment map;

determine one of a predetermined flat surface or a curved surface based on the environment map;

cause a display to display a menu on the one of the predetermined flat surface or the curved surface, when the detected tilt indicates that the apparatus is tilted in a downward direction; and

cause the display to display an output image in accordance with a selection from the menu.

2. The apparatus according to claim 1, wherein the one of the predetermined flat surface or the curved surface is a surface of a ground, a table, or a floor.

3. The apparatus according to claim 1, wherein the circuitry is configured to cause the display to display the output image when the detected tilt of the apparatus exceeds a predetermined value.

4. The apparatus according to claim 1, wherein the menu includes an item corresponding to navigation, game, or mail.

5. The apparatus according to claim 1, wherein the output image is related to a navigation image.

6. The apparatus according to claim 1, wherein the circuitry is further configured to

acquire the environmental map based on an image of a surrounding environment captured by the image device;

and

detect the one of the predetermined flat surface or the curved surface in the captured image of the surround environment on which the menu is displayed.

7. The apparatus according to claim 1, wherein items included in the menu are displayed non-linearly.

8. The apparatus according to claim 1, wherein the selection from the menu is made by an operating object.

9. The apparatus according to claim 8, herein the operating object is a part of the user.

10. A method comprising:

detecting a tilt of an apparatus with a sensor;

acquiring an environment map;

determining, by processing circuitry of the apparatus, one of a predetermined flat surface or a curved surface based on the environment map;

causing, by the processing circuitry, a display to display a menu on the one of the predetermined flat surface or the curved surface, when the detected tilt indicates that the apparatus is tilted in a downward direction; and

causing, by the processing circuitry, the display to display an output image in accordance with a selection from the menu.

11. The method of claim 10, wherein the one of the predetermined flat surface or the curved surface is a surface of a ground, a table, or a floor.

12. The method of claim 10, further comprising:

causing, by the processing circuitry, the display to display the output image when the detected tilt of the apparatus exceeds a predetermined value.

13. The method of claim 10, wherein the menu includes an item corresponding to navigation, game, or mail.

14. The method of claim 10, wherein the output image is related to a navigation image.

15. A non-transitory computer-readable medium including computer-readable instructions that, when executed by a computer, cause the computer to perform a method of an apparatus, the method comprising:

detecting a tilt of the apparatus with a sensor;

acquiring an environment map;

determining one of a predetermined flat surface or a curved surface based on the environment map;

causing a display to display a menu on the one of the predetermined flat surface or the curved surface, when the detected tilt indicates that the apparatus is tilted in a downward direction; and

causing the display to display an output image in accordance with a selection from the menu.

16. The non-transitory computer-readable medium of claim 15, wherein the one of the predetermined flat surface or the curved surface is a surface of a ground, a table, or a floor.

17. The non-transitory computer-readable medium of claim 15, wherein the method further comprises:

causing the display to display the output image when the detected tilt of the apparatus exceeds a predetermined value.

18. The non-transitory computer-readable medium of claim 15, wherein the menu includes an item corresponding to navigation, game, or mail.

19. The non-transitory computer-readable medium of claim 15, wherein the output image is related to a navigation image.