IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD AND PROGRAM

• US 20110234879A1
• Filed: 03/16/2011
• Published: 09/29/2011
• Est. Priority Date: 03/24/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: KASHITANI, Tatsuki
• Granted Patent: US 8,502,903 B2
• Time in Patent Office: Days
• US Class Current: 348/333.02
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

There is provided an image processing device including a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in an input image based on an environment map, a superimposition display image generating unit which generates a superimposition display image by setting superimposition display data at the position of the object determined by the superimposition display position determining unit, an image superimposing unit which superimposes the superimposition display image on a visual field of a user, an operating object recognizing unit which recognizes an operating object imaged in the input image, and a process executing unit which executes a process corresponding to an item selected based on a position of the operating object recognized by the operating object recognizing unit.

Claims

1. An image processing device comprising:

a feature data storage unit which stores feature data indicating a feature of appearance of an object;

a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user and an item position which is a position of an item forming the superimposition display data;

an environment map generating unit which generates an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit;

a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map;

a superimposition display image generating unit which generates a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit;

an image superimposing unit which superimposes the superimposition display image on the visual field of the user;

an operating object recognizing unit which recognizes an operating object imaged in the input image; and

a process executing unit which executes a process corresponding to the item selected based on a position of the operating object recognized by the operating object recognizing unit.

2. The image processing device according to claim 1,

wherein the operating object recognizing unit recognizes a foot imaged in the input image as the operating object.

3. The image processing device according to claim 2,

wherein the operating object recognizing unit performs matching between a shoe registration image which is an image of a shoe which has been registered in advance and the input image, and when the operating object recognizing unit determines that a shoe which matches with the shoes in the shoe registration image is imaged in the input image, the operating object recognizing unit recognizes the shoe as the operating object.

4. The image processing device according to claim 2,

wherein, when a user wears the imaging device on his/her head, the operating object recognizing unit determines whether or not a foot imaged in the input image is coming in from a side which is the nearest from the user among respective sides that form the input image, and when the operating object recognizing unit determines that the foot is coming in from the side which is the nearest from the user, the operating object recognizing unit recognizes the foot as the operating object.

5. The image processing device according to claim 2,

wherein the operating object recognizing unit determines whether or not a shoe with a mark, which has been subjected to predetermined marking in advance, is imaged in the input image, and when the operating object recognizing unit determines that the shoe with a mark is imaged in the input image, the operating object recognizing unit recognizes the shoe as the operating object.

6. The image processing device according to claim 2,

wherein the process executing unit determines whether or not a contact is detected by a contact sensor attached to the foot, and when the contact is detected by the contact sensor, the process executing unit executes a process corresponding to the item selected based on a position of the foot.

7. The image processing device according to claim 2,

wherein the process executing unit determines whether or not the operating object recognized by the operating object recognizing unit has stopped at substantially the same position for a predetermined time period, and when the process executing unit determines that the operating object has stopped at substantially the same position for the predetermined time period, the process executing unit executes a process corresponding to the item selected based on a position of the operating object.

8. The image processing device according to claim 1,

wherein the superimposition display data storage unit stores other superimposition display data which is displayed subsequently to the superimposition display data,

wherein, when the item is selected by the process executing unit, the superimposition display image generating unit generates a new superimposition display image by further setting the other superimposition display data, and

wherein, when the item is selected by the process executing unit, the image superimposing unit further superimposes the other superimposition display data on the new superimposition display image.

9. The image processing device according to claim 1,

wherein the operating object recognizing unit recognizes a hand imaged in the input image as the operating object.

10. The image processing device according to claim 1,

wherein the image superimposing unit superimposes the superimposition display image on the visual field of the user and causes a display unit to display the superimposition display image.

11. The image processing device according to claim 10,

wherein, when the image superimposing unit detects, by using a sensor which detects the gradient of the imaging device, that a user wearing the imaging device on his/her head tilts the head in a downward direction at a gradient exceeding a predetermined value, the image superimposing unit causes the display unit to display the superimposition display image, and when the image superimposing unit does not detect that a user wearing the imaging device on his/her head tilts the head in a downward direction at a gradient exceeding the predetermined value by using the sensor which detects the gradient of the imaging device, the image superimposing unit restricts displaying the superimposition display image, the display being performed by the display unit.

12. The image processing device according to claim 1,

wherein the superimposition display position determining unit determines, as the position of the object having the predetermined flat surface or curved surface, a position of an object having a flat surface that extends in a substantially horizontal direction.

13. The image processing device according to claim 1, further comprising

a position estimating unit which estimates a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of the object represented by the environment map,

wherein the superimposition display position determining unit determines the position of the floor surface as the position of the object further based on the position of the floor surface or the wall surface estimated by the position estimating unit.

14. The image processing device according to claim 13,

wherein the feature data includes data indicating, for one or more points on the surface of each object, whether each point is likely to come in contact with the floor surface or the wall surface in the real space, and

wherein the position estimating unit estimates the position of the floor surface or the wall surface in the real space further based on the feature data.

15. An image processing method performed by an image processing device which includes a feature data storage unit which stores feature data indicating a feature of appearance of an object, a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user and an item position which is a position of an item forming the superimposition display data, an environment map generating unit, a superimposition display position determining unit, an image superimposing unit, an operating object recognizing unit, and a process executing unit, the image processing method comprising the steps of:

generating, by the environment map generating unit, an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit;

determining, by the superimposition display position determining unit, a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map;

generating, by the superimposition display image generating unit, a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit;

superimposing, by the image superimposing unit, the superimposition display image on the visual field of the user;

recognizing, by the operating object recognizing unit, an operating object imaged in the input image; and

executing, by the process executing unit, a process corresponding to the item selected based on a position of the operating object recognized by the operating object recognizing unit.

16. A program for causing a computer to function as an image processing device which includes

a feature data storage unit which stores feature data indicating a feature of appearance of an object,

a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user and an item position which is a position of an item forming the superimposition display data,

an environment map generating unit which generates an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit,

a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map,

a superimposition display image generating unit which generates a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit,

an image superimposing unit which superimposes the superimposition display image on the visual field of the user,

an operating object recognizing unit which recognizes an operating object imaged in the input image, and

a process executing unit which executes a process corresponding to the item selected based on a position of the operating object recognized by the operating object recognizing unit.

17. An image processing device comprising:

a feature data storage unit which stores feature data indicating a feature of appearance of an object;

a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user;

an environment map generating unit which generates an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit;

a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map;

a superimposition display image generating unit which generates a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit; and

an image superimposing unit which superimposes the superimposition display image on the visual field of the user and causes a display unit to display the superimposition display image.

18. The image processing device according to claim 17,

wherein, when the image superimposing unit detects, by using a sensor which detects the gradient of the imaging device, that a user wearing the imaging device on his/her head tilts the head in a downward direction at a gradient exceeding a predetermined value, the image superimposing unit causes the display unit to display the superimposition display image, and when the image superimposing unit does not detect that a user wearing the imaging device on his/her head tilts the head in a downward direction at a gradient exceeding the predetermined value by using the sensor which detects the gradient of the imaging device, the image superimposing unit restricts displaying the superimposition display image, the display being performed by the display unit.

19. The image processing device according to claim 18, further comprising

a self-position detecting unit which dynamically detects a position of the image processing device based on the input image and the feature data,

wherein, when a distance from the position of the image processing device detected by the self-position detecting unit to the position of the object determined by the superimposition display position determining unit exceeds a predetermined value, the image superimposing unit restricts displaying the superimposition display image, the display being performed by the display unit.

20. The image processing device according to claim 17,

wherein the superimposition display position determining unit determines, as the position of the object having the predetermined flat surface or curved surface, a position of an object having a flat surface that extends in a substantially horizontal direction.

21. The image processing device according to claim 20,

wherein the superimposition display position determining unit determines, as the position of the object having the flat surface that extends in a substantially horizontal direction, a position of at least a floor surface, a table top, and a staircase.

22. The image processing device according to claim 17,

wherein, when the superimposition display image generating unit detects, by using a sensor which detects rotation of the imaging device, that the user wearing the imaging device on his/her head rotates the head in a substantially horizontal direction, the superimposition display image generating unit moves the superimposition display data set in the superimposition display image by changing a setting position of the superimposition display data in accordance with a degree of the rotation.

23. The image processing device according to claim 17, further comprising

a position estimating unit which estimates a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of the object represented by the environment map,

wherein the superimposition display position determining unit determines the position of the floor surface as the position of the object further based on the position of the floor surface or the wall surface estimated by the position estimating unit.

24. The image processing device according to claim 23,

wherein the feature data includes data indicating, for one or more points on the surface of each object, whether each point is likely to come in contact with the floor surface or the wall surface in the real space, and

wherein the position estimating unit estimates the position of the floor surface or the wall surface in the real space further based on the feature data.

25. An image processing method performed by an image processing device which includes a feature data storage unit which stores feature data indicating a feature of appearance of an object, a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user, an environment map generating unit, a superimposition display position determining unit, a superimposition display image generating unit, and an image superimposing unit, the image processing method comprising the steps of:

generating, by the environment map generating unit, an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit;

determining, by the superimposition display position determining unit, a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map;

generating, by the superimposition display image generating unit, a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit; and

superimposing, by the image superimposing unit, the superimposition display image on the visual field of the user, and causing, by the image superimposing unit, a display unit to display the superimposition display image.

26. A program for causing a computer to function as an image processing device which includes

a feature data storage unit which stores feature data indicating a feature of appearance of an object,

a superimposition display data storage unit which stores superimposition display data which is to be a source of an image that is superimposed on a visual field of a user,

an environment map generating unit which generates an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the feature data storage unit,

a superimposition display position determining unit which determines a position of an object having a predetermined flat surface or curved surface out of an object imaged in the input image based on the environment map,

a superimposition display image generating unit which generates a superimposition display image by setting the superimposition display data at the position of the object determined by the superimposition display position determining unit, and

an image superimposing unit which superimposes the superimposition display image on the visual field of the user and causes a display unit to display the superimposition display image.