Information processing method, information processing apparatus, and image sensing apparatus

• US 20070132662A1
• Filed: 05/23/2005
• Published: 06/14/2007
• Est. Priority Date: 05/27/2004
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morita, Kenji
• Granted Patent: US 7,952,594 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

The position and orientation of the viewpoint of an observer (100) are acquired. The position and orientation of a stylus (120) are acquired. A list image is laid out near the position of the stylus (120). An image of a virtual space after laying out the list image, which is seen in accordance with the position and orientation of the viewpoint, generated. The generated image is output to the display screen of an HMD (110).

Claims

1. An image processing method of generating an image of a virtual space formed from a virtual object including at least one part, characterized by comprising:

a first acquisition step of acquiring a position and orientation of a viewpoint of an observer;

a second acquisition step of acquiring a position and orientation of a pointing device which is worn by the observer on a hand to execute various kinds of operations;

a layout step of laying out a list image to display a list of pieces of information about the parts near the position acquired in the second acquisition step;

a virtual space image generation step of generating the image of the virtual space after laying out the list image, which is seen in accordance with the position and orientation of the viewpoint; and

an output step of outputting the image generated in the virtual space image generation step to a predetermined display device.

2. The method according to claim 1, characterized in that in the layout step, the list image is laid out while making a plane of the list image, on which the pieces of information about the parts are described, visible from the position and orientation of the viewpoint.

3. The method according to claim 1, characterized in that in the layout step, the list image is laid out at a position that internally divides a line segment connecting the position acquired in the first acquisition step and the position acquired in the second acquisition step to t:

(1-t).

4. The method according to claim 1, characterized by further comprising a layout control step of controlling processing in the layout step in accordance with an instruction whether to display the list image to switch whether to contain the list image in the virtual space image.

5. The method according to claim 1, characterized by further comprising a conversion step of converting the list image into a semitransparent image in accordance with an instruction to change the list image to the semitransparent image.

6. The method according to claim 1, characterized by further comprising

a determination step of determining on the basis of the position and orientation of the viewpoint and a position of the virtual object whether the virtual object is present in a direction of line of sight of the viewpoint, and

a transparency control step of, when it is determined in the determination step that the virtual object is present, making a transparency of the list image higher than that when it is determined in the determination step that the virtual object is not present.

7. The method according to claim 1, characterized by further comprising

a distance calculation step of calculating a distance between the position of the pointing device and the position of the virtual object, and

a list image generation step of generating the list image to display the list of pieces of information about the parts up to a layer level corresponding to the distance calculated in the distance calculation step in a hierarchical structure of the parts included in the virtual object.

8. The method according to claim 1, characterized in that the list image is an image to display a list of pieces of information about, of the parts included in the virtual object, a part at a position closest to the position of the pointing device acquired in the second acquisition step.

9. The method according to claim 1, characterized in that in the virtual space image generation step, when the list image overlaps a hand region in the physical space image acquired in the physical space image acquisition step, the image of the virtual space is generated on the basis of priority data to designate which of the hand region and the list image should be rendered in front.

10. The method according to claim 9, characterized by further comprising a designation step of designating which of the hand region and the list image should be rendered in front,

wherein in the designation step, designated contents are set to the priority data.

16. A program characterized by causing a computer to execute an image processing method of claim 1.

11. An image processing method of generating a virtual image corresponding to a position and orientation of an observer and compositing the virtual image with a physical image corresponding to the position and orientation of the observer, including

acquiring the position and orientation of the observer,

generating the virtual image in accordance with the position and orientation of the observer, and

compositing the physical image corresponding to the position and orientation of the observer with the virtual image, characterized by comprising:

acquiring a position of pointing means operated by the observer;

identifying a part pointed by the pointing means;

generating a part information virtual image to display information about the identified part; and

determining a layout position of the part information virtual image in accordance with the position of the pointing means.

12. The method according to claim 11, characterized in that the layout position is determined in accordance with the position of the pointing means and the position and orientation of the observer.

13. The method according to claim 11, characterized in that the information about the part comprises a list of a plurality of parts associated with the identified part.

17. A program characterized by causing a computer to execute an image processing method of claim 11.

14. An image processing apparatus of generating an image of a virtual space formed from a virtual object including at least one part, characterized by comprising:

first acquisition unit adapted to acquire a position and orientation of a viewpoint of an observer;

second acquisition unit adapted to acquire a position and orientation of a pointing device which is worn by the observer on a hand to execute various kinds of operations;

layout unit adapted to lay out a list image to display a list of pieces of information about the parts near the position acquired by said second acquisition unit;

virtual space image generation unit adapted to generate the image of the virtual space after laying out the list image, which is seen in accordance with the position and orientation of the viewpoint; and

output unit adapted to output the image generated by said virtual space image generation unit to a predetermined display device.

15. An image processing apparatus of generating a virtual image corresponding to a position and orientation of an observer and compositing the virtual image with a physical image corresponding to the position and orientation of the observer, comprising:

acquiring unit adapted to acquire the position and orientation of the observer,

generating unit adapted to generate the virtual image in accordance with the position and orientation of the observer, and

compositing unit adapted to composite the physical image corresponding to the position and orientation of the observer with the virtual image, characterized by comprising;

acquiring unit adapted to acquire a position of pointing means operated by the observer;

identifying unit adapted to identify a part pointed by the pointing means;

generating unit adapted to generate a part information virtual image to display information about the identified part; and

determining unit adapted to determining a layout position of the part information virtual image in accordance with the position of the pointing means.