System, image processing apparatus, and information processing method

• US 20060170652A1
• Filed: 01/27/2006
• Published: 08/03/2006
• Est. Priority Date: 01/31/2005
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Yamamoto, Hiroyuki, Bannai, Yuichi, Ono, Eita
• Granted Patent: US 7,843,470 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G09B 23/30 Anatomical models G09B23/281 - G09B23/288 take precedence
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F

Abstract

An image (200) displayed on a display device (23a, 23b) includes a mixed reality space viewed from an operator, i.e., a physical object (42), virtual object (43), stylus (41a), and pointer (41d).

Claims

1. A system characterized by comprising:

first acquisition unit adapted to acquire a position/orientation of a viewpoint of a first observer;

generation unit adapted to generate an image of a virtual space viewed from the viewpoint having the position/orientation acquired by said first acquisition unit;

first manipulation unit used by the first observer to manipulate a virtual object;

second manipulation unit used by a second observer to manipulate the virtual object, the second observer remote-controlling the manipulation of the virtual object by the first observer;

second acquisition unit adapted to acquire an image of a physical space viewed from the viewpoint; and

output unit adapted to output an image formed by superimposing the image generated by said generation unit on the image acquired by said second acquisition unit to a head mounted display worn by the first observer and a head mounted display worn by the second observer,

wherein said generation unit generates the image of the virtual space on which manipulation results by said first manipulation unit and said second manipulation unit are reflected.

2. The system according to claim 1, characterized by further comprising layout unit adapted to lay out a virtual object representing said second manipulation unit in the virtual space so as to make a position/orientation relationship between the virtual object and the viewpoint of the first observer substantially equal to a position/orientation relationship between the viewpoint of the second observer and said second manipulation unit.

3. The system according to claim 1, characterized by further comprising layout unit adapted to lay out, in a space where the first observer exists, a virtual object representing said second manipulation unit in a position/orientation of said second manipulation unit in a coordinate system defined in a space where the second observer exists.

4. The system according to claim 1, characterized in that each of said first manipulation unit and said second manipulation unit comprises a stylus held by a corresponding one of the first observer and the second observer in a hand.

5. An image processing apparatus characterized by comprising:

first acquisition unit adapted to acquire a position/orientation of a viewpoint of a first observer;

generation unit adapted to generate an image of a virtual space viewed from the viewpoint having the position/orientation acquired by said first acquisition unit;

second acquisition unit adapted to acquire an image of a physical space viewed from the viewpoint; and

output unit adapted to output an image formed by superimposing the image generated by said generation unit on the image acquired by said second acquisition unit to a head mounted display worn by the first observer and a head mounted display worn by a second observer who remote-controls manipulation of the virtual object by the first observer;

wherein said generation unit generates the image of the virtual space on which manipulation results by first manipulation unit used by the first observer to manipulate the virtual object and said second manipulation unit used by the second observer to manipulate the virtual object are reflected.

6. An information processing method of generating a 3D composited image by compositing a physical image and a virtual image representing a virtual object, characterized by comprising:

acquiring a right physical image and a left physical image from a viewpoint of a first observer;

acquiring a position/orientation of the viewpoint of the first observer;

acquiring a position/orientation of a first pointing unit used by the first observer to manipulate a virtual object;

acquiring a position/orientation of a second pointing unit used by a second observer to manipulate the virtual object;

controlling information of the virtual object on the basis of one of the position/orientation of the first pointing unit and the position/orientation of the second pointing unit;

generating a right virtual image and a left virtual image corresponding to the position/orientation of the viewpoint of the first observer on the basis of the controlled information of the virtual object;

generating a right composited image and a left composited image by compositing the physical images and the virtual images; and

presenting the composited image to the first observer and the second observer.

7. The method according to claim 6, characterized in that

the position/orientation of the first pointing unit is a position/orientation in a world coordinate system,

the position/orientation of the second pointing unit is obtained by transforming a relative position/orientation to a viewpoint of the second observer into a position/orientation in the world coordinate system on the basis of the position/orientation of the viewpoint of the first observer, and

the information of the virtual object is controlled on the basis of the position/orientation of the first pointing unit in the world coordinate system and the position/orientation of the second pointing unit in the world coordinate system.

8. The method according to claim 6, characterized by further comprising:

generating a first pointing unit virtual image based on the position/orientation of the first pointing unit and a second pointing unit virtual image based on the position/orientation of the second pointing unit; and

generating the composited image by compositing the first pointing unit virtual image, the second pointing unit virtual image, and the virtual image of the virtual object with the physical image.

9. A program characterized by causing a computer to execute an information processing method of claim 6.

10. An information processing method of causing a second user to share a mixed reality space image in which a virtual object is superimposed on a space where a first user exists, characterized by comprising:

a mixed reality space image acquisition step of acquiring a mixed reality space image based on an image of a first image sensing unit worn by the first user and a virtual object image based on a position/orientation of the first image sensing unit;

an event information acquisition step of acquiring event information by the first user for the virtual object;

a second user viewpoint position/orientation information acquisition step of acquiring viewpoint position/orientation information of the second user; and

a generation step of generating a virtual object image corresponding to the event information on the basis of the viewpoint position/orientation information of the second user,

wherein the method has a first mode to present a first image to the second user and a second mode to present the virtual object image to the second user.

11. The method according to claim 10, characterized in that the method has a third mode to present the virtual object image corresponding to the viewpoint position/orientation information of the second user to the second user without depending on the event information.

12. A program characterized by causing a computer to implement an information processing method of claim 10.

13. An information processing system for sharing, from a remote site, an operator mixed reality space in which a virtual object is superimposed on a space where an operator exists, characterized by comprising:

stereoscopic image transmitting unit adapted to transmit a stereoscopic image of the operator mixed reality space from a viewpoint of the operator to an instructor,

virtual object sharing means for causing the operator and the instructor to share the virtual object; and

stereoscopic means for presenting the stereoscopic image to the operator and the instructor,

wherein the system has a mode to cause the instructor to share the stereoscopic image from the operator viewpoint and a mode to allow the instructor to manipulate the virtual object from a viewpoint separated from the operator viewpoint.

14. The system according to claim 13, characterized in that the mode to allow the instructor to manipulate the virtual object from the viewpoint separated from the operator viewpoint includes a mode to share the virtual object and a mode not to share the virtual object.

15. The system according to claim 13, characterized by further comprising switching unit adapted to switch between the mode to cause the instructor to share the stereoscopic image from the operator viewpoint and the mode to allow the instructor to manipulate the virtual object from the viewpoint separated from the operator viewpoint.

16. The system according to claim 13, characterized by further comprising identification unit adapted to identify the mode of the instructor by using said stereoscopic unit of the operator.

17. The system according to claim 13, characterized by further comprising display unit adapted to display a viewpoint position of the instructor by using said stereoscopic unit of the operator and displaying the viewpoint position of the operator by using said stereoscopic unit of the instructor.