Interactive image generation method and apparatus utilizing a determination of the visual point position of an operator

• US 6,437,794 B1
• Filed: 04/30/1996
• Issued: 08/20/2002
• Est. Priority Date: 05/15/1995
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Yamamoto, Hiroyuki, Ohshima, Toshikazu, Uchiyama, Shinji
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Padmanabhan, Mano
• Application Number:
  US08/640,081
• Time in Patent Office: 2,303 Days
• Field of Search: 345/437, 345/436, 345/433, 345/419, 345/126, 345/326, 345/158, 345/978, 345/648, 345/649, 345/619, 345/700, 345/702
• US Class Current: 345/619
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

In an interactive image generation method and apparatus, excellent operability is provided when the operator selects a figure to be operated on from among a plurality of figure units displayed on the picture surface of a display unit. In such a method and apparatus, respective units for measuring the visual point and the direction of the line of sight of the operator observing the display device are provided, and a figure unit displayed at a position viewed by the operator on the picture surface is identified. Accordingly, the operator can be sure of correctly selecting the target figure by a direct operation of looking at the figure, and therefore excellent operability can be obtained.

Claims

1. An image generation method comprising the steps of:

displaying a figure data expressed in first three-dimensional coordinates on a display screen;

measuring a position of an operator'"'"'s head in second three-dimensional coordinates;

providing an instruction specifying that a three-dimensional figure be selected;

converting the second three-dimensional coordinates into the first three-dimensional coordinates;

identifying a line-of-sight direction along which the operator is looking in the converted second three-dimensional coordinates;

identifying a displayed three-dimensional figure in the first three-dimensional coordinates intersected by the identified direction in response to the instruction based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line-of-sight;

calculating coordinates of a closest point; and

registering the closest point obtained in said calculating step as an operating point.

2. A method according to claim 1, wherein, in said position measuring step and said direction identifying step, the position of the point and the direction are measured, respectively, for a dominant eye of the operator.

3. A method according to claim 1, wherein, in said direction identifying step, the forward direction of the operator'"'"'s head is approximated as the direction along which the operator is looking.

4. An image generation method according to claim 1, further comprising the steps of:

coordinate transforming information relating to respective coordinates of the displayed figure identified in said identifying step and the operating point registered in said registering step, using the operating point as an origin for the coordinate transformation;

generating an image of a figure based on transformed information produced in said transforming step, using the position of the point measured in said measuring step as an origin for projection; and

displaying the image generated in said generating step.

5. A method according to claim 4, further comprising the step of determining whether or not the operator has issued an instruction to register the position calculated in said calculating step as the origin for the coordinate transformation in said transforming step, wherein said registrating step includes registering the position based on the determination made in said determining step.

6. A method according to claim 5, wherein said determining step further comprises the steps of:

detecting bending angles of respective finger joints of a hand of the operator; and

determining whether or not the bending angle of each of the finger joints is within a preset angle range,

wherein, when the finger joints have been determined to be in a specific state in said finger-joint-state determining step, it is determined that an instruction to register the position calculated in said calculating step as the operating point, has been performed.

7. A method according to claim 5, wherein said determining step further comprises the steps of:

inputting a voice instruction given by the operator; and

recognizing the voice instruction input in said inputting step,

wherein, when it has been determined that the voice instruction recognized in said recognizing step is the same as a preset voice instruction indicating instruction of registration, it is determined that an instruction has been given to register the position calculated in said calculating step as the operating point.

8. A method according to claim 4, wherein said displaying step includes displaying a figure which serves as an index while superposing it on the image generated in said generating step at the position calculated in said calculating step.

9. An image generation apparatus comprising:

a display device, arranged to display a figure data expressed in first three-dimensional coordinants on a display screen;

a visual-point-position measurer, arranged to obtain a position of an operator'"'"'s head in second three-dimensional coordinates;

an instructor, arranged to provide an instruction specifying that a three dimensional figure be selected;

a converter, arranged to convert the second three-dimensional coordinates into the first three-dimensional coordinates;

a line-of-sight direction measurer, arranged to identify a direction along which the operator is looking in the converted second three-dimensional coordinates;

an object determiner, arranged to identify a displayed three-dimensional figure in the first three-dimensional coordinates intersected by the identified direction in response to the instruction of the instructor based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line of sight;

an operating-point registerer, arranged to register a position as an operating point;

a figure-coordinate transformation processor, arranged to perform a coordinate transformation on information relating to respective coordinates of the displayed figure identified by said object determiner, and the operating point registered by said operating-point registerer, using the operating point as an origin for the coordinate transformation;

an image generator, arranged to generate an image of a figure based on the transformed information provided by said figure-coordinate transformation processor, using the position of the visual point obtained by said visual-point-position measurer as an origin for projection; and

an image display arranged to display the image generated by said image generator.

10. An apparatus according to claim 9, wherein said sight-line-direction measurer approximates the forward direction of the operator'"'"'s head as the direction of the line of sight.

11. A figure designation apparatus comprising:

a display arranged to display figure data expressed in first three-dimensional coordinates on a display screen for a plurality of three-dimensional figures;

a memory, arranged to store coordinate information of each of said plurality of three-dimensional figures;

a visual-point-direction measurer, arranged to obtain coordinates of a position of an operator'"'"'s head in second three-dimensional coordinates;

an instructor, arranged to provide an instruction that a three-dimensional figure be selected;

a convertor, arranged to convert the second three-dimensional coordinates into the first three-dimensional coordinates;

a line-of-sight direction measurer, arranged to identify a line-of-sight direction along which the operator is looking in the converted second three-dimensional coordinates;

an adjuster, arranged to adjust, in response to the instruction provided by the instructor, the coordinates of the figures displayed on said display and the coordinates of a position based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line-of sight; and

a selector, arranged to select one of the figures displayed on said display on the basis of the coordinates adjusted by said adjuster and said direction of the line-of-sight.

12. An apparatus according to claim 11, wherein said visual-point-direction measurer and said line-of sight-direction measurer measure the position of the visual point and the direction, respectively, for a dominant eye of the operator.

13. An apparatus according to claim 11, wherein said line-of-sight-direction measurer approximates a forward direction of the operator'"'"'s head as the direction of the line-of-sight.

14. An apparatus according to claim 11, further comprising an instruction recognizer, arranged to determine whether or not the operator has issued an instruction to register the position obtained by said visual-point-direction measurer as the coordinates to be adjusted by said adjuster.

15. An apparatus according to claim 14, wherein said instruction recognizer further comprises:

an angle measurer, arranged to detect bending angles of respective finger joints of a hand of the operator; and

a hand-joint-state determiner for determining whether or not the bending angle of each of the finger joints is within a preset angle range,

wherein, when the finger joints have been determined to be in a specific state by said finger-joint-state determiner, it is determined that an instruction has been given to register the position obtained by said visual-point-position measurer.

16. An apparatus according to claim 14, wherein said instruction recognizer further comprises:

a voice inputter, arranged to input a voice instruction issued by the operator; and

a voice recognizer, arranged to recognize the voice instruction input by said voice inputter,

wherein when it has been determined that the voice instruction recognized by said voice is the same as a preset voice instruction indicating an instruction for registration, it is determined that an instruction has been given to register the position obtained by said visual-point-direction measurer.

17. A figure designation method comprising the steps of:

displaying a figure data expressed in first three-dimensional coordinates for a plurality of three-dimensional figures on a display;

storing coordinate information of each of the plurality of three-dimensional figures in a memory;

obtaining a position of an operator'"'"'s head in second three-dimensional coordinates;

providing an instruction;

converting the second three-dimensional coordinates into the first three-dimensional coordinates;

identifying a line-of-sight direction along which the operator is looking in the converted second three-dimensional coordinates;

adjusting, in response to the instruction, the coordinates of the figures displayed on the display and the coordinates of a position based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line-of-sight; and

selecting one of the figures displayed on the display on a basis of the coordinates adjusted in said adjusting step and the direction of the line-of-sight.

18. A method according to claim 17, wherein the position of the visual point and the direction of line-of-sight for a dominant eye of the operator are measured in said obtaining step and said identifying step, respectively.

19. A method according to claim 17, wherein said identifying step approximates a forward direction of the operator'"'"'s head as the direction of the line-of-sight.

20. A method according to claim 19, further comprising the steps of:

detecting bending angles of respective finger joints of a hand of the operator; and

determining whether or not the bending angle of each of the finger joints is within a preset angle range,

wherein, when the finger joints have been determined to be in a specific state in said determining step, it is determined that an instruction has been given to register the position obtained in said obtaining step.

21. An method according to claim 19, further comprising the steps of:

inputting a voice instruction issued by the operator; and

recognizing the voice instruction input in said inputting step,

wherein when it has been determined that the voice instruction recognized in said recognizing step is the same as a preset voice instruction indicating an instruction for registration, it is determined that an instruction has been given to register the position obtained in said obtaining step.

22. A method according to claim 17, further comprising a step for determining whether or not the operator has issued an instruction to register the position obtained in said obtaining step as the coordinates to be adjusted in said adjusting step.

23. A storage medium containing computer readable program comprising code which upon execution will cause an image generation apparatus to perform the steps comprising:

displaying a figure data expressed in first three-dimensional coordinates on a display screen;

measuring a position of an operator'"'"'s head in second three-dimensional coordinates;

providing an instruction specifying that a three-dimensional figure is to be selected;

converting the second three-dimensional coordinates into the first three-dimensional coordinates;

identifying a line-of-sight direction along which the operator is looking in the converted second three-dimensional coordinates;

identifying a displayed three-dimensional figure in the first three-dimensional coordinates intersected by the identified direction in response to the instruction based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line-of sight;

calculating the coordinates of the closest point;

registering the closest point obtained in said calculating step as an operating point;

coordinate transforming information relating to respective coordinates of the displayed figure identified in said identifying step and the operating point registered in said registering step, using the operating point as an origin for the coordinate transformation;

generating an image of a figure based on transformed information produced in said transforming step, using the position of the point measured in said measuring step as an origin for projection; and

displaying the image generated in said generating step.

24. A storage medium containing computer readable program comprising code which upon execution will cause a figure designation apparatus to perform the steps comprising:

displaying a figure data expressed in first three-dimensional coordinates for a plurality of three-dimensional figures on a display;

storing coordinate information of each of the plurality of three-dimensional figures in a memory;

obtaining coordinates of a position of an operator'"'"'s head in second three-dimensional coordinates;

providing an instruction;

converting the second three-dimensional coordinates into the first three-dimensional coordinates;

adjusting, in response to the instruction, the coordinates of the figures displayed on the display and the coordinates of a position based on the position of the operator'"'"'s head in the converted second three-dimensional coordinates and the direction of the line-of-sight; and

selecting one of the figures displayed on the display on a basis of the coordinates adjusted in said adjusting step and the direction of the line-of-sight.