Image processing apparatus and image processing method, and image processing program and recording medium of the same

• US 20030234823A1
• Filed: 05/12/2003
• Published: 12/25/2003
• Est. Priority Date: 05/17/2002
• Status: Active Grant

• Current Assignee: Columbia Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Kubota, Akira, Sato, Shigemi
• Granted Patent: US 6,862,006 B2
• Time in Patent Office: Days
• US Class Current: 715/848
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

An image processing apparatus and an image processing method are provided which are capable of changing display from a virtual space image to a real space image or vice versa, or combining them based on the movement speed of the user'"'"'s point of gaze while the user is moving in a virtual space. An image processing program and a medium recording the same are also provided. In the image processing apparatus, when the movement of a head mounted display (HMD) in the virtual space is detected (Step S111), the movement speed is calculated (Step S112) and it is determined whether the movement speed is equal to or greater than akm/h. When the movement speed is slower than akm/h, an updated virtual space image reflecting the movement of the HMD is created for display (Steps S114 and S118). When the movement speed is equal to or greater than akm/h, the real image then being taken is displayed as it is (Steps S115 and S118).

Claims

1. An image processing apparatus comprising:

an image pickup unit adapted to take an image in real space seen from a user'"'"'s point of gaze;

an image creating unit adapted to create an image in virtual space corresponding to the real space image taken by the image pickup unit;

a display unit adapted to be able to display the real space image taken by the image pickup unit and the virtual space image created by the image creating unit;

a speed detecting unit adapted to detect the speed of movement of the user'"'"'s point of gaze; and

a display control unit adapted to change at least a part of an image to be displayed on the display unit from one of the real space image and the virtual space image to the other, based on the movement speed detected by the speed detecting unit.

2. The image processing apparatus according to claim 1, wherein the display control unit changes the image to be displayed from the real space image to the virtual space image or from the virtual space image to the real space image when the movement speed detected by the speed detecting unit is greater than a predetermined value.

3. An image processing apparatus comprising:

an image pickup unit adapted to take an image in real space seen from a user'"'"'s point of gaze;

an image creating unit adapted to create an image in virtual space corresponding to the real space image taken by the image pickup unit;

a display unit adapted to be able to display the real space image taken by the image pickup unit and the virtual space image created by the image creating unit;

a speed detecting unit adapted to detect the speed of movement of the user'"'"'s point of gaze; and

a display control unit adapted to change the display ratio of the real space image and the virtual space image based on the movement speed detected by the speed detecting unit and combine these images for display.

4. The image processing apparatus according to claim 3, wherein the display control unit causes the display unit to display the virtual space image when the movement speed detected by the speed detecting unit is slower than a first threshold value, the display control unit makes the display unit to display the real space image when the movement speed is greater than a second threshold value, and the display control unit changes the ratio of the real space image and the virtual space image based on the movement speed, combines these images, and causes the display unit to display the combined image when the movement speed is between the first threshold value and the second threshold value.

5. The image processing apparatus according to claim 3 further comprising:

a finger position detecting unit adapted to detect a finger position; and

a tactile and kinesthetic sensation presenting unit having a tactile and kinesthetic sensation generating unit adapted to give a tactile and kinesthetic sensation to a finger when the finger position detected by the finger position detecting unit matches the position of a virtual object displayed in the virtual space image,

wherein the image creating unit creates a virtual finger image at the position of the finger detected by the finger position detecting unit in the virtual space, and

the display unit displays the virtual space image including the virtual finger image.

6. The image processing apparatus according to claim 4 further comprising:

a finger position detecting unit adapted to detect a finger position;

a tactile and kinesthetic sensation presenting unit having a tactile and kinesthetic sensation generating unit adapted to give a tactile and kinesthetic sensation to a finger when the finger position detected by the finger position detecting unit matches the position of a virtual object displayed in the virtual space; and

a drive control unit adapted to control activation of the tactile and kinesthetic sensation generating unit so that a finger pad receives a sensation corresponding to contact when the finger position detected by the finger position detecting unit matches the position of the virtual object displayed in the virtual space image in the case where the movement speed is between the first threshold value and the second threshold value,

wherein the image creating unit creates a virtual finger image at the position of the finger detected by the finger position detecting unit in the virtual space, and

the display unit displays the virtual space image including the virtual finger image.

7. The image processing apparatus according to claim 4 further comprising:

a finger position detecting unit adapted to detect a finger position;

a tactile and kinesthetic sensation presenting unit having a tactile and kinesthetic sensation generating unit adapted to generate a tactile and kinesthetic sensation given to a finger when the finger position detected by the finger position detecting unit matches the position of a virtual object displayed in the virtual space; and

a drive control unit adapted to control the tactile and kinesthetic sensation generating unit so that a finger pad cannot receive a sensation corresponding to a contact even when the finger position detected by the finger position detecting unit matches the position of the virtual object displayed in the virtual space, in the case where the movement speed is between the first threshold value and the second threshold value,

wherein the image creating unit creates a virtual finger image at the finger position detected by the finger position detecting unit in the virtual space, and

the display unit displays the virtual space image including the virtual finger image.

8. The image processing apparatus according to claim 3, wherein the image pickup unit has two CCD cameras corresponding to the user'"'"'s point of gaze.

9. The image processing apparatus according to claim 3, wherein the image creating unit updates and creates the virtual space image at predetermined time intervals based on the speed of the user'"'"'s point of gaze detected by the speed detecting unit.

10. The image processing apparatus according to claim 3, wherein the display unit has a position detecting unit adapted to detect the position of the user'"'"'s point of gaze, and

the speed detecting unit includes an operating unit adapted to calculate the speed of the user'"'"'s point of gaze from the positions of the user'"'"'s point of gaze detected by the position detecting unit and the time required to move between positions.

11. The image processing apparatus according to claim 10, wherein the position detecting unit is a position detecting unit of an orthogonal coil system having a first coil, a second coil and a third coil, and their central axes are orthogonal to each other.

12. The image processing apparatus according to claim 10, wherein the finger position detecting unit has gyro sensors for detecting an angular velocity about X-, Y- and Z-axes orthogonal to each other.

13. The image processing apparatus according to claim 3, wherein the speed detecting unit comprises:

a three-dimensional acceleration sensor for sensing an acceleration in three directions orthogonal to each other, and

an operating unit adapted to calculate speed by at least one of integrating and adding the output of the acceleration sensor.

14. The image processing apparatus according to claim 3, wherein the display unit is a product to be worn on a face for use.

15. The image processing apparatus according to claim 3, wherein the display unit and the image pickup unit are configured as one unit, which further comprises:

at least one of;

an image creating unit;

a speed detecting unit;

a display control unit; and

a drive control unit; and

a communication unit adapted to send and receive at least one of real image data, virtual space image data, position data, and drive control data.

16. The image processing apparatus according to claim 3 further comprising a storing unit adapted to store at least one of the real space image data taken by the image pickup unit, virtual space image data created by the image creating unit, movement speed detected by the speed detecting unit, and position data detected by the position detecting unit.

17. An image processing method comprising the steps of:

taking an image in real space seen from a user'"'"'s point of gaze;

creating a virtual space image corresponding to the taken real space image;

displaying at least one of the taken real space image and the created virtual space image;

detecting the speed of the user'"'"'s point of gaze; and

changing at least a part of the image to be displayed from one of real space image and virtual space image to the other for display, based on the detected movement speed.

18. The image processing method according to claim 17, wherein the changing and displaying step changes the image to be displayed from the real space image to the virtual space image or from the virtual space image to the real space image when the detected movement speed is greater than a predetermined value.

19. An image processing method comprising the steps of:

taking an image in real space seen from a user'"'"'s point of gaze;

creating a virtual space image corresponding to the taken real space image;

displaying at least one of the taken real space images and the created virtual space image;

detecting the speed of the user'"'"'s point of gaze; and

changing the ratio of the real space image and the virtual space image and combining them for display, based on the detected movement speed.

20. The image processing method according to claim 19, wherein the combining and displaying step causes the virtual space image to be displayed when the movement speed of the user'"'"'s point of gaze is smaller than a first threshold value, causes the real space image to be displayed when the movement speed is greater than a second threshold value, and determines a ratio of the real space image and the virtual space image based on the movement speed and combines them for display when the movement speed is between the first threshold value and the second threshold value.

21. The image processing method according to claim 19 further comprising the steps of:

detecting the finger position to display a finger image in the virtual space image, and

generating a tactile and kinesthetic sensation given to a finger when the detected finger position matches the position of a virtual object displayed in the virtual space image.

22. The image processing method according to claim 20 further comprising the steps of:

detecting a finger position to display a finger image in the virtual space image,

generating a tactile and kinesthetic sensation given to the finger when the detected finger position matches the position of a virtual object displayed in the virtual space image, and

controlling the generation of the tactile sense and the kinesthetic sense so that a finger pad receives a sensation corresponding to the degree of contact of the detected finger position with the position of the virtual object displayed in the virtual space image in the case where the movement speed is between the first threshold value and the second threshold value.

23. The image processing method according to claim 20, further comprising the steps of:

detecting a finger position and displaying an image of the finger in the virtual space image;

generating a tactile and kinesthetic sensation given to a finger when the detected finger position matches the position of a virtual object displayed in the virtual space image; and

controlling the generation of the tactile sense and the kinesthetic sense so that even when the detected finger position matches the position of the virtual object displayed in the virtual space image, the finger pad can not receive a sensation corresponding to the degree of contact in the case where the movement speed is between the first threshold value and the second threshold value.

24. The image processing method according to claim 19 further comprising the step of detecting the position of the user'"'"'s point of gaze before the speed detecting step,

wherein the speed detecting step includes the step of calculating the speed of the user'"'"'s point of gaze based on the different detected positions of the user'"'"'s point of gaze and the time required to move between the different positions.

25. The image processing method according to claim 19, wherein the speed detecting step includes the step of detecting an acceleration of the user'"'"'s point of gaze and at least one of integrating and adding the acceleration to calculate the speed.

26. An image processing program for an information processing device for creating a virtual space image and displaying the virtual space image to be visually recognized by a user, the program making a computer of the information processing device to function as:

image pickup means for taking an image in real space seen from a user'"'"'s point of gaze;

image creating means for creating a virtual space image corresponding to the real space image taken by the image pickup means;

display means for displaying the real space image taken by the image pickup means and the virtual space image created by the image creating means;

speed detecting means for detecting the speed of the user'"'"'s point of gaze; and

display controlling means for changing at least a part of the image to be displayed on the displaying means from one of the real space image and the virtual space image to the other, based on the movement speed detected by the speed detecting means.

27. The image processing program according to claim 26, wherein the display controlling means includes a program code ordering change of the image to be displayed from the real space image to the virtual space image or from the virtual space image to the real space image when the movement speed detected by the speed detecting means is greater than a predetermined value.

28. An image processing program for an information processing device for creating a virtual space image and displaying the virtual space image so as to be visually recognizable by a user, the program making a computer of the information processing device to function as:

image pickup means for taking an image in real space seen from a user'"'"'s point of gaze;

image creating means for creating a virtual space image corresponding to the real space image taken by the image pickup means;

display means for displaying the real space image taken by the image pickup means and the virtual space image created by the image creating means;

speed detecting means for detecting the speed of the user'"'"'s point of gaze; and

display controlling means for changing the display ratio of the real space image and the virtual space image based on the movement speed detected by the speed detecting means and combining them for display.

29. The image processing program according to claim 28, wherein the display controlling means includes a program code making the displaying means to display the virtual space image when the movement speed detected by the speed detecting means is slower than a first threshold value, making the displaying means to display the real space image when the movement speed is greater than a second threshold value, and determining the ratio of the real space image and the virtual space image to be displayed based on the movement speed when the movement speed is between the first threshold value and the second threshold value, combining these images, and making the displaying means to display the combined image.

30. The image processing program according to claim 28 further comprising a program code for making the computer of the information processing device to function as:

finger position detecting means for detecting a finger position;

finger displaying means for displaying a finger image at the finger position detected by the finger position detecting means in the virtual space; and

tactile and kinesthetic sensation generating means for generating a tactile and kinesthetic sensation in a finger when the finger position detected by the finger position detecting means matches the position of a virtual object displayed in the virtual space image.

31. The image processing program according to claim 29 further comprising a program code making the computer of the information processing device to function as:

finger position detecting means for detecting a finger position;

finger displaying means for displaying a finger image at the finger position detected by the finger position detecting means in the virtual space image;

tactile and kinesthetic sensation generating means for generating a tactile and kinesthetic sensation given to a finger when the finger position detected by the finger position detecting means matches a position of a virtual object displayed in the virtual space image; and

drive controlling means for controlling the generation of the tactile sense and the kinesthetic sense so that a finger pad can receive a sensation corresponding to a contact when the finger position detected by the finger position detecting means matches the position of the virtual object displayed in the virtual space image in the case where the movement speed is between the first threshold value and the second threshold value.

32. The image processing program according to claim 29 further comprising a program code making the computer of the information processing device to function as:

finger position detecting means for detecting a finger position;

finger displaying means for displaying a finger image at the finger position detected by the finger position detecting means in the virtual space;

tactile and kinesthetic sensation generating means for generating a tactile and kinesthetic sensation in a finger when the finger position detected by the finger position detecting means matches the position of a virtual object displayed in the virtual space; and

drive controlling means for controlling the generation of the tactile sense and the kinesthetic sense so that even when the detected finger position matches the position of the virtual object displayed in the virtual space image, the finger pad can not receive a sensation corresponding to the degree of contact in the case where the movement speed is between the first threshold value and the second threshold value.

33. The image processing program according to claim 28, further comprising a program code making the computer of the information processing device to function as position detecting means for detecting the position of the user'"'"'s point of gaze,

wherein the speed detecting means includes a program code for calculating the speed of the user'"'"'s point of gaze based on the different positions of the user'"'"'s point of gaze detected by the position detecting means and the time required to move between these different positions.

34. The image processing program according to claim 28, wherein the speed detecting means includes a program code to detect the acceleration of the user'"'"'s point of gaze and at least one of integrate and add the acceleration for calculating the speed.

35. A recording medium recording the image processing program according to claim 28 for allowing a computer to execute the program, the recording medium being readable by the computer.