Image processing method and image processing apparatus

• US 20060244820A1
• Filed: 03/30/2006
• Published: 11/02/2006
• Est. Priority Date: 04/01/2005
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Suzuki, Masahiro, Katayama, Akihiro, Tanimura, Kaname, Morita, Kenji
• Granted Patent: US 7,868,904 B2
• Time in Patent Office: Days
• US Class Current: 348/14.130
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

On a captured image corresponding to acquisition time t of acquisition of the captured image, a virtual-space image based on the position and/or orientation at time that depends on the time t and a lag d is superimposed. An image on which the virtual-space image is superimposed is displayed. (S602-S606).

Claims

1. An image processing method comprising:

a first storage step of storing a captured image of each frame of moving images acquired from an imaging device which captures moving images in a physical space, in association with the time at which the image has been captured;

a second storage step of storing the position and/or orientation of the imaging device acquired from a sensor measuring the position and/or orientation of the imaging device, in association with the time of acquisition of the position and/or orientation;

a lag setting step of setting data d representing a lag;

a generation step of generating an image by superimposing a virtual-space image based on a position and/or orientation that depend on time t or a lag d on the captured image whose acquisition time corresponds to time t;

a display step of displaying the image generated at the generation step; and

a determination step of determining a lag d on the basis of a user instruction.

2. The image processing method according to claim 1, wherein the generation step generates, for each of times t=t1, t2, . . . , T, an image obtained by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t−

d) on a captured image stored in association with the acquisition time t of the captured image.

3. The image processing method according to claim 1, wherein for the acquisition time t of the captured image, if the relation of time (t−

d) to the acquisition time ta and tb of the position and/or orientation stored at the second storage step is such that ta<

(t−

d)<

td,

the generation step interpolates the position and/or orientation at the time (t−

d) by using a position and/or orientation stored in association with the time ta and the position and/or orientation stored in association with the time tb, and,

for each of times t=t1, t2, . . . , T, generates an image obtained by superimposing a virtual-space image viewed from a viewpoint having the interpolated position and/or orientation on the captured image stored in association with the acquisition time t of the captured image.

4. The image processing method according to claim 1, wherein the generation step generates a plurality of images corresponding to times t=t1, t2, . . . , T respectively;

and

the display step displays the plurality of images generated at the generation step as a list.

5. The image processing method according to claim 1, wherein the generation step generates a plurality of images corresponding to times t=t1, t2, . . . , T respectively;

and

the display step displays the plurality of images generated at the generation step as moving images.

7. The image processing method according to claim 1, wherein the generation step generates a plurality of images in accordance with d=d0, d1, . . . , D respectively.

8. The image processing method according to claim 7, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other;

if the set time t1 is equal to any of the acquisition time stored in association with the captured images at the first storage step, the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t1−

d) on a captured image stored in association with the time t1.

9. The image processing method according to claim 7, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other; and

if the set time t1 is not equal to any of the acquisition time stored at the first storage step but the acquisition time t2 is closest to the time t1 among the acquisition time stored, the generation step, for each d=d0, d1, . . . , D, generates an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t2−

d) on the captured image stored in association with the time t2.

10. The image processing method according to claim 7, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured image in association with each other;

if the relation of the set time t1 to the acquisition time t2 and t3 stored at the first storage step is such that t2<

t1<

t3,

the generation step generates a captured image at the time t1 with interpolation using the captured image stored in association with the time t2 and the captured image stored in association with the time t3,

generates a position and/or orientation at the time t1 with interpolation by using the position and/or orientation stored in association with the time t2 and the position and/or orientation stored in association with the time t3, and

generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation generated at the generation step on the captured image generated at the image generation step.

11. The image processing method according to claim 7, further comprising a time setting step of setting a plurality of times t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other; and

the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to time t1 for each of the set times t1.

12. The image processing method according to claim 1, further comprising a time setting step of setting time t1, wherein:

the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to the set time t1; and

the display step displays the images generated at the generation step;

the image processing method further comprising;

a second time setting step of setting, at the time (t1−

d) corresponding to a position and/or orientation used when an image specified among images displayed at the display step has been generated at the generation step, time ds=ds1, ds2, . . . dsD that are close to the d;

a second generation step of generating, for each of ds=ds1, ds, . . . , dsD, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

ds) on the captured image corresponding to the set time t1; and

a second display step of displaying the images generated at the second generation step.

15. The image processing method according to claim 1, wherein the lag setting step acquires the lag based on an instruction from a user.

17. A program for causing a computer to perform the image processing method according to claim 1.

18. A computer-readable storage medium on which the program according to claim 17 is stored.

6. An image processing apparatus comprising:

first storage unit adapted to store a captured image of each frame of moving images acquired from an imaging device which captures moving image in a physical space, in association with the time at which the image has been captured;

second storage unit adapted to store the position and/or orientation of the imaging device acquired from a sensor measuring the position and/or orientation of the imaging device, in association with the time of acquisition of the position and/or orientation;

lag setting unit adapted to set data d representing a lag;

generation unit adapted to generate an image by superimposing a virtual-space image based on a position and/or orientation that depends on time t and a lag d on the captured image whose acquisition time corresponds to time the t;

display unit adapted to display the image generated by said generation unit; and

determination unit adapted to determine a lag d on the basis of a user instruction.

13. The image processing apparatus according to claim 6, wherein said generation unit generates a plurality of images in accordance with each d=d0, d1, . . . , D respectively.

14. The image processing apparatus according to claim 6, further comprising

a time setting unit for setting time t1, wherein:

said generation unit generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to the set time t1; and

said display unit displays the images generated by said generation unit;

said image processing apparatus further comprising;

second time setting unit adapted to set, at the time (t1−

d) corresponding to a position and/or orientation used when an image specified among images displayed by said display unit has been generated by said generation unit, time ds=ds1, ds2, . . . dsD that are close to the d;

second generation unit adapted to generate, for each ds=ds1, ds2, . . . , dsD, an image by superimposing a virtual-space image viewed from the viewpoint having a position and/or orientation corresponding to time (t1−

ds) on a captured image corresponding to the set time t1; and

second display unit adapted to display the images generated by said second generation unit.

16. The image processing apparatus according to claim 6, wherein said lag setting unit acquires the lag based on an instruction from a user.