Image processing method and image processing apparatus

• US 7,868,904 B2
• Filed: 03/30/2006
• Issued: 01/11/2011
• Est. Priority Date: 04/01/2005
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Tanimura, Kaname, Morita, Kenji, Suzuki, Masahiro, Katayama, Akihiro
• Primary Examiner(s): GOOD JOHNSON, MOTILEWA
• Application Number:
  US11/392,600
• Publication Number:
  US 20060244820A1
• Time in Patent Office: 1,748 Days
• Field of Search: 345/633, 345/629, 345/632
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

On a captured image corresponding to acquisition time t of acquisition of the captured image, a virtual-space image based on the position and/or orientation at time that depends on the time t and a lag d is superimposed. An image on which the virtual-space image is superimposed is displayed. (S602-S606).

Claims

1. An image processing method comprising:

a first storage step of storing a captured image of each frame of moving images of a physical space acquired from an imaging device which captures the moving images, in association with the time at which the captured image has been captured;

a second storage step of storing positions and/or orientations of the imaging device at respective times acquired from a sensor measuring the positions and/or orientations of the imaging device, in association with the time of acquisition of the positions and/or orientations;

a lag setting step of setting data d representing a lag;

a generation step of generating, for each time t, an image by superimposing a virtual-space image, based on the position and/or orientation whose acquisition time is based on time t and the lag d set in the lag setting step, on the captured image whose acquisition time corresponds to time t;

a display step of displaying multiple of the respective images generated for the respective times t; and

a determination step of determining a value for the lag d on the basis of a user instruction.

2. The image processing method according to claim 1, wherein the generation step generates, for each of times t=t1, t2, . . . , T, an image obtained by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t−

d) on a captured image stored in association with the acquisition time t of the captured image.

3. The image processing method according to claim 1, wherein for the acquisition time to t of the captured image, if the relation of time (t−

d) to the acquisition time ta and tb of the position and/or orientation stored at the second storage step is such that ta<

(t−

d)<

tb,

the generation step interpolates the position and/or orientation at the time (t−

d) by using a position and/or orientation stored in associated with the time ta and the position and/or orientation stored in association with the time tb, and,

for each of times t=t1, t2, . . . , T, generates an imaged obtained by superimposing a virtual-space image viewed from a viewpoint having the interpolated position and/or orientation on the captured image stored in association with the acquisition time t of the captured image.

4. The image processing method according to claim 1, wherein the generation step generates a plurality of images corresponding to times t=t1, t2, . . . , T respectively;

and

the display step displays the plurality of images generated at the generation step at a list.

5. The image processing method according to claim 1, wherein the generation step generates a plurality of images corresponding to times t=t1, t2, . . . , T respectively;

and

the display step displays the plurality of images generated at the generation step as moving images.

6. The image processing method according to claim 1, wherein the generation step generates a plurality of images in accordance with d=d0, d1, . . . , D respectively.

7. The image processing method according to claim 6, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other;

if the set time t1 is equal to any of the acquisition time stored in association with captured images at the first storage step, the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t1−

d) on a captured image stored in association with the time t1.

8. The image processing method according to claim 6, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other; and

if the set time t1 is not equal to any of the acquisition time stored at the first storage step but the acquisition time t2 is closest to the time t1 among the acquisition time stored, the generation step, for each d=d0, d1, . . . , D, generates an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation stored in association with time (t2−

d) on the captured image stored in association with the time t2.

9. The image processing method according to claim 6, further comprising a time setting step of setting time t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured image in association with each other;

if the relation of the set time t1 to the acquisition time t2 and t3 stored at the first storage step is such that t2<

t1<

t3,

the generation step generates a captured image at the time t1 with interpolation using the captured image stored in association with the time t2 and the captured image stored in association with the time t3,

generates a position and/or orientation at the time t1 with interpolation by using the position and/or orientation stored in association with the time t2 and the position and/or orientation stored in association with the time t3, and

generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation generated at the generation step on the captured image generated at the image generation step.

10. The image processing method according to claim 6, further comprising a time setting step of setting a plurality of times t1, wherein:

the first storage step stores captured images of a plurality of frames and the acquisition time of the captured images in association with each other; and

the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to time t1 for each of the set times t1.

11. The image processing method according to claim 1, further comprising a time setting step of setting time t1, wherein:

the generation step generates, for each d=d0, d1, . . . , D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to the set time t1; and

the display step displays the images generated at the generation step;

the image processing method further comprising;

a second time setting step of setting, at the time (t1−

d) corresponding to a position and/or orientation used when an image specified among images displayed at the display step has been generated at the generation step, time ds=ds1, ds2, . . . , dsD that are close to the d;

a second generation step of generating, for each of ds=ds1, ds, . . . , dsD, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

ds) on the captured image corresponding to the set time t1; and

a second display step of displaying the images generated at the second generation step.

12. The image processing method according to claim 1, wherein the lag setting step acquires the lag based on an instruction from a user.

13. A non-transitory computer-readable storage medium storing a computer-executable program for making a computer execute the image processing method according to claim 1.

14. An image processing apparatus comprising:

first storage unit adapted to store a captured image of each frame of moving images of a physical space acquired from an imaging device which captures the moving image, in association with the time at which the captured image has been captured;

second storage unit adapted to store the positions and/or orientations of the imaging device at respective times acquired from a sensor measuring the positions and/or orientations of the imaging device, in association with the time of acquisition of the positions and/or orientations;

lag setting unit adapted to set data d representing a lag;

generation unit adapted to generate, for each time t, an image by superimposing a virtual-space image, based on the position and/or orientation whose acquisition time is based on time t and the lag d set in the lag setting unit, on the captured image whose acquisition time corresponds to time t;

display unit adapted to display multiple of the respective images generated, for the respective times t, by said generation unit; and

determination unit adapted to determine a value for the lag d on the basis of a user instruction.

15. The image processing apparatus according to claim 14, wherein said generation unit generates a plurality of images in accordance with each d=d0, d1, . . . , D respectively.

16. The image processing apparatus according to claim 14, further comprising a time setting unit for setting time t1, wherein:

said generation unit generates, for each d=d0, d1, . . . D, an image by superimposing a virtual-space image viewed from a viewpoint having a position and/or orientation corresponding to time (t1−

d) on the captured image corresponding to the set time t1; and

said display unit displays the images generated by said generation unit;

said image processing apparatus further comprising;

second time setting unit adapted to set, at the time (t1−

d) corresponding to a position and/or orientation used when an image specified among images displayed by said display unit has been generated by said generation unit, time ds=ds1, ds2, . . . dsD that are close to the d;

second generation unit adapted to generate, for each ds=ds1, ds2, . . . dsD, an image by superimposing a virtual-space image viewed from the viewpoint having a position and/or orientation corresponding to time (t1−

ds) on a captured image corresponding to the set time t1; and

second display unit adapted to display the image generated by said second generation unit.

17. The image processing apparatus according to claim 14, wherein said lag setting unit acquires the lag based on an instruction from a user.