INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND STORAGE MEDIUM

• US 20200133388A1
• Filed: 10/28/2019
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Takemoto, Kazuki
• Granted Patent: US 11,275,434 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

An information processing apparatus supplies, an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display an image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit. The information processing apparatus includes a generation unit configured to generate an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined time from a time when the image display apparatus starts to move in the captured image of the real space including the specific object, and a control unit configured to shift a position at which the image generated by the generation unit is displayed on the display unit based on a change in a position and/or an orientation of the image display apparatus.

Claims

1. An information processing apparatus that supplies, to an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display an image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit, the information processing apparatus comprising:

a generation unit configured to generate an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined period from a time when the image display apparatus starts to move in the captured image of the real space including the specific object; and

a control unit configured to shift a position at which the image generated by the generation unit is displayed on the display unit based on a change in a position and/or an orientation of the image display apparatus.

2. The information processing apparatus according to claim 1, wherein the control unit shifts a position at which a second image to be supplied to the display unit is displayed based on a delay time caused by a processing time for at least a part of processing from the image capturing to the display.

3. The information processing apparatus according to claim 1, wherein the predetermined period is a difference between a first time when the image capturing apparatus performs image capturing and a second time when the position and/or the orientation of the image display apparatus is measured.

4. The information processing apparatus according to claim 1, wherein the generation unit generates an image depicting an area obtained by extracting the specific object from the captured image of the real space at a position based on a result of the estimation.

5. The information processing apparatus according to claim 3,

wherein the specific object is a hand, and

wherein the generation unit generates an image depicting the hand extracted from a skin color area in the captured image of the real space at a position based on a result of the estimation.

6. The information processing apparatus according to claim 5,

wherein the captured image of the real space is a stereoscopic image, and

wherein the generation unit generates an image depicting the specific object at a position based on a result of the estimation based on a three-dimensional shape of the specific object extracted from the stereoscopic image.

7. The information processing apparatus according to claim 5,

wherein the captured image of the real space is a stereoscopic image, and

wherein the generation unit acquires a three-dimensional shape of the specific object from the stereoscopic image and generates an image depicting the object in a three-dimensional manner at a position based on a result of the estimation.

8. The information processing apparatus according to claim 1, further comprising a position/orientation acquisition unit configured to acquire a result of measuring the position and/or the orientation of the image display apparatus,

wherein the control unit controls an amount of shift of the image generated by the generation unit based on the result obtained by the position/orientation acquisition unit.

9. The information processing apparatus according to claim 1, further comprising an estimation unit configured to estimate a position of the specific object after a predetermined time from a time when the image display apparatus starts to move based on a previous image captured by the image capturing unit and a delay time caused by a processing time for at least a part of processing from the image capturing to the display,

wherein the generation unit generates an image depicting the object at the position of the object after the predetermined time by using a result of the estimation by the estimation unit.

10. The information processing apparatus according to claim 9,

wherein the specific object is a hand,

wherein the estimation unit further estimates a joint angle of a finger of the hand, and

wherein the generation unit generates an image obtained by changing the finger of the hand based on a result of the estimation by the estimation unit.

11. The information processing apparatus according to claim 10, wherein the estimation unit performs the estimation based on a learned model for estimating the joint angle of the finger from the image captured by the image capturing unit.

12. The information processing apparatus according to claim 1, further comprising an extraction unit configured to extract the specific object from the captured image of the real space,

wherein the generation unit generates an image depicting the specific object extracted by the extraction unit in a three-dimensional manner at a position based on a result of the estimation.

13. The information processing apparatus according to claim 12,

wherein the specific object includes a hand and an object held by the hand, and

wherein the extraction unit extracts the specific object from the captured image of the real space based on a skin color area and a feature set adjacent to the skin color area.

14. The information processing apparatus according to claim 1, wherein the image display apparatus is a head-mounted display.

15. An information processing apparatus that supplies, to an image display apparatus including a display unit configured to display an image generated using an image of a virtual space, an image to be displayed on the display unit, the information processing apparatus comprising:

a generation unit configured to generate an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined time from a time when the image display apparatus starts to move in the virtual space; and

a control unit configured to shift a position at which the image generated by the generation unit is displayed on the display unit based on a change in a position and/or an orientation of the image display apparatus.

16. An information processing apparatus that supplies, to an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display an image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit, the information processing apparatus comprising:

an estimation unit configured to estimate a position at which a specific object included in the captured image of the real space is present after a predetermined time from a time when the image display apparatus starts to move; and

a control unit configured to cause the display unit to display a combined image of an image obtained by capturing the real space in which a position at which the image is displayed is shifted based on a change in a position and/or an orientation of the image display apparatus, and an image depicting the specific object at the position estimated by the estimation unit.

17. A storage medium storing a program for causing a computer to execute an information processing method for supplying, to an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display the image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit, the information processing method comprising:

generating an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined time from a time when the image display apparatus starts to move in the captured image of the real space including the specific object; and

performing control to shift a position at which the generated image is displayed based on a change in a position and/or an orientation of the image display apparatus.

18. An information processing method for supplying, an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display the image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit, the information processing method comprising:

generating an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined time from a time when the image display apparatus starts to move in the captured image of the real space including the specific object; and

performing control to shift a position at which the generated image is displayed based on a change in a position and/or an orientation of the image display apparatus.

19. An information processing system including an information processing apparatus that supplies, to an image display apparatus including an image capturing unit configured to capture an image of a real space, and a display unit configured to display the image generated using the image captured by the image capturing unit, an image generated using the image captured by the image capturing unit, the information processing apparatus comprising:

a generation unit configured to generate an image depicting a specific object at a position at which the specific object is estimated to be present after a predetermined time from a time when the image display apparatus starts to move in the captured image of the real space including the specific object; and

a control unit configured to shift a position at which the image generated by the generation unit is displayed on the display unit based on a change in a position and/or an orientation of the image display apparatus.