Image processing apparatus, image processing method, and program

• US 10,068,382 B2
• Filed: 03/15/2017
• Issued: 09/04/2018
• Est. Priority Date: 10/27/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaino, Akihiko, Fukuchi, Masaki, Kashitani, Tatsuki, Ooi, Kenichiro, Guo, Jingjing
• Primary Examiner(s): Thompson, James A
• Application Number:
  US15/459,711
• Publication Number:
  US 20170186242A1
• Time in Patent Office: 538 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An information processing system that acquires video data captured by an image pickup unit; detects an object from the video data; detects a condition corresponding to the image pickup unit; and controls a display to display content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit.

Claims

1. An information processing apparatus comprising:

a storage configured to store computer program instructions; and

a processor configured to execute the computer program instructions to cause the processor to;

acquire video data of a real space captured by a camera;

detect a real object in the real space and recognize a three-dimensional position and posture of the information processing apparatus based on the acquired video data;

identify a virtual object based on the detected real object;

determine a position of the virtual object according to the detected real object; and

control display of the virtual object at the determined position based on the recognized three-dimensional position and the recognized posture of the information processing apparatus, wherein the virtual object continues to be displayable when the detected real object is no longer detectable in the video data.

2. The information processing apparatus of claim 1, wherein the processor is configured to execute the computer program instructions to cause the processor to control the display of the virtual object by selecting one of plural sets of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

3. The information processing apparatus of claim 1, wherein the processor is configured to execute the computer program instructions to cause the processor to control the display of the virtual object by selecting a first set of drawing data or a second set of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

4. The information processing apparatus of claim 3, wherein the first set of drawing data includes additional information that does not correspond to any information included in the second set of drawing data.

5. The information processing apparatus of claim 4, wherein the additional information included in the first set of drawing data corresponds to a constituent part of the virtual object.

6. The information processing apparatus of claim 4, wherein the additional information included in the first set of drawing data corresponds to additional details of the virtual object.

7. The information processing apparatus of claim 3, wherein the first set of drawing data corresponds to a near situation, and the second set of drawing data corresponds to a far situation.

8. The information processing apparatus of claim 3, wherein the processor is further configured to execute the computer program instructions to cause the processor to download the first set of drawing data and the second set of drawing data for rendering the virtual object from an exterior server.

9. The information processing apparatus of claim 1, wherein the processor is configured to execute the computer program instructions to cause the processor to:

perform a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired video data; and

determine the position of the virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.

10. The information processing apparatus of claim 1, farther comprising the camera.

11. An information processing method performed by an information processing apparatus, the method comprising:

acquiring video data of a real space captured by a camera;

detecting a real object in the real space and recognizing a three-dimensional position and posture of the information processing apparatus based on the acquired video data;

identifying a virtual object based on the detected real object;

determining a position of the virtual object according to the detected real object; and

controlling, by circuitry of the information processing apparatus, display of the virtual object at the determined position based on the recognized three-dimensional position and the recognized posture of the information processing apparatus, wherein the virtual object continues to be displayable when the detected real object is no longer detectable in the video data.

12. The information processing method of claim 11, wherein controlling the display of the virtual object comprises selecting a first set of drawing data or a second set of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

13. The information processing method of claim 12, wherein the first set of drawing data includes additional information that does not correspond to any information included in the second set of drawing data.

14. The information processing method of claim 13, wherein the additional information included in the first set of drawing data corresponds to a constituent part of the virtual object.

15. The information processing method of claim 13, wherein the additional information included in the first set of drawing data corresponds to additional details of the virtual object.

16. The information processing method of claim 12, wherein the first set of drawing data corresponds to a near situation, and the second set of drawing data corresponds to a far situation.

17. The information processing method of claim 11, further comprising:

performing a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired video data; and

determining the position of the virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.

18. A non-transitory computer-readable medium including computer program instructions that, when executed by an information processing apparatus, cause the information processing apparatus to perform a method, the method comprising:

acquiring video data of a real space captured by a camera;

detecting a real object in the real space and recognizing a three-dimensional position and posture of the information processing apparatus based on the acquired video data;

identifying a virtual object based on the detected real object;

determining a position of the virtual object according to the detected real object; and

controlling display of the virtual object at the determined position based on the recognized three-dimensional position and the recognized posture of the information processing apparatus, wherein the virtual object continues to be displayable when the detected real object is no longer detectable in the video data.

19. The non-transitory computer-readable medium of claim 18, wherein controlling the display of the virtual object comprises selecting a first set of drawing data or a second set of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

20. The non-transitory computer-readable medium of claim 19, wherein the first set of drawing data includes additional information that does not correspond to any information included in the second set of drawing data.

21. The non-transitory computer-readable medium of claim 18, wherein the method performed by executing the computer program instructions further comprises:

performing a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired video data; and

determining the position of the virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.