Image processing apparatus, image processing method, and program

• US 10,453,266 B2
• Filed: 08/01/2018
• Issued: 10/22/2019
• Est. Priority Date: 10/27/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaino, Akihiko, Fukuchi, Masaki, Kashitani, Tatsuki, Ooi, Kenichiro, Guo, Jingjing
• Primary Examiner(s): Thompson, James A
• Application Number:
  US16/051,893
• Publication Number:
  US 20180357829A1
• Time in Patent Office: Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An information processing system that acquires video data captured by an image pickup unit; detects an object from the video data; detects a condition corresponding to the image pickup unit; and controls a display to display content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit.

Claims

1. An information processing method, comprising:

acquiring image data of a real space captured by a camera;

recognizing a first real object and subsequently recognizing a second real object from the acquired image data; and

controlling, by circuitry of an information processing apparatus, display of a first virtual object associated with the recognized first real object and a second virtual object associated with the recognized second real object based on a camera position or posture, including, in response to recognition of the second real object when the first virtual object is being displayed,

displaying the second virtual object together with the first virtual object in response to a distance from the first real object to the camera position being less than a predetermined distance threshold when the second real object is recognized, or

displaying the second virtual object to replace the display of the first virtual object in response to the distance from the first real object to the camera position being greater than the predetermined distance threshold when the second real object is recognized.

2. The information processing method of claim 1, wherein the first virtual object and the second virtual object belong to a same type.

3. The information processing method of claim 1, wherein the first real object and the second real object belong to a same type.

4. The information processing method of claim 1, further comprising:

recognizing a three-dimensional position and posture of the information processing apparatus based on the acquired image data; and

determining a position of the first virtual object according to the recognized first real object.

5. The information processing method of claim 4, wherein the controlling the display of the first virtual object comprises selecting a first set of drawing data or a second set of drawing data for rendering the first virtual object based on the recognized three-dimensional position of the information processing apparatus.

6. The information processing method of claim 4, further comprising:

performing a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired image data; and

determining the position of the first virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.

7. An information processing apparatus, comprising:

a storage configured to store computer program instructions, and

a processor configured to execute the computer program instructions to cause the processor to;

acquire image data of a real space captured by a camera;

recognize a first real object and subsequently recognize a second real object from the acquired image data; and

control display of a first virtual object associated with the recognized first real object and a second virtual object associated with the recognized second real object based on a camera position or posture, including, in response to recognition of the second real object when the first virtual object is being displayed,

displaying the second virtual object together with the first virtual object in response to a distance from the first real object to the camera position being less than a predetermined distance threshold when the second real object is recognized, or

displaying the second virtual object to replace the display of the first virtual object in response to the distance from the first real object to the camera position being greater than the predetermined distance threshold when the second real object is recognized.

8. The information processing apparatus of claim 7, wherein the first virtual object and the second virtual object belong to a same type.

9. The information processing apparatus of claim 7, wherein the first real object and the second real object belong to a same type.

10. The information processing apparatus of claim 7, wherein the circuitry is configured to:

recognize a three-dimensional position and posture of the information processing apparatus based on the acquired image data; and

determine a position of the first virtual object according to the recognized first real object.

11. The information processing apparatus of claim 10, wherein the circuitry is configured to control the display of the first virtual object by selecting a first set of drawing data or a second set of drawing data for rendering the first virtual object based on the recognized three-dimensional position of the information processing apparatus.

12. The information processing apparatus of claim 10, wherein the circuitry is configured to:

perform a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired image data; and

determine the position of the first virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.

13. An information processing apparatus, comprising:

a storage configured to store computer program instructions, and

a processor configured to execute the computer program instructions to cause the processor to;

acquire image data of a real space captured by a camera;

recognize an object from the acquired image data;

recognize a three-dimensional position and posture of the information processing apparatus based on the acquired image data;

generate a three-dimensional map data based on the acquired image data; and

control display of a virtual object that is associated with the recognized real object based on the three-dimensional map data, including

in response to recognition of another recognized object,

displaying the virtual object together with another virtual object, the another virtual object corresponding to the another recognized object, in response to a distance from the recognized real object to the camera position being less than a predetermined distance threshold when the another object is recognized, and

displaying the another virtual object to replace the display of the virtual object in response to the distance from the recognized real object to the camera position being greater than the predetermined distance threshold when the another object is recognized, and

controlling a movement of the virtual object based on the posture of the information processing apparatus.

14. The information processing apparatus of claim 13, wherein a speed of movement of the virtual object is slower than a speed of a change in a field of view of the camera caused by a posture change of the information processing apparatus.

15. The information processing apparatus of claim 13, wherein the circuitry is configured to control the display of the virtual object by selecting one of plural sets of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

16. The information processing apparatus of claim 13, wherein the circuitry is configured to:

perform a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired video data; and

determine the position of the virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.

17. The information processing apparatus of claim 13, further comprising the camera.

18. An information processing method, comprising:

acquiring image data of a real space captured by a camera;

recognizing an object from the acquired image data;

recognizing a three-dimensional position and posture of the information processing apparatus based on the acquired image data;

generating a three-dimensional map data based on the acquired image data; and

controlling, by circuitry of an information processing apparatus, display of a virtual object that is associated with the recognized real object based on the three-dimensional map data, including

in response to recognition of another recognized object,

displaying the virtual object together with another virtual object, the another virtual object corresponding to the another recognized object, in response to a distance from the recognized real object to the camera position being less than a predetermined distance threshold when the another object is recognized, and

displaying the another virtual object to replace the display of the virtual object in response to the distance from the recognized real object to the camera position being greater than the predetermined distance threshold when the another object is recognized, and

controlling a movement of the virtual object based on the posture of the information processing apparatus.

19. The information processing method of claim 18, wherein a speed of movement of the virtual object is slower than a speed of a change in a field of view of the camera caused by a posture change of the information processing apparatus.

20. The information processing method of claim 18, wherein the controlling the display of the virtual object comprises selecting one of plural sets of drawing data for rendering the virtual object based on the recognized three-dimensional position of the information processing apparatus.

21. The information processing method of claim 18, further comprising:

performing a simultaneous localization and mapping process to dynamically recognize a three-dimensional structure of a peripheral environment of the camera based on the acquired video data; and

determining the position of the virtual object in association with the recognized three-dimensional position and the recognized posture of the information processing apparatus based on the recognized three-dimensional structure of the peripheral environment of the camera.