Image processing apparatus, image processing method, and program

• US 9,292,974 B2
• Filed: 09/04/2012
• Issued: 03/22/2016
• Est. Priority Date: 10/27/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaino, Akihiko, Fukuchi, Masaki, Kashitani, Tatsuki, Ooi, Kenichiro, Guo, Jingjing
• Primary Examiner(s): Thompson, James A
• Application Number:
  US13/824,140
• Publication Number:
  US 20140225919A1
• Time in Patent Office: 1,295 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An information processing system that acquires video data captured by an image pickup unit; detects an object from the video data; detects a condition corresponding to the image pickup unit; and controls a display to display content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit.

Claims

1. An information processing system comprising:

an image pickup circuit configured to acquire video data;

a storage configured to store content; and

a processor configured to

detect an object from the video data;

detect a condition corresponding to the image pickup circuit;

retrieve content associated with the object based on the object detected from the video data; and

control a display to display the content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup circuit, the controlling including controlling the display to display the content associated with the object when the object is detected in the acquired video data, and controlling the display to continue displaying the content associated with the object when the object is no longer detected or when the object is partially observable in the acquired video data.

2. The information processing system of claim 1, wherein

the object detected from the video data is at least one of a shape, symbol, character string, design, object, part of an object and an image displayed on a display.

3. The information processing system of claim 1, wherein

detecting a condition corresponding to the image pickup circuit includes detecting at least one of a position and a posture of the image pickup circuit.

4. The information processing system of claim 3, wherein the processor is further configured to

detect at least one of a position and a posture of the detected object.

5. The information processing system of claim 4, wherein the processor is further configured to

control the position on the display at which the content associated with the object is displayed based on the at least one of the detected position and the detected posture of the image pickup circuit and the at least one of the detected position and the detected a posture of the detected object.

6. The information processing system of claim 1, further comprising:

a sensor that detects the condition corresponding to the image pickup circuit.

7. The information processing system of claim 6, wherein

the sensor includes at least one of a positioning sensor, an acceleration sensor and a gyrosensor.

8. The information processing system of claim 1, further comprising:

a memory that stores information corresponding to at least one object in association with information corresponding to the content, wherein

the information corresponding to the content includes at least one of a marker identifier, a related content identifier and a size corresponding to the content to be displayed.

9. The information processing system of claim 1, further comprising:

a memory that stores information corresponding to the detection of the object, wherein

the information corresponding to the detection of the object includes at least one of an object identifier, an object position, an object posture, an object detection time and a time corresponding to when the object was no longer detected in the acquired video data.

10. The information processing system of claim 9, wherein the processor is further configured to

control the position on the display at which the content associated with the object is displayed based on the condition corresponding to the image pickup circuit and the information corresponding to the detection of the object.

11. The information processing system of claim 1, wherein the processor is further configured to

control the display to move the content associated with the object on the display based on a change in at least one of a position and a posture of the image pickup circuit.

12. The information processing system of claim 1, wherein

the detected condition corresponding to the image pickup circuit includes at least one of a position and a posture of the image pickup circuit, and

the processor is further configured to control the display to stop displaying the content associated with the object when at least one of the position and the posture of the image pickup circuit satisfies a predetermined condition.

13. The information processing system of claim 12, wherein the processor is further configured to

control the display to display an indication that the display is to stop displaying the content associated with the object when at least one of the position and the posture of the image pickup circuit satisfies the predetermined condition.

14. The information processing system of claim 12, wherein

the predetermined condition corresponds to a distance between the image pickup circuit and the detected object, and

the processor is further configured to control the display to stop displaying the content associated with the object when the distance between image pickup circuit and the detected object exceeds a predetermined threshold value.

15. The information processing system of claim 12, wherein

the predetermined condition corresponds to a difference in an angle between an optical axis of the image pickup circuit and an axis extending from the image pickup circuit to the detected object, and

the processor is further configured to control the display to stop displaying the content associated with the object when the difference in angle exceeds a predetermined threshold value.

16. The information processing system of claim 12, wherein

the predetermined condition corresponds to a time elapsed from when the object was first detected in the acquired video data, and

the processor is further configured to control the display to stop displaying the content associated with the object when time elapsed exceeds a predetermined threshold value.

17. The information processing system of claim 12, wherein

the predetermined condition corresponds to a time elapsed from when the object was last detected in the acquired video data, and

the processor is further configured to control the display to stop displaying the content associated with the object when time elapsed exceeds a predetermined threshold value.

18. The information processing system of claim 1, wherein the processor is configured to control the display to start displaying the content associated with the object when the object is detected in the acquired video data and when a distance between the object and the image pickup circuit is less than a predetermined threshold value.

19. An information processing method performed by an information processing system, the method comprising:

acquiring video data captured by an image pickup unit;

detecting an object from the video data;

detecting a condition corresponding to the image pickup unit;

retrieving content associated with the object based on the object detected from the video data; and

controlling a display to display the content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit, the controlling including controlling the display to display the content associated with the object when the object is detected in the acquired video data, and controlling the display to continue displaying the content associated with the object when the object is no longer detected or when the object is partially observable in the acquired video data.

20. The information processing method of claim 19, wherein the controlling includes controlling the display to start displaying the content associated with the object when the object is detected in the acquired video data and when a distance between the object and the image pickup circuit is less than a predetermined threshold value.

21. A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a method, the method comprising:

acquiring video data captured by an image pickup unit;

detecting an object from the video data;

detecting a condition corresponding to the image pickup unit;

retrieving content associated with the object based on the object detected from the video data; and

controlling a display to display the content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit, the controlling including controlling the display to display the content associated with the object when the object is detected in the acquired video data, and controlling the display to continue displaying the content associated with the object when the object is no longer detected or when the object is partially observable in the acquired video data.

22. The non-transitory computer-readable medium of claim 21, wherein the controlling includes controlling the display to start displaying the content associated with the object when the object is detected in the acquired video data and when a distance between the object and the image pickup circuit is less than a predetermined threshold value.