Display control device and display control method

• US 10,049,497 B2
• Filed: 07/11/2016
• Issued: 08/14/2018
• Est. Priority Date: 11/13/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kasahara, Shunichi, Rekimoto, Junichi
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US15/206,435
• Publication Number:
  US 20160321843A1
• Time in Patent Office: 764 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

There is provided a display control device including a matching section configured to match a first image or sensor data output from a first imaging device or a sensor worn on a head of a first user, to a second image output from a second imaging device worn on a part other than the head of the first user, a sight estimation section configured to estimate a region corresponding to a sight of the first user in the second image, on the basis of a result of the matching, and a display control section configured to generate an image expressing the sight of the first user using the second image on the basis of a result of the estimation of the sight, and display the image expressing the sight of the first user toward a second user that is different from the first user.

Claims

1. An information processing apparatus, comprising:

at least one Central Processing Unit (CPU) configured to;

receive sight information corresponding to a direction of a first display terminal of a first user;

receive a background image from a wide-angle camera, wherein the wide-angle camera captures the background image having a view angle wider than a view angle of a display of the first display terminal; and

send display information to a second display terminal of a second user, wherein the second display terminal displays a marking image within the background image based on the display information, wherein the marking image is based on the sight information, and the marking image indicates a viewing perception of the first user in the background image,

wherein the first display terminal is separate from the wide-angle camera such that the direction of the first display terminal is changed independently from a direction of the wide-angle camera, and

wherein a viewpoint of the background image is moved based on a movement of the wide-angle camera in a real space independently of the first display terminal.

2. The information processing apparatus according to claim 1, wherein the at least one CPU is further configured to receive sensor data associated with the sight information, and wherein the sensor data is output from a sensor worn on a head of the first user.

3. The information processing apparatus according to claim 2, wherein the received sensor data corresponds to movement information of the head of the first user.

4. The information processing apparatus according to claim 3, wherein the sight information is associated with a direction of the head of the first user.

5. The information processing apparatus according to claim 4, wherein the wide-angle camera is mountable on a part of a body of the first user other than the head of the first user, and wherein the background image is associated with a direction of the body of the first user.

6. The information processing apparatus according to claim 2, wherein the sensor includes a motion sensor.

7. The information processing apparatus according to claim 1, wherein the at least one CPU is further configured to receive a sight image as the sight information, and wherein the sight image is output from an imaging device of the first user.

8. The information processing apparatus according to claim 2, wherein the at least one CPU is further configured to:

synchronize a first time of a first image, associated with the first display terminal, to a second time of the background image, wherein the first time is synchronized with the second time based on the sensor data of the first display terminal; and

determine a region, within the background image, that matches the first image, wherein the region is determined based on the synchronization.

9. The information processing apparatus according to claim 1, wherein the least one CPU is at least a part of a network server.

10. The information processing apparatus according to claim 9, wherein the network server is configured to:

receive annotation information from the second user; and

add an annotation to the background image based on the annotation information.

11. The information processing apparatus according to claim 9, wherein the network server is configured to send graphical user interface information to the second display terminal, wherein the second display terminal displays at least one graphical user interface for input of an annotation, and wherein the at least one graphical user interface is displayed based on the graphical user interface information.

12. The information processing apparatus according to claim 1, wherein the marking image includes a contour having at least one of a rectangle shape or a circle shape, and wherein the at least one of the rectangle shape or the circle shape substantially corresponds to the viewing perception of the first user.

13. The information processing apparatus according to claim 12, wherein the second display terminal initiates the display of the marking image based on a user input on the second display terminal.

14. The information processing apparatus according to claim 13, wherein the second display terminal is a hand-held terminal that includes a touch panel, wherein the touch panel receives a gesture input as the user input.

15. A display control method comprising:

in at least one central processing unit (CPU);

receiving sight information corresponding to a direction of a first display terminal of a first user;

receiving a background image from a wide-angle camera, wherein the wide-angle camera captures the background image having a view angle wider than a view angle of a display of the first display terminal; and

sending display information to a second display terminal of a second user, wherein the second display terminal displays a marking image within the background image based on the display information, wherein the marking image is based on the sight information, and the marking image indicates a viewing perception of the first user in the background image,

wherein the first display terminal is separate from the wide-angle camera such that the direction of the first display terminal is changed independently from a direction of the wide-angle camera, and

wherein a viewpoint of the background image is moved based on a movement of the wide-angle camera in a real space independently of the first display terminal.

16. A non-transitory computer-readable medium having stored thereon computer-executable instructions which when executed by a computer, cause the computer to execute operations, the operations comprising:

receiving sight information corresponding to a direction of a first display terminal of a first user;

receiving a background image from a wide-angle camera, wherein the wide-angle camera captures the background image having a view angle wider than a view angle of a display of the first display terminal; and

sending display information to a second display terminal of a second user, wherein the second display terminal displays a marking image within the background image based on the display information, wherein the marking image is based on the sight information, and the marking image indicates a viewing perception of the first user in the background image,

wherein the first display terminal is separate from the wide-angle camera such that the direction of the first display terminal is changed independently from a direction of the wide-angle camera, and

wherein a viewpoint of the background image is moved based on a movement of the wide-angle camera in a real space independently of the first display terminal.