Display method and display apparatus

• US 10,303,945 B2
• Filed: 12/16/2016
• Issued: 05/28/2019
• Est. Priority Date: 12/27/2012
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Aoyama, Hideki, Oshima, Mitsuaki, Nakanishi, Koji, Maeda, Toshiyuki, Ueki, Akihiro, Miyoshi, Kengo, Mukai, Tsutomu
• Primary Examiner(s): Chen, Yu
• Application Number:
  US15/381,940
• Publication Number:
  US 20170206417A1
• Time in Patent Office: 893 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2358/00 Arrangements for display data security
• CPC Class Codes: G09G 2360/16 Calculation or use of calculated indices related to luminance levels in display data
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2370/18 Use of optical transmission of display information
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04B 10/1149 Arrangements for indoor wireless networking of information
• CPC Class Codes: H04B 10/116 Visible light communication
• CPC Class Codes: H04J 3/0635 Clock or time synchronisation in a network timer in protocols H04L69/28
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 1/0061 Error detection codes
• CPC Class Codes: H04L 1/0071 Use of interleaving interleaving per se H03M13/27
• CPC Class Codes: H04L 7/0033 : Correction by delay
• CPC Class Codes: H04L 7/0091 : Transmitter details
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72412 : using two-way short-range wireless interfaces
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A display method is for a display apparatus to display an image, and includes: obtaining a captured display image and a decode target image by an image sensor capturing an image of a subject; obtaining a light ID by decoding the decode target image; transmitting the light ID to a server; obtaining, from the server, an AR image and recognition information which are associated with the light ID; recognizing a region according to the recognition information as a target region from the captured display image; and displaying the captured display image in which the AR image is superimposed on the target region.

Claims

1. A display method for a display apparatus to display an image, the display method comprising:

(a) obtaining a captured display image by using an image sensor with a first exposure time;

(b) obtaining light identification information by visible light communication with a subject, wherein the subject transmits the light identification information by change in luminance by causing at least one light emitting element to blink;

(c) transmitting the light identification information to a server;

(d) obtaining, from the server, an augmented reality image and recognition information which are associated with the light identification information, wherein the recognition information indicates a location of a target region within the captured display image;

(e) recognizing the target region within the captured display image using the recognition information; and

(f) displaying the captured display image in which the augmented reality image is superimposed on the target region,

wherein

the obtaining of the light identification information includes;

obtaining a decode target image by using the image sensor with a second exposure time which is shorter than the first exposure time; and

obtaining the light identification information by decoding the decode target image.

2. The display method according to claim 1, wherein

the subject is an object illuminated by a transmitter which transmits a signal by changing luminance,

the augmented reality image is a video which includes images, and

in (f), the video is displayed, starting with one of, among the images, an image which includes the object and a predetermined number of images which are to be displayed around a time at which the image which includes the object is to be displayed.

3. The display method according to claim 2, wherein

the predetermined number of images are ten frames.

4. The display method according to claim 2, wherein

the object is a still image, and

in (f), the video is displayed, starting with an image same as the still image.

5. The display method according to claim 1, wherein

the recognition information is reference information for locating a reference region of the captured display image, and

in (e), the reference region is located from the captured display image, based on the reference information, and the target region is recognized from the captured display image, based on a position of the reference region.

6. The display method according to claim 1, wherein

the recognition information includes reference information for locating a reference region of the captured display image, and target information indicating a relative position of the target region with respect to the reference region, and

in (e), the reference region is located from the captured display image, based on the reference information, and a region in the relative position indicated by the target information is recognized as the target region from the captured display image, based on a position of the reference region.

7. The display method according to claim 6, wherein

the reference information indicates that the position of the reference region in the captured display image matches a position of a bright line pattern region in the decode target image, the bright line pattern region including a pattern formed by bright lines which appear due to exposure lines included in the image sensor being exposed.

8. The display method according to claim 6, wherein

the reference information indicates that the reference region in the captured display image is a region in which a display is shown in the captured display image.

9. The display method according to claim 8, wherein

the reference region is an outer frame portion having a predetermined color in an image displayed on the display.

10. The display method according to claim 1, wherein

in (f), a first augmented reality image which is the augmented reality image is displayed for a predetermined display period, while preventing display of a second augmented reality image different from the first augmented reality image.

11. The display method according to claim 10, wherein

in (f), decoding a decode target image newly obtained is prohibited during the predetermined display period.

12. The display method according to claim 11, wherein

(f) further includes:

measuring an acceleration of the display apparatus using an acceleration sensor during the predetermined display period;

determining whether the measured acceleration is greater than or equal to a threshold; and

displaying the second augmented reality image instead of the first augmented reality image by no longer preventing the display of the second augmented reality image, when the measured acceleration is determined to be greater than or equal to the threshold.

13. The display method according to claim 1, wherein

(f) further includes:

determining whether a face of a user is approaching the display apparatus, based on image capturing by a face camera included in the display apparatus; and

displaying a first augmented reality image which is the augmented reality image while preventing display of a second augmented reality image different from the first augmented reality image, when the face is determined to be approaching.

14. The display method according to claim 1, wherein

(f) further includes:

determining whether a face of a user is approaching the display apparatus, based on an acceleration of the display apparatus measured by an acceleration sensor; and

displaying a first augmented reality image which is the augmented reality image while preventing display of a second augmented reality image different from the first augmented reality image, when the face is determined to be approaching.

15. The display method according to claim 1, wherein

the captured display image and the decode target image are obtained by the image sensor capturing an image which includes a plurality of displays each showing an image,

in (e), a region in which, among the plurality of displays, a transmission display that is transmitting the light identification information is shown is recognized as the target region from the captured display image, and

in (f),

first subtitles for an image displayed on the transmission display are superimposed on the target region, as the augmented reality image, and

second subtitles obtained by enlarging the first subtitles are further superimposed on a region larger than the target region of the captured display image.

16. The display method according to claim 15, wherein

(f) further includes:

determining whether information obtained from the server includes sound information; and

preferentially outputting sound indicated by the sound information over the first subtitles and the second subtitles, when the sound information is determined to be included.

17. A display apparatus which displays an image, the display apparatus comprising:

a processor; and

a memory in which a computer program is stored, wherein

the computer program causes the processor to perform the display method according to claim 1 when the computer program is executed by the processor.

18. A non-transitory recording medium in which a computer program is stored, the computer program causing a processor to perform the display method according to claim 1 when the computer program is executed by the processor.

19. The display method according to claim 1, wherein

the displaying superimposes the augmented reality image only on the captured display image among the captured display image and the decode target image.

20. The display method according to claim 1, wherein

the obtaining of the light identification information includes:

setting the second exposure time of the image sensor which has a plurality of exposure lines, so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to each of the plurality of exposure lines included in the image sensor appears according to the change in luminance of the subject;

obtaining the decode target image including a plurality of bright lines, by capturing the subject that changes in luminance by the image sensor with the set second exposure time;

obtaining the light identification information by demodulating data specified by a pattern of the plurality of bright lines included in the obtained decode target image.

21. A display method, comprising:

(a) obtaining a captured image by an image sensor, at a first exposure time, capturing an image of, as a subject, a still image illuminated by a transmitter which transmits a signal by changing luminance by causing at least one light emitting element to blink;

(b) obtaining the signal by visible light communication with the still image which is illuminated by the transmitter which transmits the signal by changing luminance by causing the at least one light emitting element to blink; and

(c) reading a video corresponding to the obtained signal from a memory, superimposing the video on a target region corresponding to the still image in the captured image, and displaying, on a display, the captured image in which the video is superimposed on the target region, the video including a plurality of images, a leading image of the plurality of images being the same as the still image, wherein

in (c), the video is displayed, starting with the leading image same as the still image, by superimposing the leading image on the target region corresponding to the still image at the start of displaying the video,

the obtaining of the signal includes;

obtaining a decode target image by using the image sensor with a second exposure time which is shorter than the first exposure time; and

obtaining the signal by decoding the decode target image.

22. The display method according to claim 21, wherein

the still image includes an outer frame having a predetermined color,

the display method further comprising:

recognizing the target region from the captured image, based on the predetermined color, wherein

in (c), the video is resized to a size of the recognized target region, the resized video is superimposed on the target region in the captured image, and the captured image in which the resized video is superimposed on the target region is displayed on the display.

23. The display method according to claim 21, wherein

the obtaining of the signal includes:

setting the second exposure time of the image sensor which has a plurality of exposure lines, so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to each of the plurality of exposure lines included in the image sensor appears according to the change in luminance of the subject;

obtaining the decode target image including a plurality of bright lines, by capturing the subject that changes in luminance by the image sensor with the set second exposure time;

obtaining the signal by demodulating data specified by a pattern of the plurality of bright lines included in the obtained the decode target image.