Display method and display apparatus

• US 10,521,668 B2
• Filed: 04/12/2019
• Issued: 12/31/2019
• Est. Priority Date: 12/27/2012
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Aoyama, Hideki, Oshima, Mitsuaki, Nakanishi, Koji, Maeda, Toshiyuki, Ueki, Akihiro, Miyoshi, Kengo, Mukai, Tsutomu
• Primary Examiner(s): Chen, Yu
• Application Number:
  US16/383,286
• Publication Number:
  US 20190236369A1
• Time in Patent Office: 263 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2358/00 Arrangements for display data security
• CPC Class Codes: G09G 2360/16 Calculation or use of calculated indices related to luminance levels in display data
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2370/18 Use of optical transmission of display information
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04B 10/1149 Arrangements for indoor wireless networking of information
• CPC Class Codes: H04B 10/116 Visible light communication
• CPC Class Codes: H04J 3/0635 Clock or time synchronisation in a network timer in protocols H04L69/28
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 1/0061 Error detection codes
• CPC Class Codes: H04L 1/0071 Use of interleaving interleaving per se H03M13/27
• CPC Class Codes: H04L 7/0033 : Correction by delay
• CPC Class Codes: H04L 7/0091 : Transmitter details
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72412 : using two-way short-range wireless interfaces
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A display method is for a display apparatus to display an image, and includes: obtaining a captured display image by an image sensor included in a terminal device; obtaining a light ID by visible light communication with a subject; obtaining an AR image and recognition information which are associated with the light ID from a memory included in the terminal device; recognizing a target region within the captured display image using the recognition information; and displaying the captured display image in which the AR image is superimposed on the target region.

Claims

1. A display method for a display apparatus to display an image, the display method comprising:

obtaining a captured display image by using an image sensor included in a terminal device with a first exposure time;

obtaining light identification information by visible light communication with a subject, wherein the subject transmits the light identification information by change in luminance by causing at least one light emitting element to blink;

obtaining an augmented reality image and recognition information which are associated with the light identification information from a memory included in the terminal device;

recognizing a target region within the captured display image using the recognition information; and

displaying the captured display image in which the augmented reality image is superimposed on the target region,

wherein

the obtaining of the light identification information includes;

obtaining a decode target image by using the image sensor with a second exposure time which is shorter than the first exposure time; and

obtaining the light identification information by decoding the decode target image.

2. The display method according to claim 1, wherein

the recognition information indicates a location of the target region within the captured display image.

3. The display method according to claim 1, wherein

the subject is an object illuminated by a transmitter which transmits a signal by changing luminance,

the augmented reality image is a video which includes images, and

in the displaying of the captured display image, the video is displayed, starting with one of, among the images, an image which includes the object and a predetermined number of images which are to be displayed around a time at which the image which includes the object is to be displayed.

4. The display method according to claim 3, wherein

the predetermined number of images are ten frames.

5. The display method according to claim 3, wherein

the object is a still image, and

in the displaying of the captured display image, the video is displayed, starting with an image same as the still image.

6. The display method according to claim 1, wherein

the recognition information is reference information for locating a reference region of the captured display image, and

in the recognizing of the target region, the reference region is located from the captured display image, based on the reference information, and the target region is recognized from the captured display image, based on a position of the reference region.

7. The display method according to claim 1, wherein

the recognition information includes reference information for locating a reference region of the captured display image, and target information indicating a relative position of the target region with respect to the reference region, and

in the recognizing of the target region, the reference region is located from the captured display image, based on the reference information, and a region in the relative position indicated by the target information is recognized as the target region from the captured display image, based on a position of the reference region.

8. The display method according to claim 7, wherein

the reference information indicates that the position of the reference region in the captured display image matches a position of a bright line pattern region in the decode target image, the bright line pattern region including a pattern formed by bright lines which appear due to exposure lines included in the image sensor being exposed.

9. The display method according to claim 7, wherein

the reference information indicates that the reference region in the captured display image is a region in which a display is shown in the captured display image.

10. The display method according to claim 9, wherein

the reference region is an outer frame portion having a predetermined color in an image displayed on the display.

11. The display method according to claim 1, wherein

in the displaying of the captured display image, a first augmented reality image which is the augmented reality image is displayed for a predetermined display period, while preventing display of a second augmented reality image different from the first augmented reality image.

12. The display method according to claim 11, wherein

in the displaying of the captured display image, decoding a decode target image newly obtained is prohibited during the predetermined display period.

13. The display method according to claim 12, wherein

the displaying of the captured display image includes:

measuring an acceleration of the display apparatus using an acceleration sensor during the predetermined display period;

determining whether the measured acceleration is greater than or equal to a threshold; and

displaying the second augmented reality image instead of the first augmented reality image by no longer preventing the display of the second augmented reality image, when the measured acceleration is determined to be greater than or equal to the threshold.

14. The display method according to claim 1, wherein

the displaying of the captured display image includes:

determining whether a face of a user is approaching the display apparatus, based on image capturing by a face camera included in the display apparatus; and

displaying a first augmented reality image which is the augmented reality image while preventing display of a second augmented reality image different from the first augmented reality image, when the face is determined to be approaching.

15. The display method according to claim 1, wherein

the displaying of the captured display image includes:

determining whether a face of a user is approaching the display apparatus, based on an acceleration of the display apparatus measured by an acceleration sensor; and

displaying a first augmented reality image which is the augmented reality image while preventing display of a second augmented reality image different from the first augmented reality image, when the face is determined to be approaching.

16. The display method according to claim 1, wherein

the captured display image and the decode target image are obtained by the image sensor capturing an image which includes a plurality of displays each showing an image,

in the recognizing of the target region, a region in which, among the plurality of displays, a transmission display that is transmitting the light identification information is shown is recognized as the target region from the captured display image, and

in the displaying of the captured display image,

first subtitles for an image displayed on the transmission display are superimposed on the target region, as the augmented reality image, and

second subtitles obtained by enlarging the first subtitles are further superimposed on a region larger than the target region of the captured display image.

17. The display method according to claim 1, wherein

the displaying superimposes the augmented reality image on the captured display image among the captured display image and the decode target image.

18. The display method according to claim 1, wherein

the obtaining of the light identification information includes:

setting the second exposure time of the image sensor which has a plurality of exposure lines, so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to each of the plurality of exposure lines included in the image sensor appears according to the change in luminance of the subject;

obtaining the decode target image including a plurality of bright lines, by capturing the subject that changes in luminance by the image sensor with the set second exposure time;

obtaining the light identification information by demodulating data specified by a pattern of the plurality of bright lines included in the obtained decode target image.

19. A display apparatus which displays an image, the display apparatus comprising:

a processor; and

a memory in which a computer program is stored, wherein

the computer program causes the processor to perform the display method according to claim 1 when the computer program is executed by the processor.

20. A non-transitory recording medium in which a computer program is stored, the computer program causing a processor to perform the display method according to claim 1 when the computer program is executed by the processor.