Overly non-video content on a mobile device

• US 10,609,308 B2
• Filed: 12/28/2018
• Issued: 03/31/2020
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven
• Primary Examiner(s): Khan, Usman A
• Application Number:
  US16/235,582
• Publication Number:
  US 20190141260A1
• Time in Patent Office: 459 Days
• Field of Search: 348239
• CPC Class Codes: H04N 21/23892 involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Methods, systems, and devices are described for presenting non-video content through a mobile device that uses a video camera to track a video on another screen. In one embodiment, a system includes a video display, such as a TV, that displays video content. A mobile device with an integrated video camera captures video data from the TV and allows a user to select an area in the video in order to hear/feel/smell what is at that location in the video.

Claims

1. A mobile device comprising:

a video camera;

a display;

a processor; and

a memory communicatively coupled with the processor and storing computer-readable instructions that, upon execution by the processor, cause the mobile device to;

generate, by at least using the video camera, video data of video content presented on a video display;

track the video content based on the video data, the tracking comprising determining an identifier of the video content and a temporal position of the video content indicating timing of a presented portion of the video content; and

present, on the display of the mobile device, overlay content over at least a portion of the tracked video content based on the identifier and the temporal position.

2. The mobile device of claim 1, wherein the overlay content is selected based on the identifier of the video content, and wherein the presenting of the overlay content is time synchronized with the tracked video content based on the temporal position of the video content.

3. The mobile device of claim 1, wherein the overlay content is selected by at least:

determining a plurality of overlay contents associated with the temporal position and

selecting the overlay content from the plurality of overlay contents based on the identifier of the video content.

4. The mobile device of claim 1, wherein tracking the video content comprises determining a position and an orientation of the video content relative to the mobile device and a size of the video content, and wherein the overlay content is presented based on one or more of the size, the position, or the orientation of the video content.

5. The mobile device of claim 1, wherein the video content is presented on the video display by a video source, wherein tracking the video content comprises determining a viewing angle between the mobile device and the video source based on the video data, wherein the overlay content is selected based on the viewing angle.

6. The mobile device of claim 1, wherein the execution of the computer-readable instructions further causes the mobile device to store a low-resolution version of the video content, and wherein the tracking of the video content comprises image matching based on a hash of the low-resolution version of the video content.

7. The mobile device of claim 1, wherein the video content is presented on the video display by one or more computing devices, wherein the execution of the computer-readable instructions further causes the mobile device to receive the overlay content from at least one of the one or more computing devices.

8. The mobile device of claim 7, wherein the overlay content is received based on the presenting of the video content on the video display.

9. The mobile device of claim 8, wherein the presenting of the overlay content is time synchronized with the tracked video content based on the temporal position of the video content.

10. The mobile device of claim 7, wherein the overlay content is received based on communications between the mobile device and the at least one of the one or more computing devices, wherein the communications are based on one or more of the identifier of the video content or the temporal position of the video content.

11. The mobile device of claim 10, wherein the communications comprise the identifier of the video content, and wherein the overlay content and an additional overlay content are received from the at least one of the one or more computing devices based on the identifier of the video content, and wherein presenting the overlay content comprises selecting the overlay content from received overlay contents based on the temporal position of the video content.

12. The mobile device of claim 7, wherein the video content is also received from the at least one of the one or more computing devices, and wherein the tracking of the video content is further based on the received video content.

13. The mobile device of claim 1, wherein a plurality of overlay contents are pre-loaded onto the mobile device, and wherein presenting the overlay content comprises:

selecting the overlay content from the plurality of overlay contents based on the identifier and the temporal position of the video content; and

automatically presenting the overlay content based on a user setting for automatic presentation of overlay contents.

14. The mobile device of claim 1, wherein the execution of the computer-readable instructions further causes the mobile device to:

receive a user selection of a subportion of an image in the video content;

access the overlay content associated with the subportion of the image and synchronized with the temporal position of the video content, wherein the overlay content is presented on the display at substantially the same time as the video content is captured using the video camera.

15. A method implemented on a computer system that comprises a mobile device, the method comprising:

generating, based on a video camera of the mobile device, video data of video content displayed on a video display by a video source;

tracking the video content based on the video data, the tracking comprising determining an identifier of the video content and a temporal position of the video content indicating timing of a presented portion of the video content; and

presenting, on a display of the mobile device, overlay content over at least a portion of the tracked video content based on the identifier and the temporal position.

16. The method of claim 15, wherein the overlay content is selected based on the identifier of the video content, and wherein the presenting of the overlay content is time synchronized with the tracked video content based on the temporal position of the video content.

17. The method of claim 15, wherein the overlay content is selected, based on the temporal position of the video content, from a plurality of overlay contents associated with the identifier of the video content.

18. One or more non-transitory computer-readable storage media comprising instructions that, upon execution on a computer system that comprises a mobile device, cause the computer system to perform operations comprising:

generating, based on a video camera of the mobile device, video data of content displayed on a video display by a video source;

tracking the video content based on the video data, the tracking comprising determining an identifier of the video content and a temporal position of the video content indicating timing of a presented portion of the video content; and

presenting, on a display of the mobile device, overlay content over at least a portion of the tracked video content based on the identifier and the temporal position.

19. The one or non-transitory computer-readable storage media of claim 18, wherein the mobile device receives the overlay content from the video source based on the presenting of the video content on the video display, and wherein the presenting of the overlay content on the display of the mobile device is time synchronized with the tracked video content based on the temporal position of the video content.

20. The one or non-transitory computer-readable storage media of claim 18, wherein the overlay content is received from the video source based on communications between the mobile device and the video source, wherein the communications are based on one or more of the identifier of the video content or the temporal position of the video content.