Overlay non-video content on a mobile device

• US 10,171,754 B2
• Filed: 09/11/2017
• Issued: 01/01/2019
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven
• Primary Examiner(s): Khan, Usman A
• Application Number:
  US15/700,607
• Publication Number:
  US 20170374294A1
• Time in Patent Office: 477 Days
• Field of Search: 348239
• CPC Class Codes: H04N 21/23892 involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Methods, systems, and devices are described for presenting non-video content through a mobile device that uses a video camera to track a video on another screen. In one embodiment, a system includes a video display, such as a TV, that displays video content. A mobile device with an integrated video camera captures video data from the TV and allows a user to select an area in the video in order to hear/feel/smell what is at that location in the video.

Claims

1. A mobile device comprising:

a video camera;

a display;

a processor; and

a memory communicatively coupled with the processor and storing computer-readable instructions that, upon execution by the processor, cause the mobile device to;

capture, by at least using the video camera, video content displayed on a video display, the video content having a marker that includes time code, the time code identifying a temporal position corresponding to a portion of time in the video content;

track the video content based on the marker;

receive a user selection of a subportion of an image in the video content;

access non-video content associated with the subportion of the image and synchronized with the temporal position of the video content using the time code; and

present, on the display, the non-video content associated with the subportion of the image at substantially the same time as the video content is captured using the video camera.

2. The mobile device of claim 1, wherein the time code of the marker is presented with a pattern in the video content, wherein the pattern changes over time, wherein a presented change to the pattern corresponds to the time code, identifies a progression of the video content based on the time code, and does not identify the non-video content, and wherein the progression indicates the temporal position of the video content.

3. The mobile device of claim 1, wherein the video content is provided from a video game console to the video display, and wherein the computer-readable instructions further cause the mobile device to transmit data associated with the non-video content to a game controller communicatively coupled with the video game console.

4. The mobile device of claim 1, wherein tracking the video content comprises determining, based on the marker, a position and an orientation of the video content relative to the mobile device and a size of the video content.

5. The mobile device of claim 4, wherein the non-video content is accessed based on at least the position or orientation of the video content.

6. The mobile device of claim 1, wherein the marker comprises an on-screen marker that is presented at the video display, and wherein the presented on-screen marker comprises a binary pattern that identifies the video content and the time code.

7. The mobile device of claim 1, wherein the computer-readable instructions further cause the mobile device to store a low-resolution version of the video content, and wherein the presentation of the non-video content is time synchronized with the presentation of the video content by at least performing image matching based on a hash of the low-resolution version of the video content.

8. The mobile device of claim 1, wherein the marker is presented the video display at a frequency not perceptible to the human eye, and wherein the computer-readable instructions further cause the mobile device to identify the marker based on an application of a Fourier transform to the video content.

9. The mobile device of claim 1, wherein the marker further includes an audio marker, and wherein the temporal position is identified based on the audio marker.

10. The mobile device of claim 1, wherein the video content is provided from a video game console to the video display, and wherein accessing the non-video content comprises receiving the non-video content from the video game console based on the marker.

11. The mobile device of claim 1, wherein non-video content is pre-loaded onto the mobile device.

12. The mobile device of claim 1, wherein the non-video content is accessed and presented in response to the user selection of the subportion of the image.

13. A computer-implemented method comprising:

capturing, by a video camera of a mobile device, video content displayed on a video display, the video content having a marker that includes time code, the time code identifying a temporal position corresponding to a portion of time in the video content;

tracking, by the mobile device, the video content based on the marker;

receiving, by the mobile device, a user selection of a subportion of an image in the video content;

accessing, by the mobile device, non-video content associated with the subportion of the image and synchronized with the temporal position of the video content using the time code; and

presenting, by the mobile device on a display of the mobile device, the non-video content associated with the subportion of the image at substantially the same time as the video content is captured using the video camera.

14. The computer-implemented method of claim 13, wherein the time code of the marker is presented in the video content with a pattern that changes over time, wherein a presented change to the pattern corresponds to the time code, identifies a progression of the video content based on the time code, and does not identify the non-video content, and wherein the progression indicates the temporal position of the video content.

15. The computer-implemented method of claim 13, wherein the marker includes one or more markers each having a different value.

16. The computer-implemented method of claim 13, wherein the non-video content includes at least one of text, animations, graphic enhancements, advertisements, and hyperlinks.

17. A non-transitory computer-readable storage medium comprising instructions that, upon execution on a mobile device, cause the mobile device to perform operations comprising:

capturing, by a video camera of the mobile device, video content displayed on a video display, the video content having a marker that includes time code, the time code identifying a temporal position corresponding to a portion of time in the video content;

tracking the video content based on the marker;

receiving a user selection of a subportion of an image in the video content;

accessing non-video content associated with the subportion of the image and synchronized with the temporal position of the video content using the time code; and

presenting, on a display of the mobile device, the non-video content associated with the subportion of the image at substantially the same time as the video content is captured using the video camera.

18. The non-transitory computer-readable storage medium of claim 17, wherein the time code of the marker is presented in the video content with a pattern that changes over time, wherein a presented change to the pattern corresponds to the time code, identifies a progression of the video content based on the time code, and does not identify the non-video content, and wherein the progression indicates the temporal position of the video content.

19. The non-transitory computer-readable storage medium of claim 17, wherein the marker comprises an on-screen marker that is presented at the video display, and wherein the presented on-screen marker comprises a binary pattern that identifies the video content and the time code.

20. The non-transitory computer-readable storage medium of claim 17, wherein the marker further includes an audio marker, and wherein the temporal position is identified based on the audio marker.