Overlay video content on a mobile device

• US 8,730,354 B2
• Filed: 07/13/2010
• Issued: 05/20/2014
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey R., Osman, Steven, Zhu, Yunpeng
• Primary Examiner(s): KHAN, USMAN A
• Application Number:
  US12/835,645
• Publication Number:
  US 20120013770A1
• Time in Patent Office: 1,407 Days
• Field of Search: 348/239
• US Class Current: 348/239
• CPC Class Codes: H04N 21/23892 involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Methods, systems, and devices are described for providing overlay content to be placed over captured video content for display on a mobile device. In one embodiment, a system includes a video display, such as a TV, that displays video content. A mobile device with an integrated video camera and display may capture video data which includes the video content from the video display. The location of the video content within the captured video data may be tracked. The mobile device may access overlay content, and display the accessed overlay content over the tracked video content.

Claims

1. A system for augmented video, the system comprising:

a video source configured to provide video content to a video display; and

a mobile device with an integrated video camera and display, the mobile device configured to;

capture video data including the video content from the video display through an image sensor of the integrated video camera, the video content from the video display having at least one marker identifying a temporal position within the video content, the temporal position corresponding to at least a portion of time in the video content;

track the video content in the captured video data;

access overlay content responsive to the temporal position of the video content, the temporal position identified by the at least one marker; and

display the accessed overlay content with the tracked video content.

2. The system of claim 1, wherein the mobile device is configured to display the accessed overlay content over at least a portion of the tracked video content.

3. The system of claim 1, wherein the video source comprises a device selected from the group consisting of a video disc player and a video game console.

4. The system of claim 1, wherein the video content displayed on the video display includes a marker formatted to allow the mobile device to track the video content in the captured video data.

5. The system of claim 1, wherein a physical marker off the video display includes a marker formatted to allow the mobile device to track the video content in the captured video data.

6. The system of claim 1, wherein the overlay content is selected based at least in part on an identification of the video content.

7. The system of claim 6, wherein the mobile device is further configured to identify the video content.

8. The system of claim 1, wherein the video content displayed on the video display includes a marker identifying the video content.

9. The system of claim 1, wherein the mobile device is configured to track the video content by identifying a position and orientation of the video content relative to the mobile device.

10. The system of claim 1, further comprising:

an overlay content datastore, remote from the video display and mobile device and in communication with the mobile device, configured to store the overlay content, wherein the mobile device accesses the overlay content from the overlay content datastore.

11. The system of claim 1, wherein the video source is configured to store the overlay content, and the mobile device accesses the overlay content from the video source.

12. A computer-implemented method for augmented video, the method comprising:

capturing video data including video content from a video display, the video data captured through an image sensor of a video camera integrated into a mobile device, the video content from the video display having at least one marker identifying a temporal position within the video content, the temporal position corresponding to at least a portion of time in the video content;

tracking the video content in the captured video data;

accessing overlay content responsive to the temporal position of the video content, the temporal position identified by the at least one marker; and

displaying the accessed overlay content with the tracked video content.

13. The computer-implemented method of claim 12, wherein the tracking the video content in the captured video data comprises:

locating one or more markers in the video content, the markers formatted to allow the video content in the captured video data to be tracked.

14. The computer-implemented method of claim 12, further comprising:

identifying the video content in the identified video content; and

selecting the overlay content based at least in part on the temporal position in the identified video content.

15. The computer-implemented method of claim 12, wherein,

the video content displayed on the video display includes a marker formatted to allow the video content to be tracked and the video content to be identified;

the marker changes over time to allow a temporal position in the identified video content to be identified; and

the overlay content includes at least one of text, animations, graphic enhancements, advertisements, hyperlinks, and other information.

16. A device for augmenting video content, the device comprising:

a processor module;

a display coupled with the processor module;

a video camera coupled with the processor module, and configured to capture video data through an image sensor, including video content from a video display distinct from the device having at least one marker identifying a temporal position within the video content the temporal position corresponding to at least a portion of time in the video content; and

memory including instructions that, when executed by the processor module, cause the device to;

track the video content in the captured video data;

access overlay content responsive to the temporal position of the video content, the temporal position identified by the at least one marker; and

output the accessed overlay content with the tracked video content on the display.

17. A computer-implemented method for augmented video, the method comprising:

receiving video data captured through an image sensor of a video camera integrated into a mobile device, the video data including video content captured from a video display;

identifying the video content captured from the video display;

identifying a temporal position within the video content, the temporal position corresponding to at least a portion of time in the video content; and

accessing overlay content to be displayed over the video content, the overlay content selected based at least in part on the identified video content and the temporal position.

18. The computer-implemented method of claim 17, further comprising:

tracking the video content in the captured video data, wherein the tracking comprises identifying a relative position and orientation of the video content.