Supplemental content on a mobile device

• US 9,832,441 B2
• Filed: 07/20/2012
• Issued: 11/28/2017
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven
• Primary Examiner(s): Flynn, Nathan
• Assistant Examiner(s): Huerta, Alexander Q
• Application Number:
  US13/554,963
• Publication Number:
  US 20130183021A1
• Time in Patent Office: 1,957 Days
• Field of Search: 725 37- 61, 725 62, 725131
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/403 Connection between platform and handheld device
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: G07F 17/3211 Display means
• CPC Class Codes: G07F 17/3216 Construction aspects of a gaming system, e.g. housing, seats, ergonomic aspects
• CPC Class Codes: G07F 17/3223 Architectural aspects of a gaming system, e.g. internal configuration, master/slave, wireless communication
• CPC Class Codes: G07F 17/323 wherein the player is informed, e.g. advertisements, odds, instructions
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 9/87 Regeneration of colour television signals H04N9/80 takes precedence

Abstract

Methods, systems, devices, and software are described for providing supplemental content for presentation on a mobile device that identifies a video and a portion within the video. In one embodiment, a system includes a mobile device with an integrated video camera that tracks a display playing a movie. The mobile device automatically identifies the current scene in the movie and then accesses supplemental non-video content related to the identified scene. The accessed supplemental non-video content (e.g., audio, tactile, olfactory data) is then presented to the user at the same time the movie is played.

Claims

1. A system for augmenting a video, the system comprising:

means of a mobile device, the means for inputting a video segment from a video presentation that includes video segments;

a processor of the mobile device; and

a memory of the mobile device, the memory storing computer-readable instructions that, upon execution by the processor, cause the mobile device to;

identify a time variable marker in the video segment based on a presentation of the video segment, wherein the time variable marker comprises a presented pattern in the presentation of the video segment, wherein;

the presented pattern changes over time,

a presented change to the presented pattern corresponds to a time code, identifies the video presentation, identifies a progression of the video presentation based on the time code, and does not identify supplemental non-video content, and

the progression indicates a temporal position of the video segment relative to another video segment from the video presentation;

determine a match of the video segment to the video presentation based on the time variable marker in the video segment;

select potential supplemental non-video contents based on the match;

determine the temporal position of the video segment from the presented change to the presented pattern;

select the supplemental non-video content from the potential supplemental non-video contents based on the temporal position of the video segment; and

present the supplemental non-video content at substantially a same time as the video segment, wherein a presentation of the supplemental non-video content is time synchronized with the presentation of the video segment based on the presented change to the presented pattern of the time variable marker.

2. The system of claim 1, wherein the means for inputting the video segment comprises a video input port.

3. The system of claim 1, wherein the means for inputting the video segment comprises an integrated video camera configured to capture the video segment from a video display.

4. The system of claim 3, wherein the mobile device is configured to track the video display by identifying a position and orientation of the video segment relative to the mobile device.

5. The system of claim 3, wherein the time variable marker is formatted to allow the mobile device to track the video display by capturing the time variable marker using the integrated video camera.

6. The system of claim 1, wherein the supplemental non-video content is audio content associated with the video segment and the temporal position within the video presentation.

7. The system of claim 1, wherein the supplemental non-video content is tactile content associated with the video segment and the temporal position within the video presentation.

8. The system of claim 1, wherein the supplemental non-video content is determined from a video game console, the video presentation is a video game, and the supplemental non-video content comprises first player-specific information for the video game for a user of the mobile device.

9. The system of claim 8, wherein the video segment corresponds to a presentation of a video game, and further comprising a second mobile device, the second mobile device configured to:

identify at least one marker in the video segment, wherein the at least one marker comprises video game information;

match the video segment to the video presentation based at least in part on the at least one marker in the video segment;

determine a game status of the video game within the video presentation based at least in part on the at least one marker in the video segment;

determine second player-specific information for the video game for a user of the second mobile device based on the game status, the second player-specific information different from the first player-specific information; and

present the determined second player-specific information through the second mobile device at substantially the same time as the video segment.

10. The system of claim 1, wherein the time variable marker is selected from the group consisting of a pattern of dots, binary codes, digital watermarks, and encoded images.

11. The system of claim 1, wherein the mobile device has a stored low-resolution version of the video presentation, and wherein the presentation of the supplemental non-video content is time synchronized with the presentation of the video segment by at least performing image matching based on a hash of the stored low-resolution version of the video presentation.

12. The system of claim 1, wherein the video segment is presented at a remote display, wherein the mobile device is further configured to determine a view angle of the mobile device relative to the remote display, and wherein the potential supplemental non-video contents are available to the mobile device based on the view angle.

13. The system of claim 12, wherein the view angle identifies a user profile of the mobile device, and wherein the potential supplemental non-video contents are customized based on the user profile being identified according to the view angle.

14. The system of claim 1, wherein the video presentation is available from a content source and is presented on a display according to a display rate, and wherein the mobile device is configured to identify at time intervals the presented change to the presented pattern of the time variable marker on the display, wherein the time intervals correspond to predefined time intervals that are different from the display rate and that the content source uses to update the time variable marker in the video presentation on the display.

15. The system of claim 1, wherein the time variable marker is presented at a frequency not perceptible to the human eye, and wherein the time variable marker is identified based on an application of a Fourier transform to the video segment.

16. The system of claim 1, wherein the time variable marker further comprises an audio marker, and wherein the temporal position is identified based on the audio marker.

17. A computer-implemented method for augmenting video, the method comprising:

identifying a time variable marker in a video segment from a scene of a video presentation that includes video segments, wherein the time variable marker comprises a presented pattern in the video segment, wherein;

the presented pattern changes over time,

a presented change to the presented pattern corresponds to a time code, identifies the video presentation, identifies a progression of the video presentation based on the time code, and does not identify supplemental non-video content, and

the progression indicates a temporal position of the video segment;

determining a match of the video segment to the video presentation based on the time variable marker;

selecting potential supplemental non-video contents based on the match;

determining the scene that comprises the video segment from the video presentation based on the temporal position of the video segment;

selecting the supplemental non-video content from the potential supplemental non-video contents based on the scene; and

presenting the supplemental non-video content at substantially a same time as the video segment, wherein a presentation of the supplemental non-video content is time synchronized with the presentation of the video segment based on the presented change to the presented pattern of the time variable marker.

18. A computer-implemented method for augmented video, the method comprising:

transmitting a time variable marker over a first channel of communication comprising video game content configured to be displayed on a video display, the time variable marker tracked by a first mobile device associated with a first player and tracked by a second mobile device associated with a second player, wherein the time variable marker comprises a presented pattern that changes over time and that is set based on a time code corresponding to a progression of the video game content, wherein;

a presented change to the presented pattern corresponds to the time code, identifies the video game content, identifies a progression of the video game content based on the time code, and does not identify player-specific data,

the progression indicates a temporal position of a video game segment of the video game content relative to another video game segment of the video game content, and

the time variable marker is processed by the first mobile device to allow the first mobile device to synchronize with the video game content;

accessing first player-specific video or non-video data from a first datastore in response to the first mobile device indicating that a first tracking by the first mobile device of the time variable marker, wherein the accessed first player-specific data is supplemental to, synchronized with, and associated with the video game content, for the first player based on a time synchronization from the presented change to the presented pattern of the time variable marker;

transmitting over a second channel of communication the accessed first player-specific data to the first mobile device for presentation;

accessing second player-specific video or non-video data from a second datastore in response to the second mobile device indicating that a second tracking by the second mobile device of the time variable marker, wherein the accessed second player-specific data is supplemental to and associated with the video game content, for the second player based on the time synchronization from the presented change to the presented pattern of the time variable marker; and

transmitting over the second channel of communication the accessed second player-specific data to the second mobile device for presentation.

19. The computer-implemented method of claim 18, wherein the first player-specific data is information available to the first player and unavailable to the second player.

20. The computer-implemented method of claim 18, wherein the first and second player-specific data is accessed from a storage medium or a video game console.