System to augment a visual data stream with user-specific content

• US 8,392,450 B2
• Filed: 02/08/2011
• Issued: 03/05/2013
• Est. Priority Date: 02/08/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Autonomy Corporation PLC (HP Inc.)
• Inventors: Blanchflower, Sean Mark, Lynch, Michael Richard
• Primary Examiner(s): MENG, JAU SHYA
• Application Number:
  US13/023,495
• Publication Number:
  US 20120203799A1
• Time in Patent Office: 756 Days
• Field of Search: None
• US Class Current: 707/769
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18

Abstract

User profile information for many users may be stored in a user profile database associated with a server computer connected to a network. Based on information received from a mobile computing device, the server computer selects augment information from an augment information database coupled with the user profile database. The information received from the mobile computing device is related to frames of a video stream being captured by the mobile computing device and geographical information associated with these frames. The augment information is also selected based on the user profile associated with the mobile computing device. The augment information is to be transmitted from the server computer to the mobile computing device using wireless communication and is used by the mobile computing device to overlay the frames of the video stream being captured by the mobile computing device.

Claims

1. A computer-implemented method for generating relevant augment information based on user profiles, the method comprising:

storing user profile information for a plurality of mobile computing devices in a user profile database accessible from a network;

receiving identity information from a first mobile computing device connected to the network using wireless communication, the identity information corresponding to a user profile associated with the first mobile computing device;

decomposing one or more objects contained in a first frame into geometric shapes and identifying other visual distinctive features of the one or more objects to assist in determining points of interest contained within the first frame;

decomposing one or more objects contained in a second frame into geometric shapes and identifying other visual distinctive features of the one or more objects to assist in determining the points of interest contained within the second frame;

relating patterns from a series of frames, including the first and second frames, in a video stream to assist in determining the points of interest contained within each of the series of frames;

receiving characteristics information on the points of interest included in the series of frames of the video stream being captured by the first mobile computing device and geographic information associated with the series of frames of the video stream from the first mobile computing device using wireless communication, where the characteristics information of the points of interest at least include patterns of the geometric shapes and the other distinctive visual features and combining the characteristics information and the geographical information to match to characteristics of objects that are stored in an object database in order to identify the points of interest captured in the video stream;

selecting relevant augment information from an augment information database coupled with the user profile database, wherein the relevant augment information is selected based on the user profile associated with the first mobile computing device; and

using the relevant augment information in the first mobile computing device to enable

     1) overlaying,

     2) highlighting, or

     3) combination, of the points of interest in the series of frames of the video stream with the relevant augment information.

2. The method of claim 1, wherein the identity information from the first mobile computing device is associated with a subscriber identity module (SIM) of the first mobile computing device.

3. The method of claim 1, wherein the user profile information is generated, stored and updated based on how the first mobile computing device is used by a user over time as captured by and received from the first mobile computing device.

4. The method of claim 3, wherein the user profile information comprises personal information of the user.

5. The method of claim 1, wherein the relevant augment information is selected from a plurality of augment information that relate to the characteristics information.

6. The method of claim 5, wherein the characteristics information comprises patterns of the geometric shapes and associated color information as well as any distinctive symbols.

7. The method of claim 6, further comprising:

applying a set of filters to the patterns of the geometric shapes and the associated color information to determine whether a known object can be matched, the known object stored in the object database coupled with the augment information database; and

based on having a match with the known object, identifying the plurality of the augment information associated with the known object.

8. The method of claim 7, wherein the set of filters is applied using distributed processing with two or more server computers.

9. A server computer system to generate relevant augment information based on user profiles, the system comprising:

a user profile database to store user profile information for a plurality of mobile computing devices in a user profile database accessible from a network;

a user profile engine to receive identity information from a first mobile computing device connected to the network using wireless communication, the identity information corresponding to a user profile associated with the first mobile computing device;

an object recognition engine to decompose one or more objects contained in a first frame and a second frame into geometric shapes and identifying other visual distinctive features of the one or more objects to assist in determining points of interest contained within the first and second frame, relate patterns from a series of frames, including the first and second frames, in a video stream to assist in determining the points of interest contained within each of the series of frames, and receive characteristics information on the points of interest included in the series of frames of the video stream being captured by the first mobile computing device and geographic information associated with the series of frames of the video stream from the first mobile computing device using wireless communication, where the characteristics information of the points of interest at least include patterns of the geometric shapes and the other distinctive visual features and combining the characteristics information and the geographical information to match to characteristics of objects that are stored in an object database in order to identify the points of interest captured in the video stream; and

an augment engine to select relevant augment information from an augment information database coupled with the user profile database, wherein the relevant augment information is selected based on the user profile associated with the first mobile computing device,

wherein the relevant augment information in the first mobile computing device is used to enable

     1) overlaying,

     2) highlighting, or

     3) combination, of the points of interest in the series of frames of the video stream with the relevant augment information.

10. The system of claim 9, wherein the identity information from the first mobile computing device is associated with a subscriber identity module (SIM) of the first mobile computing device.

11. The system of claim 9, wherein the user profile information is generated, stored and updated based on how the first mobile computing device is used by a user over time as captured by and received from the first mobile computing device.

12. The system of claim 11, wherein the user profile information comprises personal information of the user.

13. The system of claim 9, wherein the relevant augment information is selected from a plurality of augment information that relate to the characteristics information.

14. The system of claim 13, wherein the characteristics information comprises patterns of the geometric shapes and associated color information as well as any distinctive symbols.

15. The system of claim 14, further comprising:

a set of filters applied to the patterns of the geometric shapes and the associated color information to determine whether a known object can be matched, the known object stored in the object database coupled with the augment information database; and

based on having a match with the known object, identifying the plurality of the augment information associated with the known object.

16. The system of claim 15, wherein the set of filters is applied using distributed processing with two or more server computers.

18. The application of claim 16, wherein the user profile information is generated, stored and updated based on how the first mobile computing device is used by a user over time as captured by and received from the first mobile computing device.

19. The application of claim 16, further comprising:

a routine coded to apply a set of filters to the characteristics information including patterns of the geometric shapes and associated color information to determine whether a known object can be matched, the known object stored in the object database coupled with the augment information database; and

the routine coded to identify the plurality of the augment information associated with the known object based on having a match with the known object.

20. The application of claim 16, wherein the set of filters is applied using distributed processing with two or more server computers.

17. An application coded to operate on a computing system to generate relevant augment information based on user profiles, the application comprising:

instructions to be executed by a processor in the computing system, where any portion of the application coded in software is stored on a non-transitory computer readable medium;

a user profile database to store user profile information for a plurality of mobile computing devices in a user profile database accessible from a network;

a user profile algorithm configured to receive identity information from a first mobile computing device connected to the network using wireless communication, the identity information corresponding to a user profile associated with the first mobile computing device;

an object recognition algorithm to decompose one or more objects contained in a first frame and a second frame into geometric shapes and identifying other visual distinctive features of the one or more objects to assist in determining points of interest contained within the first and second frame, relate patterns from a series of frames, including the first and second frames, in a video stream to assist in determining the points of interest contained within each of the series of frames, and receive characteristics information on the points of interest included in the series of frames of the video stream being captured by the first mobile computing device and geographic information associated with the series of frames of the video stream from the first mobile computing device using wireless communication, where the characteristics information of the points of interest at least include patterns of the geometric shapes and the other distinctive visual features and combining the characteristics information and the geographical information to match to characteristics of objects that are stored in an object database in order to identify the points of interest captured in the video stream; and

an augment algorithm to select relevant augment information from an augment information database coupled with the user profile database, wherein the relevant augment information is selected based on the user profile associated with the first mobile computing device,

wherein the relevant augment information in the first mobile computing device is used to enable

     1) overlaying,

     2) highlighting, or

     3) combination, of the points of interest in the series of frames of the video stream with the relevant augment information.