System to augment a visual data stream based on a combination of geographical and visual information

• US 8,488,011 B2
• Filed: 02/08/2011
• Issued: 07/16/2013
• Est. Priority Date: 02/08/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Longsand Limited (Open Text Corporation)
• Inventors: Blanchflower, Sean Mark, Lynch, Michael Richard
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Berhan, Ahmed A
• Application Number:
  US13/023,463
• Publication Number:
  US 20120200743A1
• Time in Patent Office: 889 Days
• Field of Search: 348/239, 348/211.11, 348/211.12, 348/211.99, 348/211.1, 348/159, 348/333.02, 348/333.05, 348/211.2, 348/211.3, 345/633
• US Class Current: 348/211.3
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 21/254 Management at additional data server, e.g. shopping server, rights management server arrangements for maintenance or administration in data networks H04L41/00; network services using third party service providers H04L67/53
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4355 involving reformatting operations of additional data, e.g. HTML pages on a television screen optimising the visualization of content for information retrieval from the Internet G06F16/9577; adaptation of message content in packet-switching networks H04L51/066; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

An mobile computing device includes a video capturing module to capture a video stream, a global positioning system (GPS) module to generate geographical information associated with frames of the video stream to be captured by the video capturing device, and a video processing module to analyze the frames of the video stream and extract features of points of interest included in the frames. The video processing is configured to transmit the features of the points of interest and the geographical information to a server and to receive augment information from the server computer using wireless communication. The video processing module uses the augment information to overlay the frames of the video stream to generate an augmented video stream.

Claims

1. A client mobile computing system, comprising:

a processor, a memory, a built in battery to power the mobile computing device, built-in video camera, a video capturing module, a global positioning system (GPS) module, a video processing module, a display screen for the mobile computing device, and built-in Wi-Fi circuitry to wirelessly communicate with a server computer connected to network;

where the video capturing module is coupled with the processor and configured to capture a video stream;

where the GPS module is coupled with the video capturing module and configured to generate geographical information associated with frames of the video stream to be captured by the video capturing module;

where the video processing module is coupled with the video capturing module and configured to analyze the frames of the video stream and extract features of points of interest included in the frames, where the video processing module is further configured to combine the features of the points of interest and the geographical information as well as relate patterns from a series of two or more frames in the video stream to reduce an amount of possible points of interest in order to identify what are the points of interest captured in the video stream, wherein the video processing module is configured to

     1) overlay,

     2) highlight, or

     3) combination of both the points of interests in the frames of the video stream with the augment information to generate an augmented video stream; and

a display module coupled with the video processing module and configured to display the augmented video stream on a display screen, where the video processing module is configured to detect whether a user has chosen to activate a highlighted point of interest to view the augment information associated with the augmented video stream being displayed on the display screen of the client mobile computing system.

2. The system of claim 1, wherein the features of the points of interest comprises geometrical shapes associated with the points of interest and color associated with each of the geometrical shapes, and wherein each of the geometrical shapes is associated with a pattern of X-Y coordinates.

3. The system of claim 2, further comprising a direction sensor coupled with the video camera and configured to sense direction information of the video camera, wherein the video processing module is further configured to cause transmission of the direction information of the camera to the server computer, wherein the features of the points of interest are transmitted as texts.

4. The system of claim 3, wherein the video processing module is further configured to perform compression operations prior to causing the transmission of the features of the points of interest, the geographical information, and the direction information of the video camera to the server computer.

5. The system of claim 4 wherein the compression operations use DivX compression technology.

6. The system of claim 5, wherein the video processing module is further configured to select the server computer among multiple server computers based on the location and workload of the server computer.

7. The system of claim 6, wherein the augment information comprises at least one of audio information, textual information, and video information.

8. The system of claim 6, wherein the features of the points of interest are associated with a person or an object included in the frames of the video stream.

9. The system of claim 1, further comprising an audio processing module coupled with the video processing module and configured to generate and detect chirp signals, and wherein the chirp signals are used to generate direction augment information to direct a first user toward a position of a second user based on a chirp signal associated with a mobile computing device of the second user.

10. A computer-implemented method for generating augmented video streams, the method comprising:

relating patterns from a series of two or more frames in a video stream with a module in a mobile computing device to reduce an amount of possible points of interest in order to identify what are the points of interest captured in the video stream;

identifying characteristics information of points of interest included in the series of two or more frames of a video stream being captured;

transmitting the characteristics information and geographical information associated with the frames to a server computer connected to a network using wireless communication;

receiving augment information from the server computer, the augment information related to the points of interest included in the frames of the video stream;

overlaying the augment information onto the frames of the video stream to generate an augmented video stream; and

enabling the augmented video stream to be viewable on a display screen and displaying the augmented information when activated by a user of the mobile computing device.

11. The method of claim 10, wherein the characteristics information is transmitted as texts.

12. The method of claim 10, further comprising performing compression using DivX compression technology prior to transmitting the characteristics information to the server computer.

13. The method of claim 10, wherein the characteristics information of the points of interest includes geometrical shapes and associated color information, and wherein each of the geometrical shapes is associated with a pattern of X-Y coordinates.

14. The method of claim 10 wherein the server computer is selected from a group of servers in the network, and wherein server selection criteria include server proximity and server workload.

15. The method of claim 10, further comprising transmitting video camera direction information to the server computer, wherein the video camera direction information is associated with a video camera used to capture the video stream, and wherein overlaying the augment information onto the frames of the video stream comprises outputting an audio portion of the augment information to a speaker.

16. The method of claim 10, further comprising:

exchanging chirp signals with another mobile computing device to determine a distance from a location of a current mobile computing device to the other mobile computing device;

receiving direction augment information from the server computer; and

presenting the direction augment information on the display screen of the current mobile computing device, wherein the direction augment information overlays a video stream being played on the display screen.

17. A non-transitory computer-readable media that stores instructions, which when executed by a machine, cause the machine to perform operations comprising:

detecting that a video stream is being captured by a video camera;

identifying features of objects included a central area of frames of the video stream;

generating geometrical shapes of the features of the objects and patterns associated with each of the geometrical shapes;

combining the patterns of the geometrical shapes, color of each geometrical shape, and geographical information associated with the frames as well as relating patterns from a series of two or more frames in the video stream with a module in a mobile computing device to reduce an amount of possible points of interest in order to identify what are the points of interest captured in the video stream;

receiving augment information, the augment information related to the objects included in the frames of the video stream; and

overlaying the augment information onto the frames of the video stream, where a user of the mobile computing device activates a point of interest to cause the augment information to be displayed on the mobile computing device.

18. The computer-readable medium of claim 17, further comprising:

using a set of filters on the objects included in the frames of the video stream to eliminate non-centered objects included in the frames as possible points of interest unless those objects are contained within a series of two or more consecutive frames.

19. The computer-readable medium of claim 17, wherein the features of the objects are associated with a person, and wherein the augment information received from the server computer comprises information about the person.

20. The computer-readable medium of claim 19, further comprising exchanging chirp signals with another mobile computing device to determine distance from a current mobile device to the other mobile device, and wherein based on said exchanging of the chirp signals the augment information includes direction augment information.