System to augment a visual data stream based on a combination of geographical and visual information

• US 8,953,054 B2
• Filed: 07/11/2013
• Issued: 02/10/2015
• Est. Priority Date: 02/08/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Longsand Limited (Open Text Corporation)
• Inventors: Blanchflower, Sean Mark, Lynch, Michael Richard
• Primary Examiner(s): Berhan, Ahmed A
• Application Number:
  US13/940,069
• Publication Number:
  US 20130307873A1
• Time in Patent Office: 579 Days
• Field of Search: 348/207.99, 348/333.05, 348/211.3
• US Class Current: 348/211.3
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 21/254 Management at additional data server, e.g. shopping server, rights management server arrangements for maintenance or administration in data networks H04L41/00; network services using third party service providers H04L67/53
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4355 involving reformatting operations of additional data, e.g. HTML pages on a television screen optimising the visualization of content for information retrieval from the Internet G06F16/9577; adaptation of message content in packet-switching networks H04L51/066; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

According to an example, a computing device includes a memory on which is stored machine readable instructions that may cause a processor to access a video stream, generate geographical information associated with frames of the video stream, extract features of points of interest in the frames, transmit the extracted features of the points of interest and the geographical information to a server computer that is to use the extracted features of the points of interest and the geographical information to identify augment information, receive the augment information from the server computer, and augment the video stream with the augment information to generate an augmented video stream.

Claims

1. A computing device comprising:

a video camera to capture a video stream;

a display screen to display the captured videos stream;

a processor; and

a memory on which is stored machine readable instructions that are to cause the processor to;

access the video stream captured by the video camera;

extract features of points of interest in frames of the captured video stream;

relate patterns from a series of two or more of the frames to reduce an amount of points of interest in the frames that the server computer is to use in identifying the augment information;

transmit the extracted features of the points of interest and the related patterns to a server computer that is to use the extracted features of the points of interest to identify augment information;

receive the augment information from the server computer;

augment the captured video stream with the augment information to generate an augmented video stream; and

display the augmented video stream on the display screen.

2. The computing device according to claim 1:

wherein the machine readable instructions are further to cause the processor to;

generate geographical information associated with the frames of the video stream; and

transmit the geographical information to the server computer, wherein the server computer is to use the extracted features of the points of interest and the geographical information to identify the augment information.

3. The computing device according to claim 1, wherein the augment information is related to the points of interest in the frames.

4. The computing device according to claim 1, wherein the features of the points of interest comprise geometrical shapes associated with the points of interest and color associated with each of the geometrical shapes, and wherein each of the geometrical shapes is associated with a pattern of X-Y coordinates.

5. The computing device according to claim 4, further comprising:

a direction sensor coupled with the video camera, wherein the video camera is to sense direction information of the video camera; and

wherein the machine readable instructions are further cause the processor to transmit the direction information of the video camera to the server computer.

6. The computing device according to claim 5, wherein the machine readable instructions are further to compress the extracted features of points of interest and the direction information prior to transmission of the extracted features of points of interest and the direction information.

7. The computing device according to claim 1, wherein the augment information comprises at least one of audio information, textual information, and video information.

8. The computing device according to claim 1, wherein the features of the points of interest are associated with at least one of a person and an object included in the frames of the video stream.

9. The computing device according to claim 1, wherein the machine readable instructions are further to cause the processor to generate and detect chirp signals, wherein the chirp signals are used to generate direction augment information, and wherein the direction augment information is to be displayed on the display screen with the video stream.

10. A method for generating augmented video streams, said method comprising:

capturing a video stream with a video camera of a computing device;

displaying the captured video stream on a display screen of the computing device;

extracting features of points of interest in frames of the captured video stream;

relating patterns from a series of two or more of the frames to reduce an amount of points of interest in the frames that the server computer is to use in identifying the augment information;

transmitting the extracted features of the points of interest and the related patterns to a server computer that is to use the extracted features of the points of interest to identify augment information related to the points of interest in the frames;

receiving, in the computing device, the augment information from the server computer;

augmenting, by a processor, the captured video stream with the augment information to generate an augmented video stream; and

displaying the augmented video stream on the display screen.

11. The method according to claim 10, further comprising:

generating geographical information associated with the frames of the captured video; and

wherein transmitting the extracted features further comprises wirelessly transmitting the extracted features and the geographical information to the server computer over a network.

12. The method according to claim 10, said method further comprising:

sensing a direction of the video camera in capturing the video stream; and

transmitting information pertaining to the direction of the video camera to the server computer, wherein the server computer is to further use the information pertaining to the direction of the video camera in identifying the augment information.

13. The method according to claim 10, further comprising:

exchanging chirp signals with another computing device to determine a distance from a location of the processor to the another computing device;

transmitting information related to the determined distance from the location of the processor to the another computing device to the server computer, wherein the server computer is to identify direction augment information from the determined distance;

receiving the direction augment information from the server computer; and

displaying the direction augment information on a display screen with the augmented video stream.

14. A non-transitory computer readable storage medium on which is stored machine readable instructions that when executed by a processor cause the processor to:

capture a video stream with a video camera of a computing device;

display the captured video stream on a display screen of the computing device;

extract features of points of interest in frames of the captured video stream;

relate patterns from a series of two or more of the frames to reduce an amount of points of interest in the frames that the server computer is to use in identifying the augment information;

transmit the extracted features of the points of interest and the related patterns to a server computer that is to use the extracted features of the points of interest to identify augment information related to the points of interest in the frames;

receive, in the computing device, the augment information from the server computer;

augment the captured video stream with the augment information to generate an augmented video stream; and

display the augmented video stream on the display screen.

15. A computer comprising:

a processor; and

a memory on which is stored machine readable instructions that are to cause the processor to;

receive characteristics and geographical information of objects in frames of a video stream accessed in a computing device;

use the characteristics and the geographical information to reduce a set of known objects that potentially match the objects in the frames of the video stream;

recognize the objects from the reduced set of objects that potentially match the objects;

determine augment information associated with the recognized objects; and

transmit the augment information to the computing device, wherein the computing device is to augment the video stream with the augment information.

16. The computer according to claim 15, wherein the machine readable instructions are further to cause the processor to access at least one of an object database and a facial recognition database to reduce the set of known objects that potentially match the objects and recognize the objects from the reduced set of objects.

17. The computer according to claim 15, wherein the computing device is to capture the video stream, and wherein the machine readable instructions are further to cause the processor to receive information pertaining to a direction in which the computing device is facing during capture of the video stream and to use the direction information to reduce the set of known objects that potentially match the objects in the frames of the video stream.

18. The computer according to claim 15, wherein the machine readable instructions are further to cause the processor to:

verify an identity of the computing device;

determine a profile associated with the identity of the computing device in response to the identity of the computing device being verified; and

determine the augment information as augment information that is associated with the recognized objects and based on the determined profile.

19. The computer according to claim 15, wherein the characteristics of the frames comprise features of points of interest in the frames extracted by the computing device.

20. The computer according to claim 15, wherein the characteristics of the frames comprise patterns from a series of two or more of the frames as related by the computing device.

21. A method for determining augment information for a video stream accessed by a computing device, said method comprising:

receiving characteristics and geographical information of objects in frames of the video stream from the computing device;

using the received characteristics and geographical information of the objects in the frames of the video stream to reduce a set of known objects that potentially match the objects in the frames of the video stream;

recognizing the objects from the reduced set of objects that potentially match the objects;

determining, by a processor, augment information associated with the recognized objects; and

transmitting the augment information to the computing device, wherein the computing device is to augment the video stream with the augment information.

22. The method according to claim 21, wherein using the received characteristics and geographical information of the objects to reduce the set of known objects further comprises using the received characteristics and geographical information of the objects to reduce the set of known objects from the known objects stored in at least one of an object database and a facial recognition database.

23. The method according to claim 21, wherein the computing device is to capture the video stream, said method further comprising:

receiving information pertaining to a direction in which the computing device is facing during capture of the video stream; and

using the direction information to reduce the set of known objects that potentially match the objects in the frames of the video stream.

24. The method according to claim 21, further comprising:

verifying an identity of the computing device;

determining a profile associated with the identity of the computing device in response to the identity of the computing device being verified; and

determining the augment information as augment information that is associated with the recognized objects and based on the determined profile.

25. The method according to claim 21, wherein the geographical information of the objects comprises information identifying a geographical location at which the video stream was captured, and wherein using the received characteristics and geographical information of the objects to reduce the set of known objects further comprises using the geographical information of the objects to filter out known objects that do not correspond to the geographical information of the objects.

26. The method according to claim 21, wherein determining the augment information associated with the recognized objects further comprises accessing a database that stores information identifying which of a plurality of augmented information is associated with of a plurality of recognized objects.

27. A non-transitory computer readable storage medium on which is stored machine readable instructions that when executed by a processor cause the processor to:

receive characteristics and geographical information of objects in frames of the video stream from the computing device;

use the received characteristics and geographical information of the objects in the frames of the video stream to reduce a set of known objects that potentially match the objects in the frames of the video stream;

recognize the objects from the reduced set of objects that potentially match the objects;

determine augment information associated with the recognized objects; and

transmit the augment information to the computing device, wherein the computing device is to augment the video stream with the augment information.