SYSTEM TO CORRELATE VIDEO DATA AND CONTEXTUAL DATA

• US 20170161382A1
• Filed: 01/28/2016
• Published: 06/08/2017
• Est. Priority Date: 12/08/2015
• Status: Active Application

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Ouimet, Kirk, Zheng, Yantao, Sehn, Timothy Michael, Yan, Rong, Spiegel, Evan
• Application Number:
  US15/008,885
• Publication Number:
  US 20170161382A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

In some embodiments, a method of processing image data may include receiving environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data; processing the environmental data to generate metadata; time stamping the metadata using the capture time data; receiving video data and video time data at a processor; correlating the metadata to the video data using the capture time data and the video time data; receiving a search query; and/or identifying a frame within the video data by performing a search of the metadata using the search criterion.

Claims

1. A method of processing image data, the method comprising:

receiving environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data;

processing the environmental data to generate metadata;

time stamping the metadata using the capture time data;

receiving video data and video time data at a processor, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;

correlating the metadata to the video data using the capture time data and the video time data;

receiving a search query, including a search criterion, at the processor;

identifying a frame within the video data by performing a search of the metadata using the search criterion; and

including the identified frame in a search result.

2. The method of claim 1, wherein the environmental data comprise audio data, and the processing of the environmental data includes identifying a sound in the audio data that is associated with at least one of a place, object and event and generating the metadata based on the at least one place, object or event.

3. The method of claim 2, wherein identifying the sound comprises generating an audio fingerprint using at least a portion of the audio data, and comparing the audio fingerprint to a plurality of stored audio fingerprints.

4. The method of claim 2, wherein identifying the sound comprises filtering the audio data to extract filtered audio data.

5. The method of claim 4, wherein the extracted filtered audio data comprise background audio data and the method includes using the background audio data to identify the at least one place, object or event.

6. The method of claim 1, wherein the environmental data comprise audio data, the method further including processing of the environmental data includes performing voice recognition to identify a word including audio data and generating the metadata to include the word.

7. The method of claim 1, wherein the environmental data comprises location data, the method further comprising processing of the environmental data to identify at least one place, object or event using the location data, and generating the metadata to include the at least one place, object or event.

8. The method of claim 1, including performing image recognition with respect to the frame, and tagging the at least one frame with a tag indicative of an object recognized within the frame, the identifying of the at least one frame performed by matching the search criterion and the tag.

9. The method of claim 1, including performing characteristic recognition with respect to the frame, and tagging the frame with a tag indicative of a characteristic recognized in the frame, the identifying of the frame being performed by matching the search criterion and the tag.

10. The method of claim 1, comprising receiving segments of the video data by a messaging system as part of respective video messages, and combining the segments to constitute the video data.

11. The method of claim 10, wherein the respective video messages are ephemeral messages.

12. A system, the system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, configure the system to;

receive environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data;

process the environmental data to generate metadata;

time stamp the metadata using the capture time data;

receive video data and video time data at a processor, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;

correlate the metadata to the video data using the capture time data and the video time data;

receive a search query, including a search criterion, at the processor;

identify a frame within the video data by performing a search of the metadata using the search criterion; and

include the identified frame in a search result.

13. The system of claim 12, wherein the environmental data comprise audio data, and the processing of the environmental data includes identifying a sound in the audio data that is associated with at least one of a place, object and event and generating the metadata based on the at least one place, object or event.

14. The system of claim 13, wherein identifying the sound comprises generate an audio fingerprint using at least a portion of the audio data, and comparing the audio fingerprint to a plurality of stored audio fingerprints.

15. The system of claim 13, wherein identifying the sound comprises filtering the audio data to extract background audio data and the instructions, when executed by the at least one processor, configure the system to use the background audio data to identify the at least one place, object or event.

16. The system of claim 12, wherein the environmental data comprise audio data, and the instructions, when executed by the at least one processor, further configure the system to identify a word included in the audio data and to generate the metadata to include the word.

17. The system of claim 12, wherein the environmental data comprises location data, and the instructions, when executed by the at least one processor, further configure the system to process the environmental data to identify at least one place, object or event using the location data, and to generate the metadata to include the at least one place, object or event.

18. The system of claim 12, wherein the instructions, when executed by the at least one processor, further configure the system to perform characteristic recognition with respect to the frame, and to tag the frame with a tag indicative of a characteristic recognized in the frame, the identifying of the frame being performed by matching the search criterion and the tag.

19. The system of claim 12, wherein:

the instructions, when executed by the at least one processor, further configure the system to receive segments of the video data by a messaging system as part of respective video messages, and to combine the segments to constitute the video data; and

the respective video messages are ephemeral messages.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

receive environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data;

process the environmental data to generate metadata;

time stamp the metadata using the capture time data;

receive video data and video time data at a processor, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;

correlate the metadata to the video data using the capture time data and the video time data;

receive a search query, including a search criterion, at the processor;

identify a frame within the video data by performing a search of the metadata use the search criterion; and

include the identified frame in a search result.