Apparatus, System, and Method for Annotation of Media Files with Sensor Data

• US 20130330055A1
• Filed: 02/21/2012
• Published: 12/12/2013
• Est. Priority Date: 02/21/2011
• Status: Active Grant

• Current Assignee: National University of Singapore
• Original Assignee: National University of Singapore
• Inventors: Zhang, Ying, Zimmermann, Roger, Kim, Seon Ho, Arslan Ay, Sakire, Seo, Beomjoo, Shen, Zhijie, Wang, Guanfeng, Hao, Jia
• Granted Patent: US 9,509,968 B2
• Time in Patent Office: Days
• US Class Current: 386/240
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6181 involving transmission via a mobile phone network arrangements for providing broadcast or conference services to substation in data switching networks in combination with wireless systems H04L12/189; wireless uplink channel access H04W74/004
• CPC Class Codes: H04N 21/8193 dedicated tools, e.g. video decoder software or IPMP tool
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 9/79 Processing of colour television signals in connection with recording

Abstract

Embodiments of methods for multimedia annotation with sensor data (referred to herein as Sensor-rich video) includes acquisition, management, storage, indexing, transmission, search, and display of video, images, or sound, that has been recorded in conjunction with additional sensor information (such as, but not limited to, global positioning system information (latitude, longitude, altitude), compass directions, WiFi fingerprints, ambient lighting conditions, etc.). The collection of sensor information is acquired on a continuous basis during recording. For example, the GPS information may be continuously acquired from a corresponding sensor at every second during the recording of a video. Therefore, the acquisition apparatus generates a continuous stream of video frames and a continuous stream of sensor meta-data values. The two streams are correlated in that every video frame is associated with a set of sensor values. Note that the sampling frequency (i.e., the frequency at which sensor values can be measured) is dependent on the type of sensor. For example, a GPS sensor may be sampled at 1-second intervals while a compass sensor may be sampled at 50 millisecond intervals. Video is also sampled at a specific rate, such as 25 or 30 frames per second. Sensor data are associated with each frame. If sensor data has not changed from the previous frame (due to a low sampling rate) then the previously measured data values are used. The resulting combination of a video and a sensor stream is called a sensor-rich video.

Claims

1. A method comprising:

obtaining information at an electronic device, the information comprising;

a location of the electronic device;

a direction that a surface of the device points to;

a level of light that the device is exposed to; and

an acceleration of the device'"'"'s motion; and

recording the information.

2. The method of claim 1, wherein the information is recorded in a video format.

3. The method of claim 1, further comprising associating the information with a time reference.

5. The method of claim 1, wherein the information is continuously obtained at a first rate, and the information is continuously recorded at a second rate.

6. The method of claim 5, further comprising determining whether the first rate matches the second rate.

7. The method of claim 6, further comprising, if the first rate is lower that the second rate, calculating data values for the information when at an information recordation time, no information is obtained.

8. The method of claim 7, wherein the calculation of data values uses information obtained prior to the recordation time and information obtained after the recordation time.

9. The method of claim 1, further comprising:

determining an accuracy of the information at a recordation time; and

recalculating a value for the information, if the information is inaccurate.

10. The method of claim 1, wherein the accuracy of the information at a recordation time is determined by information obtained prior to the recordation time and after the recordation time.

11. The method of claim 1, further comprising identifying target information from the continuously recorded information.

12. The method of claim 11, further comprising extracting the target information from the continuously recorded information.

13. The method of claim 1, further comprising identifying a target object in the continuously recorded information.

14. The method of claim 13, further comprising calculating a visible distance from the electronic device to the target object.

4. The method of claim 4, wherein associating the information with a time reference comprises associating measurement time of the location, direction, level of light, and acceleration information.

15. An electronic device, the electronic device configured to:

continuously obtain information, the information comprising;

a location of the electronic device;

a direction that a surface of the device points to;

a level of light that the device is exposed to; and

an acceleration of the device'"'"'s motion; and

continuously record the information.

16. The electronic device of claim 15, wherein the information is recorded in a video format.

17. The electronic device of claim 15, further configured to associate the information with a time reference.

18. The electronic device of claim 17, wherein associating the information with a time reference comprises associating measurement time of the location, direction, level of light, and acceleration information.

19. The electronic device of claim 15, wherein the information is continuously obtained at a first rate, and the information is continuously recorded at a second rate.

20. The electronic device of claim 19, further configured to determine whether the first rate matches the second rate.

21. The electronic device of claim 20, further configure to calculate data values for the information when at an information recordation time, no information is obtained when the first rate is lower that the second rate.

22. The electronic device of claim 21, wherein the calculation of data values uses information obtained prior to the recordation time and information obtained after the recordation time.

23. The electronic device of claim 15, further configured to:

determine an accuracy of the information at a recordation time; and

recalculate a value for the information when the information is inaccurate.

24. The electronic device of claim 23, wherein the accuracy of the information at a recordation time is determined by information obtained prior to the recordation time and after the recordation time.

25. The electronic device of claim 15, further configured to identify target information from the continuously recorded information.

26. The electronic device of claim 25, further configured to extract the target information from the continuously recorded information.

27. The electronic device of claim 15, further configured to identify a target object in the continuously recorded information.

28. The electronic device of claim 17, further configured to calculate a visible distance from the electronic device to the target object.

29. A system comprising:

a location sensor configure to measure a location of the system;

a direction sensor configured to measure a direction of the system;

a light sensor configured to measure a level of light that the system is exposed to;

an accelerometer configured to measure an acceleration of the system'"'"'s motion;

a recording device configured to continuously record information, the information comprising the location, direction, level of light and acceleration associated to the system.

30. The system of claim 29, wherein the information is recorded in a video format.

31. The system of claim 29, further configured to associate the information with a time reference.

32. The system of claim 31, wherein associating the information with a time reference comprises associating measurement time of the location, direction, level of light, and acceleration information.

33. The system of claim 29, wherein the information is continuously obtained at a first rate, and the information is continuously recorded at a second rate.

34. The system of claim 29, further configured to determine whether the first rate matches the second rate.

35. The system of claim 34, further configured to calculate data values for the information when at an information recordation time, no information is obtained when the first rate is lower that the second rate.

36. The system of claim 35, wherein the calculation of data values uses information obtained prior to the recordation time and information obtained after the recordation time.

37. The system of claim 29, further configured to:

determine an accuracy of the information at a recordation time; and

recalculate a value for the information when the information is inaccurate.

38. The system of claim 37, wherein the accuracy of the information at a recordation time is determined by information obtained prior to the recordation time and after the recordation time.

39. The system of claim 29, further configured to identify target information from the continuously recorded information.

40. The system of claim 39, further configured to extract the target information from the continuously recorded information.

41. The system of claim 29, further configured to identify a target object in the continuously recorded information.

42. The system of claim 41, further configured to calculate a visible distance from the electronic device to the target object.