Methods and systems for generating and joining shared experience

• US 8,493,353 B2
• Filed: 04/13/2011
• Issued: 07/23/2013
• Est. Priority Date: 04/13/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Longsand Limited (Open Text Corporation)
• Inventors: Blanchflower, Sean Mark, Hayhurst, Simon
• Primary Examiner(s): Ng, Fan
• Application Number:
  US13/086,358
• Publication Number:
  US 20120263154A1
• Time in Patent Office: 832 Days
• Field of Search: 345/173, 348/333.01
• US Class Current: 345/173
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A method for implementing shared experiences using mobile computing devices comprises capturing an image of a trigger along with background information associated with the trigger in a video frame using a video camera built into a first mobile computing device equipped with a display screen and a wireless communication module; identifying characteristics information of the trigger and the associated background information; transmitting the characteristics information of the trigger and the associated background information to a server computing device connected to a network using wireless communication; receiving an augmented reality corresponding to the characteristics information of the trigger and the associated background information from the server computing device; and presenting the augmented reality on the display screen of the first mobile computing device, wherein the augmented reality associated with the first mobile computing device is shared and time synchronized with an augmented reality previously transmitted by the server computing device to a second mobile computing device.

Claims

1. A system to enable sharing an augmented reality, the system comprising:

a trigger server configured to receive characteristics information from a first and second mobile computing devices using wireless communication, the first and second mobile computing devices located in close proximity to one another, the trigger server further configured to generate signature information based on the characteristics information received from the first and second mobile computing devices, the characteristics information associated with a trigger, each of the first and second mobile computing devices configured with a built-in video camera to capture an image of the trigger, the trigger server coupled with a trigger database configured to store information about a plurality of triggers; and

an augmented reality server coupled with the trigger server and configured to use the signature information to identify an augmented reality, the augmented reality server coupled with an augmented reality database configured to store information about a plurality of augmented realities, wherein the same identified augmented reality is to be transmitted to the first mobile computing device and the second mobile computing device, and wherein a user of the first mobile computing device shares the identified augmented reality with a user of the second mobile computing device by viewing the identified augmented reality on a display screen of the first mobile computing device while the second user views the identified augmented reality on a display screen of the second mobile computing device, wherein the augmented reality server and the trigger server in the system are implemented in logic blocks of electronic hardware, software coding, and any combination of the two, where portions implemented in software coding are stored in an executable format by a processor on one or more non-transitory machine-readable mediums.

2. The system of claim 1, wherein each of the first and second mobile computing devices is equipped with a global position system (GPS) device, and wherein GPS information from the first and second mobile computing device is used to determine whether they are located in close proximity to one another.

3. The system of claim 1, wherein the identified augmented reality as viewed on the display screen of the first mobile computing device is time synchronized with the identified augmented reality as viewed on the display screen of the second mobile computing device.

4. The system of claim 1, wherein the identified augmented reality is displayed within a bounding area of the display screen of the first or the second mobile computing device, the bounding area defined by an area that bounds the trigger.

5. The system of claim 1, wherein the trigger includes one or more trigger objects and associated background information,

wherein the trigger server is configured to present the information about the plurality of triggers and the information about the plurality of augmented realities to the first mobile computing device,

wherein the trigger server is configured to determine whether the signature information generated based on the characteristics information received from the first mobile computing device is a new signature or a known signature, and to store the signature information when it is a new signature,

wherein the augmented reality server is configured to associate the new signature with an existing augmented reality as stored in the augmented reality database, and

wherein the augmented reality database is configured to store information generated using markerless tracking, and wherein information received by the trigger server from the first and second mobile computing devices includes orientation information.

6. A computer-generated method for generating a shared experience of augmented reality on a display of each participants mobile computing device, the method comprising:

identifying visual characteristics information of trigger objects and background scene around the trigger objects captured in the frames of a video stream being recorded on a mobile computing device with a built in video camera application, video camera lens and microphone, and display panel;

transmitting the characteristics information of the trigger objects and the background, as well as geographical information of the location of the trigger objects over to a server connected to a network using wireless communication;

generating a visual signature composed of the visual characteristics information of the trigger objects and background;

presenting one or augmented reality scenarios on a display a user of a first mobile computing device may select to become a part of;

receiving a selection of an augmented reality scenarios from the user of the first mobile computing device;

sending the augmented reality scenario by the server to overlay the scenario over the trigger objects and background included in the frames of the video stream;

receiving visual characteristics information from the video stream of a second mobile device matching the visual signature; and

enabling the augmented video stream with the augmented reality scenario to be viewable on a first display screen of the first mobile computing device and a second display screen of the second mobile computing device based on both mobile devices wirelessly sending up the same visual signature to a server.

7. The computer generated method of claim 6, where the augmented reality scenario is time synchronized on both the first and second display to be showing the same augmented reality scenario at the period of time within that augmented reality scenario on both the first and second display.

8. The computer generated method of claim 6, where the client mobile computing device, includes:

a processor, a memory, a built in battery to power the mobile computing device, built-in video camera and display screen for the mobile computing device, and built-in Wi-Fi circuitry to wirelessly communicate with a server computer connected to network;

a video capturing module coupled with the processor and configured to capture a video stream, where the video capturing module is resident on the mobile device and part of an application that offers a sub-program to enter an augmented reality experience, once the user of the mobile device activates both the application that offers the sub-program to enter an augmented reality experience, the video module captures and sends the visual signature to the server to compare the visual signature with any other instances of visual signatures stored in a temporary database of the current unique visual signatures and then when a match exists the server broadcasts the augmented reality scenario to the all of the mobile devices that sent up the visual signature and the augmented reality scenario is time synchronized to be in sync with all of the mobile devices displaying that augmented reality scenario on the display of that mobile device; and

a global positioning system (GPS) module coupled with the video capturing module and configured to generate geographical information associated with frames of the video stream to be captured by the video capturing module;

where the GPS location is used to reduce the amount of trigger items to compare against when comparing the transmitted visual characteristics;

a video processing module coupled with the video capturing module and configured to analyze the frames of the video stream and extract features of the trigger items and background included in the frames, the video processing module further configured to cause transmission of the features of the points of interest and the geographical information to the server computer and to receive of augment information from the server computer, where the transmitted visual characteristics include the characteristics of the trigger items themselves including color, shape(s), patterns, and other visual features, the orientation relative to each other and distance between multiple trigger items, potentially a bounding box image of the multiple trigger objects and background; and

wherein the video processing module is configured to overlay the trigger objects in the frames of the video stream with the augment reality scenario to generate an augmented video stream displayed on each display of the two or more mobile devices.

9. The computer generated method of claim 8, where once a user 1) activates an icon or other object within the augmented reality scenario, or 2) performs a pre-configured anticipated implicit behavior that the application resident on the mobile device is detecting for, then the application can capture that activation and transmit that action back up to the server so that the augmented reality scenario being broadcast no longer merely overlays objects being captured in the video stream, the content of the augmented reality scenario being broadcast and displayed on the display of the mobile device substitutes and replaces the video stream being displayed on each display of the two or more mobile devices, where the display module coupled with the video processing module is configured to display the augmented video stream, and

where a first comparator on the server compares incoming trigger objects to the database of known trigger items, where an amount of known trigger items to compare to when analyzing the visual signature may be constrained by 1) category, 2) type, 3) geographic location and any combination of the three, and

where a second comparator on the server compares an incoming visual signature from a given second mobile device to the visual signatures stored in the temporary database of visual signatures and checks when a match of visual signatures occurs, and then when a match does occur, the server broadcasts the same augmented reality scenario being already broadcast to the first mobile device, synchronized in time reference within the augmented reality scenario, to the all of the mobile devices with that matching visual signature so that all the users of the mobile devices are sharing the same augmented reality experience.

10. A computer-implemented method for sharing an augmented reality on display screens of mobile computing devices, the method comprising:

capturing an image of a trigger along with background information associated with the trigger in a video frame using a video camera built into a first mobile computing device equipped with a display screen and a wireless communication module;

identifying characteristics information of the trigger and the associated background information;

transmitting the characteristics information of the trigger and the associated background information to a server computing device connected to a network using wireless communication;

receiving an augmented reality corresponding to the characteristics information of the trigger and the associated background information from the server computing device; and

presenting the augmented reality on the display screen of the first mobile computing device, wherein the augmented reality associated with the first mobile computing device is shared and time synchronized with an augmented reality previously transmitted by the server computing device to a second mobile computing device.

11. The method of claim 10, wherein the server computing device is configured to generate signature information based on the characteristics information of the trigger and the associated background information, and wherein the server computing device is further configured to identify the augmented reality associated with the signature information.

12. The method of claim 11, wherein the augmented reality is presented by overlaying the augmented reality onto the portion of the display screen of the first mobile computing, the portion of the display screen defined by a bounding area that bounds the trigger.

13. The method of claim 12, wherein, prior to the first mobile computing device, the second mobile computing device is configured to transmit to the server computing device the characteristic information of the trigger and the associated background information as captured by a video camera built into the second mobile computing device.

14. The method of claim 13, wherein the server computing device is configured to generate signature information based on the characteristics information and the associated background information transmitted by the second mobile computing device, to determine whether the signature information associated with the first mobile computing device is similar to the signature information associated with the second mobile computing device, and based on the two signature information being similar, to transmit the same augmented reality to the first mobile computing device as previously transmitted to the second mobile computing device.

15. The method of claim 14, wherein the server computing device is configured to transmit information about existing triggers to one or more of the first and second mobile computing devices.

16. The method of claim 15, wherein the server computing device is configured to transmit information about existing augmented realities to one or more of the first and second mobile computing devices.

17. The method of claim 16, wherein the information about the existing augmented realities is categorized.

18. A non-transitory computer-readable media that stores instructions, which when executed by a machine, cause the machine to perform operations comprising:

transmitting characteristics information of a trigger to a server computing device connected to a network using wireless communication;

receiving an augmented reality corresponding to the characteristics information of the trigger from the server computing device; and

enabling the augmented reality to be displayed within a bounding area of a display screen of a first mobile computing device, wherein the augmented reality associated with the first mobile computing device is shared and time synchronized with an augmented reality previously transmitted by the server computing device to a second mobile computing device, the second mobile computing device having previously transmitted characteristic information of the same trigger to the server computing device, wherein the first and second mobile computing device are located within a close proximity of one another.

19. The non-transitory computer-readable media of claim 18, further comprising:

receiving information about known triggers and corresponding augmented reality information from the server computing device; and

enabling an image of the trigger to be captured in a frame of a video stream using a video camera of the first mobile computing device, wherein the characteristics information of the trigger is generated by analyzing the image of the trigger.

20. The non-transitory computer-readable media of claim 19, wherein the trigger includes one or more trigger objects and wherein the bounding area is defined as an area that bounds the one or more trigger objects.