SYSTEM AND METHOD FOR IMPLEMENTING A PROFICIENCY-DRIVEN FEEDBACK AND IMPROVEMENT PLATFORM
A system and method for implementing a proficiency-driven feedback and improvement platform is disclosed. A particular embodiment includes: establishing a data connection with a learner; generating a distribution of questions to determine the learner'"'"'s level of proficiency on a topic, the topic being composed of a set of codes and counter-codes; determining the learner proficient on the topic after the learner has demonstrated proficiency on each code and counter-code within the topic; and distributing questions to the learner on codes and counter-codes within the topic until the learner has demonstrated proficiency on each code and counter-code within the topic. A particular embodiment further includes: providing the learner feedback on work according to a set of criteria; and ensuring that only users who have first demonstrated proficiency on one or more criteria in the set of criteria can give feedback on those criteria.
- 1-2. -2. (canceled)
- 3. A method for network-enabled proficiency-driven feedback and improvement for learners, the method comprising:
establishing, by use of a data processor and a data network, a data connection with a learner; generating a distribution of questions to determine the learner'"'"'s level of proficiency on a criterion associated with one or more topics; determining the learner proficient on the criterion after the learner has demonstrated proficiency by correctly answering a predetermined threshold of questions presented on the criterion, the learner determined proficient on the criterion being designated a proficient user; and enabling the proficient user to provide, via the data network, feedback to other learners on the criterion for which the proficient user has demonstrated proficiency.
- View Dependent Claims (4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
This patent application is a continuation patent application drawing priority from U.S. non-provisional patent application Ser. No. 15/706,701; filed Sep. 16, 2017. This present non-provisional patent application draws priority from the referenced patent application. The entire disclosure of the referenced patent application is considered part of the disclosure of the present application and is hereby incorporated by reference herein in its entirety.
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings that form a part of this document: Copyright 2016-2019 NoRedlnk Corp., All Rights Reserved.
This patent application relates to computer-implemented software systems, computer-network enabled systems, and user interfaces, according to one embodiment, and more specifically to a system and method for implementing a proficiency-driven feedback and improvement platform.
Helping learners make demonstrated gains in a given subject area has long been constrained by the time, effort, and expertise required to effectively assess learners'"'"' work and help them improve. These factors are among the most significant deterrents to learning.
The example embodiments disclosed herein provide a proficiency-assessment platform that generates a distribution of questions to evaluate a learner'"'"'s level of proficiency on a set of juxtaposing sub-skills (referred to herein as “codes” and “counter-codes”), such that a learner will continue to answer questions on each code and counter-code until the learner has demonstrated proficiency on all of them, in order to most accurately determine the learner'"'"'s level of proficiency on a topic composed of that set of sub-skills.
The example embodiments disclosed herein also provide a proficiency-driven feedback and improvement platform to allow learners to submit work and receive feedback from other learners who have first demonstrated proficiency through the platform on one or all of a set of criteria related to that work.
The various embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which:
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. It will be evident, however, to one of ordinary skill in the art that the various embodiments may be practiced without these specific details.
In various example embodiments, a system and method for implementing a proficiency-driven feedback and improvement platform is disclosed. In various example embodiments, a computer-implemented tool is described to provide such a proficiency-driven feedback and improvement platform. The example embodiments disclosed herein allow learners to submit work into a digital platform, practice concepts designed to help them strengthen a particular set of skills, receive feedback from other learners who have demonstrated proficiency on some or all of those skills and/or feedback from an instructor, give feedback themselves on criteria on which they have demonstrated proficiency, and leverage the feedback they have received to revise and improve their own work.
Referring now to
Referring now to
While the various example embodiments described herein may refer to learners and instructors/qualified experts, it should be understood that the present invention is not limited to any particular learning structure or hierarchy, nor is it limited to any one particular skill. The present invention can be applied with or without any instructor, and it may be applied in any situation where feedback is useful, including without limitation various forms of writing, reading, speaking, mathematics, science, language acquisition or improvement, critical thinking, and performing and creative arts. It may be effectively applied in any context where visual, written, audio, or other forms of feedback can help one improve.
The example embodiments provide a system and method for improving a learner'"'"'s work that is unique. In contrast to other solutions, the example embodiments disclosed herein improve performance by providing a proficiency-driven feedback and improvement system that:
- removes the instructor/qualified expert as the sole source of feedback. In many learning environments, learners only receive one source of feedback from an instructor who may simultaneously have to provide feedback to many other learners, causing the feedback to be rushed and of low quality, the learners to wait for an extended time period receive it, and the resulting performance gains from the feedback to be inadequate
- teaches learners how to analyze work according to a guided set of criteria. The disclosed example embodiments improve learners'"'"' skills as they first practice concepts designed to strengthen their understanding and then further enhance those skills through the act of giving feedback on the criteria on which they have demonstrated proficiency.
- provides learners with a higher level of qualitative feedback. Unlike fully auto-graded solutions, which are more adept at pointing out surface-level errors on learners'"'"' original work, the disclosed embodiments foster in-depth, more valuable feedback to learners.
The disclosed proficiency-driven feedback and improvement platform ensures that learner feedback is accurate and of high quality. In various example embodiments, either or both of two main concepts are involved in this process of proficiency-driven feedback and improvement: 1) a proficiency-assessment process, and 2) a proficiency-driven feedback and improvement process. These processes and their related systems and technical implementation are described in more detail below. It should be understood that these two processes can be used independently or together as appropriate.
In various example embodiments, a system and method for implementing a proficiency-driven feedback and improvement platform are disclosed. In the various example embodiments described herein, a computer-implemented tool or software application (app) implements the proficiency-driven feedback and improvement platform 101. As described in more detail below, a computer or computing system on which the described embodiments can be implemented can include personal computers (PCs), portable computing devices, laptops, tablet computers (e.g., iPad™), personal digital assistants (PDAs), personal communication devices (e.g., cellular telephones, smartphones, or other wireless devices), network computers, set-top boxes, servers, mainframe computers, wearable computing devices, Internet-of-Things (IoT) devices, or any other type of computing, data processing, communication, networking, or electronic system. The network can include any local-area network or wide-area network, such as the Internet.
Referring again to
One or more of the plurality of learner devices 120, the plurality of instructor/qualified expert user devices 130, and/or the plurality of online instructional resources 135 can be grouped together and provided by one or more third party providers operating at various locations in a digital ecosystem. It will be apparent to those of ordinary skill in the art that the plurality of learner devices 120, the plurality of instructor/qualified expert devices 130, and the plurality of online instructional resources 135 can be enabled by any of a variety of networked third party information providers, websites, network-accessible information nodes, or other network resources as described in more detail below. In a particular embodiment, a resource list maintained at the host site 110 can be used as a registry or list of all learner devices 120, instructor/qualified expert devices 130, and online instructional resources 135, which users or the host site 110 may visit/access and from which users or the host site 110 can obtain or submit information content. The host site 110, learner devices 120, instructor/qualified expert devices 130, or online instructional resources 135, and user platforms 140 may communicate and transfer data and information in the data network ecosystem shown in
Networks 115 and 114 are configured to couple one computing device with another computing device. Networks 115 and 114 may be enabled to employ any form of computer readable media for communicating information from one electronic device to another. Network 115 can include the Internet in addition to LAN 114, wide-area networks (WANs), direct connections, such as through a universal serial bus (USB) port, other forms of computer-readable media, or any combination thereof. On an interconnected set of LANs, including those based on differing architectures and protocols, a router and/or gateway device acts as a link between LANs, enabling messages to be sent between computing devices. Also, communication links within LANs typically include twisted wire pair or coaxial cable, while communication links between networks may utilize analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communication links known to those of ordinary skill in the art. Furthermore, remote computers and other related electronic devices can be remotely connected to either LANs or WANs via a wireless link, WiFi, Bluetooth™, satellite, or modem and temporary telephone link.
Networks 115 and 114 may further include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, and the like, to provide an infrastructure-oriented connection. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, and the like. Networks 115 and 114 may also include an autonomous system of terminals, gateways, routers, and the like connected by wireless radio links or wireless transceivers. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of networks 115 and 114 may change rapidly and arbitrarily.
Networks 115 and 114 may further employ a plurality of access technologies including 2nd (2G), 2.5, 3rd (3G), 4th (4G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, and the like. Access technologies such as 2G, 3G, 4G, and future access networks may enable wide-area coverage for mobile devices, such as one or more of client devices 141, with various degrees of mobility. For example, networks 115 and 114 may enable a radio connection through a radio network access such as Global System for Mobile communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), CDMA2000, and the like. Networks 115 and 114 may also be constructed for use with various other wired and wireless communication protocols, including TCP/IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, EDGE, UMTS, GPRS, GSM, UWB, WiFi, WiMax, IEEE 802.11x, and the like. In essence, networks 115 and 114 may include virtually any wired and/or wireless communication mechanisms by which information may travel between one computing device and another computing device, network, and the like. In one embodiment, network 114 may represent a LAN that is configured behind a firewall (not shown), within a business or educational facility data center, for example.
The learner devices 120, instructor/qualified expert devices 130, and/or the online instructional resources 135 may include any of a variety of providers or consumers of digital data. The digital data can be transported in any of a family of file formats and associated mechanisms usable to enable a host site 110 and/or a user platform 140 to receive information from or transfer information to learner devices 120, instructor/qualified expert devices 130, and/or online instructional resources 135 over the network 115. In various embodiments, the file format can be a HyperText Markup Language (HTML) format, a WordPress™ format, a Microsoft™ Word text format, a Microsoft™ Excel spreadsheet format, a CSV (Comma Separated Values) format, or the like; however, the various embodiments are not so limited, and other file formats and transport protocols may be used. For example, data formats other than Excel or CSV or formats other than open/standard formats can be supported by various embodiments. Any electronic file format, such as Microsoft™ Access Database Format (MDB), Portable Document Format (PDF), audio (e.g., Motion Picture Experts Group Audio Layer 3-MP3, and the like), video (e.g., MP4, and the like), and any proprietary interchange format defined by specific sites can be supported by the various embodiments described herein. Moreover, learner devices 120, instructor/qualified expert devices 130, and/or the online instructional resources 135 may provide or consume a variety of different data sets.
The client devices of user platform 140 may also include at least one client application that is configured to receive information content and/or control data from another computing device via a wired or wireless network transmission. The client application may include a capability to provide and receive textual data, graphical data, video data, audio data, and the like. Moreover, client devices of user platform 140 may be further configured to communicate and/or receive a message, such as through a Short Message Service (SMS), direct messaging (e.g., Twitter™), email, Multimedia Message Service (MMS), instant messaging (IM), internet relay chat (IRC), mIRC, Jabber, Enhanced Messaging Service (EMS), text messaging, Smart Messaging, Over the Air (OTA) messaging, or the like, between another computing device, and the like.
Referring again to
Referring again to
The proficiency-driven feedback and improvement system 200 may include a proficiency-assessment module 210. The proficiency-assessment module 210 can be configured to perform the processing as described herein. Initially, the proficiency-assessment module 210 can be configured to establish a data connection with at least one of the plurality of learner devices 120, at least one of the plurality of instructor/qualified expert devices 130, and optionally one or more of the plurality of online instructional resources 135. The proficiency-assessment module 210 can be configured to generate and display on a user interface of a user platform 140 information content and at least one user-selectable input object or icon associated with available user-selectable option elements of a pre-defined set of user-selectable option elements as described in more detail below.
The proficiency-assessment module 210 can be further configured to gather and process information pertaining to each learner. The learner information is processed to assess each learner'"'"'s level of understanding. In various example embodiments, a proficiency-assessment process uses adaptive technology to gauge each learner'"'"'s understanding of concepts as he or she works toward proficiency. This process is unique in its ability to accurately assess learners'"'"' proficiency of intertwined and related skills.
Referring now to
Referring now to
Referring now to
- the relationship 410 between juxtaposing skills, referred to here as “codes” and their “counter-codes,” within the topic. These sets of codes and counter-codes 410 may be assessed together to ensure that learners do not falsely extrapolate patterns.
- the necessary threshold 412 for demonstrating proficiency of each skill. The threshold for demonstrating proficiency 412 may vary based on the difficulty and complexity of each skill.
- the relative frequency 414 to assess each skill. This relative frequency ratio 414 is also intended to mirror the desired emphasis in the topic.
- the difficulty of all questions available on the site. The proficiency-assessment module 210 can calculate the difficulty of questions based on a collection (e.g., billions) of prior attempts from learners on answering the questions.
In various example embodiments, the proficiency-assessment module 210 may use the inputs described above to generate a customized set of questions for each learner on a topic. The questions can be presented to each learner by the proficiency-assessment module 210 via the user interface of a user platform 140 as described above. The customized set of questions may be generated and presented for each learner such that each learner receives a mix of questions from codes and their counter-codes. The questions presented may also vary in difficulty depending on learner performance, such that learners see easier questions after making errors and harder questions after answering correctly. Learners may ultimately interact with progressively more difficult questions on each code or counter-code until they have demonstrated a desired level of proficiency on each one. The learner may continue to work on the topic until he or she has demonstrated proficiency on all codes and counter-codes within the topic.
In various example embodiments, the proficiency-assessment module 210 ensures that:
- learners are not “finished” until they have shown understanding of all necessary codes and their counter-codes.
- learners see gradually more difficult questions as they answer correctly, and are guided toward slightly easier questions when they are struggling.
- learners see questions from all codes in the topic in randomized fashion so that learners cannot “hack” their way to completion (e.g., by answering all questions on one code, where the answer always follows a particular pattern, and then answering all questions on another code, where the answer always follows the opposite pattern). In various example embodiments, the proficiency-assessment module 210 can oscillate between codes, emphasizing codes on which the learner has not demonstrated proficiency (while also assessing any counter-codes, even if the learner has demonstrated proficiency on those), and still ensuring that there is no predictable pattern to the questions being displayed.
- as learners gain proficiency on individual codes in a topic, they still see questions from a given code until they have demonstrated proficiency on that code and all its counter-codes. In a given topic, if there are two sets of codes and counter-codes, one called set A and one called set B (see
FIG. 6for reference), learners will continue to see questions from all codes and counter-codes in Set A until learners demonstrate proficiency on all codes and counter-codes in Set A.
As such, the proficiency-assessment module 210 uses each topic'"'"'s codes and counter-codes to generate customized sets of questions presented to learners via the user interface for assessing the skills and level of proficiency of each learner on one or more topics of instruction.
Referring now to
Referring now to
Referring again to
In various example embodiments, through the proficiency-driven feedback and improvement system 200, learners are guided through the process of submitting and/or editing work, practicing concepts that help them strengthen a particular skill or set of skills, evaluating the work of other learners (e.g., submitting input on a binary, ternary, or Likert scale, annotating or highlighting work, and submitting free-form comments), receiving feedback from other learners, and/or revising their own work as necessary. As shown in
- enables learners to submit work or update and submit their work (process block 750).
- enables learners to practice skills through a series of exercises on relevant concepts. The exercises may be specifically configured for the skill level and/or prior performance of each learner. While learners practice, the platform assesses each learner'"'"'s level of proficiency (process block 752).
- enables learners to evaluate the submissions of other learners using a guided set of criteria. The platform may only allow learners to evaluate work on the concepts on which they have proven proficient (process block 754).
- enables learners to review feedback from other learners and to submit revisions to their prior work (process block 756).
As shown in
Referring again to
In many example embodiments, a proficiency-driven feedback and improvement process may also be used as part of the proficiency-driven feedback and improvement system 200. The proficiency-driven feedback module 230, provided for this purpose, may be configured to determine and enforce a policy wherein a learner can only evaluate or provide feedback on those criteria on which he or she has demonstrated proficiency, helping to ensure that learners receive accurate and high-quality feedback. Proficiency can be evaluated using a variety of different techniques and processes as described herein.
In some embodiments of the process implemented by proficiency-driven feedback module 230, learners evaluate their own work using the same criteria that they would use to evaluate the work of other learners. These embodiments may be configured such that learners must evaluate their own work before the proficiency-driven feedback module 230 reveals feedback on their work from other learners. Some example embodiments of the proficiency-driven feedback and improvement system 200 maintain a visually consistent experience and/or utilize similar language across the practice and evaluation phases, maximizing the transference of skills from practice to evaluation of other learners to self-evaluation.
As configured and directed by the user via a computer-generated user interface as described herein, various example embodiments may automatically: 1) present and enable selection and configuration of a plurality of criteria; 2) present and enable configuration and selection of a plurality of prompt options or other guiding questions; 3) manage a variety of other operations related to the generation, configuration, and execution of a system and method for implementing a computer proficiency-driven feedback and improvement platform. In various example embodiments, the computer-generated user interface can be illustrated in several screen snapshots as illustrated herein and described below.
It will be apparent to those of ordinary skill in the art in view of the disclosure herein that the web app implementation of the example user interfaces shown in
Referring now to
Referring now to
The example mobile computing and/or communication system 700 includes a data processor 702 (e.g., a System-on-a-Chip (SoC), general processing core, graphics core, and optionally other processing logic) and a memory 704, which can communicate with each other via a bus or other data transfer system 706. The mobile computing and/or communication system 700 may further include various input/output (I/O) devices and/or interfaces 710, such as a touchscreen display, an audio jack, and optionally a network interface 712. In an example embodiment, the network interface 712 can include one or more radio transceivers configured for compatibility with any one or more standard wireless and/or cellular protocols or access technologies (e.g., 2nd (2G), 2.5, 3rd (3G), 4th (4G) generation, and future generation radio access for cellular systems, Global System for Mobile communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), LTE, CDMA2000, WLAN, Wireless Router (WR) mesh, and the like). Network interface 712 may also be configured for use with various other wired and/or wireless communication protocols, including TCP/IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, Bluetooth™, IEEE 802.11x, and the like. In essence, network interface 712 may include or support virtually any wired and/or wireless communication mechanisms by which information may travel between the mobile computing and/or communication system 700 and another computing or communication system via network 714.
The memory 704 can represent a machine-readable medium on which is stored one or more sets of instructions, software, firmware, or other processing logic (e.g., logic 708) embodying any one or more of the methodologies or functions described and/or claimed herein. The logic 708, or a portion thereof, may also reside, completely or at least partially within the processor 702 during execution thereof by the mobile computing and/or communication system 700. As such, the memory 704 and the processor 702 may also constitute machine-readable media. The logic 708, or a portion thereof, may also be configured as processing logic or logic, at least a portion of which is partially implemented in hardware. The logic 708, or a portion thereof, may further be transmitted or received over a network 714 via the network interface 712. While the machine-readable medium of an example embodiment can be a single medium, the term “machine-readable medium” should be taken to include a single non-transitory medium or multiple non-transitory media (e.g., a centralized or distributed database, and/or associated caches and computing systems) that stores the one or more sets of instructions. The term “machine-readable medium” can also be taken to include any non-transitory medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the various embodiments, or that is capable of storing, encoding or carrying data structures utilized by or associated with such a set of instructions. The term “machine-readable medium” can accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.
As described herein for various example embodiments, a system and method for implementing a proficiency-driven feedback and improvement platform are disclosed. In various example embodiments, a software application program is used to enable the development and presentation of information categories, information sub-categories, and information components in customizable views on the display screen of a computing or communication system, including mobile devices. As described above, in a variety of contexts, the proficiency-driven feedback and improvement system 200 of an example embodiment can be configured to automatically obtain a variety of information from one or more 3rd party sites or network resources via a data network to facilitate the user experience of searching for desired information datasets, configuring a proficiency-driven feedback and improvement platform, and sharing information, all from the convenience of a computing device or a portable electronic device, such as a smartphone. This collection of particular user-selected information datasets and customized views has traditionally been possible only via multiple, personal interactions with a plurality of different parties at different locations. The embodiments as presently disclosed and claimed enable these disparate transactions to be integrated into a single set of electronic interactions with a mobile device or other computing device. As such, the various embodiments as described herein are necessarily rooted in computer and network technology and serve to improve these technologies when applied in the manner as presently claimed. In particular, the various embodiments described herein improve the use of computing device technology and mobile device technology in combination with data network technology in the context of a proficiency-driven feedback and improvement platform via electronic means.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.