Connections: The Quarterly Journal

Volume XII, Number 1, Winter 2012

Mobile Learning in Security and Defense Organizations

Christian Glahn

Research Ethics in the Mobile Learning Environment (MoLE) m-Learning Project

Jacob Hodges and Geoff Stead

Supporting Crisis Simulations with the ARLearn Toolkit for Mobile Serious Games

Stefaan Ternier, Atish Gonsalves, Bernardo Tabuenca, Fred de Vries, and Marcus Specht

Supporting Learner Mobility in SCORM-Compliant Learning Environments with ISN Mobler Cards

Christian Glahn

Mobile Learning and ADL’s Experience API

Kristy Murray, Peter Berking, Jason Haag, and Nikolaus Hruska

The Role of Teaching History for a Nation-Building Process in a Post-Conflict Society: The Case of Macedonia

Biljana Popovska

GAO Report on Higher Education

Use of New Data Could Help Improve Oversight of Distance Education

Mobile Learning in Security and Defense Organizations

Christian Glahn [*]

Technology has been part of education and training in security and defense organizations for a long time. E-learning and advanced distributed learning (ADL) have helped to standardize, optimize, and scale education and training. Many organizations and affiliated institutions already make good use of Web-based technologies to provide training for performance support and career development. Over the past years, ADL solutions have become part of the standard procedures of many organizations. The Partnership for Peace Consortium (PfPC) plays an important role in facilitating the exchange of knowledge and experiences among ADL practitioners in defense academies. The ADL Working Group of the PfPC is home to a strong and active community that brings new technologies into the practice of education and training in security and defense organizations. In the past, the primary focus of ADL activities was creating and enhancing interoperable Web-based training modules by promoting the adoption of the Scalable Content Object Reference Model (SCORM). The joint learning management system of the PfPC that is hosted by the International Relations and Security Network (ISN) based in Zurich has become a hub for Web-based training resources that are shared and used by the entire PfPC ADL community.

The majority of Web-based ADL systems are optimized for desktop computing, reflecting the infrastructure that is most commonly available to learners and trainers in these organizations. However, with the sophistication and widespread availability of the current generation of smartphones, mobile technologies have rapidly gained relevance on a large scale. The “mobile revolution” refers to a number of technologies that support a wider range of interaction modes beyond the “keyboard–mouse–screen” interactions that are familiar to users of desktop computers. These interaction modes include support for the “responsive” arrangement of information for a wide range of screen sizes, gesture-based touch interactions, and location-based services that go beyond active manipulation of information on a personal screen.[1] These new forms of interaction are used for information filtering, but are not limited to that function. Like much other legacy information and communications technology, present ADL systems are not designed to support these new forms of interaction. Moreover, the related educational resources often appear to be unsuited for delivery to different platforms, or are not positioned to benefit from these new ways of creating, using, and sharing information. This raises concerns and doubts among stakeholders about the needs and benefits for supporting mobile technologies for education and training in security and defense.

In response to these socio-technological developments, a few mobile learning flagship projects have been launched in the ADL community. Until recently, these activities were only loosely connected. In order to connect the different activities and projects, the ISN Zurich organized a workshop on mobile learning in security and defense organizations in conjunction with the Eleventh World Conference on Mobile and Contextual Learning, held in Helsinki in October 2012.[2] The workshop brought together researchers from these flagship projects to discuss new educational approaches, organizational constraints, standardization, and scalable solutions. The findings of the workshop provided the basis for a better understanding of the different dimensions of mobile learning in the defense sector and for improved alignment of different initiatives for mutual benefit.

The articles in this issue of Connections continue this discussion by integrating the latest results of research and development projects in the area of ADL, mobile learning, and defense and security education. The insights presented by the contributions are based in research projects that are closely related to the educational practices of security and defense organizations. The previous findings already indicated that the core technological concepts are well understood and are ready for mobile learning applications. However, security and defense organizations face three key challenges for implementing mobile learning:

- Organizational regulations

- Interoperability of mobile learning solutions

- Novel approaches to educational design.

The importance of laws and organizational regulations in the security and defense sector cannot be underestimated. In order to bring mobile learning solutions from prototypes to practice, it is necessary to understand the context in which mobile learning will be applied. Specifically, defense organizations have rigid rules for how and where to use mobile devices. These rules go far beyond the level of research ethics. In their article, Jacob Hodges and Geoff Stead emphasize the need for better knowledge about how organizational, national, and international regulations influence the introduction of new educational technologies and approaches in security and defense organizations.

Integrating mobile learning with existing infrastructures is the primary challenge confronting the effort to scale up the use of mobile devices for education and training. This not only includes the user interfaces of mobile devices, but also touches on the interoperability of mobile learning solutions. Christian Glahn’s essay analyzes how organizations can leverage their existing educational resources to support mobile learning. The approach outlined in this article aims at lowering the barriers to the introduction of mobile learning through the use of SCORM-compliant learning material that is already present in many organizations. Kristy Murray and her associates go a step further, and provide an outlook into the future of SCORM by showing the relation of the Experience API (xAPI) and mobile learning.

Finally, it has become evident that mobile learning also requires revisiting the instructional design concepts for ADL. This is not only required for novel concepts such as mobile collaborative simulations, but also for more conventional concepts such as formative tests and content delivery. Stefaan Ternier and his colleagues in their article introduce the ARLearn framework for creating location-based and context-aware simulations for teams in real-world settings. This framework has been used for security training at the United Nations Refugee Agency (UNHCR) in order to provide an authentic learning experience. Glahn’s approach applies the instructional design concepts of micro learning and learning analytics to increase the continuity of learning of mobile learners. An initial proof of concept of this concept analyzed the implications of reusing existing training materials in new mobile instructional designs.

The articles in this issue provide a comprehensive overview of current findings and developments in the area of mobile learning in security and defense organizations. The authors address the key challenges and provide suggestions as well as solutions that provide a foundation for developing best practices of using mobile technologies in education and training.

Research Ethics in the Mobile Learning Environment (MoLE) m-Learning Project

Jacob Hodges and Geoff Stead [*]

Introduction

This research paper discusses the Mobile Learning Environment (MoLE) Project, a unique and ambitious effort sponsored by the U.S. Department of Defense’s Coalition Warfare Program (CWP) in partnership with over twenty nations. The mobile learning project explored the usefulness and effectiveness of using mobile technologies as a tool to support training activities in medical stability operations. This article discusses the importance of employing global research ethics and social responsibility practices in the testing and evaluating of science and technology projects. It provides an understanding of research ethics requirements and looks at how the technical challenges were applied within a global framework. Finally, it showcases an integrated application of a mobile capability in accordance with a myriad of research ethics guidelines and concludes with the accomplishment of evaluating this global capability.

Research Design

Science and technology (S&T) research has played a significant role in developing new technologies that benefit both society and the defense sector. There are many positive impacts that have resulted from such research, and the benefits have revolutionized our way of life. However, this is not always the case across all fields, and there are numerous examples of ethical misconduct in social and behavioral sciences and humanities research. Some researchers in these disciplines at times assert that regulations for the protection of human research subjects do not apply to their work in the same way that they apply to scientific or medical research. However, a close reading of most regulations regarding the involvement of human beings as research subjects will find references that state otherwise.[3] Ethical conduct is an essential element in all scientific research, and is necessary to foster collaboration, cooperation, and trust. It is imperative that research be socially responsible in order to make advancements in scientific knowledge that both protect and benefit the public.[4]

Research has been defined as a process of systematic investigation that includes research development and testing and evaluation activities that are designed to develop or contribute to generalized knowledge.[5] A human subject is defined as an individual who is or becomes a participant in research, either as a recipient of an article being tested or as a control. With such a broad definition, researchers should ensure that all moral and social dimensions are considered when a research project involves any interactions with humans. During the development stage, the project should incorporate a “gate-keeping” mechanism into the planning activities that demonstrates an endorsement of ethical practices, solid research methodologies, and applicable professional standards. For cooperative research, the project’s planning activities need to adhere to each of the institution or country’s research ethics requirements to ensure that the project takes the moral and social dimensions into account.[6] Therefore, when conducting research, three components are required in the research design:

- The Human Research Protection Program, which ensures that the researchers promote the integrity of the research and safeguard against any misconduct

- A Data Collection Plan, which will ensure that there is a clear understanding of the research objectives and develops trust in the data collection process

- The Data Analysis and Interpretation Process, which builds ownership across the research project and provides safeguards against any misconduct or impropriety that might reflect on the researchers or organizations involved.[7]

Human Research Protection

Human subject research is research that involves a living individual about whom an investigator obtains data through interaction. This interaction may include, but is not limited to, any type of communication—such as surveys, emails, Internet, phone interviews, face-to-face conversations, etc.—between the individual and the researcher.[8] Human research also includes risk management and the achievement of research objectives in areas relating to human safety, security, legal, and regulatory compliance and governance.[9] Human research protection includes a code of ethics to preserve individual autonomy, confidentiality, integrity, privacy, security, and respect while minimizing risk and discomfort to the research subject. Any data collection procedure from an individual, directly or indirectly, should incorporate this ethical code within its informed consent document. This written document is viewed as a truthful and respectful conversation that outlines the research approach, and sets forth the rights and responsibilities of both the researcher and the individual subject.[10]

Data Collection

Ethical issues in data collection refer to the need to guard against the collection of harmful or identifying information. To ensure that unnecessary data is not collected—e.g., data that will not be used as part of the analysis or is not required for research objectives—extensive collaboration is needed among the research team to ensure the data collection strategy is understood and accepted. A concerted effort is required to guarantee for all the prospective research participants from whom the data is being collected that the information being collected will not constitute an intrusion into their personal life, the data will not contain any identifying information without consent, and that each person participating in the research has the right to not answer any questions without reproach.

Surveys and questionnaires have their own ethical issues, since the collection process, especially using technology-enabled capabilities, has the potential to link identifying information to the response. Researchers should be conscious of the potential for this to be intrusive, and should seek to minimize any intrusion. The confidentiality of the data must be respected, and positive measures must be taken to protect it.[11] This requires the research team to identify any potential risk related to the privacy of the individuals and to convey this as one of the primary components in the informed consent. Therefore, from a data collection standpoint, the informed consent should identify:

1.       How the research protects the anonymity of the individuals

2.       The testing process, including roles and responsibilities

3.       The expectations of the individual research subjects

4.       The evaluative process

5.       How data will be shared collectively to support other research initiatives

6.       How ownership of the data collection process will ensure anonymity, privacy, and confidentiality.[12]

Data Analysis and Interpretation

At the outset, the link between the terms ethical and statistical is not self-evident. The Collins English Dictionary & Thesaurus defines statistics as a “numerical fact collected and classified systematically, and the science of classifying and interpreting information.” The definition of ethics is the “conscience, moral values, principles, standards and rules of conduct.” Although there is not an apparent connection between statistics and ethics, then, the definition of ethics (i.e., rules of conduct) and statistics (i.e., fact collection and classification) is the key to the relationship.

Ethical issues may arise in data analysis and interpretation. Researchers should show caution during data interpretation. Data analysis should be based on sound statistical research practices leading to conservative data interpretation – which is to say, an interpretation that does not overreach or claim the data are more significant or important than they really are.[13] Even within this paradigm, the interpretation of the same data can take different pathways, none of which may be unethical. Differences in data interpretation may well benefit the scientific process and allow researchers to capitalize on the important debates that lead to new technologies.

Highly collaborative projects, especially those that are involved in collecting both qualitative and quantitative data, must engage co-researchers during the data analysis, interpretation, and report writing process to preserve the integrity of the project’s results and to ensure impartial interpretation of the data. This was particularly important in this project, given the global nature and cultural diversity of the participants. To ensure reciprocity, all of the organizations involved should receive some benefit from the research, as well as have an opportunity to provide input into the interpretation of the analysis. The research results should reflect openness, sensitivity, accuracy, and objectivity in the choices of analysis and dissemination, to ensure that the project respects the interests of the different groups in society.[14]

Mobile Learning Environment Project

The Mobile Learning Environment (MoLE) was a two-year Coalition Warfare Program (CWP) Project.[15] It was sponsored by the Commander, U.S. Naval Forces Europe (CNE); Commander, Naval Forces Africa (CNA); and Commander, Sixth Fleet (C6F). In addition, it was co-sponsored by the Deputy Director, Joint Staff (J-7) for Joint and Coalition Warfighting (DD J7 JCW) and the Joint Knowledge On-Line (JKO) Director. The CNE-CNA-C6F Deputy Director for Training envisioned that a mobile learning capability could help address the significant challenges associated with training and communicating in the largest maritime area of operations where there are also the challenges of low bandwidth, limited Internet connectivity, and limited infrastructure. The DD J7 JCW JKO Director viewed mobile technologies as a critical step in meeting his organization’s requirement to facilitate and provide training to the U.S. and its multinational partners.

The basic concept was that the MoLE Project would leverage the global cellular network infrastructure, mobile technologies, and emerging mobile applications/service models to build a mobile learning capability that could be integrated into the DD J7 JKO portal. It would provide the foundation for conducting a proof of concept for evaluating a mobile learning solution for meeting emerging training requirements that not only exist in the sponsoring organizations, but also in many related departments, initiatives, and partnerships.

Through the proof of concept it would demonstrate an enhanced level of interoperability and yield significant benefits to all the global partners involved by providing general and medical education and training to military and related civilian personnel of countries in need of humanitarian and civil assistance, joint exercises and force training, or other types of on-demand training. This would, in turn, be shared by the international community to support their medical education and training as well as initiate the development of a sustained capability within their own countries’ defense learning organizations.

In order to support the MoLE Project’s goals and objectives, a Testing and Evaluation Working Group was established, which consisted of representatives from each of the twenty-two participating nations. The working group was divided into three teams to address some key challenges related to research ethics, specifically: human research protection, data collection, and data analysis and interpretation.

Human Research Protection Approach

As a cooperative research project, MoLE faced several challenges, since it involved incorporating twenty-two institutional requirements as well as country-specific guidelines. During the planning phase, a human research protection team was established to ensure that the research ethical requirements of each institution were identified and that subject matter experts were involved to ensure a thorough understanding of the applicable directives.

There are several key U.S. directives related to the protection of human subjects and adherence to ethical standards in DoD-supported research, which have been cited above. Collectively, these guidelines employ the ethical principles outlined in the Belmont Report to create a foundation for protecting individuals involved in research, which includes respect for persons, education and training, informed consent, vulnerability of individuals, collaborative research, etc. However, since MoLE was a research project conducted in an established educational setting that involved “research conducted on the effectiveness of or the comparison among instructional techniques, curricula or classroom management methods,” it was exempt from a rigorous human research subject review.[16]

A review of the U.K. requirements stated that research involving human participants undertaken, funded, or sponsored by the Ministry of Defense (MoD) must meet acceptable ethical standards, and that these ethical standards are upheld by the MoD’s Research Ethics Committees. Their Joint Services Publication (JSP) sets out in the MoD instructions the requirements for the ethical conduct and treatment of human participants in research (both clinical and non-clinical) and the ethical treatment of human participants. The JSP states that the directive applies to the conduct of research to collect data on an identifiable individual’s behavior, either directly or indirectly (such as by questionnaire or observation). Thus, the rule was applicable to the MoLE Project.

In ensuring that the project met the European Union’s research requirements, several documents were used as reference, specifically the European Union’s Data Protection Requirements, EPIC’s Privacy and Human Rights report, Solveig Singleton’s article on data privacy in the United States and Europe, and EU Data Protection Directive 95/46/ED.[17] The basic ground rules for privacy state that all individuals involved in a research project need to be informed about the planned research use of collected data, regardless of the type of data collected. If a survey is planned within the project, individuals need not only to be informed of how their data is planned to be handled but also must be given the opportunity to provide appropriate authorization. In addition, the survey design must guarantee that only data specifically required for the purpose of the research project will be gathered, unless clearly stated otherwise.

After careful consideration of all the aforementioned documents and directives, and email exchanges among the international participants, it was determined that the most restrictive guidance was that from the EU; therefore, an informed consent would be required that incorporated both U.S. and EU ethical requirements. The MoLE Informed Consent was developed to include all research protocol areas: Introduction, Purpose of the MoLE Project, Duration of Participant Involvement, Procedures, Testing and Evaluation Process, Risk and Discomforts, Potential Benefits, Voluntary Participation and Withdrawal, Confidentiality, and Consent of the MoLE Individual.

Data Collection, Analysis, and Interpretation Approach

At the project kick-off meeting, the Testing and Evaluation Working Group held a rigorous session to develop a stage-gate approach to ensure that ethical practices were utilized throughout the data collection, analysis, and interpretation phases of the project. The participants were separated into three groups so that equal efforts would be placed on the data collection, analysis, and interpretation requirements. However, since this was the first meeting, the focus was changed to concentrate more on “what should be asked” and “what data should be collected” in order to achieve success, rather than on how the collected data would be analyzed or interpreted. As a result of the collaboration, the team decided to place concerted effort on:

1. What types of questions should be asked

2. The testing and evaluation process, including roles and responsibilities

3. Research expectations

4. The evaluation process

5. How the data will be shared during the analysis phase

6. How transparent data collection would ensure anonymity, privacy, and confidentiality.

Research Strategy

After this initial meeting, the Testing and Evaluation Working Group focused on addressing issues relevant to determining how the MoLE Project would measure whether it had met its goals and objectives. The working group was divided into three teams to ensure the key elements in evaluating the proof of concept were identified and resolved (i.e., human research protection, data collection, data analysis and interpretation). The three teams each collaborated via quarterly teleconferences and Web meetings, with the exception of the human research protection team, who collaborated on a weekly basis to ensure that all ethical requirements were identified and documented.

Human Research Protection

Subsequent to the review of pertinent documents, ranging from “the involvement of individuals in research” to “information and data collection requirements,” the human research protection team developed a draft MoLE Proof of Concept Informed Consent, which would serve as the written agreement between the researchers and the individuals. It was then emailed to all Testing and Evaluation Working Group members for review and comment in order to ensure a complete understanding of the research protocol. Comments were then incorporated into the document and presented at a three-day Testing and Evaluation Working Group meeting to ensure that the document reflected the actual state of the testing and evaluation process. This review process ensured that the research protocol was totally accurate, including how testing and evaluation would be carried out, what the duration of involvement and confidentiality would be, etc. The revised informed consent form was subsequently sent via email to over forty international delegates for their final review and feedback. The email also requested each delegate to consider if a version of the informed consent statement should be translated into their native language to ensure complete understanding of individual requirements and expectations. The informed consent form was thus made available in English and also translated into Spanish and French.

Data Collection, Analysis, and Interpretation

During the initial project meeting, the Testing and Evaluation Working Group members that were not on the human research protection team formed small groups to determine what types of data should be collected in order to meet project goals and objectives. Each of the groups provided recommendations on the types of data that could be collected and operational measures for determining effectiveness and performance. However, since the mobile content was still being developed, it was not yet possible to finalize these guidelines. In focusing on the core goals and objectives of the MoLE Project, the two teams determined that data collection should focus on four themes, as shown in Table 1. An email was sent to each of the working group members asking them to provide five questions they would ask if they were developing the survey. As a result, an average of 150 inputs per term of reference (in addition to over twenty other potential questions) were identified.

Table 1: MoLE Terms of Reference.

Term of Reference

Meaning

Accessibility

The degree to which a mobile training application is available to the user

Self-Efficacy

A user’s belief that he/she is capable of producing the desired outcome of the task required

Usefulness

The benefit or availability of mobile technologies in providing training

Utility

The effectiveness, or practicality, of using mobile technologies in providing training

Once the Medical Content Working Group identified the medical content, a three-day meeting of the Testing and Evaluation Working Group was convened, which included the Medical Content Working Group leads. The purpose of this meeting was to:

1. Inform the Testing and Evaluation Working Group of the medical content decision

2. Identify the scenarios and vignettes to ensure the proof of concept included both medical and non-medical participants

3. Identify the survey questions that would be asked as part of the scenarios and vignettes

4. Collaborate on the transparent data collection strategy

5. Review the MoLE PoC Informed Consent and how it fits into the data collection strategy

6. Develop the MoLE Testing and Evaluation Plan.

After the first day, the group was divided into two groups: the data collection team, and the data analysis and interpretation team. The data collection team, in collaboration with the Medical Content Working Group leads, developed a storyboard of how the data collection process (e.g., survey and transparent digital data) would integrate with the medical content. In the meantime, the data analysis and interpretation team focused on the how the data would be statistically displayed as well as on how the data could be made available to international researchers for follow-on research initiatives. At the end, each of the teams, including the human research protection team, briefed the content and technical teams on their results to enable integration into the mobile apps under development.

At the start of the second year, the Testing and Evaluation Working Group convened for a final meeting before the completion of the technical development phase to finalize the data collection, analysis, and interpretation strategy. Discussion focused on refining the process of how the proof of concept would function. Final decisions were made regarding the demographic data collection plan, how individuals would participate in the proof of concept, and the human research protection issues.

Application Development

Tribal Group was responsible for developing the mobile application based on the requirements from each of the three working groups: Learning Content, Technology and Transition, and Testing and Evaluation. The Learning Content group worked with a wide range of medical and training stakeholders to design, convert, import, and create mobile content to support the needs of the target users. The Technology and Transition group developed a cross-platform toolset that enabled mobile learning content to be deployed to apps on both Android and iOS (Apple) platforms, and worked with Joint Knowledge On-line (the US DoD e-learning platform) to integrate the mobile platform with their back-end infrastructure. The Testing and Evaluation group ensured that the testing and evaluation process was carried out as planned, specifically regarding the ethical research issues and human research protection issues.

Figure 1: MoLE Proof of Concept Process.

Evaluation Process

Participation in the MoLE Proof of Concept trial was a two-step approach. First, each organization participating in the project was required to identify at least twenty individuals to participate in the trial. Email addresses of each volunteer would be sent to the MoLE Research Ethics Coordinator, who was the only person with full access to all volunteer contact details. Using a tool developed by Tribal, the Research Ethics Coordinator would generate a unique personal identification number for each volunteer, and email each one with a welcome message containing all instructions for participation. This included a link to their registration page, their personal identification number (PIN), and additional links to support their involvement (i.e., introduction video, user guide, and detailed overview and installation guide). The email also provided a link to the MoLE Registration Site. In the Registration Site, no data could link the PIN with the email or name of the individual. On two or three occasions, volunteers sought support from the MoLE Technical Help Desk and inadvertently included their personal identification number. In such cases, they were reassigned another PIN to ensure their anonymity. Once the app was installed, they could participate in the MoLE Proof of Concept trial. When the trial had ended, the Research Ethics Coordinator deleted all databases that contained any references to email addresses and PINs.

As Figure 1 shows, the testing and evaluation process was broken down into six steps. First, an email announcement would be provided to each individual. The individual would register online, install the app, and then activate their app using their personal identification number. Each individual was afforded the opportunity to become familiar with the app before starting the evaluation.

During the online registration, each individual was required to acknowledge the MoLE PoC Informed Consent and complete the demographics questionnaire (see Figure 2). Without acknowledgement, they were unable to validate their PIN or activate the mobile app.

On completion, the PIN would be activated, and the user would be able to download the Global MedAid app from their local App Store, and register using their unique PIN. This PIN only carried a national identifier (to permit localization), and no personally identifying data.

Question

Responses

Age

less than 20, 20-29, 30-39, 40-49, 50+, no answer

Gender

Male, Female

How proficient are you in English

Beginner ... Advanced

Are you using your own personal smartphone for the purpose of this trial?

Yes, No, No Answer

How comfortable are you with using the mobile device that’s running the MoLE app? [Beginner to Advanced]

Beginner ... Advanced

Have you previously been involved in humanitarian assistance or disaster relief operations?

Yes, No, No Answer

What is your professional expertise?

Medical, Rescue, Training, E-learning, Other

Have you taken the Trafficking in Persons (CTIP) course within the last two years?

Yes, No, No Answer

Figure 2: Demographic Data.

Once the Global MedAid app was installed, volunteers were encouraged to familiarize themselves with the app and explore its features. When they were ready to begin the evaluation, they launched a specially designed “evaluation layer,” which gave them specific tasks to complete within the app, following pre-defined medical scenarios/vignettes. Their use of the app was monitored (transparent data), and their feedback on the task was collected via an in-app survey. These data were then synchronized back to the project website and collated.

Figure 3 shows a sample page from the evaluation survey. The volunteer is asked to complete a task, and on completion is asked to answer a few questions. Most of the questions were completed with a “slider,” allowing selection against a seven-point Likert scale. All questions offered the option of not responding, and required an active effort to select an answer.

Volunteers were asked to complete three vignettes/scenarios representing three different ways that the Global MedAid app might be used: one before deploying on a humanitarian mission, one en route, and one on arrival. Each scenario was structured in the same way, giving the user a series of tasks, tracking their activity, and recording their answers to specific questions. At the end, they were asked one final set of questions requiring text input, and offering a more open format for responding.

At a pre-defined date the proof of concept phase concluded, and no more tracking data or survey responses were collected. A new version of the app was subsequently released that automatically upgraded, removing the evaluation survey and registration requirement of the app.

Figure 3: MoLE PoC Response Format.

Once the proof of concept evaluation was completed, the Testing and Evaluation Working Group initiated an analysis of the data in which several group members independently conducted their own analyses. In addition to the internal analysis that was being conducted, several outside resources were used to provide independent analysis of the interpretation of the data. In areas where there were disagreements, additional analysis was conducted to help identify the potential disparities in the analysis and draw appropriate conclusions. The results of this evaluation will be reported in a separate report. This article aims to describe the process and clarify the steps taken to ensure that clearly defined research ethical standards were observed and that volunteers’ data were protected.

Volunteer Numbers

The data showed that 268 test subjects had registered. Of these, 177 (66.4 percent) started the proof of concept trial, and 137 (51.1 percent) completed the trial. These subjects came from twenty-one countries (Figure 4). Of these participants, 70.9 percent used their own mobile device, while 29.1 percent borrowed a colleague’s device. A majority (63 percent) were using an iPhone 4 or 5, and 37 percent used an Android device (running version 2.2, 2.3, 3 or 4).

The individuals’ professional expertise, based on the demographics survey, showed that 34.2 percent had medical experience, 26 percent were involved in e-learning, 25.3 percent identified their professional expertise as “other,” 11.2 percent conducted general training, 1.1 percent were involved in rescue efforts, 2.2 percent were involved in rescue operations, and 3 percent declined to answer the question. A majority of the individuals were using the app in English; German, French, Georgian, and Spanish versions were also available.

Figure 4: Proof of Concept (PoC) Participants.

The Wider Project

The primary purpose of this project was to explore the utility and effectiveness of using mobile technologies in security training and then to create a transition strategy that moves mobile learning and training into the mainstream of defense training for all of the international partners involved in the project. This process has already started, and several partners are adopting some of the system and processes that will include some of the content created by the project into their mobile training programs. An extensive publication has been produced about the project, so that learning and development, education and training, medical training, electronic learning and mobile learning, research and development, and testing and evaluation professionals can gain insight into what worked best in providing a mobile learning (m-learning) capability that was learner-centered (i.e., built on the skills and knowledge of the individual/teams), knowledge-centered (i.e., providing educational content that is factually sound), and community-centered (i.e., promoted the sharing of knowledge and collaboration).

Conclusion

The Mobile Learning Environment (MoLE) Project, as a global research initiative, demonstrated that project goals and objectives can be fulfilled by employing ethical and socially responsible practices. Given the diverse group of stakeholders, there was considerable complexity involved in developing and implementing a testing and evaluation strategy that incorporated research ethics guidelines. Although there were many challenges in integrating ethical research requirements into the technical interfaces, with careful management and an effective stage-gate approach, these challenges can be dealt with clearly and effectively.

Supporting Crisis Simulations with the ARLearn Toolkit for Mobile Serious Games

Stefaan Ternier, Atish Gonsalves, Bernardo Tabuenca, Fred de Vries, and Marcus Specht [*]

Introduction

Although the transfer of factual knowledge plays an important role when training personnel who operate in potentially dangerous conditions, mobile learning techniques to support security training are often underused. There is a great value in simulating dangerous situations in the training context. In contrast to learning from a textbook, a simulation leaves room for customization and surprise, as players do not know what will happen in advance. By experiencing a series of events that occur in a simulation, participants learn in an authentic context and are trained to respond to events as they occur. This is different from studying factual knowledge. The theory of situated learning supports this assumption, and states that learners do not learn via the plain acquisition of knowledge but rather through active participation in social contexts.[18] Organizing simulations in the real world can become very expensive and resource-intensive. For instance, a real-life military simulation can require the use of weapons and explosives to engage the participants in the scenario. Many dangerous situations require personnel with different roles (e.g., team leader, communication expert) to cooperate. Managing these different roles during a simulation requires extra work of the simulation facilitator.

One of the challenges of organizing a simulation virtually is to implement an immersive experience. Chris Dede defines “immersive learning” as learning that involves the “subjective impression that one is participating in a comprehensive, realistic experience.” [19] Such experiences can be typically found in games and simulations. Dede differentiates between the following types of immersion:

- Sensory immersion refers to the possibility to get haptic feedback via sensors and to create a realistic environment.

- Actional immersion makes it possible for the user to take actions that would normally be impossible in the real world. Such actions can lead to new insights.

- Symbolic immersion focuses on making new semantic and psychological associations that are acquired via the content available in a virtual environment.

UNHCR seeks applications of new technologies to train its staff in a number of key subject areas, such as refugee law, international human rights law, operations planning, and security risk management in order to protect and meet the needs of humanitarian agencies and forcibly displaced persons. Some of these topics can be delivered through conventional e-learning delivery methods, such as self-study modules and webinar-based coaching. However, training humanitarian staff for complex scenarios, such as managing hostage situations, relies more on immersive and participatory simulation exercises and role-playing activities. For the application of mobile devices in a role-playing game, we are particularly interested in actional immersion. By enabling the participants to make decisions during a crisis simulation, the simulation will follow a different course, taking into account the decisions the participants make in particular situations.

Mobile devices play an important role in crisis simulations that involve role-playing. Although they do not match the immersive feeling a participant gets when entering a real-world simulation (e.g., flying a full-scale flight simulator), mobile devices are very well suited to orchestrate a flow of events. In a mobile serious game, the mobile device acts both as a remote control and a display. The device offers the participant a perspective to the game that is properly adapted to the user’s role and context. At the same time, the device is used to provide the game with input. Users record audio and video with their smartphone, provide answers to open and multiple-choice questions, or make the game controller aware of their current context (such as their current location).

ARLearn is a toolkit for designing and playing mobile serious games, and is especially useful for mobile scenarios including field trips and role-playing games. The combination of a serious game engine with a mobile platform has produced a toolkit with significant potential for simulating crisis situations. The Open University of The Netherlands (OUNL) and the UNHCR have collaboratively developed a serious game to simulate hostage situations with the ARLearn application, and have trained UNHCR staff with it.[20] The next section of this essay covers the ARLearn features and software architecture, and how they support a mobile simulation. The following section includes a UNHCR case description that illustrates how the application has been used in different educational scenarios in which learners are to explore their environment and collect data, commonly addressed by the concept of fieldwork.[21]

ARLearn Toolkit

ARLearn, originally a tool for audio augmented reality, has grown from a standalone smartphone app into a fully-fledged mixed reality application platform including support for field trips, serious gaming, augmented virtuality, and notification systems.[22] In order to support the creation of simulations, ARLearn builds on two important concepts:

·         A game is a blueprint for a simulation, and defines the game artifacts, user roles, and the logic that combines these artifacts.

·         A run refers to a concrete play of the game and defines users grouped in teams. Within a run, the actions of these users, their responses and progress are tracked. For each game, an arbitrary amount of runs can be created.

ARLearn implements a simple data model that enables the definition of several kinds of media artifacts, including multiple choice messages, video messages, and audio messages. Media artifacts are bound to a context that can be defined by a location and/or a timestamp. The context defines where or when in the game messages have to appear. Furthermore, a flexible dependency mechanism enables the author to define the game logic. Through dependencies, the author specifies conditions for making media artifacts appear or disappear. An author can, for instance, define that sixty seconds after all players have read the introduction message, a video message will become available. Media artifacts can be configured so that they will disappear about three minutes after the artifact has been read.

Dependencies

Dependencies enable a game author to define the game logic. An ARLearn media artifact defines two dependencies:

- When an appearOn dependency is specified, the media artifact is not visible when the run starts. Only after the dependency is satisfied is the media artifact rendered visible.

·   The disappearOn dependency defines when an artifact will become invisible.

Dependencies do not evaluate to Boolean values (“true” or “false”), but rather evaluate to a timestamp when the condition is satisfied. If the system cannot determine when the dependency will satisfy, the dependency evaluates to an infinite number (i.e., the biggest number that can be represented by the system). Evaluating to a timestamp enables authors to specify time constraints and allows the mobile client to schedule a timer. ARLearn is currently equipped with three kinds of dependencies:

- An action-based dependency refers to a game action. Every message, for instance, specifies a “read action.” This action is triggered a soon as a user reads the message. Open questions or multiple-choice questions trigger an action when a response has been given.

- A time-based dependency specifies a time relative to another dependency. With this dependency, one can stipulate the time that must pass after an action has occurred. This mechanism allows for encoding a natural sentence like “thirty seconds after the artifact has been read.”

- Combined dependencies (AND/OR) combine two or more dependencies. A game author can thus use expressions to specify multiple conditions that need to be fulfilled. An AND-based dependency will evaluate its operands and will return the maximum timestamp. An OR-based dependency does the same, but returns the lowest timestamp.

These dependencies support flexible game logic. Take the example of defining a countdown for answering a question. Consider the case where a player is given three minutes to provide an answer. If the player does not succeed to answer within that time, the question disappears. Such a countdown clock is implemented via a disappearOn dependency. This combines a time-based and an action-based dependency to express a predicate that specifies “180000 milliseconds after the action ‘answer_given’ has been issued for the question.” As long as the action has not been triggered, the action-based dependency will evaluate to an infinite amount of time. The time-based dependency adds three minutes to this value.

Notifications

Notifications are a central element of ARLearn, both from a technical and conceptual point of view. In a typical multi-player game, players will receive messages as a result of actions taken by other players. For instance, as soon as the first responder arrives on a scene, all other players receive a message that informs them. The ARLearn game engine will send notifications to all players when a media item becomes visible.

Technically, ARLearn supports three mechanisms to deliver notifications to the player’s device. Efficiently organizing notifications was an important achievement, since energy-efficient mobile devices cannot continuously poll for newly available media artifacts.

The Google Channel application programming interface (API) is a JavaScript solution for sending asynchronous notifications (so-called long polling). With the channel API, an application client subscribes to a common channel and listens for updates. This solution typically works well in a browser, but is less suited for mobile devices because the solution is not energy efficient, as it requires the app to run its own notification channel as a background service. Furthermore, long polling requires the client to be continuously online. Typically, mobile users can enter and leave networked places and therefore need an application that is optimized for offline usage.

Apple Push Notification (APN) and Google Cloud Message (GCM) deliver messages to iOS and Android devices respectively. Both technologies rely on one communication channel that the device maintains for all applications, and is thus much less resource-intensive. The ARLearn client for iPhone relies on APN, while the Android app is currently being migrated from the Google Channel API to GCM. The ARLearn authoring environment, a JavaScript browser application, features a module to track the user and to be notified immediately when new responses are produced. This module thus builds on the Google Channel API.

Roles

Media artifacts can be bound to roles, meaning that they will only be visible to players that have the same role assigned. For instance, a message “answer an incoming call from journalist” can be bound to the role “communication officer.” Content that is not bound to a role is visible to all players. Dependencies can be used in combination with roles. This is useful when content needs to appear or disappear when a player in a given role performs an action. This enables expression actions like “make an information package available to all users, when the team leader enters the control room.”

Tagging and Scanning

Often, mobile serious games require actions that are bound to a user’s location. Although a GPS sensor comes in handy here, a very useful alternative is checking in to a location by scanning a tag. ARLearn supports scanning both QR codes and RFID tags. QR codes require the user to actively open the scanner to scan the tag by pointing the camera to it, while an RFID tag only requires touching it with the phone. The latter does not oblige the user to open the scanner for scanning the object, an action that relies on more user interaction cycles with the mobile app. The advantage to scanning over automatic recognition of a user’s location via GPS is twofold. Firstly, GPS is often not reliable. In places with tall buildings, reflective windows, and narrow streets, GPS signals are often very weak. When users are indoors, the signal is too inaccurate, even if used in combination with network-based location tracking. Secondly, allowing the user to scan a tag to reveal his location puts the player in control. The player actively decides to inform the system that he entered a room, and gets direct feedback that this action was registered.

The UNHCR Hostage-Taking Pilot Project

UNHCR operates in over 120 countries, and its staff members often work in hazardous locations. To better equip its staff, UNHCR conducts a number of security management training programs worldwide. The UNHCR Global Learning Center (GLC) in Budapest organizes these trainings. The typical approach is a workshop organized over a three to five-day period in which different aspects of security risk management are addressed. These workshops cover policy-based information, such as standard operating procedures, delivered through self-study e-learning modules done in advance of the course, followed by instructor-led workshops. The workshop also includes immersive simulation exercises, for example hostage taking, bomb threats, and other security-related scenarios.

For many years a role-playing game has been part of these workshops. In this game learners are split into groups representing the different roles that are needed in a real security situation. As a team, each group implements and carries out the procedures that have been introduced during the e-learning modules and the workshop. In the case of the hostage-taking scenario, a role-playing game is a highly immersive experience for the learners, in which they have to manage stress, act quickly, collaborate and negotiate in order to “save the hostage.” Running the game is an intensive exercise, not only for the participants, but also for the organizers and facilitators. It requires a lot of concentration and effort by a facilitator to encourage the whole team to engage in the chaotic development of a hostage situation. This limits the number of participants that can simultaneously join a game. Since the game is carried out at a rapid pace, it can be difficult to have an all-inclusive debriefing in which all roles on all teams receive appropriate feedback. The debriefing and reflection phase of the activity is a major learning point. Debriefing allows learners to reflect on what they learned, the challenges and risks associated with hostage situations, and their personal capabilities. Facilitators recognize that this is one of the most important learning moments, but it is frequently difficult to capture all the important points from the simulation due to the speed at which events occur. This aspect has led to the development of this project, in an effort to address this shortcoming.

As an alternative to the original game, a version was developed by applying concepts of mobile serious games and blended instructional design principles.[23] Using the ARLearn toolkit made it possible to address the following training issues:

- Enabling the creation of different reusable variations of a game design for emergency security response, covering initially the hostage situation and potentially other cases

- Enabling “on the fly” messaging to participants and real-time assessments of activities

- Semi-automatic management of the game, thus enabling more participants to experience the role-playing exercise.

- Creating a log of responses and interactions throughout the game, which can be used by the trainer to provide feedback during the debriefing session.

Game Design

The first ARLearn pilot in UNHCR was implemented at the Security Management Learning Program (SMLP) workshop in December 2011 in Entebbe, Uganda. The seventeen participants were senior staff members, including heads of offices, who are responsible for managing operations and ensuring the security of the staff members in their country offices.

A pre-game survey was conducted to examine how comfortable the participants were with smartphones. The majority used their mobile phone frequently, either constantly (nine of the seventeen) or daily (six). Eleven participants used their phone to access the Internet and email. Six of them constantly accessed the Internet with their smartphones. This survey showed that most of the participants were comfortable with a smartphone, although most of them had not used a touch-screen based device prior to the simulation.

The workshop was organized in the conference room of a hotel in Entebbe. Internet connectivity was limited and sporadic, and electricity cuts were quite frequent. In Entebbe, the game was played using nine smartphones simultaneously. Three runs were created for the game, with three roles participating in each run. Each role was assigned to two or three players.

The game is designed to train the participants on the response procedures to be initiated immediately when a staff member is taken hostage. A Hostage Incident Management (HIM) team is deployed eventually in such situations, but it can take time for this team to arrive, and country offices need to know how to respond prior to their arrival.

The players participated in this game taking one of the following three roles: head of office, security officer, and staff welfare officer. The hostage-taking simulation was designed so that players in all roles play the same game but have to react differently based on their roles. The game is organized in five phases:

Phase 1: Notification of the Incident. The game starts with a plea for help by Jerry Khan, a fictitious UNHCR employee who has been taken hostage. This video message features a blindfolded actor and creates an authentic context. This message is broadcast to all the roles. Next, players make a decision on what to do next, depending on their specific role. The head of office (Role A), for instance, can decide to “notify the Designated Officer (DO),” while a staff welfare officer (Role C) should select the option to “contact senior management.” Depending on the decisions they make, they receive feedback on whether these were good choices.

Phase 2: Assembling the Team. In the next phase, the head of office is informed by the DO that a hostage incident management team will be dispatched. In the meantime, they need to contact the security advisor (Role B) and staff welfare officer (Role C) and ask them to assemble in headquarters for a planning session.

Phase 3: Planning. When the facilitator observes that the team has assembled, an audio recording of the DO requesting the team to work out a reception plan is sent out. The team is next assigned to work out this plan on a flip chart and to capture a photo of the plan with their mobile device and submit this as soon as they are ready. Next, the participants are asked to split up and go to their individual rooms.

Phase 4: Responding. In this phase, Role A and Role C participants are to respond to calls from a journalist and a distressed family member, respectively. Meanwhile, the security officer (Role B) receives a message from the DO with the task to prepare a Proof of Life (POL) question.

Phase 5: Negotiating. In this last phase, all roles gather together again. This is triggered by a message from the hostage takers. In this phase, a negotiation with the hostage takers is simulated. The game ends with the message that the Hostage Incident Management (HIM) team has arrived and is ready to take over the negotiations.

Figure 1:   UNHCR Screenshots, from Left to Right: Map View, Message View,

Sample Assignment.

Results

Overall, the game was well received, and the participants were able to successfully complete the runs. During the run, there were three electricity cuts that caused the wireless router to reset. However, the participants were able to continue the exercise despite these challenges.

The participants found the ARLearn simulation very useful, and once they got over the initial technical obstacles were able to respond to the Notification, Assembling, Planning, Responding, and Negotiating exercises. Of the seventeen participants, fourteen answered positively on the question regarding the usefulness of mobile phones in a simulation exercise (one response was negative, and two were mixed). As both the exercise and the use of smartphones in training was new to the participants, the expressed learning outcomes varied, from fourteen persons learning from the game itself, to two people expressing that they mainly learned about using the smartphone, and one reporting no learning.

From a content perspective, the participants were able to understand the critical nature of the exercise and the tasks associated with the immediate response of a hostage-taking scenario. They learned the importance of coordination, which was enabled and assessed through the game. The game added more realism through the “Jerry Khan” hostage video, calls from distressed family members, and pressure that was applied by the demands of the hostage takers. In addition, the facilitators played a role in moderating and pacing the flow of the game through the use of the manual triggers. It created good inter-dependencies among the roles, and showed that the leadership of the head of office (Role A) plays a key role in ensuring that the team delivers.

There were also some surprising outcomes, with participants realizing the importance of efficient and rapid information sharing in a crisis that is enabled through well-connected and ICT-enabled offices. Participants also found the exercise highly stimulating, as they played the game within the device, moved around, interacted with each other, and responded to the various assessments. The learning from the ARLearn exercise was later referred to in the workshop, for example on the formation of the proof of life (POL) questions.

There were a number of technical challenges with the exercise, especially related to the problems of dependence on the wireless Internet. The live Internet enables role inter-dependencies and the ability to use manual triggers, but it requires the devices to be constantly connected to the network. The participants were also initially uneasy with the devices, and the initial challenges were more related to the usability of the devices than the game itself. As the participants got more comfortable with the devices they were able to focus on the real objectives of the exercise.

This game script was implemented in two phases. In November 2011, a dry run was organized in Budapest with staff members of the organizations. In December 2011, the actual pilot was organized in Entebbe. Here, three game runs were run at the same time, featuring three roles per run.

Although no summative and quantitative evaluation was organized for this pilot, the game organizer provided the following formative feedback:

- Being able to run the game on Android devices has many advantages. Although this was not the case for the UNHCR pilot, ARLearn games can be run on personal devices, implementing a “bring your own device” (BYOD) strategy. The ability to leverage a widespread operating system makes the solution much cheaper compared to AR solutions that need special hardware. In this particular case, however, learners had to become familiar with the device (touchscreen, Android OS, etc.). Therefore it was suggested to create a demo run to enable participants to get accustomed to the device.

- The script was implemented with both manual and automatic triggers for items to appear. Although manual triggers offer some degree of flexibility, future pilots should have more automatic triggers and fewer dependencies on a network connection. The unreliable wireless network can sometimes result in manual triggers not arriving at a device. Automatic triggers have the advantage that they are cached on the mobile device, thus lowering the workload for the operator.

·   Decreasing the dependency on Internet connection will make the game easier to export to other countries, since game organizers will not have to acquire large numbers of SIM cards or deal with wireless settings on the devices.

Conclusion and Discussion

This article presents a brief discussion of an open source and flexible architecture for mobile serious games. From an end-user point of view, creating a game is easy via a Web-based authoring environment. Players can join a game with little effort, and can download the ARLearn mobile app from Google Play. Leveraging a BYOD strategy makes organizing role-playing games—such as the UNHCR case—much cheaper to realize, as compared to acquiring specialized and expensive hardware. Furthermore, since game logic can be scripted, the cost of organizing and administering a serious game can be significantly reduced.

In the pilot organized with the employees of UNHCR, games were created with both manual and automatic triggers. For instance, in one of the first pilots, a game facilitator counted whether all participants had entered a room, and then manually triggered new content to be sent to all game participants. In the case of an automatic trigger, the dependency mechanism is responsible for making new content available. Although manual triggers offer some flexibility from the point of view of the facilitator, we have discovered that they make the game less scalable and place too large a workload on the game facilitator.

Future Work

The ARLearn tool has proved to be useful in enhancing the “realism” of UNHCR’s hostage-taking simulation exercises. UNHCR’s security training team plans to implement further security training scenarios in 2013–14 with the tool including but not limited to:

- Death of a staff member

- Car accident involving injuries

- Kidnapping of a staff member (terrorist, politically or financially motivated)

- Refugee threatening suicide

- Staff member injured in a mine strike that occurs with a team on a mission in a remote location

- Angry demonstration that turns into a riot outside the office

- Staff member(s) detained in a refugee camp by a group of angry refugees

- Arrest of a staff member

- Bomb blast on the street outside the office – gate destroyed and casualties among guards, but no major damage to the office itself

- Loss of the office.

Beyond security training, UNHCR also plans to use the ARLearn tool to train on scenarios in the field of emergency response, refugee protection, and program management.

One of the ongoing developments in ARLearn is the integration of displays into a serious game. Displays can add many features to a game play.

- Content display. Similar to broadcasting media artifacts to mobile devices, a content display makes artifacts appear on a static display. Rather than sending all players in a room an individual message, the static display can be used to show an assignment to everyone. Content display can be valuable in game synchronization points, and will also tie into the dependency system. For instance, a movie can be started as soon as all participants have checked in at the location.

- Ambient information. The information that is displayed relates to the environment and visualizes the game state. Showing a map of the area with the location of all players adds extra information to a player’s perspective. Other options include showing the score of all players, a message board that shows the latest ten messages, and a photo pane that shows the pictures that were recently taken by players.

- Classroom response system. A display can also facilitate lectures to gather input from an audience. Here a typical scenario is voting. A question is shown first on the display; after voting with the mobile devices, the results show up on the display.

Currently, ARLearn provides an export/import feature for games. This provides a way of sharing games, although it is rather cumbersome. We plan to extend the metadata of both games and media artifacts. This will enable not only sharing games as whole, but also the sharing of individual smaller-grained media artifacts. Studies have revealed an inverse relationship between the size of a resource and its reusability.[24] There is thus a potential in sharing fine-grained components like a video or an audio recording. Giving mobile game authors location-based search access to these fine-grained components can speed up the creation of a serious game.

Supporting Learner Mobility in SCORM-Compliant Learning Environments with ISN Mobler Cards

Christian Glahn [*]

Introduction

Over the last decade, mobile information technologies have become a ubiquitous part of daily life. Mobile learning research has been going on for less than ten years, given that the smartphone revolution only started in 2006.[25] As such, this field is among the newest research areas in educational technology. Given the overwhelming success of smart mobile devices on the global scale, this technology appears to be well suited to extending the reach and continuity of educational programs.

Mobile technologies have become increasingly relevant for education and training in security and defense organizations not only because of the market success of mobile phones and other portable devices but also because many mobile technologies have become part of the standard infrastructure in these organizations. Mobile technologies are part of the information networks that characterize the professional environments of soldiers, policemen, fire fighters, and other security workers. An example of such a networked infrastructure in the defense sector is the “Gladius” System that integrates infantry and vehicle-based weapon systems in the German military.[26] Further development towards “network-enabled” combat systems is currently in process.[27] These examples illustrate that the scope of technological change represented by the mobile revolution goes far beyond the availability and use of mobile phones.

Education and training in security and defense organizations are challenged by mobile technologies and the new relevance of mobility from four perspectives:

- Technological

- Socio-technological

- Professional complexity

- Organizational.

This essay emphasizes the organizational perspective. Many organizations have already made substantial investments in learning management systems (LMSs) for advanced distributed learning (ADL) infrastructure and in developing appropriate educational material. Significant investments have also been made in the training of instructors and authors to make good use of the available ADL solutions; indeed, many organizations have a rich pool of educational resources available in the SCORM format.[28] The introduction of mobile technologies into education and training is particularly challenging because many organizations have not yet completed the initial adoption of conventional ADL solutions. This raises the question for security and defense organizations of whether they have to reiterate the process and create new educational resources and programs if they want to introduce scalable mobile learning solutions.

Mobile Learning and SCORM

The sharable content object reference model (SCORM) is the most prominent interoperability standard for educational material that can be used for Web-based training. Originally developed by the ADL Co-labs in the United States, it has emerged as the industry standard for exchanging Web-based training material. SCORM introduced interoperability standards for educational material that made the solution independent from the underlying delivery platform. SCORM is one of the core elements for sustaining investments into the development of educational material. The “reference model” has been widely adopted for packaging and exchanging educational material between so-called “run-time” environments and “learning management systems.” SCORM specifies three aspects of ADL solutions:

- Content packaging, exchange, and delivery

- Content arrangement and sequencing

- Interactive content and data persistency.

The central elements of SCORM are “content objects.” The terminology of SCORM refers to content objects as “sharable content objects,” or in shorthand, “SCOs.” SCOs are those educational resources that can be shared across ADL courses, compared to non-reusable contents, such as submissions to discussion forums or student presentations, which cannot be shared across courses. SCOs can be text documents, videos, audio files, interactive multi-media, as well as tests and assessments.

The most prominent aspect of SCORM is content packaging. This aspect defines how related educational resources need to be packaged, so that learning management systems produced by different vendors can consistently export and import educational material. Content packaging in SCORM includes not only the storage format for educational materials for distribution, but also the internal logical structure of and between SCOs.

The second aspect of SCORM defines the arrangement and sequencing of SCOs in the run-time environment. There are two levels of arrangement and sequencing. The first level of sequencing is based on the internal logic of the content package. This is similar to the arrangement of print material into chapters and sections. The second level is the dynamic arrangement of SCOs, depending on how the learners use them. This is typically referred to as sequencing, and allows the definition of complex and adaptive interactions as well as simple forms of personalized presentation.

The third aspect of SCORM defines how interactive SCOs can store and retrieve data from the run-time environment, as well as how different interactive SCOs can share data through a standardized application programming interface (API). This aspect of SCORM provides a unified approach that empowers content authors to create complex and integrated learning experiences that consider how learners interact with the learning material. This part of SCORM extends the data exchange protocols of existing content formats, such as the one defined by the IMS Question and Testing Interoperability (IMS QTI) specification.[29] IMS QTI specifies a standard interface for delivering tests and assessments as well as collecting the results.

One important characteristic of SCORM is that the specification is agnostic with respect to the capabilities of the delivery platform or the devices that learners use to access and interact with the learning material. This does not mean that every SCORM package can and will run on any compliant delivery platform, but that it is the responsibility of the content author to design, select, and arrange SCOs appropriately for any kind of SCORM-compliant run-time environment. This characteristic suggests the possibility of SCORM-compliant run-time environments for mobile learning experiences, and several solutions have been analyzed and discussed elsewhere.[30] Most of these solutions focus on SCORM’s content packaging and delivery aspect. These solutions discuss the automatic adaptation of SCOs to the capabilities of the device used by the learner. The underlying presumption is that learners in conventional and mobile learning environments will perform the same learning activities.

Nakabayashi, et al., present an approach for arranging and sequencing SCOs depending on the device capabilities.[31] The authors’ presumption is that learners perform different learning activities on mobile devices and in conventional ADL environments. Therefore, the approach separates SCOs for mobile, conventional, and mixed delivery modes.

Degani, et al., analyze the new requirements of mobile learning for content delivery and educational scenarios.[32] This analysis indicates differences in use cases for SCOs in mobile learning contexts and in conventional ADL run-time environments. The authors identify new use cases for SCOs that are currently unsupported by SCORM and suggest a mobile-SCORM (or m-SCORM) extension that primarily focuses on the sequencing and interaction aspects of SCORM.

The central limitation of the approaches of both groups of authors mentioned above is that both approaches require extensions of the SCORM specification, which carries with it a corresponding reduction in interoperability. In order to be able to benefit from these extensions, a run-time environment has to be substantially extended in its functional capabilities. For security and defense organizations, such extensions create a significant barrier for adoption, because the organizations rely on external vendors who need to implement these features into their products. Furthermore, the present approaches require the development of new educational material.

Challenges for Research

The research presented in this contribution addresses the need for scalable solutions that require no conceptual changes to existing educational material and can be achieved with minimal extensions of learning management systems. Interoperability standards were specifically considered in the present work in order to minimize the organizational risk posed by introducing new educational concepts.

The present research addressed the challenge of applying SCORM-compliant learning material for supporting mobile learning and extending the continuity of learning. The objective was to identify whether the reuse of existing learning material for mobile learning experiences can be achieved by adapting existing SCORM concepts. This challenge relates to the SCORM aspects of content delivery, interactive resources, and data persistency.

Educational Design Underpinnings

The research and application development has been guided by observations and conceptual differences between the uses of mobile technologies and conventional desktop computing. In order to understand the differences between mobile learning and conventional ADL solutions, it is necessary to define the unique characteristics of mobile learning. Mobile learning can be characterized as the processes (personal and public) of coming to know through exploration and conversation across multiple contexts, among people and interactive technologies.[33] This definition avoids mentioning portable devices, while it highlights the relevance of context as a key educational dimension that is specific to mobile learning.

From the perspective of educational design, contexts are dynamic elements of mobile learning processes. This is contrary to the approaches of conventional ADL systems that consider the context of learning as constant throughout a learning activity (similarly, SCORM does not explicitly consider the context of accessing and using SCOs). However, two contextual factors are central to the design of many ADL solutions: learning time and Internet connectivity.

SCORM-compliant solutions typically presume that learners have sufficient time for learning and are connected via stable broadband services to the Internet. However, mobile learners often experience a lack of needed data connectivity when they have the time for learning, or they find that other tasks are more important than learning when the data connectivity is available. Although this mutual dependency of time for learning and availability of data connectivity is not explicitly required by the SCORM specification, it is frequently reinforced by the design of compliant run-time systems.

Closely related to these two factors—time for learning and data connectivity—is a direct consequence of dynamic contexts: interruption. Whenever a context changes, the learning experience is interrupted. This marks a fundamental paradigm shift in educational technology. Conventional ADL solutions aim to minimize interruptions and distractions in order to support the learners’ focus on achieving the learning objectives. The interruptions of the learning process that persist are generally considered to be accidental incidents that can be ignored. For mobile learning, on the other hand, minimizing interruption is out of the question. Instead, interruption is one of the core principles of mobile applications, because interruptions can and will happen at any time.[34]

Instead of addressing these interruptions for generic SCORM modules, this article focuses on representing appropriate educational approaches within the concepts of SCORM. The solution is based on the instructional design concept of “micro learning.” [35] The core idea of micro learning is to minimize the organizational traits for learning and to provide short learning activities. Micro learning has three defining characteristics:

- No or short preparation for learning

- Short, self-contained learning activities

- Immediate assessment and feedback.

Within this concept, a learning activity is a task presented to the learner. Each task is self-contained, which means that it has no sub-tasks and creates no immediate dependencies to other tasks. “Short” refers to the time that is required to complete a task. Different from conventional educational design, micro-learning activities are not directly related to the achievements of learning objectives. Learners can achieve learning objectives only through repetition and in combination with other micro-learning activities.

The concept of micro learning is suited for mobile learning because it addresses procedural interruptions. Learners can interrupt and take up the learning at any time. Due to the short completion time and self-contained nature of each task, however, the impact of procedural interruptions remains limited.

Micro learning emphasizes the individual learning activity, but it does not conceptualize the overarching learning process. However, learners require feedback on the individual learning activity and on the overarching learning process for motivation and self-regulation.[36] Previous research has indicated that relatively simple statistical metrics are beneficial for learners to orient and organize their learning.[37] As micro learning already measures the learning performance for activity-level feedback, more complex perspectives on personal experiences can be aggregated from this data across activities. Two types of feedback metrics can be differentiated based on monitoring activities: performance metrics and effort metrics.[38]

Performance metrics refer to the qualitative level of success in completing the learning activities. Performance metrics include the number of correct responses or the speed of completing test items. The metrics are qualitative because they provide information about the quality of the learning in relation to the learning objectives. Effort metrics refer to the quantitative aspects of learning. These metrics indicate the effort a person invests in learning. Effort metrics are quantitative metrics because they only refer to the quantity of activities that were performed; they provide little information about the achievement of learning goals. Both metrics provide valuable information for evaluating learning and self-motivation. Both elements have been identified as essential elements of self-regulating and self-managing learning processes.[39] Such “learning analytics” can be used to broaden the perspective on the learning process beyond the level of single learning activities.[40]

Integration with SCORM Concepts

The core limitation of previous SCORM-compliant mobile learning solutions was the need to create new learning material for these solutions. In order to overcome this limitation, it was necessary to identify those SCO types that support the characteristics of micro learning. Furthermore, it was required that sufficient SCOs were already being used by security and defense organizations.

Test items are the only SCO type that supports all the characteristics of micro learning. Test items are questions that are used in tests, assessments, and exams. Typically, test items are stored in so-called “test item stores” or “question pools.” Tests and exams refer to these question pools for randomizing and personalizing the questions that have to be solved by the learners. The handling and exchange of test items, test item stores, and tests between ADL systems is specified by IMS QTI, which is a valid SCO format within SCORM. The presented solution uses IMS QTI test items outside of an assessment context in order to support self-practicing.

All IMS QTI test items have four elements:

1. A challenge

2. A response definition

3. Assessment rules for the response

4. Performance feedback.

The challenge is a question that a learner needs to answer. In micro learning terms, this refers to the task description. The response definition defines the interaction rules for providing the answer to the challenge. IMS QTI defines thirty-two different interaction types for responding to a challenge.[41] The specification defines for each interaction type the rules for assessing the learners’ responses. This allows automatic identification of fully correct, partially correct, and incorrect responses. For the different levels of correctness, IMS QTI allows the provision of predefined multi-media feedback.

The Mobler Cards App

This article introduces the Mobler Cards app for the Ilias LMS.[42] The app is a prototype for demonstrating the feasibility of introducing novel mobile learning concepts by building on existing educational material. Mobler Cards is a variation of the flash card learning concept that uses test questions for repetitive practice  on smartphones. The

Figure 1: Mobler Cards Core Interaction Flow.

unique feature of Mobler Cards is that it synchronizes itself with an LMS while offering all functions regardless of the connectivity of the learners’ devices.

After learners install the app on their smartphone, they connect to the LMS. After authentication, Mobler Cards identifies appropriate learning resources from the courses in which a learner is enrolled. For each of the learners’ courses, the app has two modes: a practice mode and a statistics mode.

The practice mode offers the typical flash card learning experience of a question and an answer extended by the immediate performance assessment and feedback that is required by the micro-learning concept. In order to be able to provide direct feedback on the learning performance, the learners have to show that they are able to answer the question correctly. This is a conceptual change compared to the original flash card learning approach, in which the learners have to imagine the correct response to a challenge while they can easily access the correct answer. To assess the learning performance, Mobler Cards relies on the assessment rules that the LMS provides for the test item. Each response can have three levels of correctness: “excellent,” if the correct answer has been provided; “partially correct,” if some parts of the response were correct; and “wrong,” if the provided response did not match the correct answer at all. Based on these levels, the app calculates and stores a score for the test item. In addition to the calculated feedback, the learners can compare their response with the correct solutions and can access predefined multi-media feedback, if it is available. The Challenge-Response-Feedback loop of Mobler Cards is illustrated in Figure 1.

The minimalistic concept of micro learning helps learners to directly analyze their learning performance through immediate task-related feedback. However, this approach does not allow learners to relate to the overarching learning objectives. Mobler Cards’ statistics mode serves this purpose and allows the learners to analyze their performances at the course level. Four analytical measures are provided to the learners: the number of questions handled during a twenty-four-hour period; the average score that has been achieved during the same period; the progress toward answering all questions correctly; and the average time for answering each question. The difference between the average score and the progress is that the average score includes partially correct answers as well as fully correct answers, while the progress measure includes only fully correct responses. In addition to these performance-based learning analytics, the app offers two learning badges that are based on the effort of learners using the app. The first learning badge indicates that the learner handled all available questions for a course. The second badge is awarded after the learner answers a large number of questions in one sitting. For both learning badges, the performance score is irrelevant. Figure 2 shows the Mobler Cards’ interfaces for performance metrics and learning badges.

Figure 2: Mobler Cards Interfaces for Performance Statistics and Learning Badges.

Mobler Cards is loosely connected to an LMS. The app works independent from the normal interaction patterns that are implemented by the LMS, but it authenticates learners with the LMS. For authenticated learners, the app synchronizes educational resources and activity monitoring data with the LMS, which allows course moderators to monitor the activities of mobile learners. The loose integration creates some platform and system independency, so learners can configure the app to use their preferred LMS. In order to achieve this flexibility, Mobler Cards relies on three Web services and open interoperability standards. The first service is the authentication service. This service is based on the OAuth protocol, which allows secure authentication and session validation without password transmission and constant session keys. The second service is the question pool service that selects the test item pools for the courses of the authenticated learner. This service provides only access to information that is accessible by the learners and sends data by using the IMS QTI Information Model. Finally, the experience tracking service collects data about the learning performance. This service accepts and stores monitoring data from the Mobler Cards app in compliance with the Experience API.[43] The system architecture of Mobler Cards is shown in Figure 3.

Figure 3: Mobler Cards Service Architecture.

Application Design Requirements

Mobler Cards is designed to complement existing Web-based ADL courses with exercises for repetitive practice. Three core requirements were considered for the app. These requirements are key for scaling up mobile learning in security and defense organizations.

First, the app needs to be integrated with the underlying LMS. Besides avoiding the hosting and maintenance of additional systems and infrastructure, this automatically allows the app to integrate mobile learning into existing education and training programs. Furthermore, the reuse of existing infrastructure allows the utilization of existing SCORM-compliant learning material whenever possible without additional overhead. This requirement also defines that no mobile learning activities are disconnected from other course activities.

Second, the app has to minimize the overhead for content authors. This lowers a significant barrier for scaling mobile learning in security and defense organizations by enabling content authors to use their knowledge of Web-based courses and Web-based assessment. This can be achieved by reusing the authoring capabilities of the LMS for content creation. A side effect of this is that it allows the repurposing of appropriate components of existing SCORM packages for mobile learning. Being able to use existing learning material for initial courses can significantly reduce the barriers to providing mobile learning offerings on a broad scale. Instead of the necessity for creating entirely new educational resources, this approach relies on adapting existing educational material.

Third, the app has to provide full flexibility for mobile learners in order to support the continuity of learning. From the learners’ perspective, an appropriate solution enables learning in suitable moments as they occur. These learning opportunities can vary in their duration and context. These opportunities include moments such as waiting for a bus or commuting on the train. This requirement also considers the issue of Internet connectivity as a factor in the learning experience. This means that learners should be able to access the learning material during extensive offline phases as well as when they are fully connected.

Mobler Cards optimizes the time frame that is available for learning by hiding most administrative tasks from the learners. This includes authentication, data synchronization, and course navigation. Furthermore, Mobler Cards allows the learners to access supportive features such as learning statistics at any time. This feature requires that all functions have to be implemented in the app instead of being provided by the LMS.

Proof of Concept

Mobler Cards has been used with the PFPC LMS at the ISN Zürich. The proof of concept addressed the validity of the first two requirements under real-world conditions and analyzed the feasibility of reusing existing educational material as well as the implications of adapting resources for mobile learning. The present study is based on two courses: “Introduction to NATO” and “Building Defense Organizations.” Both courses are available as SCORM packages that contain primarily text resources and test items.

The proof of concept should provide insights for extending SCORM content with mobile learning features. The proof of concept was separated into two parts. The first part analyzed whether existing test item stores can be used by the app. The second part analyzed the procedures for transforming SCORM content so it can be used with Mobler Cards.

The “Building Defense Organizations” course is used to teach basic knowledge about organizational structures and management strategies. The core course structure has been built with the Ilias SCORM editor and was extended with IMS QTI-compliant tests that use a test item store. The test item store for this course consists only of test items of the types “multiple choice,” “single choice,” and “sorting.” This allowed the use of these test item stores directly in Mobile Cards without modifications.

This part of the proof of concept reused existing learning material for mobile learning. Although all test items were successfully displayed within Mobler Cards, the analysis identified differences in the minimal design requirements for test items that were designed only for assessment and those that can be used for repetitive practice. The following content-related problems were identified:

- The main information of the test item is contained within the challenge or question

- Answering the questions correctly is relatively easy even with no or limited knowledge of the subject

- The feedback is not related to the subject of the question but to the performance of the learner

- Several test items treated multiple aspects within the same test item

- Long answer options of similar lists of terms.

The first three problems are related to the intended use and the new application scenario: while the intended use focused on a scenario that allows learners to provide answers only once, the new application scenario repeatedly confronts the learner with the same question. Learners have only a single chance to respond to a test item in the original scenario. The test was intended to identify whether a learner has mastered the main text resources of the course or not. Therefore, the test items were not very difficult, and when it was present, the additional feedback simply referred to the correctness of the response. The new application scenario allows learners to repeat test items as often as they wish. This provides the opportunity to present more challenging test items and feedback that is related to the subject matter of the course.

The last two problems are related to the display size and the mode of interaction. In conventional Web-based training scenarios on desktop computers it is relatively easy to read and distinguish between complex questions and answer options. Complex question-answer settings typically benefit from the simultaneous availability of complete information. In mobile learning scenarios, the screen real estate is far more limited, and learners typically cannot access information in the question and from the answer options at the same time. This problem increases with the complexity of the test item: the longer a question gets, the harder it is for learners to memorize it correctly while responding.

The “Introduction to NATO” course provides a general overview about the history and structure of NATO. The course has been created with the SCORM editor of the Ilias LMS. The test items for this course were either single- or multiple-choice items, but were embedded as interactive text. Therefore, this course had no test item store that can be used by Mobler Cards.

This part of the proof of concept addressed strategies for transforming existing and creating additional educational resources for mobile learning. This process included the following steps:

1. Transforming existing exercises into test items in a course-wide test item store. This step separates text material from test items. Many SCORM-compliant authoring solutions treat test items simply as interactive text content and not as a different resource type. Therefore, it is necessary to separate the different types of SCOs.

2. Extending the test item store with additional test items within the same logic as the existing exercises. This step extends the pool of test items by creating variations of the original test items based on the subject matter information in the available text resources. The objective of this step is to stimulate the learners’ attention rather than encouraging the memorization of answer patterns.

3. Identifying gaps in the exercises with respect to covered subject matter and creating new test items. This step seeks for gaps in the existing test items and the text resources. New test items were created whenever a gap was identified. The objective of this step is to cover all learning objectives with the test items.

4. Creating appropriate feedback and enriching test items. The final step focused on providing meaningful information to the learners that supports the answering of a test item. For incorrect or partially correct answers, the relevant passage of the text resource was provided to the learner. For correct answers, additional background information related to the test item is included.

Conclusions and Implications for Practice

The objective of the presented study was to analyze the constraints of reusing SCORM-compliant educational resources for mobile learning. This approach is considered as appropriate for lowering the barriers to mobile learning in security and defense organizations because they can directly integrate mobile learning into their existing ADL and blended learning strategies. Prior research suggested a direct translation of SCORM concepts of Web-based training into the mobile realm. This project analyzed the implications of transforming conventional Web-based training material for supporting mobile learners. Instead of treating mobile learning as a technical problem of content delivery, this contribution analyzed the underpinnings of the instructional design for mobile learning and grounded the technology development on these insights. This has resulted in the Mobler Cards app that applies the concept of micro learning to test item stores of SCORM modules for supporting individual practice.

Although the initial proof of concept showed the applicability of the solution in a real-world environment, it also indicated conceptual differences between testing for assessment and testing for practice. This article analyzed the required steps for adapting and extending the available SCOs for mobile learning without creating new requirements for the structure of educational resources. This can be considered to lower the barrier for adopting mobile learning in security and defense organizations because the demand for the creation of new content is relatively low.

Mobile Learning and ADL’s Experience API

Kristy Murray, Peter Berking, Jason Haag, and Nikolaus Hruska [*]

Introduction

The Advanced Distributed Learning (ADL) Initiative’s Sharable Content Object Reference Model (SCORM®) has been a staple of online learning standards since 2001. The SCORM specification was created by ADL to address interoperability challenges that existed prior to the wide adoption of touchscreen mobile devices such as smartphones and tablets. A new interoperability standard is needed to support training opportunities on mobile devices. ADL’s Experience API (xAPI) can now provide an option for mobile devices to support traditional online training scenarios as well as new types of informal learning opportunities.[44] However, a mobile SCORM capability involves more than simply ensuring technical compatibility with a new technology. It has new implications for instructional design as well as the potential to improve the overall learning experience.

Mobile learning is now a ubiquitous educational technology, one that introduces both exciting capabilities and complexity into the learning design process. However, there are very few guidelines for developing mobile learning. As a growing number of mobile innovations become available in the learning space, education and training technology thought leaders are now interested in how to effectively design programs for a variety of mobile learning scenarios. ADL is currently leading a project that will develop an instructional design framework along with guidelines and best practices to better support mobile learning design.

Mobile Learning

ADL believes that mobile learning should be viewed as a way to augment the learner through the use of ubiquitous technology and information, anytime and anywhere. Unlike other learning technologies, mobile learning is unique in that it can accommodate both formal and informal learning, in collaborative or individual learning modes. Many of the existing definitions of mobile learning in the education and training community are too learner-centric or too device-centric. However, ADL believes that both the learner and the device should be taken into consideration in order to provide a more flexible perspective on mobile learning. ADL currently describes mobile learning as “Leveraging ubiquitous mobile technology for the adoption or augmentation of knowledge, behaviors, or skills through education, training, or performance support while the mobility of the learner may be independent of time, location, and space.” [45] While mobile learning is not appropriate in all instances, we believe that it should be considered as a potential tool for any organization’s learning and training support infrastructure.

Overview of the xAPI

While the SCORM was successful in meeting the high-level requirements to solve the challenges within Web-based training systems, it was created prior to the widespread use of other learning environments and platforms such as mobile devices, intelligent tutoring systems, virtual worlds, games, and other social networking tools that augment the performance of today’s learner beyond formal training situations. Further, SCORM content was designed to be accessed and tracked via a learning management system (LMS). Mobile learning provides new opportunities to capture more than just a learner’s assessment score or course completion status. Learners, as well as education and training practitioners, expect new types of learning data to be captured and used within the aforementioned learning environments, in order to provide a personalized learning experience. They expect to learn informally and collaboratively, and to be able to use social networks as part of the learning experience. Users expect that their learning experiences will earn them credit, regardless of whether the learning activities are browser-based or not; thus there is a new requirement to free learners from being tied to an LMS. The xAPI gives learners, instructional developers, and instructors the opportunity to track and access data that far exceeds the current capabilities afforded by the SCORM.

Learning Activity “Streams” and the Learning Record Store

The data from social networks such as Facebook or Twitter are delivered in the form of “streams” that can be widely applied and syndicated in many contexts. The xAPI is an integrated approach to generate and capture learning stream data and then organize the data into meaningful learning contexts. It is an interoperable way to encapsulate and exchange learning data through the use of a learning-based activity stream. These activity stream data include defined actors, verbs, and activities associated with the learning experience so that the data exchanged maintain contextual meaning.

A simple example of an activity stream that relates to traditional web-based learning is: “I (actor) completed (verb) the Information Assurance course (activity).” An example that reflects a modern informal learning scenario could be “Jill (actor) posted (verb) to the Project Management course student forum (activity).” It is expected that learning communities will develop standard options for each of the three elements to address their own domain-specific requirements (e.g., the medical community, government, higher education).

Another element of the xAPI is the Learning Record Store (LRS), which is the experience tracking and storage component of ADL’s service-based approach and vision toward the Training and Learning Architecture (TLA). The platform-independent LRS design allows flexibility in that it may be a stand-alone service or a complementary component of a traditional LMS. A goal of the TLA is to ensure that past investments in SCORM can be maintained, while offering the benefits of platform neutrality, intermittent or disconnected network scenarios, and the capability to move learning out of the desktop browser.

The LRS will leverage additional services for content brokering, user profiles, and competency networks to build a customized suite of TLA services. In addition, the LRS allows authorized systems to retrieve previously recorded activity stream statements, which enables the development of advanced third-party reporting and data analytics tools. The xAPI also moves beyond the single-learner approach, allowing for team-based exercises, collaboration, and direct instructor intervention. This enables group learning, informal learning, and social learning – on any device or platform.

ADL’s Mobile Training Implementation Framework (MoTIF) Project

The mobile platform presents both unique opportunities and constraints for the design of learning content. Generally, these possibilities and limitations are not accounted for in traditional instructional design models. Many education and training practitioners are creating new mobile content and converting existing e-learning courses without any consideration of alternative learning methods (e.g., performance support) or leveraging the capabilities of the mobile platform (e.g., cameras, sensors, GPS). They are simply shrinking distributed learning courses down to fit the smaller screen.[46]

This is similar to what happened when the Internet was first introduced to the general public. Principles from print-based desktop publishing were applied directly to Web design, with no adjustment for the unique constraints and capabilities of the new medium. Therefore, many usability issues ensued, and opportunities for interactive engagement were lost.

ADL’s MoTIF project will explore new types of learning and design approaches that take advantage of the capabilities of the mobile platform. The project will produce both theories and practical interventions as its outcomes. The interventions will include such things as strategies, materials, products, and guidelines, but will also advance our knowledge about the characteristics of these interventions and the processes involved in designing and developing them. The specific types of outcomes that this project expects to produce include, but are not limited to, the following:

- Domain Theories: theories about the context and outcomes within the instructional design domain and mobile learning paradigm

- Design Framework: a workflow process and examples that will serve as a set of design guidelines for determining a mobile learning solution or strategy

- Design Methodologies: guidelines for how to implement the framework and the expertise that is required.

MoTIF is currently in its first phase. ADL will include community involvement to ensure that the results of the effort have durability and traction in the global learning space.

Mobile Learning and the xAPI

ADL’s MoTIF project is predicated on the current trend of people wanting to learn, explore, and interact with the world via mobile devices in new ways and more frequently than ever before. The xAPI enables opportunities for capturing the activities from diverse learning experiences. One of the primary goals of MoTIF is to focus on the unique capabilities of the mobile platform for learning and leverage features such as the mobile phone’s camera. For instance, if a learning design was predicated on students taking pictures of examples and counterexamples of a particular phenomenon, and then sharing and discussing these with fellow students, the xAPI enables the many activities in this learning design to be recorded and tracked in an LRS. If a training stakeholder wanted to know which students shared pictures, he or she could easily set up an xAPI statement that asserted that “Student 1 (actor) uploaded (verb) a picture to the student sharing repository (activity).”

Focusing on new learning opportunities that are enabled by, and most appropriate for, the mobile platform is another goal of the MoTIF project. Performance support can be thought of as an alternative learning opportunity. Instructional designers familiar with the SCORM typically choose to create intentional or preplanned learning (such as training courses) or performance support. With performance support, there is no explicit intention for the learner to retain knowledge or skills; they are simply tools to be used for “just in time” help in performing a task at that particular moment.

Mobile performance support apps provide some excellent examples of mobile learning, because users can access them anytime, anywhere. This freedom from a “tethered” desktop computer obviously lends itself well to all types of performance support. Mobile performance support apps are usually not delivered within an LMS, and they are often used in disconnected mode, perhaps as a stand-alone mobile app. The xAPI allows performance support tools to be tracked. A mobile performance support tool simply needs to be instrumented with the xAPI protocol in order to write data to the LRS related to where, how, and when the tool is used. Stakeholders can analyze these data to determine design modifications that may be needed to improve the effectiveness of the mobile tool, as well as to refine job skills and knowledge requirements.

In addition to mobile performance support, new approaches to instruction have emerged on the mobile platform. MoTIF intends to focus on these as well. These new approaches embrace the notion that people are learning from teachers, by themselves, and through collaboration in a community via social networks. The opportunity presented by this new paradigm is that the xAPI can track and capture learning artifacts in order to assess the learning, link the learner to experts or mentors, and provide recommendations based on granular learning data mined over time. In addition, the xAPI can capture a teacher’s activity in his or her Personal Learning Networks in a formal or blended environment. All of this interaction and collaboration is greatly enabled by mobile devices.

Conclusion

The learning approaches enabled by the xAPI can influence the design process, but a rigorous design process should, in turn, lead instructional designers to consider these xAPI-enabled approaches. It is particularly true that performance support can influence, and is influenced by, the design process. Mobile learning opens the door to a wide variety of effective performance support solutions. The xAPI enables these solutions to be assessed for effectiveness and monitored for usage. The analysis phase of the instructional system design process should not simply deal with what is to be trained, but whether mobile performance support can enhance the solution.

Exposing data through the xAPI provides a means for interoperability but also allows for innovation of learning content, experiences, and systems that is not easily afforded in the current learning model. The combination of mobile learning and the xAPI represents a new and powerful synergy. The xAPI permits the extensive capabilities of the mobile platform to be used to create new kinds of learning experiences and a much wider adoption of mobile-based performance support.

To find out how you can participate in the MoTIF project, contact the ADL Mobile Team at adlmobile@adlnet.gov.

The Role of Teaching History for a Nation-Building Process in a Post-Conflict Society: The Case of Macedonia

Biljana Popovska [*]

Introduction

The theoretical framework of this article is based on several published works whose content deals with history teaching as a key mechanism of justice in transitional societies.[47] Then, it draws from the work of the Center for Democracy and Reconciliation in Southeast Europe and their project “Clio in the Balkans” and the Joint History Textbook Project. In addition, there are materials from interviews with Macedonian and Albanian history teachers, experts, and government representatives selected from the participants in the Macedonian project presented at a United States Institute of Peace conference in Washington, D.C. in November 2005.

Unite or Divide?

In societies recovering from violent conflict, questions of how to deal with the past are sensitive, especially when they involve memories of widespread victimization, death, and destruction. It is very often the case that, in the wake of violence, political leaders and others seem to prefer social amnesia to the study of their society’s recent history, as they try to “move forward” and promote stability.

Therefore, the question arises of whether the teaching of history could help transitional societies become more democratic, and whether it can contribute to the development of empathy for, or even social cohesion among, former enemies in societies in which some groups were marginalized or were deprived of certain rights. Going further, can history teaching reinforce other transitional justice processes, such as truth telling and legal accountability for crimes that had been committed in the past? Finally, can teaching history promote belief in the rule of law, resistance to a culture of impunity, and greater trust in public institutions, including schools themselves?

The United States Institute of Peace conference convened on this topic in Washington, D.C. in November 2005 raised the issue of the content of post-conflict history education, which raised additional concerns about developing and adopting new history curricula. The issues to be considered include:

- Who decides what version(s) of history will be taught?

- What impact do those choices have on promoting stable, cohesive, and tolerant societies?

- What is the relationship between the (re)writing of history by academic historians and the development of secondary-school history textbooks?

- What impact do transitional justice processes have on the development of new secondary-school history textbooks and the way history is actually taught in schools?

One particularly problematic issue for post-conflict school systems in divided, multiethnic, and multilingual societies is determining which languages will be used to instruct schoolchildren. Although it is important for children in a multilingual country to learn the language (and, by extension, the culture) of other main groups of citizens in addition to their own mother tongue, having too many official languages in the schools can promote semi-literacy, poor performance, high repetition, and high dropout rates (as is seen in many African countries). At the same time, the rising importance of English as a lingua franca in the global marketplace is increasingly influencing language policies. Ethnic segregation or integration of schools also is an important structural aspect of education. When different ethnic groups are educated separately within the national education system, and especially when one ethnic (or gender) group receives more educational resources than another, such arrangements can convey important overt or hidden messages to students. Cole and Barsalou’s report says that some educational systems (such as Macedonia’s) permit the use of different history texts in ethnically segregated classrooms. In this case, history instruction in Macedonia is the same for Albanians and Slavs – but only in the sense that each group separately learns a remarkably similar history of victimization by the other, and each claims the same distinctions, such as a longer presence in the region. Another challenge is the decreasing priority given to the teaching of history and the humanities by post-conflict societies intent on preparing their students to compete in                                          the global marketplace, with more emphasis being placed on subjects seen to have practical value, such as foreign languages, math, science, technology, and vocational training. Thus the potential for schools to promote social reconstruction through history education in post-conflict societies is not being fully realized.

In post-conflict countries that receive substantial foreign attention, post-conflict reconstruction increasingly tends to be a transnational process, although “insiders,” or locals, are the ones who will have to live with (and take responsibility for) the long-term results of reconstruction and reform work. Outsiders who work on history education reform tend to be from nongovernmental organizations (NGOs) rather than transnational organizations or foreign governments, although some academics from foreign universities also are becoming involved. Often, however, powerful outside actors, particularly funders, view education as a domestic issue that “insiders” are best qualified to tackle. However, they consider other transitional justice processes, such as trials and elections, more deserving of their time and support. On the positive side, outsiders can get insiders engaged in reform processes that are too touchy for locals to handle on their own by bringing together groups that are otherwise disinclined to work together.

Pedagogical reform is attractive as a strategy because it may be less controversial or threatening than attempts immediately after conflict to change historical narratives through curriculum reform. But pedagogy reform is most effective when combined with curriculum reform.[48]

Transitional Justice and Reforming History Education

History education at the secondary school level also conceptually fits into some aspects of the work of transitional justice, which is another reason that Elizabeth Cole makes the argument that it should have “a place at the table” of transitional justice, meaning that it should be a part of transitional justice planning. Beyond the possibility that educational initiatives—both school-based and non-school-based (the latter would include museums, monuments and other cultural projects)—can help to carry forward the work of transitional justice mechanisms, the question remains of how to allow ordinary people to take ownership of processes of transitional justice. How does creating new approaches to history education relate to the specific goals of transitional justice? The purpose of history education in the modern state has generally been to transmit ideas of citizenship and both the idealized past and the promised future of the community. History and civics textbooks in most societies present an “official” story highlighting narratives that shape contemporary patriotism. Education helps articulate relations between state and society and sets the boundaries and terms of citizenship. If the representation of a group’s past is now recognized as an integral part of its identity, and this identity includes not only how one views one’s own group but also the groups designated as “other” or as adversaries, then understandings of history are crucial to a society’s ability to reckon with the difficult past for the sake of a more just future.

Cole also posits that revisions in history education methodology, as well as content, can serve the goal of deepening democracy by enhancing critical thinking and empathy skills, the willingness to question simplistic models and the ability to disagree about interpretations of the past, and their implications for present social issues without resorting to violence. Teaching, which presents history to students as an academic discipline with widely accepted standards and methodologies, rather than as a political tool or expression of nationalism, can help make the study of history “at its best … not simply a collection of facts, not a politically sanctioned listing of indisputable ‘truths,’ but an ongoing means of collective self-discovery about the nature of our society.” [49]

Cole also points out that one of the main insights from the USIP “Unite or Divide?” conference is that reforming the way history is taught in a time of political transition should take priority in many contexts over the history curriculum and over content revision, although pedagogy and teacher training are still very much under-addressed when history education reform is under discussion. Traditionally, history education has been a conservative project, closely controlled at the national level, intended to build group cohesion and loyalty to the state, and in the past it has focused on military and political history and the role of political and military elites (generally men). The emphasis was on learning facts, not on inquiry, discussion, debate, nor on drawing connections between problems in contemporary society and the past. Finally, the historical narrative was presented as something fixed whose legitimacy was unquestionable: there was little attempt to expose students to history as a field of science, to historiography, to the ways in which historical knowledge is constructed and understandings of which might allow students “to understand why accounts of the same event might differ.” [50]

While this has changed dramatically in Western Europe and North America, and is beginning to change in other regions, it is generally true of history content and pedagogy in transitional countries prior to and during conflict and/or dictatorship.

Secondary school history is much more politically sensitive than university-level history or the work of academic historians, and often lags behind them in terms of progressiveness and risk taking, because younger students are seen as highly impressionable and politically “pure.”

It is partially for this reason—although also for more practical reasons, such as limited resources and the complex procedures of program and curriculum reforms and textbook creation, production, and distribution—that the reformation of history textbooks in the aftermath of massive human rights abuses tends to take a long time. Elizabeth Jelin, another author Cole cites, suggests that the processes of incorporation of difficult issues into the education system have a very strong institutional component, since they require reaching a minimum consensus and an institutionally legitimized version of what took place (Jelin was writing specifically about the case of post-conflict Argentina). If the political conflict is not yet resolved, it is impossible to elaborate such a version of past events.

Other obstacles to reforming history education are more practical than political, but no less important to consider. One such obstacle is the low priority placed on history as well as literature, arts and social studies, compared to education in math, science, technology and vocational training in many parts of the world. This is true for developing countries with scarce resources for education as much as it is for developed nations.

Cole further points out the conclusions of the participants in the 2005 “Unite or Divide?” conference, who included South African, Rwandan, Macedonian, Colombian, Northern Irish, Sri Lankan, and Lebanese educators:

[H]istory teachers generally are under enormous pressure in post-conflict societies to play too many roles – from psychologist and guidance counselor to conflict resolution expert and mediator. Education reformers, especially those from the outside, also typically expect teachers to be agents of fundamental social change. Yet evidence from Northern Ireland shows that teachers are not comfortable being leading agents of social change, and they doubt that anything they teach can counter what the history students learn at home. In the most extreme cases, in highly charged political contexts where adopting new teaching approaches or texts may lead to threats to teachers’ physical safety, they will be especially likely to shy away from innovation.[51]

Finally, Cole points out that some surveys indicate that history as a school subject is one of the most unpopular subjects with students, and that adults also have unfavorable memories of their history classes – despite the fact that nonacademic vehicles for history, such as movies, popular histories, the creation of family genealogies, and historical sites are popular. This may indicate that reforming the history classroom to support the work of transitional justice will be a very difficult process. But it may also show that the problem is not that the past cannot be made compelling for students, but rather that it has rarely been done thus far.

However, there are some promising methodologies that lie in the rapidly expanding area of new media. Interactive multimedia websites can capture historical material in a variety of multilayered printed, visual, audio, and video formats, allowing students to hear the voices of witnesses to recent history and of scholars who use history in their work, including archaeologists and forensic anthropologists. These scholars are able to discuss how they use evidence to arrive at historical conclusions and describe how they conduct their own research through a vast array of sources, including newspapers, museums, official government and nonofficial and nongovernmental sites, in order to create their own primary documents by posting their own accounts of current events and to design and curate their own exhibitions online. New technological approaches to history teaching, fortunately, are not the only sources of hope for a wider role for history education in transitional justice. Two globalizing tendencies in history education (although not without problems of their own) may contribute to a history education that can better complement the work of transitional justice processes. These are “social-sciencization (an increasing focus on contemporary history and on society as opposed to the state) and a steady globalization.” [52] The global spread of these two tendencies is well attested by a recent report on dramatic changes in Chinese history textbooks. The new books focus more on society, economics, culture, and international history, and less on ideology, leaders, wars, and political history. In his comments on the new textbooks, education professor and textbook author Zhou Chunsheng refers to global trends in history education: “History does not belong to emperors or generals … it belongs to the people. It may take some time for others to accept this, naturally, but a similar process has long been underway in Europe and the United States.” [53] As the article points out, gains in the area of giving students access to the voices and experiences of many new actors are offset by other problems: Chinese history textbooks do not yet address disastrous periods in the history of the People’s Republic (such as the mass starvation that accompanied Chairman Mao’s “Great Leap Forward”), and may in fact deflect students’ attention from recent political events. But the new approaches to history may allow Chinese students to gain some of the skills and perspectives to approach, at some later point in their lives, the difficult past their parents and grandparents lived through.

Cole posits that another promising sign for the likelihood that history education can be the subject of meaningful and effective reforms is the gradual increase in attention paid to it and the increase in experts from a variety of disciplines (academic history, education, human rights, public health, and child and adolescent psychology) working in the field. There has been an increase in international learning about the field of history education, as there was in earlier decades in transitional justice in the development of NGOs, and of women’s and indigenous rights. Valuable studies—some of the first large-scale comparative ones in the field—were carried out by the Human Rights Center at the University of California, Berkeley, in close collaboration with local human rights and education specialists. At the level of transnational institutions, UNESCO has supported studies of education and its connection to post-conflict social reconstruction. In Europe, the Georg Eckert Institute, Euroclio, and the Thessaloniki-based Center for Democracy and Reconciliation in Southeast Europe have all been active in spearheading collaborative international projects to examine and reform history textbooks, curricula, and teaching practices. Conferences and exchanges for educators from different transitional societies to share their experiences and approaches, often including South African, Northern Irish, German, and Polish educators, have become more common.

Cole concludes her article with several recommendations, drawn primarily from the Carnegie Council study and the USIP-sponsored conference mentioned above. The first is that educators and historians should be involved from the beginning in planning transitional justice interventions. Educators as well as legal scholars and political and religious leaders should be given a stake in the work of transitional justice.

One prime topic might be to investigate what the relationship between historical/ history textbook commissions and history education reform can tell us about truth commissions and history education, since more is known about changes in history education in the aftermath of historical commissions. Although historical commissions are not yet counted among mainstream transitional justice mechanisms, they have been used in long-term efforts to reckon with the past, particularly in Europe, and are currently being tried in the context of several historical conflicts. Textbooks have been revised as a result of the findings of historical commissions, and are often cited as important components of furthering long-term reconciliation between Germany and several World War II-era victim groups, particularly France, Poland, and the Czech Republic. There is ample documentation of the changes in narratives that young people in Germany, France, Poland, and the Czech Republic have learned as a result of history education reform, particularly through the studies done by the Georg Eckert Institute, which both consults on history education reform as well as carries out research on history textbooks as they relate to conflict and democracy. The evidence from European historical commissions and history education reform may well be useful in trying to assess the effects that history education revision, as a long-term follow up to truth and reconciliation commissions (TRCs), may have on intergroup relations.

School Textbook Revision and Stability

Mirela-Luminita Murgescu, in an article titled “Rewriting School Textbooks as a Tool of Understanding and Stability,” says that in spite of all particularities of the different countries in the region, there are obviously some common problems of history teaching in all Southeast European countries.[54] The tendency to present an ethnocentric vision of history, which presents “the nation” as the main historical actor even for periods when it was not relevant, or for processes that happened at a local or regional level, has been stressed in many countries by the didactic division between national history and world history, which are taught in different grades, as distinct disciplines. Equally significant is the tendency to present all of history as a continuous heroic struggle of the home nation, which had to resist the hostility of the rest of the world. Such a sharp division between “we” and “they” allows textbook authors to explain present dissatisfactions as the outcome of the malicious action of external factors.

Another sensitive point in textbooks, especially in history textbooks, is the inability to rise above a one-sided vision of history. In this perspective, each textbook presents exclusively the version of history of its own political, national, or ethnic community. It is usually only the experience of one’s own ethnic group with other groups that is presented, but not the experiences the “others” have with “us.” The experiences of Croats during the interwar period in Yugoslavia are not included in Serbian textbooks, just as the experience of Serbian families during the Ustasha regime are at best presented at a superficial level in Croat textbooks.[55]

Another common feature of history teaching in the region in general is the authoritarian pattern of the teaching process, where the pupils are asked only to learn what the teacher teaches them, while the teaching of analytical patterns and of critical discussion are almost completely absent. It is obvious that such an outdated way of teaching history is less effective, yet the most simplistic historical information still influences the younger generation, and the lack of analytical abilities increases the likelihood that the youngsters will accept uncritically the most simplistic historical narratives.

Murgescu recommends that the first task should be the removal of erroneous, false, exaggerated and/or offending statements about other nations, peoples, social and ethnic groups. In this respect there exists significant experience in Western Europe, and the South East European countries also made some progress in the late 1980s and the 1990s.[56]

Removing the offending elements is just one part of the job. In fact, an expurgated history is less attractive, and the pupils will learn from their parents what they no longer learn from school. Therefore, there is a crucial need to insert something in the place of the hatred that had been removed. A first step in this direction would be to create a database of sources and narratives showing instances of cooperation between the neighboring peoples, the benefits of mutual understanding, and the disadvantages of hostility. Then, each party would be encouraged to include in the curricula and in the textbooks episodes or sources from this database.

Improvement of curricula and textbooks should include mainly history, but also other identity-forming disciplines such as civics, geography, literature, and religion, which often disseminate hostility even more effectively than history. For this purpose, the crucial level of action is that of the national school authorities (usually ministries and various national agencies).

In the past, and in other parts of Europe, there have been bilateral textbook conferences backed by the education ministries of the countries involved. These could provide a model for similar conversations in South East Europe. An institution similar to a think tank could also be formed in the region to consider the issue of textbook reform. Such an institution should include distinguished scholars with a wide knowledge of South East European history and culture, and also of European and world history. These scholars should be asked to design and discuss a list of “positive” elements that should be included in the curricula and textbooks; the final list should include short presentations for each entry, in order to allow non-specialists to realize the educational significance of such an approach. Murgescu mentions that the Center for Democracy and Reconciliation in Southeastern Europe, based in Thessaloniki, has already established two committees, which might provide the core of the think tanks that could be asked to prepare the general conference for curricula and textbook improvement in South East Europe.

Murgescu concludes by saying that it is a well-accepted fact that “images of the past commonly legitimate a present social order. It is an implicit rule that participants in any social order must presuppose a shared memory.” [57] In this respect, that the metaphor of school textbooks as the modern equivalents of the village storytellers is correct. Like story-tellers in non-literate societies, textbooks in history, geography and civics are responsible for conveying to youth what adults believe they should know about their own culture as well as that of other societies. There are, of course, many sources of socialization in modern society, but none compares to textbooks in their capacity to convey uniform, approved, even official versions of what youth should believe.[58]

Rewriting textbooks does not imply only composing new textbooks according to new values and educational aims. This is just one step in a long and difficult process. South East European societies have to manage not only a moderation of historical memory, but also the effort to encourage individual members of society to understand and support the necessity of such changes. Education must open “the ability to develop a historical consciousness” and not to distribute only, as today, “ready historical images.” [59]

Education programs should not concentrate only on history textbooks. For several years, history textbooks were a primary focus of attention, but similar importance should be placed on civics, geography, literature, music, and art classes. For instance, the literary texts selected for primary education textbooks are one of the most efficient tools for instilling not only knowledge, but also for creating attitudes and a social and historical consciousness. To conclude, an efficient educational policy regarding the rewriting of textbooks in South East Europe as a tool of understanding and stability should take into account some specific prerequisites like the political will of governments and political elites in the region to adjust their education policies according to European standards, and thus to base them on democratic values and civic attitudes. This should include both an extensive study of the collective and national memory in South East Europe, linked with analyses of educational policies and textbooks in the last two centuries, and the preparation of a set of concrete changes to be enforced.

The Case of Macedonia

How does all of the above translate to the case of Macedonia as a post-conflict society? This article has previously mentioned Prof. Violeta Petroska-Beshka’s work with history teachers in Republic of Macedonia. It is interesting to mention that the selected teachers were from particularly conflict-torn areas of Macedonia (Kumanovo, Tetovo, and Skopje), and some of them even belonged to the category of conflict-vulnerable citizens. Although ten years have passed since the conflict, I decided to interview these teachers since, aside from the actual work they did on the lesson, they went through a training program about understanding conflicts – thus, they can be said to have gone through the indirect process of a reconciliation program. However, my aim has been to see whether their perceptions have changed and how they actually view the practice of history teaching in the Republic of Macedonia. As one of the Macedonian respondents who had the role of an expert in the sessions brought up the phrases “Clio in the Balkans,” “Center for Democracy and Reconciliation in Southeast Europe,” and “Joint History Textbook Project,” it may be best to take as a starting point the work of the Center for Democracy and Reconciliation in Southeast Europe (CDRSEE).

The CDRSEE is a non-governmental, non-profit organization that seeks to foster democratic, pluralist, and peaceful societies in South East Europe by advocating principles of social responsibility, sustainable development, and reconciliation among the peoples in the region. Macedonia has been a contributor to that work through its scholars. One of the publications produced by the center, titled Clio in the Balkans, includes contributions from Macedonian scholars.[60] I shall refer to two articles, the first by Nikola Jordanovski, the second by Emilija Simoska. Their analyses should provide a full picture of history teaching in Macedonia and raise some issues for consideration.

In her Appendix on educational systems and history teaching in FYR Macedonia, Emilija Simoska says that national history is not a separate subject in any school in Macedonia, and it is only taught as a part of world history.[61] The main method of teaching is based on the pure presentation of historical information, with almost no evaluation and very little additional information, which is sometimes very difficult for the children to understand. According to Simoska, the Pedagogical Office, which is an expert body of the Ministry for Education, approves all of the textbooks used in history education. Simoska notes that the biggest problems arise with respect to the fact that the same circle of people who write the textbooks also approve them, which makes it difficult to break this circle in order to establish normal competition that would definitively result in better quality (this is the case in spite of an open bid for the writing of new textbooks). The main recent innovation has been in amending the laws on primary and secondary education to allow each school the right to use additional textbooks according to their choice.

Nikola Jordanovski analyzes the treatment of the common Yugoslav history in Macedonian school books with the intention to show how some crucial questions of the former nation’s recent history were addressed.[62] The vocabulary and the style used in the new books are inherited from the old school books, and the message has remained the same as well. Jordanovski states that interethnic violence is a subject one might think would take up more space in the schoolbooks, while there is actually only one passage on that issue in the school book for the fourth year of gymnasium, which will speak for itself when quoted in its totality: “Immediately after the occupation, the occupiers helped by the local traitors started a terrible terror through arrests, deportations, individual and group murders and massacres, mass exterminations of whole national collectivities….”

The Joint History Textbook Project. Another project of the CDRSEE related to Macedonia is the Joint History Textbook Project. As an introduction to this project I shall cite parts of the Preface to the Macedonian edition of Workbook 1 by Irena Stefovska, and will continue with analyses of the media discourse after the textbooks were launched as alternative history textbooks.

Ms. Stefovska writes, “The constant emphasizing of the ethnical, religious and language differences and not respecting the similarities resulting from the joint history leads to stereotyped understanding of the specifics of the history of the region as a whole and each of the nations separately thus creating an image that ‘our history is one and unique’.” [63] She further sates that the difference is not something that should be turned into a matter of life or death and something to help the elites rule. Rather, it is an issue we should seek an answer to on behalf of the ethics of knowledge.

Media Discourse. The media discourse that followed the launching of the textbooks in the Republic of Macedonia was rather positive, and promised that the use of the books as alternative materials would be a successful effort and would serve its purpose. However, the use of the acronym FYROM raised issues, and it was followed by defensive statements from contributors to the project saying that not only did they not have concrete input into the content, but in some cases they were even out of town when the textbooks were prepared.

The Voice of the Teachers. It was very interesting to note that none of the interviewees mentioned the use of any “Joint History Textbook Project” material. Only the Macedonian respondent who was involved in the project as an expert pointed out the importance of the project and the amount of work invested into it. Although this question was not explicitly asked, as the interview followed a semi-structured script, it allowed teachers opportunities to comment on it and talk about whether the Joint History Textbook Project had been a success in Macedonia.

One Macedonian respondent stated: “In the text books under the term culture they cover areas from the history of art (literacy and architecture) and do not use the term in its anthropological sense.” The same respondent went on: “The actual book writing was preceded by training about theory of conflicts but not getting deeper into conflicts in order to help the project participants [history teachers] make joined lessons on the assigned topics.” The respondent said that people are not aware of the stereotypes of the language used in the discourse. During the project they (project participants) did discourse analyses of how they could describe an event as interesting without expressing a particular sympathy for their own ethnic group. Gradually they all realized that the model of representing opinion is left to individual actors. In addition, the respondent said that the type of learning/teaching within the Macedonian state school system is doctrinaire and authoritarian.

One Albanian respondent stated that textbooks used for teaching historical events—especially those related to Albanian history—most often contain unconfirmed historical facts or divert some of the already established findings which are the result of previous research recognized by the world historiography. About the concept of sharing history, the following sample was obtained from an Albanian respondent: “We live with stereotypes. Young Albanians do not know [the] Macedonian language. It seems that the gap between Macedonian and Albanians is even wider after 2001.”

The final project product is a published book (one lesson of seven pages) but in a format that includes three alternative versions of the same lesson (the Macedonian version, the Albanian version, and the agreed version). The participants in the project hope that the book will become part of the official high school curriculum. They all agree that the most difficult area in which to reach a consensus has been related to the causes/reasons for the conflict in Macedonia. Some of the work was carried out through homework, and the teachers were grouped as extreme nationalists from different ethnic groups. These teachers had to exchange ideas and work on the material via e-mail correspondence.

An Albanian project participant claimed that after the project he changed a lot, and felt that it was positive experience for him as it changed his point of view towards Macedonian history teachers. However, a Macedonian expert responded as follows: “From what is invested in the teachers, there are weak results. We should all follow the motto of lifelong learning. Values and skills are important.”

When asked about the status of history as a school subject, one participant responded, “The curriculum (plans and programs) are prepared by the BRO (Bureau for Development of Education). They have been changed three times since the independence of 1991. After the conflict new textbooks have been made, and for the first time the history of Albanians in the region has been included. There is little information about Vlachs in Macedonia. The greatest portion is the national history of Macedonia, then something about Albanian history, and then comes the rest.”

In response to the question of “whether the lessons made during the ‘Understanding History’ project are used in the classrooms,” the general answer was negative. As one Macedonian respondent put it, “To use materials in a highly centralized educational system—especially history lessons, in spite of the freedom given to the teachers to be 30 % creative—is not done by the teachers. So they do not supplement their lessons. What they have to teach is mostly high politics, facts and dates, heroes. It is not very interesting for the students, and it is sometimes torturous to have to remember facts.”

Responses from teachers and parents depended on the location of the schools, which means additional research would be of use to include places affected by the conflict, where students had to leave their houses and move away, in comparison to places that did not experience conflicts, and places where there is no contact with the Albanian ethnic community. As one Macedonian participant said, “During the conflict the emotions must have been stronger but there is a time distance now.”

Conclusions

There is a quite a lot of work invested in Macedonia related to the teaching of the nation’s history. But it seems that the subject gets neglected when it comes to actual classroom practice. Albanians seem to view history teaching in terms of percentages. Some of them use additional materials to teach about Albanian history, and even use books imported from Albania. The Ministry of Education allows the use of additional materials, but it seems that teachers are not particularly eager to use them. Sometimes both Macedonian and Albanian history teachers form their opinion based on word of mouth rather than on reading books. They all agree that silence is a preferred outcome instead of heated debates. It took them a year to write just a few pages of one combined lesson on the recent past, even a lesson that contained three versions (Macedonian, Albanian, and a consensus one). Sometimes, the joint work lasts only as long as the project lasts, and it seems like there is no initiative for sustainability.

Elizabeth Cole’s message and Violeta Petroska-Beshka’s project both represent efforts to bring people of different ethnicities together to teach them to be open and embrace diversity. Macedonians and Albanians live on the same territory. How can they both look ahead and determine what would make the best state for them going forward? There are many opinions and answers to this question. Each party has its own opinion.

The efforts invested into the projects discussed above have produced some positive results, as we can see from the interviews. But we need to move to the next level. One of the interviewees said, “People will make the effort only if somebody from outside puts them together.” This means that civil society in Macedonia is as yet underdeveloped for the task at hand, so external help is necessary. This problem of collective action means that the people of Macedonia are not able to coordinate their actions. Looking to history, and particularly to civil society in history, may give Macedonia models to follow that can help the nation move forward – an effort for which a new approach to teaching the nation’s history will be necessary.

Elizabeth Cole’s recommendations and the findings drawn from the interviews are a good starting point. There is a necessity to work along these lines, and such work should be institutionalized (such as the revision of history books). Some compromises have been made in the country by signing the Framework Agreement. What is next? Ten years have passed since the period of conflict, but the potential for a relapse still exists. What is the next level? Embracing concepts of civil society is not enough. Contacts between ethnicities are not enough. Either scientific encouragement is needed, or help from outside is required to address the problem of collective action. The answer is to decide to let civil society develop and then act on that decision.

GAO Report on Higher Education

Use of New Data Could Help Improve Oversight of Distance Education [64]

Highlights

Main Findings

While distance education can use a variety of technologies, it has grown most rapidly online with the use of the Internet. Online distance education is currently being offered in various ways to students living on campus, away from a campus, and across state lines. School offerings in online learning range from individual classes to complete degree programs. Courses and degree programs may be a mix of face-to-face and online instruction – “hybrid” or “blended” instruction. Online asynchronous instruction—whereby students participate on their own schedule—is most common because it provides students with more convenience and flexibility, according to school officials. In the 2009-2010 academic year, almost half of postsecondary schools offered distance education opportunities to their students. Public 2- and 4-year schools were most likely to offer distance education, followed closely by private for-profit 4-year schools.

Students in distance education enroll mostly in public schools, and they represent a diverse population. While they tend to be older and female, and have family and work obligations, they also include students of all races, current and former members of the military, and those with disabilities. According to the most current Education data (2007-2008), students enrolled in distance education studied a range of subjects, such as business and health.

Accrediting agencies and schools assess the academic quality of distance education in several ways, but accreditors reported some oversight challenges. Federal law and regulations do not require accrediting agencies to have separate standards for reviewing distance education. As such, accreditors GAO spoke with have not adopted separate review standards, although they differed in the practices they used to examine schools offering distance education. Officials at two accreditors GAO spoke with cited some challenges with assessing quality, including keeping pace with the number of new online programs. School officials GAO interviewed reported using a range of design principles and student performance assessments to hold distance education to the same standards as face-to-face education. Some schools reported using specialized staff to translate face-to-face courses to the online environment, as well as standards developed by distance education experts to design their distance education courses. Schools also reported collecting outcome data, including data on student learning, to improve their courses.

Education has increased its monitoring of distance education but lacks sufficient data to inform its oversight activities. In 2009, Education began selecting 27 schools for distance education monitoring based on an analysis of risk factors, but it did not have data to identify schools with high enrollments in distance education, which may have impeded its ability to accurately identify high-risk schools. Between 2011 and 2013, Education’s National Center for Education Statistics (NCES) will start collecting survey data on the extent to which schools offer distance education, as well as enrollment levels. However, the department’s Office of Federal Student Aid (FSA), responsible for monitoring Title IV compliance, was not involved in the process of deciding what distance education information would be collected; therefore, it did not provide input on what types of data could be helpful in oversight. Further, FSA officials said they do not yet have a plan on how they will use the new data in monitoring.

Higher Education

As the largest provider of financial aid for postsecondary education, with about $134 billion provided to students during fiscal year 2010 under Title IV of the Higher Education Act of 1965, as amended (HEA),[65] the U.S. Department of Education (Education) has a considerable interest in distance education [66] – education that uses technology to provide instruction to students who are separated from the instructor. Students participating in distance education are eligible for federal financial aid in the same way as students taking traditional courses. This aid is available through Education in the form of grants, loans, and work study wages. While distance education has extended the opportunity to continue higher education learning and complete degrees to many students, including nontraditional students with work and family obligations as well as military personnel, its rapid growth in popularity presents challenges for ensuring both its quality and the oversight of federal funding. Congress has included a number of provisions in the HEA, as amended, such as oversight requirements for accrediting agencies, in recognition of the increasing role of distance education in higher education and to address concerns regarding the academic quality of distance education given the rapid growth in this area. Education’s Office of the Inspector General (OIG) has also raised concerns over the past few years about the ability of schools to verify student identities and ensure enrolled students are engaged in academic activities given the limited or no face-to-face contact with distance education students.

To provide a national perspective on the nature of distance education and its current level of oversight, we were asked to describe (1) the characteristics of distance education today, (2) the characteristics of students participating in distance education, (3) how the quality of distance education is being assessed, and (4) how Education monitors distance education in its stewardship of federal student aid funds.

To conduct our work, we reviewed and analyzed relevant federal laws and regulations, literature, studies, and program documents, and consulted with subject matter experts. We analyzed data from Education’s Integrated Postsecondary Education Data System (IPEDS) and the National Postsecondary Student Aid Study (NPSAS) databases to determine the school and student characteristics involved in distance education. We also reviewed information from a 2008 report by Education’s National Center for Education Statistics (NCES) and a 2010 industry report to obtain a national perspective on distance education practices and offerings at postsecondary schools. We conducted site visits to Florida, Minnesota, and Puerto Rico to visit schools and interview state agency officials. We selected these sites based on various factors, including the level of state data collected and an industry summary of states’ policies for approving distance education. We interviewed officials from higher education organizations, accrediting agencies (three regional, two national, and one specialized), and Education to determine their role in overseeing the delivery of distance education. Finally, we interviewed school officials from a nongeneralizable sample of 20 postsecondary schools to obtain more information on current practices in and the specific types of programs and coursework being offered through distance education. Our criteria for school selection included total enrollment, change in enrollment over time, school sector (public, private nonprofit, and private for-profit), and geographic diversity, among other factors. For more information on our scope and methodology, see appendix I.

We conducted this performance audit from November 2010 to November 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Background

Distance education is not a new concept, but in recent years, it has assumed markedly new forms and greater prominence. In the past, distance education generally took the form of correspondence courses— home study courses completed by mail. Distance education today can take many forms and is defined by federal law and regulation as education that uses one or more technologies (such as the Internet or audio conferencing) to deliver instruction to students who are separated from the instructor and to support regular and substantive interaction between the students and the instructor.[67] Instruction provided through the Internet—or online—may be synchronous (simultaneous or “real time”) or asynchronous, whereby students and the instructor need not be present and available at the same time (see Figure 1).

Figure 1: Illustration of Synchronous and Asynchronous Learning Methods.

Oversight

In general, for their students to be eligible for federal student aid funds under Title IV programs, schools must be legally authorized by a state, accredited by an agency recognized by Education, and be found eligible and certified by Education.[68] State governments, accrediting agencies, and Education form the program integrity triad established by Title IV of the HEA to oversee postsecondary education.[69] The state authorization role is primarily one of providing consumer protection through the state licensing process, while the accrediting agencies are intended to function as a quality assurance mechanism. In certifying a school for participation,[70] Education is responsible for determining the financial responsibility and administrative capability of schools and is also responsible for monitoring to ensure compliance with Title IV requirements.

Accrediting agencies, private educational associations set up to review the qualifications of member schools, are the primary overseers of schools’ academic quality. Accreditation is a peer review process that evaluates a school against the accrediting agency’s established standards. An institutional accrediting agency assesses a school in its entirety, including resources, admissions requirements, services offered, and the quality of its degree programs, while a programmatic accrediting agency reviews specific programs or single-purpose schools. A school’s accreditation is re-evaluated every 3 to 10 years, depending on the accrediting agency. If a school makes a substantive change to its educational programs or method of delivery from those that were offered when the agency last evaluated the school, the agency must ensure the change continues to meet standards.[71] Schools may lose accreditation if their accrediting agency determines that they no longer meet the established standards. While Education does not have the authority to dictate the specifics of an agency’s standards, the department recognizes accrediting agencies by reviewing and assessing their standards in various areas required by statute, such as student achievement, curricula, and student support services.[72]

Education’s Office of Federal Student Aid (FSA) is responsible for monitoring the over 6,000 postsecondary schools participating in Title IV programs to ensure their compliance with applicable statutory and regulatory provisions and to ensure that only eligible students receive federal student aid. The postsecondary school types include the following:

- Public schools – schools operated and funded by state or local governments, including state universities and community colleges

- Private nonprofit schools – schools owned and operated by nonprofit organizations whose net earnings do not benefit any shareholder or individual

- For-profit schools – schools that are privately owned or owned by a publicly traded company and whose net earnings can benefit a shareholder or individual.

Education fulfills its school monitoring responsibilities through four main activities. First, it determines the initial eligibility of schools to participate in the federal student aid programs, as well as recertifies that eligibility periodically. Second, as part of ensuring compliance, FSA staff conduct program reviews of a select number of schools each year where they examine school records, interview school staff and students, and review relevant student information, among other things. FSA issues reports on these reviews, which include information on areas where a school was found to be in violation of the Title IV requirements. Third, schools are required to employ independent auditors to conduct annual compliance reviews and financial audits, which are then submitted to Education. Finally, Education’s OIG conducts its own audits and investigations of schools to identify and combat fraud, waste, and abuse and makes recommendations to the department. Education may assess liabilities and/or impose fines or other sanctions on schools found in violation of Title IV requirements.

Brief History of Statutory Provisions Related to Distance Education

Over the past two decades, Congress has made several changes to the Higher Education Act of 1965 that have affected schools’ offering, and accrediting agencies review, of distance education. To combat cases of fraud and abuse at postsecondary schools that primarily delivered distance education though correspondence courses, Congress stipulated in 1992 that schools were not eligible to participate in federal student aid programs if more than 50 percent of their courses were offered by correspondence or if more than 50 percent of their students were enrolled in correspondence courses. Furthermore, students enrolled in telecommunications courses were considered enrolled in a correspondence course if the sum of the telecommunications and correspondence courses equaled or exceeded 50 percent of the total courses offered at the school.[73] In 2006, Congress excluded telecommunication courses from the first two rules and eliminated the requirement that telecommunication courses be considered correspondence courses if the sum of both exceeded 50 percent of the total courses offered, allowing schools to expand distance education offerings while maintaining their eligibility to participate in Title IV programs.[74],[75]

While Congress has required that accrediting agencies apply and enforce standards with regard to distance education offered by schools, including that such reviews be included in their scope of accreditation since 1998,[76] it added additional requirements in 2008 through the Higher Education Opportunity Act (HEOA).[77] Although not required to have separate evaluation standards, accrediting agencies that have or are seeking to include distance education in their scope of review must demonstrate to Education that they effectively address the quality of a school’s distance education program in the same areas they are required to evaluate a school’s other educational offerings.[78] The agencies must require schools offering distance education to have processes to establish that the student who registers in a distance education course or program is the same student who participates in the program.[79] Additionally, agencies must ensure that accreditation team members whose responsibilities include evaluating distance education are well trained and knowledgeable with regard to distance education.[80] Finally, all accrediting agencies are required to monitor the growth of programs at institutions that are experiencing significant enrollment growth.[81],[82]

Distance Education Has Become Common in All Sectors and Is Offered through a Range of Programs and Courses

Online Distance Education Has Grown Dramatically and Is Offered in a Variety of Ways

While distance education can use various technologies, it has grown most rapidly online with the use of the Internet to support interaction among users. With the emergence of the Internet and expansion of Internet-based communication technologies, distance education today is a common phenomenon and widely used throughout higher education. Moreover, the term “distance education” no longer connotes only instruction separated by physical distance, since many distance education courses—specifically online courses—are offered to students living on campus as well as away from a campus and across state lines. School offerings in online learning range from individual classes to complete degree programs. Individual courses as well as degree programs may also be a mix of face-to-face and online instruction – often referred to as “hybrid” or “blended” instruction. Furthermore, an online class may be synchronous (simultaneous, real-time instruction), or asynchronous, where students and the instructor are not present and available at the same time.

According to a 2008 study on distance education conducted by Education, postsecondary schools of all types offer a variety of distance education courses.[83] Specifically, for the 2006-2007 school year, 61 percent of 2-year and 4-year schools reported offering online courses, 35 percent reported hybrid/blended courses, and 26 percent reported other types of distance courses. The study also suggests that the majority of schools offering distance education used asynchronous Internet technologies. Specifically, 92 percent of the degree-granting postsecondary institutions offering distance education in 2006-2007 reported using asynchronous Internet technologies to a moderate or large extent, compared with 31 percent of schools that reported using synchronous technologies to a moderate or large extent. In our interviews at the schools we selected, officials said that online, asynchronous instruction was also their predominant method for providing distance education and that this type of instruction meets students’ need for flexible schedules. For example, over half of the school officials we interviewed noted that many students taking classes online are working adults or active duty military service members who would otherwise be unable to continue or complete their studies.

The use of distance education, particularly online learning, has grown dramatically in recent years. According to a 2010 industry survey, online enrollment in degree-granting postsecondary schools has continued to grow at rates far in excess of the growth for total enrollment in higher education.[84] Survey results indicate that over 5.6 million students were taking at least one online course during the fall 2009 term – an increase of nearly 1 million students over the number reported the previous year and an increase of 21 percent, as compared with the less than 2 percent growth in the overall higher education student population. The survey also suggests that nearly 30 percent of higher education students were taking at least one course online.

Such remarkable growth may be attributed to institutional efforts to expand access to more students, alleviate constraints on campus capacity, and the desire to capitalize on emerging market opportunities and compete with other schools. According to Education’s 2008 study on distance education, which includes online and other forms of distance education, the top four factors affecting postsecondary schools’ decisions regarding distance education offerings are (1) meeting student demand for flexible schedules; (2) providing access to college for students who otherwise would not have access due to geographic, family, or work-related reasons; (3) making more courses available; and (4) seeking to increase student enrollment. Several of these factors, such as providing access to more students, were also cited by school officials we interviewed. For example, one school we visited had increased access to education by establishing over 20 “cyber-centers,” including one on a National Guard base and another in a shopping mall where students can access computers with Internet capabilities and participate in online courses as well as complete assignments and take exams. Additionally, officials at two of the schools we interviewed noted that on-campus students were registering for online classes, instead of face-to-face classes that were otherwise full or scheduled for times of day that conflicted with their personal schedules. Furthermore, one school we interviewed provided flexibility to its students by allowing them to begin and complete courses at their own pace.[85]

While cost savings might be a factor, none of the school officials we spoke with cited cost savings as the primary reason for providing online distance education courses and programs. Moreover, they said students taking distance education courses, including online courses, are generally charged the same tuition and fees as students taking face-to-face courses.[86] These officials cited various costs associated with developing and expanding online distance education offerings, such as the purchase of hardware and software (which includes a learning management system), course development, faculty training and salaries, and the provision of student support services. They also said online instruction is not necessarily less expensive to provide, in part, because schools have to provide similar support services to both online students and classroom students – such as tutoring, library access, and (virtual) faculty office hours. For example, officials at three schools mentioned one of the major expenses associated with online distance education is providing off-hours library access or tutoring. Also, almost all the officials said it is often difficult to isolate the costs of online courses from the costs of providing traditional courses. Professors generally teach both online and face-to-face course sections, and the infrastructure developed for online distance education, such as the online learning management systems, can also be used by students and instructors participating in face-to-face instruction.

Figure 2:   Percentage of 2- and 4-Year Postsecondary Schools Offering Distance Education, by Sector.

A Wide Variety of Schools Provide a Range of Distance Education Courses and Programs

Schools of all types reported offering distance education, according to data collected by Education through its annual IPEDS survey.[87] Specifically, during the 2009-2010 school year, 46 percent of all Title IV eligible schools reported that they offered distance education opportunities to their students. Figure 2 shows the variation among these schools by sector and program length.

As shown in figure 2, public schools, both 2- and 4-year, were more likely to offer distance education opportunities than private nonprofit or for-profit schools. Among public schools, distance education was more likely to be offered at 2-year schools rather than 4-year schools. One school official we spoke with attributed this likelihood to the increased number of students at 2-year schools, given the weak economy and limited capacity at 4-year public schools.

With regard to minority-serving institutions and institutions with specific high minority concentrations,[88] IPEDS data indicate that these institutions are as likely or more likely to offer some distance education than all schools combined, with the exception of Hispanic-serving institutions. For the 2009-2010 school year, more than 60 percent of Historically Black Colleges and Universities and Tribal Colleges and Universities offered distance education opportunities to their students, compared with about 46 percent of institutions overall. Furthermore, 49 percent of Asian/Pacific Islander/Native Hawaiian-serving institutions offered distance education to their students. Among Hispanic-serving institutions, just over 30 percent of these schools were offering distance education (see fig. 3).[89]

Figure 3: Percentage of Minority-Serving Institutions Offering Distance Education.

With regard to the size of schools that offer some distance education, the IPEDS data suggest that larger schools—as defined by enrollment—are more likely to offer distance education opportunities than smaller schools. Specifically, 23 percent of schools with fewer than 1,000 students offered distance education, while 96 percent of larger schools—those with 20,000 or more students—did so (see fig. 4).

Figure 4:   Percentage of Postsecondary Schools Offering Distance Education,

by School Size, 2009-2010

The 2008 distance education study by Education provided additional insights on the extent and nature of distance education offerings by school type, sector, and size. In terms of full degree and certificate programs, the study indicated that in the 2006-2007 academic year, about a third of all degree-granting schools offered entire degree programs or certificate programs through distance education. Additionally, public schools were more likely to offer a degree or certificate program entirely through distance education than were private schools. Larger schools were also more likely to offer a degree or certificate program entirely through distance education than smaller schools (see table 1).

Table 1:    Percentage of All Title IV Degree-Granting Postsecondary Schools

Offering Degree or Certificate Programs Totally through Distance

Education in Academic Year 2006-2007

School type and size

Percentage of schools that offered college-level degree or certificate programs through distance education

All schools

32

School type

Public, 2-year

45

Public, 4-year (undergraduate and graduate programs)

58

Private for-profit, 2 year

7

Private for-profit, 4-year (undergraduate and graduate programs)

27

Private nonprofit, 4-year (undergraduate and graduate programs)

24

Size of school (by enrollment)

Less than 3,000

19

3,000 to 9,999

49

10,000 or more

67

Source: U.S. Department of Education.

Students in Distance Education Enroll Mostly in Public Schools and Represent a Diverse Population

Most Distance Education Students Attend Public Schools and Study a Range of Subjects

Our analysis of the NPSAS [90] data for the 2007-2008 academic year showed that of the estimated 5 million [91] postsecondary students who have taken distance education, participation was most common among students attending public schools. These students enrolled in a range of academic fields of study.

Most distance education students enroll at public schools. As might be expected, most undergraduate and graduate students taking distance education courses or programs were enrolled at public schools, followed by private nonprofit and private for-profit schools (see fig. 5).

Figure 5: Percentage of Students Enrolled in Distance Education, by Sector.

Figure 6:   Average Age of Undergraduate and Graduate Students, by Distance

Education Enrollment Status.

Distance education students enroll in a variety of fields of study. Both undergraduate and graduate students taking distance education courses or programs had higher rates of enrollment in the fields of business and health. Undergraduates taking distance education courses and programs also often majored in the humanities (liberal arts), while graduate students often studied education.

While Students in Distance Education Tend to Be Older and Female, and Have Family and Work Obligations, They Are Also a Diverse Population

According to our analysis of 2007-2008 NPSAS data, distance education students varied somewhat from students who did not enroll in distance education in that they tended to be somewhat older and female, and have family and work obligations.[92] Moreover, students who are participating in distance education represent a diverse population that includes students of all races, current and former members of the military, and students with disabilities.[93] Some of these characteristics are consistent with what we reported in our 2002 testimony on distance education [94] and also were corroborated in our recent interviews with selected schools for this report.

Distance education students tend to be older. As figure 6 shows, undergraduate and graduate students who took distance education courses or programs were about 3 years older, on average, when compared with students who did not take any distance education courses.

Distance education students are more often female. Women represented about 61 percent of undergraduate students who took distance education courses or programs, compared with about 56 percent of undergraduates who took no distance education, and about 57 percent of undergraduates overall. For graduate students, the percentage of students taking distance education courses or programs who were female was about 65 percent, which was higher than those who took no distance education (59 percent) and the overall percentage of graduate students who were female (61 percent).

Distance education students more often have family obligations. Figure 7 shows that undergraduate and graduate students who took distance education courses or programs were more often married and had dependents than those taking no distance education courses.

Distance education students more often work full time. A higher percentage of students who took distance education courses or programs worked full time when compared with students who did not take any distance education courses. This difference was greatest among graduate students – about 74 percent of the students who took distance education courses or programs worked full time compared with 57 percent of students who did not take any distance education courses.[95] For undergraduates, the figures were 45 percent and 31 percent, respectively.[96]

Figure 7:   Percentage of Undergraduate and Graduate Students with Family

Obligations, by Distance Education Enrollment Status.

Students of all races and ethnicities participate in distance education to some extent.[97] Postsecondary students of various races and ethnicities participated in distance education (see fig. 8).[98]

Current and former members of the military enrolled in postsecondary education participate in distance education. Forty-five percent of active duty service members, 29 percent of reservists, and 30 percent of veterans enrolled in postsecondary education took distance education courses or programs.[99] In addition, of those enrolled in postsecondary education, 42 percent of active duty service members with a disability and 29 percent of veterans with a disability took distance education courses or programs.[100] Taken together, active duty service members, reservists, and veterans represented about 7 percent of all students taking distance education courses and programs, compared with 4 percent of students who took no distance education.

Figure 8:   Percentage of Students in Each Race and Ethnicity Group Who

Participate in Distance Education.

Students with disabilities participate in distance education. Twenty-one percent of all students with disabilities, including members of the military and civilians, enrolled in distance education courses or programs.[101] Further, 25 percent of students with disabilities affecting their mobility took distance education courses or programs. Students with disabilities represented 10 percent of all students taking distance education courses and programs, while students with mobility disabilities represented about 3 percent.[102]

Many of these student characteristics were also noted by school officials we interviewed. These school officials reported that they collect data such as age, gender, and race and ethnicity of their students. The demographic data provided from schools generally showed similar student characteristics as that suggested by the 2007-2008 NPSAS data – that distance education students tend to be older and female, and have work and family obligations. Officials of at least three of the schools we selected indicated that many of their students taking classes online are veterans or students serving in the military. While at least three schools reported tracking students who identified themselves as having disabilities, at the time of our interviews, none of these schools indicated that they had determined how many of these students were taking online distance education classes. Officials at one of these schools, however, conducted some analysis after our interview and reported that about 3 percent of their students enrolled in the past year had documented disabilities. These students took, on average, 15 percent of their classes online.[103]

While most of the schools where we conducted interviews collected demographic data on their students, including those taking courses online, less than half of these schools have compared the demographics of students taking completely online courses with those taking face-to-face courses. Officials at five schools mentioned that comparing data on students can be difficult, in part, because students can take courses or degrees through a mix of instructional modalities – including completely online, hybrid/blended (mix of online and face-to-face), and completely face-to-face. For example, officials from one private nonprofit 4-year school that offers completely online as well as blended courses and degrees said that it is difficult to collect comparison data because the school’s administrative records do not differentiate online students from those who enroll in both online and campus-based courses.

Accreditors and Schools Assess the Academic Quality of Distance Education in Several Ways, but Accreditors Reported Some Oversight Challenges

Accrediting Agencies Examine the Quality of Distance Education in Various Ways but Reported Some Challenges

Accreditors we interviewed have various procedures to examine schools’ distance education programs, but some accreditors reported they face challenges.[104] Federal law and regulations require accrediting agencies to have standards that address student achievement, curricula, faculty, and student support services, among other areas. In addition, accreditors must ensure that schools have a process in place to verify registered students are doing their own work by using methods such as secure logins, passwords, proctored examinations, or other technologies. However, accrediting agencies are not required to have separate standards for distance education.[105] As such, accreditors we spoke with who accredit both distance education and face-to-face programs use the same standards for both, although they differed in the practices they used to examine schools offering distance education.

The accreditors we spoke with conduct reviews of schools’ distance education programs according to the accreditors’ own standards. For example, to address the effectiveness of a program, accreditors may review such measures as student retention rates, completion/graduation rates, student satisfaction, placement rates (if applicable), and various measures of student learning. The three regional accreditors we spoke with give schools the responsibility for determining the best way to assess student learning for both face-to-face and distance education programs. However, both national accrediting agencies and the specialized accreditor we spoke with have specific quantitative thresholds as minimum standards on various outcomes. For example, one national accreditor requires that their member schools meet specific thresholds for student retention and placement rates. Officials at this agency said they could sanction schools whose programs fall below these standards. The other national accreditor we spoke with also requires its schools to meet thresholds established for outcomes such as course completion rates, program graduation rates, student satisfaction rates, and student learning (as measured by professional licensing exams such as those for physical therapists and lawyers). One regional accreditor said it was exploring including standardized learning outcomes in its accreditation standards.[106]

As part of their periodic site visits to schools to assess the quality of academic programs, accreditors have to adapt their approach when reviewing schools with distance education. Accreditors are required to employ staff who are well-trained and knowledgeable about distance education, for example, when performing on-site reviews of schools providing distance education.[107] Officials at all six accrediting agencies we spoke with said they include such experts on their on-site review teams. At one regional accreditor we interviewed, distance education experts are tasked with specifically reviewing the quality of a school’s distance education learning infrastructure, as well as the educational effectiveness of its programs, and receive specific training to do so. To review schools’ student supports, faculty supports, and educational effectiveness, officials at another regional accreditor told us their distance education experts may use video teleconferences or e-mails to communicate with administrative staff, faculty, and students not located on campus. These experts also remotely observe interactions between students and faculty in online classes.

In addition to the periodic on-site accreditation reviews to reassess a school’s accreditation status that are required by statute,[108] accreditors are to be notified if schools make substantive changes to academic programs or their schools.[109] The main purpose of this substantive change policy is to ensure that when schools make changes, they are maintaining the same level of quality they had when last reviewed. While there are a number of circumstances that can trigger the substantive change requirement, the one most applicable to distance education is the addition of courses or programs that represent a significant departure from the existing offerings of educational programs, including method of delivery, from those that were offered when the accreditor last evaluated the school. A shift to distance education courses that constitute more than 50 percent of a program’s offerings was the substantive change threshold used by four of the six accrediting agencies we interviewed. Officials at one regional accrediting agency reported that, in calendar year 2010, the agency turned down 34 percent of initial substantive change requests for new distance education programs because of weak student learning assessments or inadequately trained faculty, among other reasons. However, they said this figure has since come down to about 16 percent because schools have had more training on how to develop a substantive change proposal.

To ensure academic integrity, the six accrediting agencies we interviewed require schools to provide evidence that they verify registered students are doing their own work. For example, officials at one regional accreditor we spoke with said they require schools to use a student identification number and password as the minimum for verifying student identity. This accreditor said most institutions also verify student identity through student interaction during the course. In addition, one national accreditor we spoke with said some schools design tests that require a login and password, and may also feature pop-up questions during tests, prompting students to enter verification information such as their address or mother’s maiden name.

While the accreditors we interviewed have a range of activities to assess the quality of distance education, a few accrediting agency officials and industry experts we spoke with also expressed some concerns and reported challenges involved in assessing the quality of distance education. These challenges were mostly related to accreditors’ capacity to keep pace with substantive changes and conduct follow-up quality reviews with schools. Officials at one regional and one national accrediting agency said they have had some difficulty keeping up with the high number of substantive change applications for new online programs. Officials representing the national accreditor said these applications have increased by about 30 percent and that the officials have had to double the number of evaluators on staff over the last 5 years. According to officials with the regional accreditor, they have increased the number of follow-up reviews to ensure that schools address concerns about meeting quality standards identified during the initial site visits. These officials reported that they withdrew one school’s accreditation for failure to demonstrate that its distance education programs met the same standards as its face-to-face programs, with respect to curriculum, resources, support, and student learning outcomes. Industry experts also acknowledged that some accreditors have limited resources and have had problems training their peer reviewers in distance education.

Schools Use a Range of Course Design Principles and Student Performance Assessments to Hold Distance Education to the Same Quality Standards as Traditional Courses

To assure that their distance education programs are accredited by federally recognized accreditors and that their students qualify for Title IV funding, officials we interviewed at 20 selected schools reported that they generally apply certain course design principles and use student performance assessments to assess the quality of the courses that make up these programs. The accreditors we spoke with require schools to have standards that address the quality of degree programs with respect to such things as student achievement, which could include such measures as course completion, licensing exams, and job placement rates, as well as student support services. A majority of school officials reported that they assess their distance education courses by the same standards they use for their traditional courses.

Officials at most of the schools we spoke with said they used instructional teams to design their distance education courses according to the schools’ standards. These teams varied in their composition and activities.[110] Some teams include specialized staff who work with faculty to translate traditional face-to-face courses to the online environment. For example, one school we visited in Florida has a 20-member instructional design team that includes instructional designers, graphic artists, multimedia technicians, and quality control coordinators. Officials at this school said the design team considers which instructional methods are most appropriate for the material delivered in each online course. For example, a psychology course may use mostly text-based storytelling, while an anthropology class may rely more heavily on video clips. Officials at an online school we spoke with stressed the need to replace face-to-face course instructors’ body language and tone of voice cues with appropriate text and video media. Besides assisting professors with designing online courses, school officials said instructional design teams also train professors in the pedagogical differences of teaching online and on the online technology used by the school.

Officials at over half of the 20 schools we interviewed also reported that, to ensure quality in the design of their courses, they had used standards and best practices, some of which were developed by distance education industry experts. For example, 5 schools subscribe to Quality Matters, a nonprofit organization that lays out principles for designing quality online and blended courses. This organization sets specific standards for learning objectives, technology, faculty-student interaction, student supports, and assessment that online courses must meet in order to receive Quality Matters certification.

In addition, school officials reported that their schools collect outcome data to help them assess the quality of courses. The types of learning outcomes that the schools reported tracking include end-of-course grades, course completion rates, and results of national professional licensing assessments. Officials at most schools we spoke with said they also used outcome data to make improvements to their courses. Officials at two schools told us they employ staff to analyze these data and make recommendations for course updates. For example, officials at one fully online school we spoke with noticed their students were performing below the national average on a section of a third-party end-of-course criminal justice test. The officials used the results of this test to strengthen the related material. According to these officials, their criminal justice students’ performance improved on that section of the exam subsequent to their course improvements. In addition to using outcome data to improve their courses, one school we spoke with in Florida had collected these types of data on their online and hybrid courses over a period of 15 years to determine which factors most influenced student success.

To meet accreditors’ requirement to verify the identities of students enrolled in their distance education courses or programs, officials at most of the schools reported using various methods. For example, most of the school officials we interviewed said they issue students a secure login and password and some also use other methods, such as proctored exams. Officials at one school said they are also starting to use audiovisual software that works as a web cam to verify the student taking an exam is the one enrolled in the course and to ensure the student is not receiving assistance. In addition to technological safeguards, officials from one school said the interaction between students and faculty is key to ensuring students are doing their own work. They said instructors become familiar with a student’s writing or communication style through online discussions or the completion of assignments, and the instructor recognizes if that style changes. Officials at one school said they cannot be completely sure that distance education students are doing all of their own work even when using these methods; officials also noted that similar challenges exist for face-to-face courses. A few schools mentioned taking further steps to combat potential fraud in their online programs. Specifically, officials at two of the completely online schools we interviewed said they conduct reviews of or request further documentation from students who register with the same e-mail addresses or telephone numbers.[111] Officials at one school we spoke with said they would like more guidance, either from Education or their institutional accrediting agency, on examples of verification and authentication systems for student identity to improve the school’s monitoring of the verification process.

Education Has Increased Its Monitoring of Distance Education but Lacks Sufficient Data to Inform Its Oversight

Education’s Office of Federal Student Aid (FSA) has recently increased its monitoring of distance education by updating its program review procedures and undertaking a risk analysis project. These efforts are in response to the expansion of distance education and the Education OIG’s identification of distance education as a high-risk area for managing student aid dollars.[112] To better monitor distance education, FSA updated and issued new program review procedures. The previous set of FSA’s procedures, issued in 2008, did not provide in-depth guidance for assessing whether a school was approved to offer distance education or if there was regular and substantive interaction between instructors and students. The new procedures on distance education provide staff with expanded guidance for assessing a school’s compliance with these requirements. FSA officials said staff have been trained on the new procedures and, as of June 2011, have been using them for program reviews.[113] All program reviews will include at least routine testing to determine basic program eligibility for schools that offer distance education, according to Education officials. Schools that offer more than half of any of their programs through distance education will also be required to undergo expanded testing for regular and substantive interaction.

Compliance with federal student aid requirements by schools offering distance education programs is difficult to assess because many of the violations Education identifies through its program reviews are not specific to distance education; for those that are, Education does not necessarily identify or code the violations as such in its database, according to an Education official. For example, violations such as a school not appropriately returning Title IV funds when a student withdraws are coded in Education’s database based on the type of violation rather than whether this violation occurred in traditional or distance education. Violations specific to distance education that are tracked by Education are related to a lack of regular and substantive interaction between instructor and students and certain accreditation issues, such as an accrediting agency being ineligible because it does not have distance education in its scope. Education reported that from October 2005 through May 2011, no program reviews or audits identified any lack of regular and substantive interaction or distance education accreditation violations.[114]

In addition to FSA updating the program review procedures, in 2009, OIG and FSA, at the request of the Deputy Undersecretary, initiated a project analyzing risk factors for noncompliance with Title IV requirements by schools offering distance education. Under this project, 27 schools were selected for review based on a set of indicators OIG and FSA considered to be high risk for noncompliance, such as a large change in the amount of federal student aid a school is receiving.[115],[116] The OIG/FSA group is conducting this project in conjunction with others in the department.[117] FSA officials were not able to estimate a date when all final project reports will be issued, but said their last program review was conducted in early August 2011. They said the results of the project, including its methods for identifying high-risk schools and the procedures used, will be evaluated to determine if any changes need to be made to FSA’s annual program reviews.

While the objective of the project was to review high-risk distance education schools, Education lacked data to adequately identify schools’ level of risk based on the extent to which they offered distance education and the amount of federal student aid they received for those programs or courses. For example, to identify high-risk schools that may be offering distance education courses and programs, one indicator Education relied on was the Department of Defense’s enrollment information on its military members. Because distance education provides the flexibility needed to fit active duty service members’ duty schedules and location, many military members are enrolled in distance education courses and programs.[118] Therefore, in its risk analysis, Education included schools that had 200 or more military members receiving tuition assistance from the Department of Defense. While Education’s IPEDS database can show which schools offer distance education, it lacks information on the extent of a school’s offerings and enrollment levels. Despite using data from multiple sources, one of the 27 schools Education originally selected for review through the risk analysis did not actually offer distance education. As a result, FSA officials said they had to substitute another school for the study. While the project is not yet complete, officials reported confidence that their study is currently based on an appropriate selection of schools.[119] Nevertheless, they acknowledged that, in selecting their target schools, they lacked sufficient data to help them identify the extent to which a school was offering distance education as well as the amount of federal dollars being spent for distance education at each school, both of which would have been significant in evaluating a school’s risk.

The Office of Federal Student Aid has plans to collect more information on distance education, but complete information on all schools may not be available for several years. Under its new Integrated Partner Management (IPM) system, which will consolidate data systems on schools receiving Title IV funds, FSA will collect information about how a school’s programs are offered. Specifically, FSA officials said when schools apply for Title IV initial certification or recertification, they will be asked to indicate whether a program is predominantly (more than 50 percent) delivered via the classroom, distance education, correspondence, or independent study. They said the IPM system is expected to be implemented in November 2012 and would eventually allow them to analyze comprehensive data about a school. For example, they will be able to match the extent to which schools offer distance education with Title IV violations identified during program reviews. However, because schools are generally required to recertify only every 6 years, officials acknowledged that it could be several years before the IPM system will contain information on all schools’ distance education offerings.[120] Therefore, distance education information on all schools may not be available through IPM until 2018.

In the meantime, Education’s NCES is expanding its IPEDS survey to provide a more in-depth picture of distance education offerings and enrollment patterns.[121] The plan by NCES to expand the IPEDS survey with regard to distance education was the result of a decision by its technical review panel to better describe postsecondary education offered throughout the nation, allow schools to compare their distance education activities with those of their peer schools, and provide valuable information to parents and students on available college programs. This expanded data collection will be conducted in phases. The 2011-2012 survey used the definition of distance education as established in 2008 and collected information about whether schools offer their programs completely through distance education. Additional new distance education questions will be added to the 2012-2013 survey. The new survey questions ask for information such as the range of a school’s offerings in distance education, the number of students enrolled either partially or entirely in distance education, and whether the students are located in or out of state in relation to the school (see fig. 9). An NCES official said the new IPEDS data are expected to be available 1 year after the survey closes but may be available earlier. For example, early release data collected during the 2011-2012 survey may be available as early as February 2012 and available publicly by November 2012.

a The classification instructional program code is a coding scheme that contains titles and descriptions of instructional programs.

Figure 9: IPEDS Changes Related to Distance Education.

Despite the prospect of more comprehensive data on schools and their distance education offerings being collected through IPEDS, FSA does not yet have specific plans to use these data for monitoring school compliance with federal student aid requirements. According to FSA officials, they intend to wait and see what information the survey yields before deciding how to make use of it.[122] Moreover, FSA indicated it was not aware of NCES’s efforts to expand the IPEDS distance education data collection and, therefore, was not involved in the planning and did not provide input during comment periods.[123] According to NCES officials, the NCES technical review panel process engages a number of stakeholders and is open to federal officials who are interested in participating.

Conclusions

Distance education, specifically online education, has been developing for a number of years and has become a part of the mainstream of higher education. This delivery mode of instruction has provided some new opportunities and access, particularly for nontraditional students and working adults who are looking to advance their careers. Moreover, it is likely to continue growing, as schools across all sectors and levels see it as a critical educational tool in meeting student needs and demand.

The growth in distance education and the sizable federal investment in higher education will challenge all segments of the triad responsible for the oversight of higher education—the states, accreditation agencies, and the federal government—in their capacity to provide consumer protection, ensure academic quality, and protect the federal investment. In response to this challenge, Education has taken steps to increase its oversight by providing its staff with expanded guidance for assessing a school’s compliance with distance education requirements and participating in the OIG/FSA risk project, which identified potential risk indicators. However, a key factor in Education’s ability to properly focus oversight on the areas of greatest risk will be the availability and use of pertinent, up-to-date data on both the extent to which schools offer distance education and the extent to which students use federal aid to attend those programs.

While FSA’s IPM system may eventually be helpful in providing Education with the opportunity to monitor distance education with better information, the expanded IPEDS data would provide relevant information much sooner. However, without a plan on how to use the new IPEDS data to identify and monitor high-risk schools, FSA may lose the opportunity to strengthen its oversight of distance education in the near term. Moreover, if FSA does not coordinate with NCES going forward, it stands to lose the opportunity to provide input on any additionally needed data that may strengthen oversight and ensure accountability in the long term.

Recommendations for Executive Action

To help Education strengthen its oversight of distance education, the Secretary of Education should direct FSA to develop a plan on how best to use the new IPEDS distance education data and provide input to NCES on future IPEDS survey work with regard to distance education.

Agency Comments and Our Evaluation

We provided a draft of this report to officials at Education for their review and comment. Education provided comments, which are reproduced in appendix III of the full text report, and technical comments, which we incorporated as appropriate. In its comments, Education agreed with our recommendation and noted that FSA will update its School Participation Team procedures to include consideration of IPEDS data on distance education for monitoring schools. Education also stated that FSA will provide input to NCES on the design and results of any future IPEDS surveys that include distance education.

We are sending copies of this report to relevant congressional committees, the Secretary of Education, and other interested parties. In addition, this report will also be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

George A. Scott

Director, Education, Workforce, and Income Security Issues

Appendix I: Objectives, Scope, and Methodology

This appendix discusses in detail our methodology for addressing the following research objectives: (1) the characteristics of distance education today, (2) the characteristics of students participating in distance education, (3) how the quality of distance education is being assessed, and (4) how Education monitors distance education in its stewardship of federal student aid funds.

To address these research questions, we reviewed relevant federal laws and regulations, literature, studies, and reports; interviewed officials from Education, representatives from all types of postsecondary schools, accreditation agencies, and distance education and industry experts; and conducted site visits to Florida, Minnesota, and Puerto Rico to interview state agency and school officials. We selected these sites based on various factors, including the level of state data collected and an industry summary of states’ policies for approving distance education. We also analyzed data from Education’s Integrated Postsecondary Education Data System (IPEDS) and the National Postsecondary Student Aid Study (NPSAS) databases to determine the school and student characteristics involved in distance education. We determined that IPEDS and NPSAS data were sufficiently reliable for the purposes of this report based on prior testing of the data from these systems in 2011.[124] The data were tested for accuracy and completeness, documentation about the data and systems used to produce the data was reviewed, and agency officials were interviewed.

To determine the current characteristics of distance education, we analyzed 2009-2010 data from Education’s IPEDS and also from a 2008 report by Education’s National Center for Education Statistics (NCES) [125] to obtain a national perspective on distance education practices and offerings at postsecondary schools. Specifically, we analyzed IPEDS data to provide information on the size, number, sector, and program length of schools offering distance education courses and programs. We used the 2008 distance education report to describe how schools are providing distance education to students, including the type of technology (Internet, video, audio, etc.) and instructional methods (asynchronous and synchronous) used, and the various types of degrees, certificates, and courses offered, including the percentage of courses offered online. In addition, we analyzed the 2010 Sloan Consortium [126] report on online education to show updated enrollment figures specific to online courses.[127] We supplemented the nationally representative data with information obtained from our interviews with industry experts and representatives at a nongeneralizable sample of postsecondary schools regarding the range of delivery and instructional techniques being used, and the type of programs and coursework offered through distance education.

To select our sample of postsecondary schools, we used enrollment data from Education’s 2009-2010 IPEDS to identify schools that were offering distance education and had significant increases in total enrollment, which may be due, in part, to increased enrollment in distance education classes or programs. Based on the schools’ percentage change in enrollment, we then selected schools by size—as defined by enrollment—as well as by sector and program length. We also considered the following factors in selecting our sample of schools:

- geographic dispersion by state

- minority serving school status (e.g., Historically Black Colleges and Universities and Hispanic-serving institutions)

- selectivity in accepting students

- industry expert or stakeholder recommendations

- extent to which distance education programs and courses are offered (totally online schools versus schools offering both campus-based and online instruction), and

- whether the schools are regionally or nationally accredited.

Based on these considerations, we selected 20 schools representing all sectors and program lengths, for site visits or phone interviews (see app. II for a list of colleges and universities we interviewed). Our selected schools break out as follows:

- 4 public 2-year schools

- 5 public 4-year schools

- 6 private nonprofit schools

- 5 private for-profit schools

After our interviews with officials from the selected schools, we conducted a content analysis on the information gathered. Interview responses and comments from officials were categorized to identify common themes. The themes were reviewed by a methodologist before all comments were categorized. One analyst coded the information and a second analyst assessed the accuracy of the coding. Disagreements between coders were resolved through discussion. We used the information gathered from these schools for illustrative purposes only. Because the schools were not selected to be representative of all postsecondary schools, the interview results are not generalizable to other postsecondary schools, including groups of schools in the same sector or program length.

To determine the characteristics of students participating in distance education courses and programs, as well as those who do not participate, we analyzed Education’s 2007-2008 NPSAS data, the most current available data.[128] These data allowed us to compare distance education students to nondistance education students on the following characteristics: age, gender, marital status, dependent status, and employment status. The data also allowed us to describe the characteristics of students enrolled in distance education, in terms of type of school attended, field of study, race, veteran status, and disability status. We supplemented this analysis with information from our interviews with selected postsecondary schools and student demographic data provided by school officials.

To determine how the quality of distance education programs is being assessed, we obtained information from accrediting agency and school officials and reviewed and analyzed federal laws and regulations related to accreditation. We interviewed officials from six accrediting agencies (three regional, two national, and one specialized [129]) and reviewed their standards and policies to determine how they are assessing the quality of distance education courses and programs. In addition, we reviewed documents from the Council for Higher Education Accreditation (CHEA) [130] website, to gain a broader understanding of accreditation. We also interviewed officials from schools in our sample to describe the specific quality assurance frameworks and the outcomes they use to assess the performance of students engaged in distance education. In addition, we interviewed an official from Quality Matters and reviewed quality standards documents provided at the interview.[131]

To determine the extent to which Education is monitoring distance education programs to ensure the protection of federal student aid funds, we reviewed relevant federal laws and regulations regarding distance education oversight requirements. We interviewed officials from Education’s Federal Student Aid office and the Office of Postsecondary Education to determine their roles in the monitoring and governance of Title IV programs, specifically with respect to distance education. In addition, we interviewed officials from NCES to learn about their IPEDS data collection efforts and Education’s Office of the Inspector General to learn about their distance education monitoring activities and findings. Finally, we reviewed agency documents, including plans to add distance education variables to the IPEDS survey, OIG testimonies and reports, and an interim status memorandum issued by the OIG/FSA Risk Project.

We conducted this performance audit from November 2010 to November 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: List of Colleges and Universities GAO Interviewed

Name of institution

Institution type

Location

Los Angeles Valley College

Public, 2-year

CA

Navarro College

Public, 2-year

TX

Seminole State College of Florida

Public, 2-year

FL

Wichita Area Technical College

Public, 2-year

KS

Morgan State University

Public, 4-year

MD

University of Central Florida

Public, 4-year

FL

University of Maryland – University College

Public, 4-year

MD

University of Minnesota – Twin Cities

Public, 4-year

MN

University of Puerto Rico – Río Piedras

Public, 4-year

PR

Baker University

Private nonprofit, 4-year

KS

Carnegie Mellon University

Private nonprofit, 4-year

PA

Columbia University in the City of New York

Private nonprofit, 4-year

NY

Inter American University of Puerto Rico

Private nonprofit, 4-year

PR

Saint Leo University

Private nonprofit, 4-year

FL

Western Governors University

Private nonprofit, 4-year

UT

American Public University System

Private for-profit, 4-year

WV

Capella University

Private for-profit, 4-year

MN

DeVry University

Private for-profit, 4-year

IL

Keiser University – Fort Lauderdale [132]

Private for-profit, 4-year

FL

National University College

Private for-profit, 4-year

PR

Source: GAO.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

George A. Scott, (202) 512-7215 or scottg@gao.gov

Staff Acknowledgments

In addition to the contact named above, Tranchau Nguyen, Assistant Director; Susan Chin, Analyst-in-Charge; Amy Anderson; Jeffrey G. Miller; and Jodi Munson Rodríguez made significant contributions to this report in all aspects of the work. Susan Bernstein contributed to writing this report. Michael Silver, Christine San, and John Mingus provided technical support, and Jessica Botsford provided legal support. Mimi Nguyen assisted with report graphics.

Related GAO Products

Distance Education: Growth in Distance Education Programs and Implications for Federal Education Policy GAO-02-1125T (Washington, D.C.: GAO, 26 September 2002); available at www.gao.gov/products/GAO-02-1125T.

Distance Education: Improved Data on Program Costs and Guidelines on Quality Assessments Needed to Inform Federal Policy, GAO-04-279 (Washington, D.C.: GAO, 26 February 2004); available at www.gao.gov/products/GAO-04-279.

Higher Education: Institutions’ Reported Data Collection Burden is Higher Than Estimated but Can Be Reduced through Increased Coordination, GAO-10-871 (Washington, D.C.: GAO, 13 August 2010); available at www.gao.gov/products/GAO-10-871.

DOD Education Benefits: Increased Oversight of Tuition Assistance Program Is Needed, GAO-11-300 (Washington, D.C.: GAO, 1 March 2011); available at www.gao.gov/products/GAO-11-300.

DOD Education Benefits: Further Actions Needed to Improve Oversight of Tuition Assistance Program, GAO-11-389T (Washington, D.C.: GAO, 2 March 2011); available at www.gao.gov/products/GAO-11-389T.

VA Education Benefits: Actions Taken, but Outreach and Oversight Could Be Improved, GAO-11-256 (Washington, D.C.: GAO, 28 February 2011); available at www.gao.gov/products/GAO-11-256.

[*]    Dr. Christian Glahn is a researcher for mobile learning solutions for security and defense organizations at the International Relations and Security Network, ETH Zürich, Switzerland. The research on which this article is based has been funded by the ADL Co-Lab and awarded by the Office of Naval Research Global (ONRG) under Grant No. N62909-12-1-7022. The views expressed herein are solely those of the author and do not represent or reflect the views of any academic, government or industry organization mentioned herein.

[1]    Ethan Marcotte, “Responsive Web Design,” A List Apart (alistapart.com); available at www.alistapart.com/articles/responsive-web-design.

[2]    Christian Glahn, ed., “Workshop Proceedings: Mobile Learning in Security and Defense Organizations,” Proceedings of the 11th World Conference on Mobile and Contextual Learning, ed. Jari Multisala, Marcus Specht, and Mike Sharples (Helsinki, 21–24 October 2012); available at http://ceur-ws.org/Vol-955/workshops/WS6Security.pdf.

[*]    Jacob Hodges is an independent advisor on innovative approaches to education and training. He served as Project Research Coordinator for the Mobile Learning Environment (MoLE) Project, and provided program management and technical support to the Project Manager, technical support to the Testing and Evaluation Working Group, and assisted the MoLE Science & Technology Coordinator in conducting S&T reviews. Geoff Stead is Head of Mobile Learning at Qualcomm. He has been active in mobile learning since 2001, innovating and building technologies to support learning, communication and collaboration across the globe.

The authors would like to acknowledge the subject matter advice on EU/UK Research Ethics provided by Dr. Andrey Girenko (Deutsche Forschungszentrum für Künstliche Intelligenz), Dr. Andrea Loesch (GIRAF PM Services), Dr. Venkat Sastry (Defense Academy of the United Kingdom), Dr. Tammy Savoie (J-4, USAF), the MoLE Testing & Evaluation Working Group Members, and the project’s international partners.

The work discussed in this article is related to Department of the Navy (DoN) Grant (N62909-11-01-7025) and two DoN Contracts (N68181-11-P-9000 and N68171-12-P-9000), which were awarded by the Office of Naval Research Global and funded by the Coalition Warfare Program. The views expressed herein are solely those of the authors, and do not represent or reflect the views of any government agency or organization that participated in the project or is mentioned herein.

[3]    Code of Federal Regulations, Title 32 – National Defense, Part 219: Protection of Human Subjects; available at http://www.law.cornell.edu/cfr/text/32/219. See also U.S. Department of the Navy, SECNAVINST 3900.39D, “Human Research Protection Program” (6 November 2006); available at www.fas.org/irp/doddir/navy/secnavinst/3900_39d.pdf.

[4]    David B. Resnik, The Ethics of Science (New York: Routledge, 1998).

[5]    National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, “Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects in Research,” Department of Health, Education, and Welfare Publication No. (OS) 78-0012 (18 April 1979); available at www.hhs.gov/ohrp/humansubjects/guidance/belmont.html.

[6]    Judith P. Swazey and Stephanie J. Bird, “Teaching and Learning Research Ethics,” in Research Ethics: A Reader, ed. Deni Elliott and Judy Stern (Hanover, NH: University Press of New England, 1997).

[7]    John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (Thousand Oaks, CA: Sage Publications, 2009).

[8]    SECNAVINST 3900.39D.

[9]    ANSI/ISO/ASQ, Quality Management Systems – Guidelines for quality management in projects (e-standard), International Standard: ANSI/ISO/ASQ Q10006-2003(E) (Milwaukee, WI: ASQ Quality Press, 2006).

[10]    Robert G. Burgess, The Ethics of Educational Research (London: Routledge Falmer, 1989).

[11]    Ibid.

[12]  Mark Israel and Iain Hay, Research Ethics for Social Scientists: Between Ethical Conduct and Regulatory Compliance (London: Sage, 2006).

[13]  Resnik, The Ethics of Science.

[14]  Burgess, The Ethics of Educational Research.

[15]  Jacob R. Hodges, Mobile Learning Environment (MoLE) Project: A Global Technology Initiative (20 February 2013); available at https://www.createspace.com/pub/simplesitesearch.search.do?sitesearch_query=Mobile+Learning+environment&sitesearch_type=STORE.

[16]  Code of Federal Regulations, Title 32 – National Defense, Part 219: Protection of Human Subjects.

[17]  Electronic Privacy Information Center (EPIC), Privacy and Human Rights Report 2006: An International Survey of Privacy Laws and Developments (Washington, D.C.: EPIC, 2007). Solveig Singleton, “Privacy and Human Rights: Comparing the United States to Europe,” Cato Institute White Paper (1 December 1999); available at www.cato.org/pubs/wtpapers/ 991201paper.html. EU Data Protection Directive 95/46/ED, 24 October 1995; available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:31995L0046:en:HTML.  For governing directives for U.S.-based researchers involved in the project, see Department of Defense (DoD) Directive 3216.2, “Protection of Human Subjects and Adherences to Ethical Standards in DoD-Supported Research” (7 January 1993); Department of Defense Instruction 3210.7, “Research Integrity and Misconduct” (14 May 2004); and Department of Defense Instruction 5400.11, “DoD Privacy Program” (8 May 2007); all available at www.dtic.mil/whs/directives/corres/ins1.html.

[*]    Dr. Stefaan Ternier, Bernardo Tabuenca, Fred de Vries, and Prof. Dr. Marcus Specht are from the Open University in The Netherlands and Atish Gonsalves is from UNHCR.

[18]    Jeanne Lave and Etienne Wenger, Situated Learning: Legitimate Peripheral Participation (Cambridge, U.K.: Cambridge University Press: 1990).

[19]    Chris Dede, “Immersive Interfaces for Engagement and Learning,” Science 323 (2009): 66–9; available at www.ncbi.nlm.nih.gov/pubmed/19119219.

[20]    “ARLearn Project Page,” 2011; available at http://code.google.com/p/arlearn.

[21]    Stefaan Ternier, Roland Klemke, Marco Kalz, Patricia van Ulzen, and Marcus Specht, “ARLearn: Augmented Reality Meets Augmented Virtuality,” Journal of Universal Computer Science 18 (2012): 2143–64.

[22]    Ibid.

[23]    Marion Gruber, Christian Glahn, Marcus Specht, and Rob Koper, “Orchestrating Learning Using Adaptive Educational Designs in IMS Learning Design,” in Sustaining TEL: From Innovation to Learning and Practice, ed. Martin Wolpers, Paul A. Krischner, Maren Scheffel, et al. (Berlin, Heidelberg: Springer Verlag, 2010).

[24]    Katrien Verbert, Jelena Jovanovic, Eric Duval, Dragan Gasevic, and Michael Meire, “Ontology-Based Learning Content Repurposing: The ALOCoM Framework,” International Journal on e-Learning 5 (2006): 67–74.

[*]    Dr. Christian Glahn – see page 1.

[25]    Reinhard Oppermann and Marcus Specht, “Adaptive Mobile Museum Guide for Information and Learning on Demand,” in Workshop on Interactive Applications of Mobile Computing (IMC 98, 1–5 November 1998, Rostock, Germany). Elliot Soloway and Cathleen A. Norris, “Using Technology to Address Old Problems in New Ways,” Communications of the ACM 41:8 (1998): 11–18.

[26]    “Infanterist der Zukunft,” Deutsches Heeresamt, Infanterieschule (2012); available at http://bit.ly/14DDVsq.

[27]    “Rheinmetall and SAAB: Creating Network-Enabled Warfighters,” Rheinmetall Defense, Press release, 30 May 2012; available at http://www.rheinmetall.ca/en/media/editor_media/rheinmetallkanada/downloads/publicrelations_1/public_relations_neu/2012_05_30_integrated_soldier_system_project.pdf.

[28]    Advanced Distributed Learning (ADL) Initiative, Sharable Content Object Reference Model (SCORM) 2004, 4th Edition Run-Time Environment (RTE), Version 1.1 (Alexandria, VA: ADL Initiative, 2004).

[29]    Wilbert Kraan, Steve Lay, and Pierre Gorissen, IMS Question & Test Interoperability Assessment Test, Section and Item Information Model, Final 2.1 (2012); available at www.imsglobal.org/question/#version2.1.

[30]    Fernando Mikic, Luis Anido, Enrique Valero, and Juan Picos, “Accessibility and Mobile Learning Standardization, Introducing Some Ideas About the Device Profile (DP),” in Second International Conference on Systems (ICONS’07), 32; Maia Zaharieva and Wolfgang Klas, “MobiLearn: An Open Approach for Structuring Content for Mobile Learning Environments,” in Web Information Systems, ed. Christoph Bussler, Suk-ki Hong, and Woochun Jun (Berlin, Heidelberg: Springer Verlag, 2004), 114–24; R. Yu-Liang Ting, “Mobile Learning: Current Trend and Future Challenges,” in Fifth IEEE International Conference on Advanced Learning Technologies (ICALT’05) (2005): 603–7.

[31]    Kiyoshi Nakabayashi, Takahide Hoshide, Masanobu Hosokawa, Taichi Kawakami, and Kazuo Sato, “Design and Implementation of a Mobile Learning Environment as an Extension of SCORM 2004 Specifications,” Seventh IEEE International Conference on Advanced Learning Technologies (ICALT 2007) (July 2007): 369–373.

[32]    Asi Degani, Geoff Martin, Geoff Stead, and Frances Wade, Mobile Learning Shareable Content Object Reference Model (m-SCORM) Limitations and Challenges, [N09-35] (Cambridge, U.K.: 2010); available at www.m-learning.org/images/stories/MobScorm.pdf.

[33]    Mike Sharples, Josie Taylor, and Giasemi Vavoula, “A Theory of Learning for the Mobile Age,” in The Sage Handbook of E-learning Research, edited by Richard Andrews and Caroline Haythornthwaite (London: Sage Publications, 2007), 221–47; available at www.lsri.nottingham.ac.uk/msh/Papers/Theory of Mobile Learning.pdf.

[34]  Alex Sbardella, “Ten Tips for Mobile UX,” Red Ant Blog; available at https://www.redant.com/articles/ten-tips-for-mobile-ux.

[35]  Gerhard Gassler, Theo Hug, and Christian Glahn, “Integrated Micro Learning – An Outline of the Basic Method and First Results,” in International Conference on Interactive Computer Aided Learning (ICL), ed. Michael E. Auer and Ursula Auer (Villach, Austria, 29 September – 1 October 2004) (CD-ROM).

[36]  Deborah L. Butler and Philip H. Winne, “Feedback and Self-Regulated Learning: A Theoretical Synthesis,” Review of Educational Research 65:3 (1995): 254–81.

[37]  Judy Kay, “Learner Know Thyself: Student Models to Give Learner Control and Responsibility,” in Control and Responsibility: International Conference on Computers in Education (AACE, 1997), 17–24.

[38]  Christian Glahn, Marcus Specht, and Rob Koper, “Smart Indicators on Learning Interactions,” in Creating New Learning Experiences on a Global Scale: LNCS 4753. Second European Conference on Technology Enhanced Learning (EC-TEL 2007), ed. Erik Duval, Ralf Klamma, and Martin Wolpers (Berlin, Heidelberg: Springer Verlag, 2007), 56–70.

[39]  Butler and Winne, “Feedback and Self-Regulated Learning: A Theoretical Synthesis.”

[40]  Dominique Verpoorten, Christian Glahn, Milos Kravcik, et al., “Personalisation of Learning in Virtual Learning Environments,” in Learning in the Synergy of Multiple Disciplines, ed. Ulrike Cress, Vania Dimitrova, and Marcus Specht (Berlin, Heidelberg: Springer Verlag, 2009), 52–66.

[41]  Kraan, Lay, and Gorissen, IMS Question & Test Interoperability Assessment Test.

[42]  “Ilias Open Source e-Learning,” www.ilias.de.

[43]  Advanced Distributed Learning (ADL) Initiative, Experience API. Draft Specification (19 October 2012); available at http://cdn3.tincanapi.com/wp-content/assets/spec/Tin-Can-API-Releasev095.pdf.

[*]    Dr. Kristy Murray is Director of the Advanced Distributed Learning (ADL) Initiative located in the Office of the Deputy Assistant Secretary of Defense (Readiness) within the U.S. Department of Defense. Peter Berking is a senior instructional designer and a contractor with Serco Inc. Jason Haag is a mobile learning expert, and a contractor with The Tolliver Group, Inc. Nikolaus Hruska is a senior software engineer and a contractor with Problem Solutions. They provide support to the ADL Initiative.

[44]    ADL Tech Team, “Resources for ‘Training and Learning Architecture’ Webinar” (2013), ADL Web site; available at http://www.adlnet.gov/resources-for-training-and-learning-architecture-webinar.

[45]    ADL Mobile Learning Team, “ADL Mobile Learning Handbook” (2012), ADL Web site; available at https://sites.google.com/a/adlnet.gov/mobile-learning-guide/basics.

[46]    iFest Panel Discussion slides, “Mobile: Thinking Outside the Course” (2011); available at www.adlnet.gov/wp-content/uploads/2011/08/brown_haag_clark_gibbens_mobile_outside_course_iFest2011.pdf

[*]    Mrs Popovska is a lecturer at the Military academy of the Republic of Macedonia that is an associate member of the State University Goce Delchev, Shtip. She holds an MA in International relations in conflict resolution from University “Sts. Cyril and Methodius,” Skopje – Faculty of Philosophy. She was a visiting research fellow at the George C. Marshall European Center for Security Studies (Summer 2013). Her area of research includes post conflict societies and transitional justice.

[47]    Elizabeth A. Cole and Judy Barsalou, Unite or Divide? The Challenges of Teaching History in Societies Emerging from Violent Conflict, Special USIP Report on a conference held on 21 November 2005 at USIP in Washington (Washington, D.C.: United States Institute of Peace, June 2006; available at http://www.usip.org/publications/unite-or-divide-challenges-teaching-history-societies-emerging-violent-conflict; Mirela-Luminita Murgescu, “Rewriting School Textbooks as a Tool of Understanding and Stability,” Southeast European and Black Sea Studies 2:1 (2002): 90–104, available at http://dx.doi.org/10.1080/14683850208454674;  and Elizabeth A. Cole, “Transitional Justice and the Reform in History Education,” International Journal on Transitional Justice 1 (2007); available at http://ijtj.oxfordjournals.org/content/1/1/115.short.

[48]    Presentation by Violeta Petroska-Beshka at the “Unite or Divide?” conference, 21 November 2005, United States Institute of Peace, Washington, D.C. Petroska-Beshka’s argument has sparked political controversies in Macedonia. An edition of the website “Balkan Insight” (No. 20, 20 February 2006) described a “storm” of controversy that erupted over Petroska-Beshka’s efforts to reform the teaching of history in Macedonia. In a typical comment, Blazhe Ristevski, then the director of the Macedonian Academy of Science and Arts, said, “As a scientist, I can’t allow that truth can be found through this kind of ‘partnership.’ It just adds more petrol to the fire between the two sides.” Ljupcho Jordanovski, at that time the Speaker of the Macedonian Parliament, said not enough time had passed since the recent conflict between Macedonian Slavs and Albanians, and objectivity was impossible because “we were all either direct or indirect participants.” Countering that such reforms are hard to undertake at any time, Petroska-Beshka argued, “If we don’t speak openly about these painful issues, we leave a space to create ethnically colored, opposing versions that will affect the definition of official history.” Teachers need strong support from parents, school administrators, and other authorities to teach new curricula and use new pedagogies. Such support must be ongoing, as teachers suffer from burnout, especially in high-stress situations. It is this group of teachers from Petroska-Beshka’s project that is the target population of the research part of this paper. They have been trained, and enough time has passed for us to learn something from their experience.

[49]    Eric Foner, Who Owns the Past? Rethinking the Past in a Changing World (New York: Hill and Wang, 2002), 88.

[50]    Alison Kitson, “History Teaching and Reconciliation in Northern Ireland,” in Teaching the Violent Past: History Education and Reconciliation, ed. Elizabeth A. Cole (Lanham, MD: Rowman & Littlefield, 2007).

[51]    Michael Ignatieff, The Warrior’s Honor: Ethnic War and the Modern Conscience (New York: Metropolitan Books, 1998), 173. For accounts of interviews with teachers in Guatemala and Northern Ireland who tackle “difficult issues” in their classrooms, see the chapters by Elizabeth Oglesby and Alison Kitson in Cole, ed., Teaching the Violent Past.

[52]    Jacques E.C. Hymans, “What Counts as History and How Much Does History Count? The Case of French Secondary Education,” in The Nation, Europe and the World: Textbooks and Curricula in Transition, ed. Hanna Schissler and Yasemin Nuhoglu Soysal (New York and Oxford: Berghahn Books, 2005), 61, citing David John Frank, et al., “What Counts as History: A Cross-National and Longitudinal Study of University Curricula,” in Comparative Education Review 44 (2000): 29–53.

[53]    Joseph Kahn, “Where’s Mao? Chinese Revise History Books,” in The New York Times, 1 September 2006.

[54]    See Mirela-Luminita Murgescu, “Rewriting School Textbooks as a Tool of Understanding and Stability,” Southeast European and Black Sea Studies 2:1 (2002): 90–104; available at http://dx.doi.org/10.1080/14683850208454674.

[55]    Wolfgang Höpken, “Textbooks and Reconciliation in Southeastern Europe,” in Culture and Reconciliation in Southeastern Europe, International Conference, Thessaloniki, Greece, 26–29 June 1997 (Thessaloniki: Paratiritis, 1997). Cited in Murgescu 2002, p. 96/8.

[56]  She refers to Evangelos Kofos (1999), “Textbooks: The Pendulum of ‘Loading’ and ‘Disarming’ History: The Southeastern European Test Case,” in Disarming History. International Conference on Combating Stereotypes and Prejudice in History Textbooks of Southeast Europe (Stockholm: Nykopia Tryck AB, 1999), p. 23–27. Cited in Murgescu, 2002, p. 96/8.

[57]  She refers to Paul Connerton, How Societies Remember (Cambridge: Cambridge University Press, 1989). Cited in Murgescu, 2002, p. 102/14.

[58]  She refers to Deborah S. Hutton and Howard D. Mehlinger, “International Textbooks Revision. Examples from the United States,” in Perceptions of History. International Textbooks Research on Britain, Germany and the United States, ed. Volker R. Berghahn and Hanna Schissler (Oxford: Berg, 1987), 141. Cited in Murgescu, 2002, p. 102/14.

[59]  She refers to Wolfgang Höpken, ed., Öl ins Feuer? Schulbücher, ethnische Stereotypen und Gewalt in Südosteuropa (Hannover: Verlag Hahnsche Buchhandlung, 1996), p. 120. Cited in Murgescu, 2002, p. 102/14.

[60]  Christina Koulouri, ed., Clio in the Balkans: The Politics of History Education (Thessaloniki: CDRSEE, 2002); available at www.cdsee.org/pdf/clio_in_the_balkans.pdf.

[61]  Emilija Simoska, Appendix, “FYR Macedonia,” in Clio in the Balkans, 495–97.

[62]  Nikola Jordanovski, “Between the Necessity and the Impossibility of a ‘National History’,” in Clio in the Balkans, 265–76.

[63]  Preface to the Macedonian edition by Irena Stefoska, p. 9, http://www.cdsee.org/jhp/pdf/WorkBook1_mak.pdf.

[64]    The report under the title “Higher Education: Use of New Data Could Help Improve Oversight of Distance Education” (GAO-12-39) was presented to the relevant committees in the U.S. Congress by the United States Government Accountability Office in November 2011. The full text of the original report is available at www.gao.gov/products/GAO-12-39.

[65]    20 U.S.C. § 1001 et. seq. Title IV of the Higher Education Act (20 U.S.C. §§ 1070-1099d) authorizes programs that provide financial assistance to students attending a variety of postsecondary schools.

[66]    While this report focuses on the student financial assistance provided under Title IV of the Higher Education Act, the Department of Defense (DOD) and the Department of Veterans Affairs (VA) also provide financial aid for postsecondary education. In fiscal year 2010, the DOD’s Military Tuition Assistance Program provided $ 531 million in tuition assistance to approximately 302,000 service members, while VA provided $ 9 billion in education benefits to service members and veterans.

[67]    20 U.S.C. § 1003(7) and 34 C.F.R. § 600.2.

[68]    In October 2010 (75 Fed. Reg. 66,832), Education published final regulations to, among other things, clarify what is required for an institution of higher education to be considered legally authorized by a state (see 34 C.F.R § 600.9). To clarify these requirements for distance education, the department provided that a school offering distance education programs to students in a state where the school is not physically located must meet any state requirements for it to be offering postsecondary distance education in that state (at 34 C.F.R. § 600.9(c)). The Association of Private Sector Colleges and Universities challenged Education’s state authorization regulations in the District of Columbia U.S. District Court. In July 2011, the court declined to address whether the state authorization regulations exceeded Education’s authority but vacated the paragraph related to distance education on procedural grounds, finding that Education failed to provide notice and the opportunity for comment because the paragraph was not included in the proposed regulations (2011 WL 2690406). Both parties have appealed that decision to the United States Court of Appeals for the District of Columbia Circuit.

[69]    The National Advisory Committee on Institutional Quality and Integrity (NACIQI), a body comprising higher education officials that meets periodically to advise the Secretary of Education on accreditation matters, has established a policy agenda for 2011 that includes a review of the roles and responsibilities of triad members, as well as the division of responsibility and accountability among the members. NACIQI is expected to provide the Secretary with its recommendations on this issue and others related to the reauthorization of the Higher Education Act.

[70]    In order to participate in Title IV programs, schools must apply to Education and meet minimum eligibility requirements, including those related to financial responsibility and administrative capability. Once a school’s application for participation is certified, it must reapply for recertification at least every 6 years. 20 U.S.C. § 1099c(g).

[71]    34 C.F.R. § 602.22.

[72]    20 U.S.C. § 1099b(a)(5). Education is required to publish a list of nationally recognized accrediting agencies that the Secretary determines to be a reliable authority as to the quality of education or training provided by the schools they accredit. 20 U.S.C. § 1001(c). Accrediting agencies must renew their recognition with Education at least every 5 years. 20 U.S.C. § 1099b(d).

[73]  Higher Education Amendments of 1992, Pub. L. No. 102-325, 106 Stat. 448.

[74]  Higher Education Reconciliation Act of 2005, Pub. L. No. 109-171, Title VIII, Subtitle A, 120 Stat. 4, 155 (2006).

[75]  This change was made following Education’s completion of a mandated distance education demonstration project. The project was undertaken to (1) test the quality and viability of expanded distance education programs, (2) provide increased student access to higher education, and (3) determine the specific statutory and regulatory requirements that should be altered to provide greater access to high-quality distance education programs. In 2005, Education reported to Congress that waivers of the 50 percent rule did not lead to increases in fraud and abuse of Title IV funds.

[76]  Higher Education Amendments of 1998, Pub. L. No. 105-244, 112 Stat. 1581.

[77]  Pub. L. No. 110-315, 122 Stat. 3078. Both institutions and individual programs can be accredited. When an institution is accredited, all courses and programs are covered, whereas in program accreditation, only the specific program is reviewed. However, for distance education, both types of accreditation require the agency to have distance education in its scope of review. 20 U.S.C. § 1099b(n)(3) and 20 U.S.C. § 1088(b)(3).

[78]  20 U.S.C. § 1099b(a)(4)(B).

[79]  20 U.S.C. § 1099b(a)(4)(B)(ii).

[80]  20 U.S.C. § 1099b(c)(1).

[81]  20 U.S.C. § 1009b(c)(2).

[82]  Recognized accrediting agencies that do not already have distance education within their scope of review may add distance education to their scope by notifying Education in writing. Such agencies must monitor the head count enrollment at each school they accredit, and if any school experiences an increase of 50 percent or more within 1 year, the agency must report that information to Education and also submit a report outlining the circumstances of the increased enrollment and how the agency evaluates the capacity of the school. Education submits that report to NACIQI for consideration in reviewing the agency’s change in scope. 20 U.S.C. §§ 1099b(a)(4)(B)(i)(II) and 1099b(q); 34 C.F.R. §§ 602.19(e), 602.31(d), and 602.34(c)(1).

[83]  U.S. Department of Education, Distance Education at Degree-Granting Postsecondary Education Institutions: 2006-07 (Washington, D.C.: Department of Education, December 2008). Data in this report are based on surveys sent to a nationally representative sample of approximately 1,600 Title IV eligible degree-granting postsecondary schools.

[84]  I. Elaine Allen and Jeff Seaman, Class Differences: Online Education in the United States, 2010 (Wellesley, MA: Babson Survey Research Group and the Sloan Consortium, November 2010). The sample for this analysis is composed of all active, degree-granting institutions of higher education in the United States that are open to the public. Questions for this study were included in the College Board’s Annual Survey of Colleges.

[85]  The university uses a competency-based model, where students’ progress is determined by what they know as opposed to seat time or credit hours. The focus is on ensuring students possess the skills and knowledge they need to be successful.

[86]  There were a few exceptions: Students at two schools are charged an additional technology fee to take online courses. At another school, an official said the school charged lower tuition for distance education classes when it was less expensive to deliver – such as when the courses are taken on a military base that does not charge the school for rent.

[87]  The Integrated Postsecondary Education Data System (IPEDS) is the federal government’s core postsecondary data collection program. All postsecondary schools participating in federal student financial aid programs are required to complete a group of annual surveys on a variety of topics. While IPEDS has the most up-to-date, comprehensive data on postsecondary schools, the only distance education data collected are whether an institution offers distance education opportunities to its students. No data on the extent of a school’s distance education offerings are collected. Additionally, because the IPEDS definition of distance education course has changed over time, consistent trend data are not readily available.

[88]  Minority-serving institutions are defined in law for purposes of certain grant programs authorized under Title III and Title V of the HEA. The three main types of minority-serving institutions are Historically Black Colleges and Universities, Tribal Colleges and Universities and Hispanic-serving institutions. Other types of minority-serving institutions include Alaska Native, Native Hawaiian, Asian American, Native American, and Pacific Islander-serving institutions. All institutions except Historically Black Colleges and Universities and Tribal Colleges and Universities are defined, in part, by the percentage of minority students enrolled.

[89]  Data on Historically Black Colleges and Universities and Tribal Colleges and Universities are derived directly as reported in IPEDS. GAO extracted data on other institutions from IPEDS using the enrollment percentages prescribed for the Title III or V grant programs. Separate data are not provided for Alaska Native and Native Hawaiian-serving institutions due to limitations with the 2009-2010 IPEDS data.

[90]  NPSAS is a recurring nationwide survey to collect demographic information on postsecondary students, as well as information on how they fund their education. We conducted our analysis using student-reported data from the most recent administration of the NPSAS, which covered students attending Title IV eligible schools during the 2007-2008 academic year. NPSAS surveys include a number of questions related to distance education.

[91]  Of the approximately 5 million distance education students, 4.28 million were undergraduates, 741,000 were graduate students, and 22,000 were first-professional students. A first-professional student is a student who is enrolled in a degree program that encompasses certain occupationally specific programs such as law and medicine. In total, there were an estimated 24.4 million postsecondary students, including 20.9 million undergraduates, 3.2 million graduate students, and 282,000 first-professional students in the 2007-2008 school year.

[92]  When we cite differences in student characteristics between distance education students and students who did not take any distance education courses, the differences are statistically significant at the 95 percent confidence level.

[93]  For the analyses of distance education participation among students of different races and ethnicities, military status, and disability status, first-professional students are included, as well as undergraduate and graduate students.

[94]  U.S. Government Accountability Office, Distance Education: Growth in Distance Education Programs and Implications for Federal Education Policy, GAO-02-1125T (Washington, D.C.: GAO, 26 September 2002).

[95]  Likely as a result of working full time more often, graduate students taking a distance education course or program also had higher average incomes than graduate students who took no distance education. In addition, students in the lowest income quartile (the lowest 25 percent of income) enrolled in distance education courses or programs less often than students with higher incomes.

[96]  As might be expected, distance education students more often attend school part time. For undergraduates, about 43 percent of the students who took distance education courses or programs were part-time students, while about 33 percent of the students who did not take any distance education courses were part-time students. This trend also occurred among graduate students (about 59 percent of those who took distance education courses or programs were part-time students compared with about 45 percent of those who did not take any distance education courses).

[97]  When we say all races and ethnicities, we are referring to all races and ethnicities reported by students to the NPSAS.

[98]  In addition, of all students taking distance education courses or degree programs, a greater percentage of white students took distance education courses or programs when compared to other racial and ethnic groups. White students represented about 66 percent of all students who took distance education courses or programs compared with about 61 percent of students who took no distance education courses or programs and about 62 percent of students overall.

[99]  Veterans and service members may be eligible to receive educational benefits from either the Department of Veterans Affairs or the Department of Defense when participating in a variety of programs, including distance education. See U.S. Government Accountability Office, VA Education Benefits: Actions Taken, but Outreach and Oversight Could Be Improved, GAO-11-256 (Washington, D.C.: GAO, 28 February 2011); DOD Education Benefits: Increased Oversight of Tuition Assistance Program Is Needed, GAO-11-300 (Washington, D.C.: GAO, 1 March 2011).

[100]  Disability data are student-reported and taken from NPSAS. Data on the rates at which reservists with disabilities took distance education courses or programs were not sufficiently reliable to report.

[101]  While we analyzed distance education participation for students with disabilities, we did not evaluate issues of accessibility for these students.

[102]  The percentage of students taking distance education courses and programs who had any type of disability was not significantly higher than the percentage for students taking no distance education. However, the percentage of students taking distance education courses and programs who had disabilities specifically affecting their mobility was significantly higher than the percentage for students taking no distance education. Percentage differences were not significant for students with other types of disabilities, including visual impairments, hearing impairments, learning disabilities, and mental conditions.

[103]  This is lower than the general student population, which averages about 25 percent of enrollments online. However, the percentage of online enrollments increased for students with physical impairments (26 percent) and visual impairments (35 percent).

[104]  Schools can be institutionally accredited by regional or national accrediting agencies, or both, and may also have degree programs in certain fields of study accredited by specialized accreditors. The seven federally recognized regional accrediting agencies review schools in their prescribed geographic region of states, whereas the seven federally recognized national accreditors can accredit schools across the United States. Regional accrediting agencies tend to accredit degree-granting colleges and universities, while national accrediting agencies tend to accredit non-degree-granting career schools. Nineteen of the 20 schools we interviewed were regionally accredited and 3 were nationally accredited, with 2 schools receiving both regional and national accreditation.

[105]  20 U.S.C. §§ 1099b(a)(4) and 1099b(a)(5); 34 C.F.R. §§ 602.16 and 602.17.

[106]  In addition to reviewing the roles and responsibilities of Triad members, NACIQI has been assessing ways to improve the accreditation process. In exploring ways accreditors can better use data to assess program quality, members have discussed the benefits and drawbacks of standardized learning outcomes.

[107]  20 U.S.C. § 1099b(c)(1).

[108]  Ibid.

[109]  Among other requirements for recognition by Education, accrediting agencies must have substantive change policies that meet certain requirements as prescribed by Education. 34 C.F.R. § 602.22.

[110]  In 2006-2007, about 94 percent of schools that offered distance education courses developed them in-house. U.S. Department of Education, Distance Education at Degree-Granting Postsecondary Institutions, 2006-07 (Washington, D.C.: Department of Education, December 2008).

[111]  In September 2011, Education’s OIG reported that, over the past 6 years, it had conducted numerous investigations of fraud involving distance education programs. Primarily, these investigations centered around fraud rings whereby a ringleader posing as a student facilitates enrolling other “straw students,” those who may not be eligible or do not intend on attending class, in order to collect Title IV student federal aid. For the full report, see Department of Education, Office of Inspector General, Investigative Program Advisory Report, Distance Education Fraud Rings, L42L0001 (Washington, D.C.: Department of Education, 26 September 2011).

[112]  The OIG reported that distance education might represent increased risk for student federal aid programs as early as 2000. According to the Acting Inspector General’s October 14, 2009, testimony before the Higher Education, Lifelong Learning, and Competitiveness Subcommittee, House Committee on Education and Labor, the potential for fraud in distance education stems from the difficulty in verifying student identity and ensuring that enrolled students are engaged in academic activity, given the limited or no face-to-face contact. The Acting Inspector General said that the rapid growth of distance education, combined with the paperless delivery of student aid funds (i.e., schools may not have an in-person relationship with the student), makes distance education vulnerable to fraud. Additionally, in fiscal year 2011, the OIG identified distance education as a management challenge for the department and started an audit reviewing the extent to which Education has adapted Title IV regulations to address these issues.

[113]  FSA officials said in recent years they have conducted about 200 program reviews per year. The reviews assess a school’s institutional eligibility, financial responsibility, and administrative capability for participating in Title IV student aid programs. Schools are selected for program reviews based on specific risk indicators, such as a referral or complaint, or as a result of a comprehensive compliance review, although officials said schools that do not meet these criteria may also be selected.

[114]  Specific dates covered by program reviews and audits that had issued final determinations were October 1, 2005, through May 25, 2011. Substantive interaction was not required until 2008 when the Higher Education Opportunity Act added the definition of distance education to the HEA, which included that requirement. Pub. L. No. 110-315, § 103(a)(1), 122 Stat. 3078, 3087 (2008).

[115]  FSA used several indicators to identify a school’s risk, including a change in school sector (e.g., from proprietary to private nonprofit or from private nonprofit to proprietary), an audit or investigation by the OIG, and the distribution of a high percentage of full student loans, as this may be an indicator that a school is not appropriately monitoring student withdrawals for return of student aid funds.

[116]  FSA officials said they conducted 25 reviews and the OIG is conducting 2 audits.

[117]  Representatives from the following Education departments also participated: Office of the Secretary, Office of the Undersecretary, Office of the General Counsel, and Office of Postsecondary Education.

[118]  In previous work, we found that, in fiscal year 2009, 71 percent of courses taken by military members using tuition assistance were distance education courses (see GAO-11-300).

[119]  The 27 schools were selected from an initial list of 2,710 schools identified as offering some distance education. As of September 7, 2011, FSA reported that it had finalized and issued reports for 11 schools for which there were no findings related specifically to distance education.

[120]  Schools may need to apply for recertification more frequently if they, for example, change their ownership or status. 34 C.F.R. § 600.20(b).

[121]  NCES is the primary federal entity for collecting and analyzing data on the condition of education in the United States and other nations.

[122]  According to GAO’s standards for internal control, program managers need operational data to determine whether they are meeting their agency’s goals for accountability for effective and efficient use of resources. See U. S. Government Accountability Office, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: GAO, November 1999).

[123]  An NCES official said NCES provided a comment period after the publication of the technical review panel’s suggested changes and the proposed changes were also published in the Federal Register.

[124]  U.S. Government Accountability Office, Federal Student Loans: Patterns in Tuition, Enrollment, and Federal Stafford Loan Borrowing Up to the 2007-08 Loan Limit Increase, GAO-11-470R (Washington, D.C.: GAO, 25 May 2011).

[125]  U.S. Department of Education, Distance Education at Degree-Granting Postsecondary Education Institutions: 2006-2007 (Washington, D.C.: Department of Education, December 2008).

[126]  The Sloan Consortium is a membership organization that helps schools and professionals improve the quality, scale, and breadth of online education through conferences, workshops, and research.

[127]  Allen and Seaman, Class Differences: Online Education in the United States, 2010. The Sloan Consortium, in conjunction with the Babson Survey Research Group, conducts an annual survey of a sample of degree-granting schools of higher education in the United States. For the 2010 report, 2,583 schools responded to the survey – a 57 percent response rate for the sample universe of 4,511 schools. The schools that responded represent 80 percent of higher education enrollments.

[128]  NCES collects characteristics of students studying at postsecondary schools using a nationally representative sample through the NPSAS survey every 3 to 4 years.

[129]  The three regional accrediting agencies are the Middle States Commission on Higher Education; the Western Association of Schools and Colleges, Accrediting Commission for Community and Junior Colleges; and the Western Association of Schools and Colleges, Accrediting Commission for Senior Colleges and Universities. The two national accrediting agencies are the Accrediting Council for Independent Colleges and Schools and the Distance Education Training Council. The specialized accrediting agency is the Commission on Collegiate Nursing Education.

[130]  CHEA is a national association of schools and accrediting agencies that advocates for accreditation and recognizes accreditors.

[131]  Quality Matters was started through a Fund for the Improvement of Postsecondary Education grant given to Maryland Online, a consortium of colleges and universities in Maryland. It is a fee-based, nonprofit program that certifies the quality of online and blended courses through a peer review process.

[132]  At the time of our interview with its officials in January 2011, Keiser University-Fort Lauderdale was a private for-profit school. However, shortly after our interview, the university became a private nonprofit school. Since we interviewed the officials as a private for-profit with no knowledge of the school’s forthcoming transition, we are categorizing them as a for-profit school.

Table of Contents

Mobile Learning in Security and Defense Organizations

Christian Glahn *

Research Ethics in the Mobile Learning Environment (MoLE) m-Learning Project

Jacob Hodges and Geoff Stead *

Introduction

Research Design

Human Research Protection

Data Collection

Data Analysis and Interpretation

Mobile Learning Environment Project

Human Research Protection Approach

Data Collection, Analysis, and Interpretation Approach

Research Strategy

Human Research Protection

Data Collection, Analysis, and Interpretation

Application Development

Evaluation Process

Volunteer Numbers

The Wider Project

Conclusion

Supporting Crisis Simulations with the ARLearn Toolkit for Mobile Serious Games

Stefaan Ternier, Atish Gonsalves, Bernardo Tabuenca, Fred de Vries, and Marcus Specht *

Introduction

ARLearn Toolkit

Dependencies

Notifications

Roles

Tagging and Scanning

The UNHCR Hostage-Taking Pilot Project

Game Design

Results

Conclusion and Discussion

Future Work

Supporting Learner Mobility in SCORM-Compliant Learning Environments with ISN Mobler Cards

Christian Glahn *

Introduction

Mobile Learning and SCORM

Challenges for Research

Educational Design Underpinnings

Integration with SCORM Concepts

The Mobler Cards App

Application Design Requirements

Proof of Concept

Conclusions and Implications for Practice

Mobile Learning and ADL’s Experience API

Kristy Murray, Peter Berking, Jason Haag, and Nikolaus Hruska *

Introduction

Mobile Learning

Overview of the xAPI

Learning Activity “Streams” and the Learning Record Store

ADL’s Mobile Training Implementation Framework (MoTIF) Project

Mobile Learning and the xAPI

Conclusion

The Role of Teaching History for a Nation-Building Process in a Post-Conflict Society: The Case of Macedonia

Biljana Popovska *

Introduction

Unite or Divide?

Transitional Justice and Reforming History Education

School Textbook Revision and Stability

The Case of Macedonia

Conclusions

GAO Report on Higher Education

Use of New Data Could Help Improve Oversight of Distance Education [64]

Highlights

Main Findings

Higher Education

Background

Oversight

Brief History of Statutory Provisions Related to Distance Education

Distance Education Has Become Common in All Sectors and Is Offered through a Range of Programs and Courses

Online Distance Education Has Grown Dramatically and Is Offered in a Variety of Ways

A Wide Variety of Schools Provide a Range of Distance Education Courses and Programs

Students in Distance Education Enroll Mostly in Public Schools and Represent a Diverse Population

Most Distance Education Students Attend Public Schools and Study a Range of Subjects

While Students in Distance Education Tend to Be Older and Female, and Have Family and Work Obligations, They Are Also a Diverse Population

Accreditors and Schools Assess the Academic Quality of Distance Education in Several Ways, but Accreditors Reported Some Oversight Challenges

Accrediting Agencies Examine the Quality of Distance Education in Various Ways but Reported Some Challenges

Schools Use a Range of Course Design Principles and Student Performance Assessments to Hold Distance Education to the Same Quality Standards as Traditional Courses

Education Has Increased Its Monitoring of Distance Education but Lacks Sufficient Data to Inform Its Oversight

Conclusions

Recommendations for Executive Action

Agency Comments and Our Evaluation

Appendix I: Objectives, Scope, and Methodology

Appendix II: List of Colleges and Universities GAO Interviewed

Appendix III: GAO Contact and Staff Acknowledgments

Related GAO Products