Appendix A: Full Literature Review

ISTE’s California Digital Learning Integration and Standards Guidance Project Literature Review 

Prepared by Caitlin Clause McLemore, Ed.D. 

Introduction 

ISTE’s California Distance Learning Curriculum and Instructional Guidance project, funded by the Sacramento County Office of Education (SCOE), aims to draft a curriculum and instructional guide for critical subject areas related to distance learning. Specifically, the guide aims to address factors related to distance learning such as administering meaningful assessments, deciding between asynchronous and synchronous instruction, ensuring well-being, bolstering social and emotional learning skills, determining instructional time, and understanding subject-specific guidelines. The primary goal of this literature review is to identify research-based best practices for educators in distance learning in general, along with the previously identified relevant factors. ISTE and its partners will use this literature review to inform the California Distance Learning Curriculum and Instructional Guidance project, which will include a formal, written report. 

Research Objective 

The primary research objective for this literature review was to identify research-based best practices for educators in distance learning to inform ISTE’s development of the California Distance Learning Curriculum and Instructional Guidance project. The literature review focused on critical variables related to distance learning such as administering meaningful assessments, deciding between asynchronous and synchronous instruction, ensuring well-being, bolstering social and emotional learning skills, determining instructional time, and understanding subject-specific guidelines. 

Search Methodology 

The following databases were used to locate sources for inclusion in the literature review: 

• Gale Academic OneFile 
• Google Scholar 
• Michigan Virtual Learning Research Institute Research Clearinghouse for K-12 Blended & Online Learning 
• Quality Matters (QM) Research Library 

Search Terms 

The following search terms were entered into the databases listed above to conduct a comprehensive search for relevant meta-analyses and systematic reviews: 

• “K–12” AND: 
• “Distance education” 
• “Distance learning” 
• “Online learning” 

Inclusion Criteria 

We applied the following inclusion criteria to the articles resulting from the search: 

1. Must be a meta-analysis or systematic review 

2. Must be peer-reviewed (e.g., scholarly journal article, conference proceedings) or from a reputable source (e.g., educational research organization, a government department) 

3. Must be published in 2000 or later 

4. Must include study populations of K-12 students or teachers 

5. Must relate to distance learning instructional design or pedagogical practices 

6. Must be relevant to the aims of the California Distance Learning Curriculum and Instructional Guidance project 

Due to the volume of individual research studies with relevant research, we limited our search to meta-analyses and systematic reviews. Meta-analyses collect individual research studies with a particular focus to calculate an overall effect size from the collected studies (Card, 2012). Systematic reviews gather an extensive collection of individual research studies with a particular focus to create an overall summary of information gathered from the collected studies. Meta-analyses provide a quantitative or numerical summary of relevant literature, while systematic reviews provide a qualitative or non-numerical summary of relevant literature. Meta-analyses and systematic reviews provide insight into an estimate of effect sizes and overall findings from relevant research. 

For this literature review, included studies were required to be peer-reviewed or published through a reputable source (e.g., a government or research organization). The included studies were also required to be published in 2000 or later to provide updated technological information. Additionally, study populations were required to include K-12 students or teachers, at minimum, and several focused solely on the K-12 environment. The final inclusion criteria ensured a focus on the primary goal of the literature review, to determine best practices related to distance learning instructional design and pedagogical strategies as determined by the aims of the California Distance Learning Curriculum and Instructional Guidance project. 

Search Processes 

After conducting initial database queries, a list of potential sources was reviewed to ensure they met the inclusion criteria. In addition to research specifically focused on K-12 online and blended learning, this literature review also included sources that addressed or explored elements related to the overarching project topics, such as serving students with disabilities, engaging students in online learning, and identifying teacher characteristics specific to online pedagogy. Based on these criteria, a total of 23 articles remained for consideration in the review (see References). This included six meta-analyses, 16 systematic reviews, and one source that included both a systematic review and meta-analysis (Means et al., 2010). Three of the six meta-analyses and 12 of the 16 systematic reviews focused specifically on K-12 students or teachers. The rest of the sources included K-12 students or teachers within a broader scope (e.g., K-12 and higher education or adult learners). 

Systematic reviews were published in peer-reviewed scholarly journals, by educational research organizations (e.g., Marzano Research, Michigan Virtual Learning Research Institute), in educational conference proceedings (e.g., Society for Information Technology and Teacher Education), or by the U.S. Department of Education. All meta-analyses were published in peer-reviewed scholarly journals (5) or by the U.S. Department of Education (2). 

Literature Summary 

This section includes a summary of the literature. A glossary of terms used in subsequent sections of the literature review is included in the Supplementary Materials. Table 1 includes a summary of results from meta-analyses and Table 2 includes a summary of results from systematic reviews. For more information about the characteristics and effect sizes of the meta-analyses, see Supplementary Materials. 

Table 1. Summary of Results from Meta-Analyses Included in Review 

Article	Purpose	Findings
Cavanaugh, 2001	Effectiveness of K-12 distance education on student learning outcomes	Student academic achievement is comparable in different settings of distance learning and traditional instruction.
Bernard et al., 2004*	Comparison research in distance education	No significant differences found in overall comparison of instructional settings.
Cavanaugh et al., 2004	Effect of distance learning on K-12 student outcomes	The effectiveness of distance learning and in-person classroom instruction is comparable.
Bernard et al., 2009*	Comparison research in distance education	Quality course design and student interactions have a positive impact on student learning.
Means et al., 2009*	Comparison of learning contexts (online, blended, face-to-face)	Students in blended/online learning environments performed better, on average, than students in face-to-face instruction. Differences may be due to instructional content, pedagogical methods, and time.
Means et al., 2013*	Comparison of learning contexts (online, blended, face-to-face)	Students in blended/online learning environments performed better, on average, than students in face-to-face instruction. Differences may be due to instructional content, pedagogical methods, and time.
Larwin & Erickson, 2016	Effectiveness of distance education on achievement for K-12 students with disabilities	Students with disabilities performed better online compared to those in face-to-face instruction, but worse than students without disabilities in the same online courses. Effective communication and high-quality student-teacher interactions are ultimately what impact student learning outcomes.

*included all learners (not just focused on K-12) 

Table 2. Summary of Results from Systematic Reviews Included in Review 

Article	Study purpose	Key findings
Rice, 2006	Examining student and teacher characteristics and qualities in distance education	Some learners succeed online and some do not. Understanding student success comes from a learner-focused, not tool-focused, approach. Teacher quality may be the most influential factor in student success.
Barbour & Reeves, 2009*	Exploring the benefits and challenges of virtual schools and identifying student characteristics of online learners	Learning is a social experience. Typical characteristics of online learners include autonomy and independence, but K-12 students need more guidance and structure.
Cavanaugh, Barbour, & Clark, 2009	Examining K-12 online learning including benefits and challenges, educator roles, and learning models.	Mixed results determining the advantages or disadvantages of online learning models. Online learning might result in better outcomes, but there is no guarantee of effectiveness.
Means et al., 2009*	Comparison of learning contexts (online, blended, face-to-face)	Blended/online learning environments resulted in comparable student learning outcomes when compared with face-to-face instruction.
Vasquez & Straub, 2012	Examining online learning research related to students with disabilities and teachers	Students with disabilities performed better when familiar with content, tools, and through exposure to accessible websites. Educators and researchers need to identify accessible online instructional methods.
Hasler-Waters, Barbour, & Menchaca, 2014	Describing and evaluating the effectiveness of online charter schools	Lack of evidence of effectiveness with full-time online learning as compared to traditional schools.
Barbour, 2015	To challenge the expansion of full-time, K-12 online learning programs	Lack of reliable research evidence to support the expansion of full-time online learning. Student performance in full-time charter or virtual schools sometimes indicates lower or mixed results compared to traditional schools.
Henrie, Halverson, & Graham, 2015*	Measuring student engagement in technology-mediated learning	Student engagement levels provide information about student academic and social outcomes, helps identify students who need additional support, and provides evidence of activity, course, or tool quality.
Brodersen & Melluzzo, 2017	Determining the effectiveness of differentiated learning programs on student achievement outcomes in K-12 online, blended, and face-to-face learning programs	Blended learning programs that exhibited a statistically significant effect on student achievement outcomes: Cognitive Tutor Algebra 1, LeapTrack, READ 180, Time to Know. All blended learning programs used online information to inform in-class instruction.
Martin, Ahlgrim-Delzel, & Budhrani, 2017*	Examining characteristics of synchronous online learning (e.g., content, student characteristics)	Modern technology tools provide synchronous learning opportunities that “enable communication, collaboration, and critical discussion” (p. 12).
Barbour, 2018	To explore how higher education blended and online learning research can inform K-12 blended and online learning	Learning analytics can help to evaluate pedagogical effectiveness. Online educators hold multiple roles of teacher, facilitator, and designer.
Pulham & Graham, 2018	Comparing K-12 online and blended teaching	Instructional design is critical to online learning. In blended learning, it is important to integrate online and face-to-face instruction. Successful teachers in blended learning environments exhibit a commitment to change, growth mindset, and self-reflection. Dynamic digital materials are adaptive, interactive, and personalized.
Pulham, Graham, & Short, 2018	Comparing teaching competencies for blended, online, and face-to-face learning	Research indicates that teachers need different skills to be successful in blended, online, and face-to-face learning environments, but often these differences are not acknowledged within teacher competencies or preparation programs.
Arnesen et al., 2019	Examining K-12 online learning research	K-12 online learning research includes high-quality scholars that are also influential in the broader fields of distance learning and educational technology. Top K-12 online learning research topics include assessment, creating and evaluating distance education programs, student characteristics, student achievement outcomes, and teacher preparation.
Hu et al., 2019	Examining K-12 online learning research published in the Journal of Online Learning Research	The majority of K-12 online learning research focuses on context comparison. The Journal of Online Learning Research is the leading publication for K-12 online learning research.
Arnesen et al., 2020	Examining low and uncited K-12 online learning research	Low and uncited K-12 online learning research often focuses on international and rural school settings, but also explores general benefits and challenges.
Barbour, 2020	K-12 online learning research	“It does not matter if the student is separated by time, place, path, and/or pace from their teacher or other students. As long as the appropriate conditions for learning are present, learning will occur” (p. 15).

*included all learners (not just focused on K-12) 

Research Findings 

To summarize research findings, we integrated findings from both meta-analyses and systematic reviews into one comprehensive section. Topics include general findings on K-12 distance learning, benefits, effective pedagogical strategies, asynchronous vs. synchronous learning, assessment, teacher quality, student characteristics, and challenges. Not included in the initial sections but noted in the research findings section are citations to chapters from the Handbook of Research on K-12 Online and Blended Learning (Kennedy & Ferdig, 2018). Each chapter within the Handbook serves as a literature review around key topics, highlighting research-based best practices. 

Distance learning is comparable to traditional classroom instruction 

Some sources found a modest improvement in student learning outcomes in online learning environments (Barbour & Reeves, 2009; Means et al., 2013), while others found a lack of evidence of effectiveness when comparing full-time online learning with traditional instruction (Hasler-Waters, Barbour, & Menchaca, 2014; Barbour, 2015). However, in most of the meta-analyses that compared blended or online learning to traditional classrooms, overall effect sizes were small and often close to zero (Cavanaugh, 2001; Bernard et al., 2004; Cavanaugh, 2004; Means et al., 2009). These results indicated no difference in the impact of instructional setting on student achievement, engagement, or other student learning outcomes. For more information on effect sizes, see Supplementary Materials. 

Though overall effect sizes were small, the effect sizes reported within individual studies varied widely. As such, researchers in both meta-analyses and systematic reviews surmised that factors of course quality, instructional design, meaningful interactions, pedagogical methods, and student characteristics contributed a greater impact on student learning than differences in instructional setting (Bernard et al., 2004; Cavanaugh et al., 2004; Bernard et al., 2009; Cavanaugh, Barbour, & Clark, 2009; Larwin & Erickson, 2016; Pulham & Graham, 2018). This sentiment was best expressed by Barbour (2020), who stated that “it does not matter if the student is separated by time, place, path, and/or pace from their teacher or other students. As long as the appropriate conditions for learning are present, learning will occur” (p. 15). Rather than focusing on where learning happens, the focus should be on “who is teaching, who is learning, and how that learning is accomplished” (Rice, 2006, p. 440). 

K-12 distance learning is unique 

Due to the developmental needs of young learners, K-12 distance learning cannot (or rather, should not) replicate higher education (Cavanaugh et al., 2004). For example, K-12 students may find synchronous instruction more beneficial as it is “better suited to their academic schedules and their need for spontaneous guidance and feedback” (Bernard et al., 2004, p. 33). As they are still developing as learners, K-12 students need additional guidance and support in online learning environments, particularly in developing intrinsic motivation (Vazquez & Straub, 2012) and social skills (Rice, 2006). 

Benefits of distance learning 

• The most referenced benefit to distance learning included expanding access to opportunities that students would not otherwise be able to access (Cavanaugh, 2001; Cavanaugh et al., 2004; Barbour & Reeves, 2009; Means et al., 2009; Vazquez & Straub, 2012; Means et al., 2013; Hasler-Waters, Barbour, & Menchaca, 2014; Arnesen et al., 2020). Distance learning provides opportunities to serve diverse student populations. For example, students with health issues that prevent them from attending in-person schools can take online courses, or students in rural locations have access to a wider variety of course offerings. 
• Distance learning facilitates skill development of digital age skills, such as collaboration, critical thinking, metacognition, and self-reflection (Cavanaugh et al., 2004; Means et al., 2009; Means et al., 2013). This occurs through increased opportunities for interactive (Means et al., 2013) and reality-based (Cavanaugh, 2001) learning. 
• Distance learning provides students with increased flexibility and choice (Rice, 2006; Barbour & Reeves, 2009; Means et al., 2009; Vazquez & Straub, 2012; Hasler-Waters, Barbour, & Menchaca, 2014; Pulham & Graham, 2018; Arnesen et al., 2020). With online courses, students can choose a learning path that works best for them. Additionally, students can often adjust their course schedules to accommodate other timing constraints. 
• Distance learning also increases exposure to diverse perspectives (Cavanaugh, 2001) and increases student motivation (Barbour & Reeves, 2009). 

Learning is a social experience 

Based on Interaction and Social Presence theory, online learning requires a variety of social interactions (Garrett Dikkers, 2018; Moore, 1989). Quality student interactions increase cognitive engagement, which in turn improves student outcomes including content retention (Rice, 2006; Barbour & Reeves, 2009; Bernard et al., 2009; Henrie, Halverson, & Graham, 2015, Larwin & Erickson, 2016; Garrett Dikkers, 2018). 

Student interaction types include student-student (how students interact with their peers), student-teacher (how students interact with their teachers), and student-content (how students interact with course content). 

• The most effective student-teacher interactions focused on critical and higher-order thinking, rather than content (Bernard et al., 2009). Consistent, positive relationships with teachers can improve retention rates in online learning, particularly for at-risk students (Rice, 2006). 
• High-quality, positive student-student interactions facilitate increased motivation and improved cognitive processes (Bernard et al., 2009). 

A common concern with online learning is a lack of social interaction, particularly with younger students. To address this concern, teachers should utilize technology tools such as discussion boards and synchronous sessions (e.g., chat, video conferencing) to facilitate student-student and student-teacher interactions and decrease feelings of social isolation (Rice, 2006). Students report positive feedback on communication tools like chat or instant messaging in virtual learning spaces because it allows them to communicate and socialize with others, both in formal and informal conversation with their peers and teachers (Barbour & Reeves, 2009). 

In addition to online communication tools, some social support strategies that can benefit students include “the encouragement of reflective thinking” and “dialogue, interaction and extension of ideas with feedback from peers and mentors on emerging issues” (McLoughlin, 2002, p. 152). 

Student emotional engagement includes students’ feelings about learning, as well as their social connections with other students (Henrie, Halverson, & Graham, 2015). Emotional engagement, along with behavioral and cognitive engagement, contributes to students’ academic achievement, persistence, and satisfaction as it helps students to develop social presence (Bernard et al., 2009). 

Incorporating student support structures is critical for student success in online learning. These support structures include, but are not limited to, behavioral, affective, and cognitive domains, and these can include help from peers, educators, the community, outside experts, and family/caregivers (Borup et al., 2020). Additionally, at-risk students need academic, behavioral, and mental support through a caring community, positive school climate, and connection with others (Repetto, Spitler, & Cox, 2018). 

Effective pedagogical strategies for impacting student learning outcomes in blended or online learning environments 

• Students in online learning environments benefit from individualized instruction (Cavanaugh, 2001; Vazquez & Straub, 2012; Hasler-Waters, Barbour, & Menchaca, 2014; Pulham & Graham, 2018; Arnesen et al., 2020) and a student-centered approach to learning (Rice, 2006; Pulham & Graham, 2018). 
• Student-centered learning includes: 
  • Opportunities for student-student collaboration (Means et al., 2009). 
  • Interactive lessons that facilitate digital age skills, such as collaboration, critical thinking, metacognition, and self-reflection (Cavanaugh, 2001; Means et al., 2009; Larwin & Erickson, 2016). 
  • Meaningful small group discussions (Cavanaugh, 2001; Vazquez & Straub, 2012) that use guiding questions (Means et al., 2010) based on real-life scenarios (Barbour, 2018). 
• Using digital learning materials and tools can improve student learning outcomes through dynamic, interactive, multimodal, and personalized learning experiences (Bernard et al., 2009; Pulham & Graham, 2018). These materials and tools incorporate data-informed decision-making to guide instruction and help students better understand more complex concepts in an efficient manner. Examples of digital learning materials and tools include content-specific websites (e.g., Khan Academy, Newsela), collaboration and creation tools (e.g., Flipgrid, G Suite for Education, Screencastify, WeVideo), learning management systems (e.g., Blackboard, Brightspace, Canvas, Google Classroom), and more. Additionally, using student data from digital learning tools can provide useful learning analytics for determining pedagogical effectiveness (Barbour, 2018). 
• Providing access to a learning coach or other local support is critical to student success in online learning (Borup, 2018; Barbour, 2015). Parents often serve in this role, yet they are unprepared to be successful in supporting students (Hasler-Waters, Barbour, & Menchaca, 2014). Thus, schools and teachers need to partner with students’ families to provide a community of support for online learners. 
• In blended learning, it is important to make clear curricular connections between in-person and online learning activities (Cavanaugh, 2001; Brodersen & Melluzzo, 2017; Pulham & Graham, 2018). 

Asynchronous vs. synchronous learning 

In general, effect sizes for student achievement favored asynchronous learning over face-to-face instruction but favored face-to-face instruction over synchronous learning (Bernard et al., 2004). With asynchronous learning, students exhibited improved achievement and attitudes but lower retention rates. 

For asynchronous learning, researchers found that a focus on active learning, collaboration, project-based learning, and quality course design was more impactful than which technology materials and tools were used (Bernard et al., 2004). For synchronous learning, researchers found the biggest impact on student learning when it was utilized to “enable communication, collaboration, and critical discussion” (Martin, Ahlgrim-Delzel, & Budhrani, 2017, p. 12). 

Regarding student development, younger children need more guidance and support in developing communication skills in virtual spaces (Rice, 2006; Barbour & Reeves, 2009). Teachers can utilize Vygotsky’s zone of proximal development to design activities and opportunities for social interaction that are developmentally appropriate (Barbour & Reeves, 2009). 

Teacher quality may be the most influential factor in student success, regardless of setting. 

As stated by Rice (2006), technology “cannot substitute for well-designed instruction and opportunities to engage in purposeful, interactive learning activities” (p. 440). Teacher competencies contributing to student learning include providing meaningful feedback (Cavanaugh, Barbour, & Clark, 2009), exhibiting a growth mindset, and engaging in self-reflection (Pulham & Graham, 2018). 

In addition to teaching duties, educators in online courses also hold multiple roles, depending on the setting, such as instructional designer and technology support (Barbour, 2018). An exemplar of online course design, the Florida Virtual School (FLVS) supports educators through collaborative teams that include an educator, but also instructional designer, web designers, technology support, and other support staff to ensure a successful learning experience for students (Barbour & Reeves, 2009). 

Assessment in online learning environments 

Assessment in online learning should include formative and summative assessment to determine student learning outcomes, including product, process, and progress (Barbour, 2018; Pulham & Graham, 2018). Online assessments should utilize technology tools to provide individualization and personalization utilizing multiple data sources (Pulham & Graham, 2018). Some examples of formative assessment in online learning environments include discussion participation, e-portfolios, and self-testing (Barbour, 2018). 

Characteristics of successful online learners 

Successful online learners exhibit independence through individual characteristics, such as autonomy (Barbour & Reeves, 2009), high levels of motivation (Barbour & Reeves, 2009), persistence (Cavanaugh et al., 2004; Rice, 2006), responsibility (Rice, 2006; Barbour & Reeves, 2009), and self-regulation (Cavanaugh et al., 2004; Rice, 2006). Additionally, sufficient access to technology and skills for using available technology contributed to a successful online learning experience (Rice, 2006). 

Concerning students with disabilities, researchers found that these students performed better in online learning environments compared to similar students in face-to-face instruction, but performed worse than students without disabilities participating in the same online courses (Larwin & Erickson, 2016). Instructional strategies found to help improve the learning outcomes of students with disabilities included student familiarity with the content topic and tools being used, as well as student exposure to accessible websites (Basham et al., 2018; Rice & Dykman, 2018; Vazquez & Straub, 2012). 

Student engagement levels during participation in online courses can help to identify students who need additional support, provide evidence of quality (e.g., activity, course, tool), and indicate student academic or social outcomes (e.g., achievement, interaction with others, motivation to succeed, participation; Henrie, Halverson, & Graham, 2015). 

Challenges of distance learning 

Researchers found several challenges of online and blended learning: 

• Access to the infrastructure, resources, and tools needed for participation (Rice, 2006; Barbour & Reeves, 2009; Arnesen et al., 2020). Investment in infrastructure is needed to adequately support online teaching and learning, including startup costs for implementation of blended or online learning programs. 
• Accountability of students in completing coursework and schools in providing a quality learning experience (Rice, 2006; Barbour & Reeves, 2009). With students, the retention rates in online courses are often lower, meaning students drop out of these courses at higher rates (Hasler-Waters, Barbour, & Menchaca, 2014). 
• Teacher preparedness for online teaching (Arnesen et al., 2019; Archambault & Kennedy, 2018; Dawson & Dana-Fichtman, 2018). Teachers often do not receive appropriate training for blended or online learning within teacher preparation programs. This creates a gap between teachers’ ability and comfort level to teach in a digital learning environment. 

Challenges and Considerations 

• Meta-analyses include quantitative results from a large collection of research studies, meaning information is summarized which thus reduces the methodological issues present in individual studies (Card, 2012). However, meta-analyses can vary in quality based on their inclusion criteria. 
• Some of the systematic reviews and meta-analyses focused on all learners rather than a specific focus on K-12 students. Thus, results from these studies may not be as applicable to K-12 students or teachers due to their unique developmental needs. 
• Most of the systematic reviews and meta-analyses we examined reported a wide variability of effect sizes within individual studies. Furthermore, individual studies often included numerous variables. For brevity purposes, we only included the overall effect sizes for major variables. 
• To provide a comprehensive literature review, we included sources that were not published in peer-reviewed scholarly journals. However, we only included additional studies from reliable publication sources, such as educational conference proceedings, government, and non-profit educational research organizations. 
• Although we are confident that the list of sources, systematic reviews, and meta-analyses is comprehensive, due to the broad scope of the project and its goals, this literature review is likely not exhaustive of all related literature. Rather than presenting a definitive conclusion on the effectiveness of distance education, the goal of this literature review was to identify best practices for curriculum and instruction in online learning environments. 

Conclusion 

This literature review identified several research-based best practices for educators in distance learning. In general, research findings from both quantitative and qualitative studies indicated no significant difference between instructional settings (e.g., blended, distance, in-person). Rather, course quality, instructional design, meaningful interactions, pedagogical methods, student characteristics, and support structures provide more meaningful impact on student learning outcomes (Bernard et al., 2004; Cavanaugh et al., 2004; Bernard et al., 2009; Cavanaugh, Barbour, & Clark, 2009; Larwin & Erickson, 2016; Pulham & Graham, 2018). These factors can be present (or not) in any instructional setting. As such, educators should focus their attention on effective pedagogical strategies to design high-quality, meaningful, and supportive learning experiences. In an online learning environment, this includes preparing educators for online teaching, understanding the benefits and challenges of digital spaces, balancing asynchronous and synchronous learning activities, implementing engaging and motivating activities for active learning, supporting students in personalized ways, designing meaningful formative and summative assessments, and guiding students to develop habits for successful online learning. 

References 

Archambault, L., & Kennedy, K. (2018). Teacher preparation for K-12 online and blended learning. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 221-245). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Arnesen, K. T., Hveem, J., Short, C. R., West, R. E., & Barbour, M. K. (2019). K-12 online learning journal articles: Trends from two decades of scholarship. Distance Education, 40, 32–53. https://doi.org/10.1080/01587919.2018.1553566 

Arnesen, K., Walters, S., Borup, J., & Barbour, M. K. (2020). Irrelevant, overlooked, or lost? Trends in 20 years of uncited and low-cited K-12 online learning articles. Online Learning Journal, 24(2), 187–206. https://doi.org/10.24059/olj.v24i2.2080 

Barbour, M. K. (2020). Misbehaving toddler or moody teenager: Examining the maturity of the field of K-12 online learning. Revista de Educación a Distancia (RED), 20(64). https://doi.org/10.6018/red.412821 

Barbour, M. K. (2018, February). Examining online research in higher education: What can we replicate in K-12? Michigan Virtual Learning Research Institute. https://michiganvirtual.org/research/publications/examining-online-research-in-higher-education-what-can-we-replicate-in-k-12/ 

Barbour, M. K. (2015). The disconnect between policy and research: Examining the research into full-time K-12 online learning. In D. Slykhuis & G. Marks (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2015 (pp. 1438-1445). Chesapeake, VA: AACE. https://digitalcommons.sacredheart.edu/cgi/viewcontent.cgi?article=1223&context=ced_fac 

Barbour, M. K., & Reeves, T. C. (2009). The reality of virtual schools: A review of the literature. Computers and Education, 52, 402–416. https://doi.org/10.1016/j.compedu.2008.09.009 

Basham, J. D., Blackorby, J., Stahl, S., & Zhang, L. (2018). Universal Design for Learning: Because students are (the) variable. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 477-507). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P. A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439. https://doi.org/10.3102/00346543074003379 

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79, 1243–1289. https://doi.org/10.3102/0034654309333844 

Borup, J. (2018). On-site and online facilitators: Current practice and future directions for research. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 423-441). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Borup, J., Graham, C. R., West, R.E., Archambault, L., & Spring, K. J. (2020). Academic Communities of Engagement: An expansive lens for examining support structures in blended and online learning. Educational Technology Research and Development, 68, 807-832. https://doi.org/10.1007/s11423-020-09744-x 

Brodersen, R. M., & Melluzzo, D. (2017). Summary of research on online and blended learning programs that offer differentiated learning options. Regional Educational Laboratory Central. https://files.eric.ed.gov/fulltext/ED572935.pdf 

Card, N.A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford. 

Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7(1), 73-88. 

Cavanaugh, C. S., Barbour, M. K., & Clark, T. (2009). Research and practice in K-12 online learning: A review of open access literature. International Review of Research in Open and Distance Learning, 10(1). https://doi.org/10.19173/irrodl.v10i1.607 

Cavanaugh, C., Gillan, K. J., Kromrey, J., Hess, M., & Blomeyer, R. (2004, October). The effects of distance education on K–12 student outcomes: A meta-analysis. Learning Point Associates/North Central Regional Educational Laboratory (NCREL). https://files.eric.ed.gov/fulltext/ED489533.pdf 

Darling-Hammond, L., et al. (2020). Restarting and reinventing school: Learning in the time of COVID and beyond. https://restart-reinvent.learningpolicyinstitute.org/ 

Dawson, K., & Dana-Fichtman, N. (2018). Professional development for K-12 online teachers. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 247-260). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Garrett Dikkers, A. (2018). Introduction. In K. Kennedy & R. Ferdig (Eds). Handbook of K-12 Blended and Online Learning Research (pp. 509-522). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Hasler-Waters, L., Barbour, M. K., & Menchaca, M. P. (2014). The nature of online charter schools: Evolution and emerging concerns. Educational Technology & Society, 17(4), 379-389. http://digitalcommons.sacredheart.edu/ced_fac 

Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers and Education, 90, 36–53. https://doi.org/10.1016/j.compedu.2015.09.005 

Horn, M. B., & Staker, H (2014). Blended: Using disruptive innovation to improve schools. San Francisco, CA: Jossey-Bass. 

Hu, M., Arnesen, K., Barbour, M. K., & Leary, H. (2019). A newcomer’s lens: A look at K-12 online and blended learning in the Journal of Online Learning Research. Journal of Online Learning Research, 5(2), 123-144. https://www.learntechlib.org/primary/p/195231/paper_195231.pdf 

Kennedy, K., & Ferdig, R. E. (Eds.). (2018). Handbook of research on K-12 online and blended learning. Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813 

Larwin, K. H., & Erickson, M. J. (2016). The potential impact of online/distance education for K-12 special education students: A meta-analytic investigation. International Journal of Elementary Education, 5(1), 8-16. https://doi.org/10.11648/j.ijeedu.20160501.12 

Martin, F., Ahlgrim-Delzell, L., & Budhrani, K. (2017). Systematic review of two decades (1995 to 2014) of research on synchronous online learning. American Journal of Distance Education, 31, 3-19. https://doi.org/10.1080/08923647.2017.1264807 

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. http://www.ed.gov/about/offices/list/opepd/ppss/reports.html 

Means, B., Murphy, R., & Bakia, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115, 1-47. 

Moore, M. G. (1989). Three types of interaction. American Journal of Distance Education, 3(2), 1-7. 

Pearlman, B. (2020). Maslow Before Bloom: Basic Human Needs Before Academics. 

Pulham, E. B., Graham, C. R., & Short, C. R. (2018). Generic vs. modality-specific competencies for K-12 online and blended teaching. Journal of Online Learning Research, 4(1), 33-52. https://files.eric.ed.gov/fulltext/EJ1174455.pdf 

Pulham, E., & Graham, C. R. (2018). Comparing K-12 online and blended teaching competencies: a literature review. Distance Education, 39(3), 411-432. https://doi.org/10.1080/01587919.2018.1476840 

Repetto, J. B., Spitler, C. J., Cox, P. R. (2018). Research on at-risk learners in K-12 online learning. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 163-180). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Rice, K. L. (2006). A comprehensive look at distance education in the K-12 context. Journal of Research on Technology in Education, 38, 425-448. https://doi.org/10.1080/15391523.2006.10782468 

Rice, M. F., & Dykman, B. (2018). The emerging research base on online learning and students with disabilities. In K. Kennedy & R. Ferdig (Eds). Handbook of research on K-12 online and blended learning (pp. 189-205). Pittsburgh, PA: Carnegie Mellon University ETC Press. https://figshare.com/articles/journal_contribution/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813

Vasquez, E., & Straub, C. (2012). Online instruction for K-12 special education: A review of the empirical literature. Journal of Special Education Technology, 27, 31-40. https://doi.org/10.1177/016264341202700303 

Supplementary Materials 

Glossary 

• Asynchronous learning involves students and teachers interacting with content and each other at different times. Asynchronous learning can happen within a structured schedule (e.g., weekly deadlines within a course). Asynchronous learning might include collaborating with others, completing formative assessments, consuming content (e.g., watching videos), discussion board participation, or project-based learning. 
• Synchronous learning involves students and teachers interacting with content and each other at the same time during live sessions. Synchronous learning might include text-based chat, live discussions, or video conferencing sessions (e.g., Google Meet, Zoom). 
• Blended learning includes both in-person and online learning and integrates both activities to create a comprehensive learning experience (Horn & Staker, 2014). 
• Distance/online learning can take on many forms, but in this context of this literature review we considered distance/online learning programs that facilitated formal learning experiences, were institutional-based (not self-study programs), included interactive communication (online or otherwise), and where students and teachers were physically separated from each other (Rice, 2006). 
• Online charter school: K-12 program publicly funded and governed by the state, where most of the schooling occurs online (Hasler-Waters, Barbour, & Menchaca, 2014). 
• Student engagement: A students’ commitment, effort, and investment in learning. Includes subcategories of behavior, cognition, and emotion (Henrie, Halverson, & Graham, 2015). 

Characteristics of Meta-Analyses Included in Review 

Study	Variable(s)	# primary studies	Overall sample size	Date range of primary studies
Cavanaugh, 2001	Achievement	19	929	1980-1998
Bernard et al., 2004	Achievement	232 total 11 K-12 studies	57,019	1985-2002
Bernard et al., 2004	Attitude	232 total 11 K-12 studies	35,365	1985-2002
Bernard et al., 2004	Retention	232 total 11 K-12 studies	3,744,869	1985-2002
Cavanaugh et al., 2004	Student outcomes	14	7561	1999-2004
Bernard et al., 2009	Achievement/attitude	74	N/A	1985-2006
Means et al., 2009	Context comparison	45	N/A	1996-2008
Means et al., 2009	K-12 learners	5	N/A	1996-2008
Means et al., 2013	Context comparison	45	N/A	1996-2008
Means et al., 2013	K-12 learners	5	N/A	1996-2008
Larwin & Erickson, 2016	Achievement	7	24,031 (total) 3,558 (students with disabilities)	2005-2014

Effect Sizes from Meta-Analyses Included in Review 

Study	Variable(s)	Mean effect size
Cavanaugh, 2001	Achievement	0.147
Bernard et al., 2004	Achievement	0.0128 (overall) -0.1022 (synchronous) 0.0527 (asynchronous)
Bernard et al., 2004	Attitude	-0.0812 (overall) -0.1846 (synchronous) -0.0034 (asynchronous)
Bernard et al., 2004	Retention	-0.0573 (overall) 0.0051 (synchronous) -0.0933 (asynchronous)
Cavanaugh et al., 2004	Student outcomes	-0.028
Bernard et al., 2009	Student interactions on achievement/attitude (overall)	0.38
Bernard et al., 2009	Student-student	0.49
Bernard et al., 2009	Student-teacher	0.32
Bernard et al., 2009	Student-content	0.46
Means et al., 2009	Online & blended learning vs. face-to-face classroom instruction	0.20
Means et al., 2009	K-12 learners	0.1664
Means et al., 2013	Online & blended learning vs. face-to-face classroom instruction	0.20
Means et al., 2013	K-12 learners	0.1664
Larwin & Erickson, 2016	Achievement	-0.015
Larwin & Erickson, 2016	Students with disabilities online vs. students with disabilities face-to-face	0.497
Larwin & Erickson, 2016	Students with disabilities online vs. students without disabilities online	-0.561