3.0 What are the Characteristics of the AT Evidence Base?

Overview

As a transdisciplinary profession (see Figure 2), the literature on AT can be found within and across disciplines such as special education, occupational therapy, rehabilitation, speech and language, educational technology, higher education, and more. As such, it may be important to keep in mind the parable of the blind men who encounter an elephant and draw vastly different conclusions about the nature of the beast from their limited experience. The findings of this rapid review of the AT literature reveal that the knowledge base is large, scattered, and growing. As a result, the parable may explain many different interpretations about the state of AT practice in education based on what is commonly believed versus what has been established through high-quality research that has yet to be fully assimilated by the profession.

Figure 2 Comparison of Disciplinary Relationships and Integration

 

3.1

Size of the Knowledge Base

The search and review process resulted in a corpus of 968 articles that form the basis of the corpus for this AT rapid review report. Table 3 illustrates the number of found articles by year. It should be noted that the 2019 article total is artificially inflated as prepublication preprints, labeled as 2019, will receive a new copyright date when published in 2020 or beyond. The sheer size of the knowledge base foreshadows the significant challenge stakeholders will experience in trying to stay up-to-date within this discipline.

Table 3 Search Results of Key Terms Within the Article Title
year	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19
	26	27	37	42	54	59	62	66	62	79	71	76	89	95	124

Figure 3 illustrates the number of articles found by disability. Some disability categories are low incidence which is reflected in the smaller number of articles found (e.g., deaf-blindness). In other cases, there is extensive research interest in disabilities such as autism, intellectual disabilities, and speech language communication. Somewhat problematic is the under-representation of a high incidence disability, like learning disabilities, not having a more substantial AT literature basis.

Figure 3 Articles Found by Disability Category

Figure 4 illustrates the number of articles found by AT construct. As noted earlier, AT is not a single unidimensional construct. The search terms used in this study reflect the diversity of the AT discipline such that there are broad concepts of interest (e.g., AT assessment, AT outcomes) as well as specific types of special needs and disabilities (e.g., specific learning disability). The sheer size of the classification of AT advocacy represents a coding artifact as this descriptor was used to code articles advocating a particular application of AT without scientific evidence on its efficacy (e.g., how-to practitioner articles).

Figure 4 Articles Found by AT Construct

3.2

Characteristics of the Corpus

Several characteristics of the corpus are worth noting. Table 4 illustrates the contribution of articles by country and reflects the dominance of United States authors and journals in the AT literature. However, given the economic and educational similarities of the four target English-speaking countries, the corpus appears to have relevant cross-cultural application.

Table 4 Documents Included in the Corpus by Country
Country	Number of Documents	Percentage of the Total Corpus
Australia	33	3.5%
Canada	53	5.5%
United Kingdom	44	4.5%
United States	743	76.7%
Other	95	9.8%
Total	970	100%

A second attribute of the corpus reflects the types of documents collected. The purpose of a rapid review was to cast a wide net, to discover not only peer-reviewed journal articles, but also grey literature that could take the form of articles for practitioners, book chapters, conference proceedings, doctoral dissertations, and reports/whitepapers. Table 5 summarises the types of documents collected by this project for review and analysis. The variety of dissemination outlets used in the field of AT is compatible with calls for diversifying the inputs associated with evidence reviews (Giustini, 2019).

Table 5 Types of Documents in the Corpus
Country	Number of Documents	Percentage of the Total Corpus
Article - Practitioner	320	33%
Article - Refereed Journal	455	47%
Book Chapter	23	2.4%
Cochrane Review	1	0.0%
Conference Proceedings	42	4.3%
Doctoral Dissertation	110	11.4%
Report/Whitepaper	17	1.7%
Total	968	99.8%

Early work on synthesising the special education technology literature discovered that the published literature focused more on how-to practice than research-based practice (Edyburn, 2000). While the current study discovered a significant body of practitioner-focused literature, there are clear patterns that more research has been conducted, and that over time, greater attention has been placed on efforts to critically analyse the literature to establish the efficacy of various interventions and professional practices (see Table 6). However, at this point it is difficult to describe the field of AT as an evidence-based profession given that only 10% of the literature focuses on research evidence for its interventions and practices.

Table 6 Types of Literature Reviews in the Corpus
Country	Number of Documents	Percentage of the Total Corpus
Cochrane Review	1	1%
Systematic Reviews	35	36.4%
Meta-Analyses	8	8.3%
Descriptive Reviews	45	46.8%
Scoping Reviews	7	7.2%
Total	96	199.7%

3.3

Literature Scatter

The current study found that six publications form a core of the AT literature that resulted in a capture rate of 54% of the documents discovered (see Table 7). Practically, this finding has important implications for libraries, resource centers, and professionals interested in creating a specialised collection for studying and monitoring a significant portion of the new annual contributions to the AT knowledge base. Technically, however, it should be noted that the data are skewed given the search methods that captured all of the articles in the grey literature (i.e., AT Outcomes and Benefits, Closing the Gap, doctoral dissertations) versus the other peer-reviewed journals that were discovered as a result of key word searching. For readers interested in international applications of AT, two journals on this list may be of particular interest: Disability and Rehabilitation: Assistive Technology; and Technology and Disability.

3.4

Quality of the Evidence Supporting AT Use

Whereas a rapid review typically only aggregates the documents into a descriptive summary, efforts were made to produce a preliminary assessment of the quality of the evidence found in each document. Two types of codes were assigned to each document to represent the quality of evidence presented. The first code was a numeric assignment, based on a 7-point scale, based on a preliminary analysis by the Principal Investigator to provide a weight that could be used regarding the quality of evidence (see Table 8). This type of coding is commonly used in evidence reviews but is problematic in this study because of the inadequate in-depth analysis applied to each found document that could results in classification errors.

Table 8 Evidence Quality
Level	Description	Number of Documents	Percentage of the Total Corpus
1	A systematic review or meta-analysis with effect sizes	15	1.5%
2	Evidence obtained from at least one well-designed RCT (e.g., large multi-site RCT)	9	1.0%
3	A well-designed controlled trial without randomization	0	0%
4	A well-designed experiment, case-control, or cohort study	8	1.0%
5	A systematic review	74	7.6%
6	Data from a single research study	330	34.0%
7	Opinion of authorities and/or reports of expert committees (non-data based)	532	54.9%
	Total	968	100%

Since the coding in Table 8 requires a level of technical analysis to apply and use, this information will have limited value to the majority of stakeholder groups who lack the technical and statistical background to discern the differences among the levels of evidence. Therefore, a three-point scale (see description in Figure 5) is commonly used to help practitioners apply research evidence (What Works Clearinghouse, 2020).

Figure 5 Quality of Evidence as Described in Educational Policy

In order to make the results meaningful to a wide variety of stakeholders, a second evidence code was assigned to each document (see Table 9). The recoding of the evidence on the three-point scale (emerging, moderate, or strong), or for non-databased works (demonstrates a rationale), should facilitate the use of evidence by the majority of stakeholder groups who have limited interest in the technical and statistical differences among the levels of evidence.

Table 9 Descriptive Evidence Level
Descriptor	Evidence Level	Number of Documents	Percentage of the Total Corpus
Strong	Levels 1-2	24	2.4%
Moderate	Levels 3-4	8	1%
Emerging	Levels 5-6	404	41.7%
Demonstrates a Rational	Level 7	532	54.9%
	Total	968	100%

As illustrated in Tables 8 and 9, expert opinion (i.e., demonstrates a rationale), explaining how/why AT can be used, is a function of more than half (54.9%) of the AT literature. Then, considering the contribution of individual studies, (34%), it is clear that the research evidence is a very small component (10.9%) of the professional literature thereby limiting the application of research-based evidence for informing efforts to scale AT interventions. Nonetheless, the corpus of 96 literature reviews with 30 moderate – strong evidence reviews offers an informative body of research for answering the additional research questions posed within this study.

3.5

Conclusions: AT Evidence Base

The following overall conclusions are drawn concerning the nature of the AT evidence base circa 2020:

• At this time, only a small number of AT interventions can be documented as having a moderate or strong evidence base. This finding, within the context of a rapid review of the literature study, is congruent with previous AT evidence synthesis reviews (Anttila, et al., 2012).

• The most research validated AT intervention focuses on speech, language, and communication disabilities and the use of communication systems such as augmentative and alternative communication (AAC) devices. The research evidence is strong and exceedingly clear: providing individuals with a method of communicating, the earlier the better, improves a variety of outcomes relative to independence, learning, and quality of life (Dunst, et al., 2013; Morin, et al., 2018; Romski et al., 2015).

• Improved tools and protocols for evaluating and grading the quality of AT primary research studies are consistently mentioned in evidence reviews as a critical need for the AT profession (Antilla, et al., 2012; Morin, et al., 2018; Muharib et al., 2018).

• The overall level of evidence concerning the effectiveness of AT is generally low because most primary studies have methodological limitations (e.g., insufficiently powered research designs, small numbers of subjects, inadequate descriptions of participants’ functional limitations and/or the study contexts, inadequate attention to reporting effect sizes and the confidence intervals of the observed changes). Resolving these issues will take concerted efforts by researchers, journal editors, and reviewers to apply evidence standards when judging the publication worthiness of new research studies (Scherer, Smith, & Layton, 2019; Williamson, et al., 2015).

Additional implications of the evidence base will be explored in the next chapter regarding what is presently known about AT use by pupils and students with special educational needs and disabilities in educational settings.

3.6

References

Anttila, H., Samuelsson, K., Salminen, A. L., & Brandt, A. (2012). Quality of evidence of assistive technology interventions for people with disability: An overview of systematic reviews. Technology and Disability, 24(1), 9-48.

Dunst, C. J., Trivette, C. M., Hamby, D. W., & Simkus, A. (2013). Systematic review of studies promoting the use of assistive technology devices by young children with disabilities. Practical Evaluation Reports, 5(1), 1-32. Asheville, NC: Orelena Hawks Puckett Institute.

Edyburn, D.L. (2000). 1999 in review: A synthesis of the special education technology literature. Journal of Special Education Technology, 15(1), 7-18.

Giustini, D. (2019). Retrieving grey literature, information, and data in the digital age. In, H. Cooper, L.V. Hedges, & J.C. Valentine, (Eds.). The handbook of research synthesis and meta-analysis (pp. 101-126). NY: Russell Sage Foundation.

Morin, K. L., Ganz, J. B., Gregori, E. V., Foster, M. J., Gerow, S. L., Genç-Tosun, D., & Hong, E. R. (2018). A systematic quality review of high-tech AAC interventions as an evidence-based practice. Augmentative and Alternative Communication, 34(2), 104-117.

Muharib, R., & Alzrayer, N. M. (2018). The use of high-tech speech-generating devices as an evidence-based practice for children with autism spectrum disorders: A meta-analysis. Review Journal of Autism and Developmental Disorders, 5(1), 43-57.

Romski, M., Sevcik, R. A., Barton-Hulsey, A., & Whitmore, A. S. (2015). Early intervention and AAC: What a difference 30 years makes. Augmentative and Alternative Communication, 31(3), 181-202.

Scherer, M., Smith, R. O., & Layton, N. (2019). Committing to assistive technology outcomes and synthesizing practice, research and policy. In N. Layton, & J. Borg, (Eds.), Global perspectives on assistive technology: Proceedings of the GReAT Consultation 2019, Volume 1 (pp. 196-217). Geneva, Switzerland: World Health Organization.

What Works Clearinghouse. (2020). Practice guides. Washington, DC: Author. Retrieved from https://ies.ed.gov/ncee/wwc/practiceguides

Williamson, T., Kenney, L., Barker, A. T., Cooper, G., Good, T., Healey, J., ... & Ryan, J. (2015). Enhancing public involvement in assistive technology design research. Disability and Rehabilitation: Assistive Technology, 10(3), 258-265.