MINDING THE GAP: A COMPREHENSIVE RUBRIC FOR INSTRUCTIONAL DESIGN IN E-LEARNING

Abstract

The recognition of practice in online instruction is still subject to interpretation and different approaches as a result of the rapid changes in technology and its effect on society. This paper reviews different examples of rubrics and instruments in higher education to propose a more comprehensive rubric that constitute a synthesis of how some institutions in HE approach best practice in this field. As such, the proposed comprehensive rubric is intended to support the development, remixing, sharing, and integration of online modules and courses by providing a single reference point with as wide a range as possible of potential pedagogical tools, facilities and approaches to e-learning.

Full text

ICICTE 2017 Proceedings
!
!
170
MINDING THE GAP:
A COMPREHENSIVE RUBRIC FOR INSTRUCTIONAL
DESIGN IN E-LEARNING
Martin Debattista
Institute of Tourism Studies
Malta
Abstract
The recognition of practice in online instruction is still subject to
interpretation and different approaches as a result of the rapid changes in
technology and its effect on society. This paper reviews different examples of
rubrics and instruments in higher education to propose a more comprehensive
rubric that constitute a synthesis of how some institutions in HE approach best
practice in this field. As such, the proposed comprehensive rubric is intended
to support the development, remixing, sharing, and integration of online
modules and courses by providing a single reference point with as wide a
range as possible of potential pedagogical tools, facilities and approaches to e-
learning.
Keywords: instructional design, e-learning, digital pedagogy, OER, higher
education
Literature Review: Slightly Different Paths for e-Learning
By the year 2002 the fusion between in-class instruction and online instruction
was already being recognised as a major new trend; old practices were being
transferred online, and different institutions had rather different
understandings of the concept of e-learning (Frydenberg, 2002; Graham,
Woodfield, & Harrison, 2013; Young, 2002). As late as 2011, Guri-Rosenblit
and Gros (2011) concluded there were “noticeable gaps” in e-learning
research and definition of terminology. One of the reasons for this lack of
coherence is:
The technological environment within which modern education
operates is becoming increasingly complex, offering new possibilities
but also giving rise to challenges. We have seen a continual evolution
of technologies and how they are used since the introduction of the
Internet. (Conole & Alevizou, 2010, p. 9)
De Freitas and Conole (2010) observed the trend of more global, more
networked and more mobile technology infrastructure, and these are emerging
in online pedagogy. Indeed, McLoughlin and Lee (2010) argued that this new
landscape:
Calls for the active involvement of students in defining their learning
goals and choosing both ICT tools and strategies for learning; it also
requires recognition that user and learner generated content has a
central place in a curriculum that fosters self-regulated learning. (p. 38)

ICICTE 2017 Proceedings
!
!
171
Learning in the digital age thus requires a re-thinking of teaching and learning
and not just replicating existing practices with new technology (Beetham &
Sharpe, 2013).
While in the early years of public Internet the emphasis was on the technical
foundations of e-learning, the pedagogical implications started shifting
direction from managing the logistics of e-learning to managing the content
(Govindasamy, 2002). An “interactive and constructive potential of e-
learning” was being recognised, one which contrasts with the traditional sage
on the stage approach of information transfer. However, it had to prove it was
more than just a more convenient way to access content (Garrison &
Anderson, 2011, p. 54).
The focus of course design and pedagogy has shifted from teaching to
learning, with teachers also becoming learners in the process of professional
development and engagement with their students. Rather than transferring the
passive teacher-centred form of learning to the online domain, a constructivist
approach would make the experience learner-centred, where active learning
and engagement takes place (Rovai, 2004). The emphasis in higher education
has shifted from the delivery of instruction to the production of learning (Barr
& Tagg, 1995).
Far from simply providing a more comfortable channel to access instructional
content, Nagel and Kotzé (2010) found, “When students engage in online
activities and take responsibility for the quality of interaction, they can have a
superior learning experience” (p. 218). ICT can indeed be successfully applied
to enhance the educational process, especially in making learners active
participants (Tomte & Sutherland Olsen, 2014,). On the other hand, the
digital literacy of academic staff remains a challenge (Johnson, Adams
Becker, Estrada, & Freeman, 2015), with many feeling the pressure of 24/7
online connectivity in the age of social media (Grove, 2017).
E-learning faces issues of suspicion (Casey, 2008) and quality (Jung &
Latchem, 2012) when compared to traditional instruction. However, a meta-
analysis by Means, Murphy, and Bakia (2013) on behalf of the U.S.
Department of Education, reveals a significant increase in performance for
blended learning but not for pure online learning. This confirmed earlier
conclusions by Roberts and Savio (2012).
In their literature review on the use of Web 2.0 technologies in HE, Conole
and Alevizou (2010) observed a number of key challenges proposed when
technology meets education, including the tensions around the nature of
openness, and changes in the role of educators and students. HE in Europe
seems to be responding positively to these rapid developments in e-learning,
though the adoption of e-learning and MOOCs is gradual (Gaebel, 2015).
Empirical evidence is increasing in respect to online learning and new
developments such as Open Educational Resources (OER), mobile learning,
software agents for online evaluation (Daradoumis, Bassi, Xhafa, & Caballe,
2013), bring your own device (BYOD), the flipped classroom, wearable

ICICTE 2017 Proceedings
!
!
172
technologies, adaptive learning technologies and the Internet of Things
(Johnson et al., 2015). These have attracted the attention of researchers to an
ever-higher degree in recent years. However, more research on OER and
MOOCs is required (Bozkurt et al., 2015).
The NMC Horizon Report 2017 (HE Edition) refers to a medium-term trend of
using digital tools to measure knowledge and skills acquisition in online
learning environments, including collaboration and creativity (Adams Becker
et al., 2017).
According to the Times Higher Education Teaching Survey 2017, half of
academics and 68% of administrators agree that students benefit from
digitised content, but they evidence less enthusiasm for recording lectures and
putting them online. The use of social media for instructor-student contact is
still not widespread as many academics feel the pressure of constantly being in
demand by students (Grove, 2017).
Crossbreeding between social network sites and e-learning is increasingly
observable. Though Facebook is not a replacement for VLEs in education, the
latter “could certainly learn something from Facebook and the nature of its
user created groups and networks, instant communications, alerts and
like/sharing features” (Hogg, 2013, p. 35). These VLEs, such as MOODLE
and Blackboard, had to introduce features similar to the increasingly popular
social network sites (Ronan, 2015) while some social network sites such as
LinkedIn are assuming educational roles (Hoffman, 2015). This development
and the proliferation of MOOCs are some of the most interesting but
controversial trends in higher education right now (Johnson, Adams Becker,
Estrada, & Freeman, 2014). Early research shows MOOCs are indeed
contributing to online learning, though more empirical research is needed
(Gamage, Fernando, & Perera, 2015; Glance, Forsey, & Riley, 2013).
An Overview of Four Different Rubrics
Measurement Through Rubrics
The measure of the success or failure of whether technology-driven education
delivers the promised success can be determined through rubrics based on
traditional principles, updated to cover the introduction of new technologies.
In this manner, different instructional strategies can be devised to serve the
different learning domains, including intellectual and cognitive strategies,
attitudes, etc. (Fenrich, 2008, p. 310).
Graham, Cagiltay, Byung-Ro, Craner, & Duffy (2001) refer to the Seven
Principles for Good Practice in Undergraduate Education published in 1987
as a popular framework for the evaluation of traditional classroom-based
education, based on 50 years of research. These principles have been adapted
for online education:
Principle 1: Good Practice Encourages Student-Faculty Contact
Principle 2: Good Practice Encourages Cooperation Among Students
Principle 3: Good Practice Encourages Active Learning

ICICTE 2017 Proceedings
!
!
173
Principle 4: Good Practice Gives Prompt Feedback
Principle 5: Good Practice Emphasizes Time on Task
Principle 6: Good Practice Communicates High Expectations
Principle 7: Good Practice Respects Diverse Talents and Ways of Learning
Established frameworks began to be adopted and adapted to e-learning. For
example, the rubric provided by the Quality Online Course Initiative of the
University of Illinois (http://www.ion.uillinois.edu/initiatives/qoci/index.asp)
is based on six sections, each with specific criteria, that mirror the above-
mentioned Seven Principles:
1. Instructional Design - Criteria: 1.1 Structure, 1.2 Learning
Goals/Objectives/Outcomes, 1.3 Course Information, 1.4 Instructional
Strategies, 1.5 Academic Integrity, 1.6 Use of Multimedia.
2. Communication Interaction and Collaboration - Criteria: 2.1 Activities and
Opportunities, 2.2 Organisation and Management, 2.3 Group Work.
3. Student Evaluation and Assessment - Criteria: 3.1 Goals and Objectives, 3.2
Strategies, 3.3 Grades, 3.4 Feedback, 3.5 Management.
4. Learner Support and Resources- Criteria: 4.1 Institutional/Programme
Support and Resources, 4.2 Academic Support and Resources.
5. Web Design - Criteria: 5.1 Layout Design 5.2 Use of Multimedia, 5.3 Use
of Images, 5.4 Links/Navigation, 5.5 Accessibility
6. Course Evaluation - Criteria: 6.1 Layout/Design
California State University provides a Quality Online Learning and Teaching
(QOLT) instrument to measure the effectiveness and quality of online courses
(Christie, 2014). The instrument can be used for both self-evaluation and peer
evaluation. It consists of 58 objectives in 10 sections:
Section 1. Course Overview and Introduction (8 objectives)
Section 2. Assessment of Student Learning (6 objectives)
Section 3. Instructional Materials and Resources (6 objectives)
Section 4. Students Interaction and Community (Course Design) (7 objectives)
Section 5. Facilitation and Instruction (Course Delivery) (8 objectives)
Section 6. Technology for Teaching and Learning (5 objectives)
Section 7. Learner Support and Resources (4 objectives)
Section 8. Accessibility and Universal Design (7 objectives)
Section 9. Course Summary and Wrap-up (3 objectives)
Section 10. Mobile Design Readiness (optional) (4 objectives)
The University of Malta (UoM) provides Minimum Standards for Study Units
in the VLE (2015) with a shortlist of suggested elements that must be provided
in face-to-face courses complemented by online study units (blended mode) to
meet minimum standards. This ‘advisory’ is provided by the IT Services at the
University of Malta rather than a unit responsible for academic quality
assurance or pedagogy. Indeed, quite significantly, these guidelines do not
make any reference to pedagogy for e-learning.

ICICTE 2017 Proceedings
!
!
174
The following are the elements listed in the advisory by the University of
Malta (http://www.um.edu.mt/vle/staff/minimumstandards): (a) Study Unit
Description, (b) Tutor Profile on VLE, (c) Class Announcements (provided by
default in the VLE), (d) General Q&A Forum, (e) Communication Statement
(tutor-student communication protocol), (f) Course Readings, (g) Other
Learning Resources, and (h) Assessment Outline, (exemplars and use of anti-
plagiarism software).
QualityMatters™ is a commercial product that provides course design rubrics
for different levels in the education domain. One of them is specific to higher
education and provides eight general standards that “work together to ensure
students achieve desired learning outcomes” in online and blended learning
(QualityMatters™, 2014, para 5). A score of 85% qualifies a course to receive
a QM certification for quality in course design.
The standards QualityMatters™ (2014) are: (a) Course overview and
introduction, (b) Learning Objectives (competencies), (c) Assessment and
measurement, (d) Instructional material, (e) Learner activities and learner
interaction, (f) Course technology, (g) Learner support, (h) Accessibility and
usability.
A Synthesis of the Rubrics to Formulate the New Comprehensive Rubric
Synthesising a Comprehensive Rubric
The rubrics referenced in this document all have common criteria that cover
the most basic elements that an online course should satisfy if it aspires to
provide effective teaching and learning. These common standards are (in no
particular order): instructional design, web design and technical access,
communication between tutor/s and students, interactivity and community
building, instructional resources with possible multimedia use, instructional
support, assessment, and evaluation of the instruction with learner feedback
(see Table 1).
The four institutions under analysis all cover most – if not all – the criteria
derived from the synthesis of their rubrics.
Table 1
The Criteria in the Rubrics by the Three Educational Institutions and One
Commercial Organisation Under Study.
Legend:
Assessment
Instructional resources
Instructional design
Learner support
Communication
!!
Web/tech design
!!
Intro/wrap-up/evaluation

ICICTE 2017 Proceedings
!
!
175
Univ. of Illinois
California State Univ.
Univ. of Malta
QualityMatters™ (2014)
Instructional
Design
Course Overview &
Intro
Study Unit
Description
Course Overview and
Introduction
Communication
Assessment of Student
Learning
Tutor/s Profile/s
Learning Objectives
(Competencies)
Student
Evaluation and
Assessment
Instructional Resources
Class
Announcements
Assessment and
Measurement
Learner Support
Student Interaction &
Community
General/QA
Forum
Instructional Materials
Web Design
Facilitation and
Instruction
Tutor-Student
Communication
Course Activities and Learner
Interaction
Course Evaluation
Tech for Teaching and
Learning
List of Readings
Course Technology
!
Learner Support
Digital Resources
Learner Support
!
Accessibility and
Universal Design
Assessment
Accessibility and Usability
!
Course Summary and
Wrap-up
!
!
!
Mobile Design
Readiness
!
!
In terms of ratings, the California State University (2013) and
QualityMatters™ (2014) provide their own rating scales. The former allows
the adopter of the rubric to assign the same weighting range to all criteria (one
to three points) according to the extent to which it is met or not, while the
latter sets specific number of points (one, two or three) to be awarded to any
individual standard when it is met (no points for partial or non-fulfilment).
The QualityMatters™ rubric assigns the most points (three out of a maximum
of three) to the statement of the learning objectives/competencies, assessment,
the quality of instructional resources, tutor-learner interaction, learner support,
and the ease of use of the technical platform where the virtual learning
environment resides.
The Comprehensive Rubric
The synthesis of the four rubrics just referenced has produced the following
comprehensive rubric that covers all the aspects mentioned by the four
institutions. This comprehensive rubric is not a collation of the four rubrics
but a synthesis of the separate approaches that – in the author’s view - reflects
the context of e-learning as explained at the beginning of this paper. This
rubric has not been tested in lab setting or a real-life scenario.

ICICTE 2017 Proceedings
!
!
176
There are 10 main standards, each containing specific standards.
10 Main Standards and Specific Standards
1. Instructional Design – An analysis of the learning needs and the use of
appropriate strategies and methods to meet them
1.1. Structure of Learning.
1.2. Learning Aims & Objectives - What the instructor needs to achieve
with the learning process.
1.3. Learning Outcomes - What learners need to achieve to have
successfully completed the learning process).
1.4. Instructional Strategies and Methods.
2. Course Opening – Welcoming learners
2.1. Accessibility – The instructor gives clear instructions on how to
access all elements of the online learning environment.
2.2. Role – The instructor gives clear information about his professional
role in the learning environment.
2.3. Description - A course description with pre-requisites (if any), clear
learning outcomes and what is expected of the learners is provided.
2.4. Behaviour - The learners are made aware of regulations, policies and
ethics than govern the course.
2.5. Integrity - The instructor is aware of the academic integrity needed to
facilitate learning.
2.6. Technical Competences - The learners are made aware of the
technical competences needed to successfully reach the learning
outcomes.
2.7. Ownership – The instructor gives learners the opportunity to share
their own learning goals.
3. Assessment of Learning – Determining what the learner has learnt
and subsequent accreditation
3.1. Goals and Objectives – The learners are aware of what is expected of
them when they are assessed.
3.2. Strategies – Clear, well-defined and measurable assessment of
learning outcomes suited to the level of the learners.
3.3. Grading – Grades are given in a fair and transparent manner through
appropriate assessment instruments sanctioned by the institution.
3.4. Feedback – Both instructor and learners are given the opportunity to
provide feedback related to grading.
3.5. Management – Learners have access to their grades and feedback at
all times so that they can track their learning progress.
4. Interaction and Community – The exchanges between instructor and
learners that build a community that supports teaching and learning

ICICTE 2017 Proceedings
!
!
177
4.1. Fostering – The instructor welcomes learners and gives them the
opportunity to communicate and create an online environment that
fosters peer learning and engagement.
4.2. Management – Community-building is supported by clear
instructions, rules and regulations. While the instructor facilitates
engagement, learners are invested with the ownership of community-
building.
4.3. Peer learning – Group work and other activities that foster peer
learning are encouraged and structured not only to fulfil the learning
outcomes, but also to present learners with an opportunity to learn
skills and competences that go beyond such outcomes, e.g., digital
literacy.
5. Instructional Resources for Teaching and Learning
5.1. Provision – Learning materials are either provided by the instructor or
the learners are given enough time to procure such resources. The
difference between compulsory and optional resources is to be made
clear.
5.2. Application – The instructor clearly explains how the resources are
going to be applied and utilised.
5.3. Entitlement – The instructor makes sure that the resources indicated to
fulfil the learning outcomes are open and accessible by all the learners
without unwarranted technical, financial or administrative barriers.
The use of Open Educational Resources (OER) should be encouraged.
5.4. Variety – Learning resources are varied in terms of the multimedia
content and multi-modal delivery channels to cater for the different
learning preferences of learners.
5.5. Openness – The instructor should give learners the opportunity to
suggest their own resources for adoption in the course.
5.6. Academic Integrity – The instructor promotes best practice in the use
of third party resources, including anti-plagiarism practices and sound
academic research/writing practices. The use and/or adherence to the
Creative Commons licensing framework is encouraged.
6. Learner Support – Learners enabled to achieve their maximum
potential
6.1. Instructional Support – The instructor explains his/her role in the
process.
6.2. Academic Support – Learners know how to obtain such services as
mentoring, advice and other skills that support them in achieving the
learning outcomes.
6.3. Technical Support – Learners know how to obtain technical support to
overcome potential issues in accessing the learning area and achieving
the learning outcomes.
6.4. Administrative Support – Learners know how to obtain administrative
support to overcome potential issues in accessing the learning area
and achieving the learning outcomes.

ICICTE 2017 Proceedings
!
!
178
7. Technology design – Technology is at the service of teaching and
learning
7.1. Support – All the utilised technologies and resources support the
achievement of the aims and objectives of the instructor and the
learning outcomes for learners.
7.2. Centricity – All technologies and resources used support a learner-
centric rather than an instructor-centric educational approach. The
learners must be in control and technology must assist them in
achieving the learning outcomes.
7.3. Openness – The technical infrastructure used to deliver the teaching
and learning is procured and implemented according to open
standards and formats that maximise the value for money and the
range of options to fulfil the learning outcomes and the academic
needs of faculty and learners.
7.4. Authentication – Authentication at different levels (device, software,
virtual learning environment, specific course/learning area) should
provide access to a safe and secure teaching and learning environment
with the minimum number of steps possible to access the learning
areas.
7.5. Access – The virtual learning environment/learning area is
device/platform agnostic as much as possible, thus accessible over
different software platforms, browsers and computing devices. The
instructor provides alternative resources if any of these are not easily
accessible for technical reasons related to special needs of learners.
7.6. Interface – The user interface and navigation in the learning area is
simple enough to be conducive to teaching and learning without the
need to possess advanced ICT skills and competences.
7.7. Investment – The technical requirements of the instructional resources
and the virtual learning environment/learning space do not require
learners to make any significant new investment in hardware,
software and online services to be able to access and use these
resources to fulfil the learning outcomes.
7.8. Management – Learners are aware of the rules, regulations and
policies at institutional and at learning community level that govern
the use of the technological infrastructure supporting e-learning.
8. Course evaluation – Feedback to improve teaching and learning
8.1. Entitlement – Instructors should give learners the opportunity to
provide feedback on the whole learning experience. On the other
hand, instructors should also be able to provide their feedback within
their organisation.
9. Course Closing
9.1. Assessment – Learners should have access to their grades and the
course material after the closure of the course (depending on the
institution’s access policies). The final grades should be provided
within a reasonable timeframe after the closure of the course.

ICICTE 2017 Proceedings
!
!
179
9.2. Resolution - All pending issues between the instructor and the
learners are resolved.
9.3. Archiving – The instructor makes sure the course/learning area
resources, texts, communication, etc., are backed-up or archived (in
line with the institution’s access policies) in a safe and secure way.
10 Instructional Design Cycle
10.1 Academic Review – The instructor and the organisation review the
course description, the experience gathered, and the evaluation given.
10.2 Technical Review – The instructor, with the relevant technical unit in
the organisation, reviews the performance of the technical
infrastructure used to deliver teaching and learning.
10.3 Administrative Review – The instructor, with the relevant
administrative unit/s in the organisation, reviews the administrative
processes supporting the delivery of teaching and learning.
This Comprehensive Rubric is being published under the following Creative
Commons License: Attribution ShareAlike 4.0 International (CC BY-SA 4.0).
This grant of the license does not claim to cover the four rubrics individually.
Recommended Actions in Support of Instructional Design
for e-Learning in Higher Education
The following recommendations are compiled from the literature review and
the development of the comprehensive rubric:
1. Teaching and learning should dictate the technological implementation
of supporting tools and facilities, not the other way round.
2. An educational institution wishing to provide e-learning opportunities
needs a clear vision and a strategy sustaining e-learning from an
academic, technological and administrative point of view.
3. The use of e-learning pedagogies should be promoted as an official
part of the professional work of academics, with tangible incentives
and rewards that can take different forms, including professional
recognition, financial and material rewards.
4. An educational institution wishing to implement e-learning needs to
invest in academic training and support, in technical support and a
sound technical infrastructure, and in administrative support.
5. An educational institution should take time in getting to know the
views and needs of its faculty and students.
6. A VLE should not be utilised as a simple online repository of content
or for document management.
7. E-learning should attempt to benefit from the affordances provided by
technology as guided by the digital pedagogies and experience with e-
learning, especially providing students with an element of control over
the pace, time and path of study. Otherwise it simply serves as an
extension of traditional teaching practices.
8. Student-instructor and student-student interaction, collaboration and
work through the VLE. Community building.

ICICTE 2017 Proceedings
!
!
180
9. A VLE alone does not provide a complete range of tools to facilitate
online learning, therefore complementary tools such as social media
should be sought.
10. E-learning, with MOOCs for example, is an opportunity to explore
micro-credentialing and accreditation of online learning.
Discussion
The analysis of the rubrics has confirmed the myriad of possibilities provided
by technology when applied to teaching and learning. Indeed, the resulting
comprehensive rubric is rather long, and its components may indeed
contribute to the welcome conclusion of the long-standing debate on whether
e-learning is as rigorous and effective as traditional face-to-face environments
(Casey, 2008; Jung & Latchem, 2012). Such a rubric will surely support
faculty in the ever-increasing implementation of e-learning (Gaebel, 2015).
The active involvement of students in the learning process, rendered possible
by technology, is well catered for in the comprehensive rubric. The early
emphasis of technology in e-learning has given way to more credence in the
pedagogical benefits (Garrison & Anderson, 2011; McLoughlin & Lee, 2010;
Rovai, 2004).
The lack of common definitions of e-learning and its constituents is notable in
the chosen rubrics, but there is nevertheless a common approach: empowering
the educator to empower the student in an online environment that promotes
learning.
Digital literacy of faculty and time pressures remains a challenge and even
though technology is available, the application of the elements listed in the
comprehensive rubric requires a level of digital competence from instructors
(Conole & Alevizou, 2010; Grove, 2017; Johnson et al., 2015).
MOOCS, OER, BOYD, social elements, artificial intelligence, augmented and
virtual reality are acquiring more space and attention in education, and the
comprehensive rubric must take into consideration these new instructional
approaches and updated digital pedagogies (De Freitas & Conole, 2010).
The tensions created by the implementation of technology in HE, especially
the changing role of educators and students brought by more social, more
ubiquitous and more open learning spaces, will surely bring to light any gaps
between the planned and the actual implementation of e-learning. It is up to
the educator, as a professional, to mind these gaps and bridge them.
Limits of Scope
It is not within the scope of this paper to review quality assurance processes
and administrative components, but to propose a rubric for course design and
self-review of faculty and HE institutions for a better alignment with what is
regarded as current standard best practice.

ICICTE 2017 Proceedings
!
!
181
Suggestions for Further Research
The proposed comprehensive rubric does not provide a scale for assigning
points when applied, thus giving a weighting to the elements of the rubric
perceived as more important than others. There are other rubrics on e-learning
by reputable HE institutions that could be included in the analysis. The litmus
test for the rubric is its application in real life situations. This is an excellent
opportunity for follow-up research analysing the outcomes of the its
application.
References
Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., &
Ananthanarayanan, V. (2017). NMC Horizon Report: 2017 Higher Education
Edition. Austin, Texas: New Media Consortium.
Barr, R. B., & Tagg, J. (1995). From teaching to learning: A new paradigm for
undergraduate education. Change, 27(6), 13–25.
Beetham, H., & Sharpe, R. (2013). Rethinking pedagogy for a digital age:
Designing for 21st century learning. London, United Kingdom: Routledge.
Bozkurt, A., Akgun-Ozbek, E., Yilmazel, S., Erdogdu, E., Ucar, H. Guler, E., …
Aydin, H. (2015). Trends in distance education research: A content analysis of
journals 2009-2013. International Review of Research in Open and Distributed
Learning, 16(1). 330-362.
California State University (2013). CSU QOLT instruments: Non-awards versions.
Retrieved from http://courseredesign.csuprojects.org/wp/qolt-nonawards-
instruments/
Casey, D. M. (2008). A journey to legitimacy: The historical development of
distance education through technology. TechTrends: Linking Research &
Practice to Improve Learning, 52(2), 45-51.
Conole, G., & Alevizou, P. (2010). A literature review of the use of Web 2.0 tools
in Higher Education (A report commissioned by the Higher Education
Academy). York, United Kingdom: HEA.
Christie, B. (2014). CSU Quality Online Learning and Teaching (QOLT)
Evaluation Instrument Sections and Objectives. California State University.
Retrieved from http://qolt.sfsu.edu/sites/default/files/QOLT2-
pgnumberslogoCore24CClicense111715.pdf
Daradoumis, T., Bassi, R., Xhafa, F., & Caballe, S. (2013). A review on massive e-
learning (MOOC) design, delivery and assessment. Eighth International
Conference on P2P, Parallel, Grid, Cloud and Internet Computing. doi:
10.1109/3PGCIC.2013.37
De Freitas, S., & Conole, G. (2010). Learners’ experiences: How pervasive and
integrative tools influence expectations of study. In R. Sharpe, H. Beetham, &
S. De Freitas (Eds.), Rethinking learning for the digital age: How learners
shape their own experiences (pp. 15-30). London, United Kingdom:
Routledge.
Fenrich, P. (2008). Instructional strategies. In S. Hirtz & D. M. Harper (Eds.),
Education for a digital world: Advice, guidelines, and effective practice from
around the globe (pp. 309-320). Burnaby, British Columbia, Canada:
Commonwealth of Learning.
Gaebel, M. (2015). E-learning in European higher education institutions - Results
of two EUA studies in 2014 and 2015. Brussels, Belgium: European
University Association. Retrieved from

ICICTE 2017 Proceedings
!
!
182
http://bologna-yerevan2015.ehea.info/files/Michael%20Gaebel.pdf
Gamage, D., Fernando, S., & Perera, I. (2015). Quality of MOOCs: A review of
literature on effectiveness and quality aspects. Education, 121, 2.
Garrison, D. R., & Anderson, T. (2011). E-learning in the 21st century: A
framework for research and practice. New York, NY: Taylor & Francis.
Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical foundations of
massive open online courses. First Monday, 18(5), 1-12.
Govindasamy, T. (2002). Successful implementation of e-learning pedagogical
considerations. Internet and Higher Education, 4, 287-299.
Graham, C. R., Cagiltay, K., Byung-Ro, L., Craner, J., & Duffy, T.M. (2001)
Seven principles of effective teaching: A practical lens for evaluating online
courses. Retrieved from
http://www.technologysource.org/article/274/?utm_content=buffere64be&utm
_source=buffer&utm_medium=twitter&utm_campaign=Buffer
Graham, C. R., Woodfield, W., & Harrison, J. B. (2013). A framework for
institutional adoption and implementation of blended learning in higher
education. Internet and Higher Education.18(3), 4-14.
Grove, J. (2017, February 16). THE Teaching Survey 2017: Results and analysis.
Times Higher Education (THE). Retrieved from
https://www.timeshighereducation.com/features/the-teaching-survey-2017-
results-and-analysis
Guri-Rosenblit, S., & Gros, B. (2011). E-learning: Confusing terminology,
research gaps and inherent challenges. International Journal of E-Learning &
Distance Education, 25(1), 1-17.
Hogg, S. (2013). An informal use of Facebook to encourage student collaboration
and motivation for off campus activities. In G. Mallia (Ed.), The social
classroom: Integrating social network use in education (pp. 23-29). Hershey,
PA: Information Science Reference.
Hoffman, R. (2015, October 20). Social networks will help education connect says
Reid Hoffman. Wired Magazine. Retrieved from
http://www.wired.co.uk/article/wired-world-2016-reid-hoffman
Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2014). NMC Horizon
Report: 2014 Higher Education edition. Austin Texas, The New Media
Consortium.
Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2015). NMC Horizon
Report: 2015 Higher Education edition. Austin Texas, The New Media
Consortium.
Jung, I., & Latchem, C. (2012). Quality assurance and accreditation in distance
education: Models, policies and research. Oxford, United Kingdom:
Routledge.
McLoughlin, C., & Lee, M. J. (2010). Personalised and self-regulated learning in
the Web 2.0 era: International exemplars of innovative pedagogy using social
software. Educational Technology, 26(1), 28-43.
Means, B., Murphy, R., & Bakia, M. (2013). The effectiveness of online and
blended learning: A meta-analysis of the empirical literature. Teachers
College Record, 115(3), 1-47.
Nagel, L., & Kotzé, T. G. (2010). Supersizing e-learning: What a CoI survey
reveals about teaching presence in a large online class. The Internet and
Higher Education, 13(1), 45-51.

ICICTE 2017 Proceedings
!
!
183
QualityMatters (2014). Higher Ed course design rubric. Retrieved from
https://www.qualitymatters.org/qa-resources/rubric-standards/higher-ed-rubric
Ronan, A. (2015). Whatever happened to course management systems?
Edudemic.com. Retrieved from http://www.edudemic.com/whatever-
happened-course-management-systems/
Rovai, A. P. (2004). A constructivist approach to online college learning. Internet
and Higher Education, 7, 79-93.
Tomte, C., & Sutherland Olsen, D. (2014). Exploring quality in teaching and
learning with ICT: A qualitative study (pp. 213-220). In Proceedings of the
European Distance and E-Learning Network 2014 Research Workshop.
Oxford, United Kingdom.
University of Illinois. (2015). Quality online course initiative. Retrieved from
http://www.ion.uillinois.edu/initiatives/qoci/rubric.asp
University of Malta. (2015). Minimum standards for study-units in the VLE.
Retrieved from http://www.um.edu.mt/vle/staff/minimumstandards
Young, J. R. (2002). “Hybrid” teaching seeks to end the divide between traditional
and online instruction. Chronicle of Higher Education, 48(28), A33-A34.
Retrieved from https://eric.ed.gov/?id=EJ645445
Author Details
Martin Debattista
martin.debattista@its.edu.mt