Content uploaded by S.C. dos Santos
Author content
All content in this area was uploaded by S.C. dos Santos on Jul 21, 2015
Content may be subject to copyright.
Applying PBL in Project Management Education: a
Case Study of an Undergraduate Course
Simone Santos, Gustavo Alexandre
N.E.X.T Research Group
UFPE - Centro de Informática
Recife, Brazil
scs@cin.ufpe.br, gsa@cin.ufpe.br
Ariane Rodrigues
N.E.X.T Research Group
UPE – Universidade de Pernambuco
Recife, Brazil
ariane.rodrigues@upe.br
Abstract— The Software Engineering sector has been demanding
an education model that targets real market practices more and
more exactly. This includes bearing in mind that, in the market, a
software project is subject to numerous restrictions of time,
budget and other resources required for its development. In this
context, this article describes the application of a learning
methodology based on problems, called xPBL. This methodology
consists of elements that enable a learning environment to be
built that in its essence is practical and contains real learning,
and that ensure that this is supported by processes that make it
possible to evaluate the effectiveness of the PBL approach from
various perspectives: namely, the student’s, the teacher’s and
that of the methodological approach itself. Based on this case
study, evidence of the applicability of xPBL is demonstrated as is
how the behavior should be understood of all stakeholders
involved in the process of teaching and learning in one of the
most complex disciplines of Software Engineering.
PBL; Project Management, Software Engineering Education.
I. INTRODUCTION
The coining of the term "Software Engineering" in
Computer Science has thrown down major challenges for
those who teach it. Obviously, the requirements of Software
Engineering demand a learning environment that is not only
practical but true to the reality of the market. That is, an
environment that allows its students to experience in a real
way, the software development process, its management
constraints (scope, time, cost, risks), technological limitations
and conflicts of interest between human resources, whether
these are members of a software development team or
customers seeking software solutions.
This is the context of what this article sets out to show. By
using a teaching methodology centered on the student and on
solving real problems in a real work environment, this article
describes the experience of applying PBL (Problem-Based
Learning) to a Planning and Project Management module
which is part of the curriculum of an undergraduate degree in
Information Systems (IS).
PBL was applied using a method called xPBL developed by
the NEXT (iNnovative Educational eXperience in
Technology) group as described in [1]. The elements of this
methodology as well as some of the practical studies that
preceded it are discussed briefly in Section II of this article.
It is important to point out that the NEXT group has been
making a study of and applying PBL in Computer Science
since 2007. Trialing was almost always carried out on post-
graduate courses or courses aimed at adding to professional
skills, most of which focused on enhancing specific
professional skills, for a target-audience of Computer Science
graduates, as described in [2], [3], [4] and [5].
Unlike previous research studies, this paper describes and
discusses the application of PBL methodology on a course of
the 4th semester of a still ongoing 8-semester undergraduate
degree in IS, as set out in Section III, in parallel with five
other highly relevant course modules, including Software
Engineering, Databases and Business Processes Management.
In this context, the Project Planning and Management (PPM)
course module, the object of this study, competes with the
modules of the same semester with respect to the availability,
commitment and engagement of the students. This feature
reinforces the need for a methodology supported by a real
environment and real assessment processes, thereby
continuously bringing students to the center of the learning to
which they belong.
The application of xPBL methodology is described and
discussed in Section III, and the evaluation process and its
results are presented and discussed in Section IV.
Lastly, Section V draws conclusions and makes some final
remarks on this research experience.
II. THE XPBL METHODOLOGY
The purpose of the xPBL methodology is to align methods
and tools to the management of the PBL approach so that the
principles that characterize it can be guaranteed when adopted
for teaching Computing.
On adopting the PBL approach, characteristics such as
flexibility and unpredictability should be considered,
especially with regard to their impact on managing the steps
and activities associated with the teaching and learning
process. According to [6], the essence of PBL is that it is
process-oriented. Therefore, as this is one of the main features
associated with this approach, it is important to maintain
alignment between the stages of the PBL process to ensure its
effectiveness. These assumptions led the NEXT group to
conducting research, especially with respect to drawing up
processes and metrics to plan and manage PBL. This involves,
for example, concepts associated with the 5W2H technique
[7], Deming´s PDCA (Plan, Do, Check and Act) [8], PBL
principles [9] and problem-solving models [10]. It is important
to emphasize that the applicability of xPBL in case studies
such as in [2], [3], [4] and [5] enabled experience to be
obtained of its implementation and the continuous
improvement of its processes.
To ensure PBL principles besides its educational goals are
adhered to, the xPBL methodology is based on five elements:
(1) Problem; (2) Environment; (3) Content; (4) Human capital
and (5) Process. These elements reinforce ten principles that
were established in [9]. Table I maps the ten principles
associated with each element of xPBL.
TABLE I. ELEMENTS OF XPBL AND PBL PRINCIPLES (SOURCE IN: [9]).
Mapping PBL principles and elements of xPBL
PBL Principles
xPBL
Elements
1. All learning activities are anchored on a task or a
problem;
Problem
2. The learner should feel he/she owns the problem, and is
responsible for his/her own learning;
3. The problem should be real;
6. The learning environment should stimulate and at the
same time challenge the learner´s reasoning;
4. The task and the learning environment should reflect
the reality of the professional market;
Environment
5. The learner needs to own the process used so as to work
out the solution to the problem;
Content
7. The learner should be encouraged to test his/her ideas
against alternative views and contexts;
9. The learning is collaborative and multidirectional;
Human
Capital
8. The learner should have the opportunity and support to
reflect on the content learned and the learning process;.
Process
10. PBL is supported by planning processes and
continuous monitoring.
Learning in PBL is directed towards problem solving. The
problem is always the central task to be undertaken in this
process. As seen in the mapping shown in Table I, Principles
1, 2, 3 and 4 are associated with the 'Problem' element of
xPBL. These principles support the idea that problems should
be real and complex so that students’ interest in solving them
is aroused.
Regarding the 'Environment' element of xPBL, Principle 4
supports the need to simulate a real learning environment that
reflects actual conditions in the labor market. This means that
learning is essentially functional in an environment where
students are of a mind to be immersed in practices and
activities in a collaborative and multidirectional way (together
with their peers, teachers and tutors).
Principles 5 and 7 refer to the "Content" element of xPBL,
which reflects the need to align practices to content (whether
structure, theory, foundations, model or concept). Although
learning is essentially functional, the PBL methodology
considers it essential to have the support of content as a
conceptual basis for the problems and that the methodology
must not be confused by practical experiments in which the
support of theory and monitoring the processes of how
solutions are found are not prominent.
Regarding "Human Capital", principle 9 reinforces the idea
of collaborative and multi-directional learning being fostered
by constantly engaging students, teachers, tutors and
customers with each other during the resolution process.
Finally, principles 8 and 10 associated with the element of
‘process’ emphasize the need to plan and monitor the
processes of teaching and learning continuously by having the
students reflect on their learning and conducting reviews
which are consistent with the paradigm of the method. It is
also pointed out that Principles 9 and 10 were defined by the
authors of this article in [9].
Therefore, the authors, when defining the xPBL
methodology, consider that how its elements are tackled is
essential at all stages of the PDCA cycle, above all with regard
to planning being aligned to its implementation.
Quality management of the processes in the cycle
associated with xPBL elements can also be guaranteed by
adopting techniques such as 5W2H. The five aspects
associated with W and the two with H presented in the
technique match the keywords given in Table II. According to
the authors [1], a plan of action guided by the technique sees
to it that the 5 elements are defined in a clear and organized
way because they obtain answers to questions such as:
"What?", "Who?", "Where?" "When?", "Why?", "How?" and
"How much?".
TABLE II. 5W2H TECHNIQUE.
5W2H Techinique
Questions
5W
What?
What action will be performed?
Who?
Who will perform the action?
Where?
Where will the action be performed?
When?
When will the action be performed?
Why?
Why will the action be performed?
2H
How?
How will the action be performed?
How Much?
How much does it cost to perform the action?
As an effective management technique, 5W2H sees to it that
activities associated with each xPBL element are broken
down, analyzed and summarized during the planning stage.
The authors objectively considered information about what
should be done and when, or who will conduct a certain
activity and when, while they were defining each element.
When considering the ‘Problem’ element of xPBL, the
question "what?" of the technique refers to the set of possible,
real and complex problems which are similar to real-world
projects and the demands of real customers. The purpose of
this element (related to the question “why?”, for
simplification, “why?”) must necessarily be compatible with
the educational objectives, with the content to be learned,
besides the skills and attitudes needed to solve the problem.
As in the real world, problems arise from real customers
(who?) and it is they who ask for the activities and evaluate
the team's results, when considering their needs. It is up to the
students (who?) to select the problem from among different
requests (where?) right at the start of training (when?). This
process (how?) allows students to identify and understand
requests so that they can thereafter draw up an innovative plan
for solving them with relevant arguments, thereby proving the
feasibility of the process and its complexity. The aspect of
‘how much’ is not applicable to this element.
The ‘Environment’ element refers to definitions of actions
set by coordinators, tutors and teachers (who?) before the start
of a course (when?). The main actions (what?) refer to the
process of forming teams because of the number of aspects
and personal aspects and defining the resolution process
associated with activities, roles, business models and
development models (how?). IT infrastructure and physical
infrastructure aspects should also be considered (what?) since
the environment should reflect the real conditions of the labor
market (why?). The costs for this element (how much?) are
related to acquiring resources and licenses. The methodology
recommends that small teams (5-7 members) be formed,
where aspects such as professional experience and
interpersonal skills should be considered.
In the context of Human Capital, roles and responsibilities
are defined (what?) during the planning (when?) so that those
involved know how to proceed correctly when implementing
the PBL approach (why?). Teachers and technical tutors
(who?) teach the content, coordinate activities and can be
consulted about real experience in projects (how?). The PBL
tutor (who?) monitors the entire process of solving the
problems of the teams, and records the results of the
performance and individual assessments (how?). Students and
customers should also be considered in this context (who?).
The aspect of ‘how much does it cost’ is not applicable in this
element.
As for the ‘content’ element, it is known that this is the
conceptual basis for the process of problem solving, as well as
being responsible for providing the formation of students’
knowledge (what?). The flexible learning associated with the
PBL approach requires teachers and tutors (who?) to map the
contents of different media such as books, magazines, and
specialist blogs (where?). The idea is not to follow pre-defined
routes, but rather to take up unpredictability as the rhythm of
the solution process in practice (why? and how?).
Finally, the last element of the xPBL methodology refers to
the assessment process (what?). The reference that the authors
[1] stress is the authentic assessment proposed by [3] to
evaluate the effectiveness of the approach under different
aspects by considering the rigor associated with real practices
both individually and collectively (why?). Throughout the
training (when?), formative type assessments are conducted to
verify that educational goals are being achieved. Summative
assessments are carried out at the end of educational units to
classify the students’ achievement levels as to course content
and foundations of the module as well as the interactions and
deliveries (when?). Teachers and tutors (who?) evaluate the
students (how?) under ways associated with the content, the
resolution process, and the performance and development of
interpersonal skills in the teaching and learning process. The
students (who?) evaluate themselves, their peers, as well as
teachers and tutors and the PBL methodology itself. In the
perspective of the students’ evaluations, aspects such as results
and customers’ satisfaction derived from authentic assessment
are considered. The level of customer satisfaction refers,
above all, to the service provided by the team. The results of
each evaluation are consolidated by the teacher in reports with
varying levels of visual details using graphics and comments.
The next section describes these elements applied to the
undergraduate degree with an emphasis on the management
and planning of information system projects.
III. USING XPBL IN PROJECT MANAGEMENT
UNDERGRADUATE COURSE
This section describes the application of the xPBL
methodology in the Project Management in Information
Systems module of the undergraduate course. There is a total
of 60 contact hours over 4 months, the educational goal being
to train students in good practices in Project Management.
To conduct the course, planning was carried out by the
pedagogical team comprising the teacher of the course and four
tutors: two technical tutors (with project management
experience); and two PBL tutors, responsible for monitoring
the learning process throughout the course. This plan followed
the guideline proposed by xPBL in [1]. Based on applying the
5W2H technique to the xPBL elements discussed in Section II,
the PBL approach was used to structure the module.
A. Environment
The learning environment consisted of 33 students, divided
into six teams: 3 teams of 5 students and 3 teams of 6 students.
Of the students, 70% of the class were between 19 and 21 years
old, and the rest up to 24 years old; 80% of the class were
male; and 33% of the group had no professional experience.
The criteria used to distribute the students into teams were:
professional experience, age, interest in Project Management,
in addition to the student's personal profile (artisan, guardian,
idealist and rational), which was identified by applying a
simplified version of the MBTI - Myers-Briggs Type Indicator
[13]. So as to form the teams in a more balanced way, the
following rules were established: (1) Each team consists of a
mix of MBTI profiles; (2) At least one profile of "idealist" per
team, usually related to creativity and innovation; (3) No team
has all members within a single age band; (4) At least one
female in each team; (5) At least half the team have some
professional experience; (6) Affinities are identified from the
initial dynamics.
To support the communication and dissemination of
educational materials of the course, the following tools were
adopted: Facebook, Dropbox, Google Drive and a WhatsApp
group, which enabled greater collaboration among all
stakeholders.
To plan and monitor the development of each project, the
teams adopted agile management techniques by free choice.
All that the teaching team required was that it should be
possible to monitor the process remotely and so each team used
a web tool that supported the process adopted and allowed the
progress of the project to be followed by the team of tutors via
the web, whether by monitoring Tables via document
repositories or the project site.
As for the classroom, the students had a whiteboard and a
physical space with mobile tables and chairs, which permitted
physical configurations to be arranged for groups. This was an
environment that was improvised and needed to be reorganized
at the start and finish of each class. Unfortunately, the
organization does not have appropriate rooms for PBL, which
reflect working environments found in the software industry.
As to the hardware used, a notebook was made available in the
classroom to each team and each team had ample access to the
undergraduate laboratories (around five of them and all of
which are of high quality and capacity) at the Informatics
Center throughout the day.
B. Problem
The objective of the 1st. step of the course was to bring all
students to the same level in Project Management concepts
thus seeing to it that they could become familiar with the
problem- solving approach, which is very different from
traditional classes based on presenting content on PMBOK
(Project Management Body of Knowledge), version 5.
In the 2nd. step of the course, students were responsible for
identifying and formalizing the problem to be investigated,
including ensuring the participation of a real client who was
external to the academic environment. To ensure that the
problems identified by the teams and the proposed solutions
would be compatible with the educational objectives and
training needs and skills associated with the course, teaching
staff guided the teams such that the solutions were designed
using WEB2.0 tools or platforms, for example: Salesforce,
Content Management Systems, or other environments based on
a high-level business component.
To help students understand the problem and propose a
solution, a dynamic was conducted, supported by the Delisle
problem-solving model [14]. The model sets out four aspects of
tackling a problem that should be observed: (1) Ideas, possible
solutions for solving the problem; (2) Facts, information about
the problem; (3) Hypotheses, raising learning issues to solve
the problem and; (4) Action Plan, strategies, resources,
information, everything that leads to the solution. On having a
clearer understanding of how to find solutions to the problem,
the students then move on to the step of formalizing the
problem. The problems were defined by each team using a
template with questions related to mastering the problem, its
causes, complexity, the target public's needs and so forth. To
define the proposals, questions guided the students on how to
evaluate possible solutions, without overlooking the strategy
for achieving them, available resources and the benefits to
customers. The definitions of problems were sent to the
teaching staff so that they could be evaluated under the criteria
of Adequacy, Complexity, Clarity, Relevance and Innovation,
thereby determining if they were to be selected or adapted.
C. Human Capital
The teaching staff consisted of the teacher of the course and
four tutors. The general function of these tutors was to
continually support student learning. Thus, the course received
support from: two technical tutors, whose competences and
experience were specific to the subjects of the course and; two
PBL tutors, who helped to conduct and implement the xPBL
methodology. In conducting the classes, one of the PBL tutors
was always present and; at least one technical tutor attended
the monitoring meetings of the projects.
Besides the four tutors and the teacher of the course (an
expert on Project Management and PBL), there was a team of
developers and the Project Manager, who was a student
belonging to the team itself and elected by its members.
Finally, there was the figure of the real client, a person
representing a company or individual entrepreneur, with whom
the teams had identified some demand that needed to be
satisfied by using information systems.
D. Content
The course was conducted by the teacher and technical
tutors for which the main reference was the PMBOK Guide
version 5, and speakers on agile management approaches
SCRUM [15] and KANBAN [11] were invited. Moreover, a
lecture on the PBL approach and a dynamic on the process of
problem solving according to Delisle´s model were given [14].
The course was run in 3 modules. The 1st module focused
on the overview of PMBOK v5, in order to ensure early study
of the main concepts of the Project Management area; the 2nd.
module focused on the phases of Initiating and Planning the
life cycle of projects; and the 3rd. module focused on the phases
of implementing, controlling and closing the project life cycle.
With reference to the project phases that the teams
experienced, the teacher defined the content planned for the
course. Based on the difficulties encountered in the evaluation
process (detailed in Section IV), the "on demand" content was
given to the class, thereby reinforcing the points for
improvement identified in the teams.
E. Assessment Process
The evaluation model implemented in this case study was
structured into three levels: the first targeted the evaluation of
the students according to the authentic assessment model and
used five perspectives – Content, Process, Results,
Performance and Client Satisfaction; the second targeted the
degree of maturity of the PBL approach in the students’ view;
finally, the third focused on the final evaluation of the course,
under criteria related to how the teacher planned it and
performed. Figure 1 summarizes the evaluation process and the
results obtained. The details of each of the levels of Figure
1will be presented in Section IV.
Fig. 1. Assessment Model used in the module.
IV. ANALYSIS OF RESULTS
A. Results of the Student Assessment
The assessment process was conducted in three modules:
the 1st module focused on knowledge and understanding of
project management concepts and fundamentals, and only
applied the ‘Content’ perspective; the 2nd and 3rd modules
focused on planning and project management, and considered
all the perspectives of the authentic assessment.
As for the results obtained by the students, Figures 1, 2
and 3 give graphs that show the concentration of the students’
performance in the 1st, 2nd and 3rd modules, respectively,
within the value scale of from 0 to 10.
It is worth pointing out that the target set by the course is
an average of 7 for a direct pass, and an average of 5 for a pass
subject to an evaluation of the cumulative content at the end of
the course (final evaluation).
Fig. 2. Concentration of the students’ performance in the 1st module.
Fig. 3. Concentration of the students’ performance in the 2nd module.
Fig. 4. Concentration of the students’ performance in the 3rd module.
Note that the results for the 1st module, reached solely by
assessing the Content perspective, show performances with
the highest concentration between the values of 5 to 6, unlike
the results of the 2nd. and 3rd modules, which have a
concentration between 7 to 8 and 8 to 9, respectively.
Table III shows the types of assessment conducted within
each module, and highlights the instrument used for the
assessment: an objective test, based on the PMI (Project
Management Institute) standards; the Meeting to start the
Project (Kickoff); the Project monitoring meeting or remote
monitoring (Status Report).
TABLE III. TYPES OF ASSESSMENT PER MODULE.
Module
Individual Assessment
(Summative)
Group Assessment
(Formative)
#
Content
Performance
Process
Output
Client
1
1st$EV$
'$
'$
'$
'$
2
2nd$EV$
1st$EV$$
Kickoff$
1st$SR$
Kickoff$
1st$SR$
Kickoff$
1st$SR$
3
3rd$EV$
2nd$EV$
2nd$SR$
3rd$SR$
2nd$SR$
3rd$SR$
2nd$SR$
3rd$SR$
EV – Evaluation | SR – Status Report.
Some reasons for the results of the first module confirm
the difficulties of the conventional assessment process, which
is summative (a test comprising 20 to 25 questions based on a
PMBOK overview) and focuses only on concepts and
definitions, given individually without debate or discussion.
To develop the essentials skills that a project manager
needs, the authentic assessment, recommended by xPBL
methodology [3] was applied in accordance with the five
perspectives:
1. Content, from objective evidence, aiming at concepts
and foundations of the Project Management area,
taught by the teacher;
2. Process, from assessments led by the teacher and
tutors during Status Report meetings;
3. Results, from the deliveries made in accordance with
the project planning during Status Report meetings,
evaluated by the teacher and tutors;
4. Performance, based on assessing the 180 degree
performance, where each student made a self-
assessment and was rated by all members of his/her
team, conducted by tutors;
5. Client satisfaction, based on the client’s evaluation of
solutions during Status Report meetings.
To better understand the performance of the class in the
2nd and 3rd module (Figures 3 and 4, respectively), the
following sections describe the performance of the teams
within each of the five perspectives of the authentic
assessment. To calculate the performance of each student per
module, the following formula was used:
20% * AA(Content) + 20% * AA(Process) + 20% * AA (Output)
+ 20% * AA(Performance) + 20% * AA(Client satisfaction), where
“AA” corresponds to the arithmetic mean of the scores related
to each perspective, when there is more than one score.
1) Content perspective:
In the Content perspective, the evaluation was applied
individually, based on the content and practices discussed on
the course, the main reference being PMBOK version 5, and at
sessions led by invited speakers on agile management
approaches SCRUM [15] and KANBAN [11]. In this
perspective, there were two tests of content, both objective
ones, with about 20 to 25 questions: (1) Phases of Starting and
Planning the project life cycle; (2) phases of Running,
Controlling and Closing the project life cycle. These tests
were conducted at the end of the respective project phases and
therefore were of a summative character. The value scale used
followed the traditional model of from 0 to 10.
On calculating the overall average of the class, solely
under the perspective of Content in the 1st and 2nd modules, we
obtained a performance similar to that of the 1st module, the
pass rate being about 57%. However, the chief gain from this
evaluation was that the results of each objective test enabled to
identify the points of most critical difficulty, and thus to work
on the content in class with a view to clearing up doubts and
mistakes identified in the tests.
Some other matters can be considered regarding the
students’ difficulty with learning theory and concepts.
Unfortunately, for many students, the study habit of reading
intensively about concepts and fundamentals has become a
rare practice. They are connected to social media based on
rapid and short answers. Despite complementary literature
being recommended such as presentations, articles and
podcasts, the result of the Content evaluations showed that the
students did not read the content proposed, even though this is
essential for a course based on a reference guide to items of
knowledge. Some of the reasons that the students alleged were
the competitive demands from other courses of similar
complexity and public holidays associated with the World Cup
which was held in Brazil in 2014. The tutors recorded these
considerations which will be considered when planning future
courses. These should include new initiatives to encourage
reading, such as debates and discussion sessions.
2) Process perspective:
In the perspective of ‘Process’, the teams were evaluated
by the teacher and the technical tutor who monitored the
projects during four meetings: one Kickoff and three Status
Report (SR) meetings. At the SR meetings, each team always
answered five questions: "What is the objective of your
project?"; "What's the plan?"; "What has been done?"; "What
are the strengths?"; and "What are the points of
improvement?". As criteria for evaluation in this perspective,
the following were defined: (1) Clarity in presentation; (2)
mastery of the presentation; (3) Completeness when
considering the five questions; (4) understanding of Planning.
Each indicator could take on one value from a simple scale of
five values: "1 - Insufficient; 2 - Regular; 3 - Good; 4 - Very
Good; 5 - Excellent". The graph in Figure 5 shows the teams’
performance throughout the project.
Fig. 5. Evolution of the perspective of Process.
On analyzing the graph in Figure 5, we see that the
performance of most of the teams improved throughout the
stages of the life cycle of the project, except team T1 team,
whose project manager left the team at the end of the course.
Turning to the performance of teams T2, T3 and T4, we
moreover see a significant improvement between the 1st
monitoring (Kickoff) meeting and the final delivery in Status
Report 3. Another interesting behavior observed in this chart
was the natural "relaxation" of the teams that obtain excellent
performances. It is common for teams to concentrate on other
priorities when they see that the challenges were met in full at
that moment, and thus this has an impact on future activities
and hence their performance in the following reviews.
3) Output Perspective:
As to the perspective of Output (Figure 6), this was
focused on analysis of the content of the presentations of the
projects in the monitoring meetings. These analyses were
conducted under the following criteria: (1) Overview of the
project; (2) Planning activities; (3) Carrying out activities; (4)
Strengths; (5) Points for improvement. Once again, the same
simple scale of five values was used. These evaluations were
conducted by the teacher and technical tutor.
Fig. 6. Evolution of the perspective of Output.
The graph in Figure 6 also shows an upward trend but
with slighter variations, thereby proving that, although
students have relaxed with the process of planning and
monitoring of the project after achieving excellent
performances, these continued to be focused on the delivery of
results, probably due to the culture of traditional teaching
approaches, usually based on delivery artifacts. In addition,
teams T2, T3 and T4 continued to stand out.
4) Performance Perspective:
In the perspective of Performance, eight competences
were assessed: self-initiative, commitment, collaboration,
innovation, communication, learning, planning and analysis.
Due to the subjectivity of this analysis, this perspective used a
scale of five values, with the following meanings: (1) "did not
meet expectations"; (2) "partially met them"; (3) "met them";
(4) "met them very well"; (5) "exceeded expectations". This
review was conducted by the PBL tutor and applied in the
self-assessment format and evaluation in pairs (known as the
180 degree evaluation), where each member of a team was
rated by his/her colleagues, anonymously. Since this was
undertaken by means of an online research tool, sophisticated
individual reports could be obtained for each student, which
showed the results of the assessment of colleagues in his/her
team and his/her own assessment in a consolidated and
graphic way, for each assessment criterion, including
subjective comments.
On analyzing Table IV, it can be seen that teams T2, T3
and T6 stand out with respect to the performance of their
members, in the eight perspectives mapped, as well as the
performance of their PMs. On comparing with the
perspectives of Process and Output, we see that there is a
direct relationship between the best results of Performance of
the Project Manager and the teams that stand out in these
perspectives show the PM´s influence on the conduct of the
process of management and deliverables of the project.
TABLE IV. EVALUATIONS IN THE PERSPECTIVE OF PERFORMANCE.
Criteria
T1
T2
T3
T4
T5
T6
SELF-INITIATIVE
3,6 /
3,4
3,7 /
4,1
3,9 /
4,1
3,4 /
3,7
3,6 /
3,4
4,3 /
4,4
COMMITMENT
3,8 /
3,6
3,9 /
3,9
4,2 /
4,2
3,7 /
3,6
3,6 /
3,4
4,5 /
4,6
COLLABORATION
3,7/
3,6
3,6 /
4
4,1 /
4,4
3,6 /
4
3,9 /
3,6
4,5 /
4,8
INNOVATION
3,3/
3,3
3,8 /
3,8
4 /
4,2
3,2 /
3,8
3,2 /
3,5
4,3 /
4,5
COMMUNICATION
3,6/
3,6
3,9 /
4,2
4 /
4,2
3,6 /
3,6
3,5 /
3,7
4,2 /
4,5
LEARNING
3,6/
3,3
3,9 /
4
4,1 /
4,2
3 /
3,6
3,3 /
3,2
4,3 /
4,5
PLANNING
3,6/
3,4
3,7 /
4,1
4 /
4,2
3,2 /
3,6
3,7 /
3,4
4,3 /
4,6
ANALYSIS
3,6/
3,5
3,6 /
4
4 /
4,2
3,4 /
3,6
3,6 /
3,2
4,4 /
4,5
General average
of the teams:
3,6/
3,5
3,8 /
4
4,0 /
4,2
3,4 /
3,7
3,6 /
3,4
4,4 /
4,6
Average of the
PM:
3/
1,5
4 /
4,3
4,3 /
4,3
3,6 /
3,8
3,4 /
4
4,2 /
4,5
It is worth pointing out some important points about
teams T1 and T6. The first was strongly impacted by the PM
abandoning the project at the 3rd Status Report meeting (final
delivery) after his performance had been rated by his team as
unsatisfactory in the 2nd evaluation of performance. This result
shows the importance of evaluation in pairs to give evidence
of the performance of everyone in the team. The second point
arose from the team which had high maturity in Performance
but a poor performance in content. This impacted the
performance of the perspectives of Process, which in this case
was associated with planning and monitoring the project,
which requires an understanding of PMBOK techniques.
5) Client Perspective:
Finally, the evaluation of client satisfaction was based on
the following criteria: projection of confidence in interviews;
understanding of the problems; clarity of presentation; quality
of the solutions proposed; level of planning. This assessment
used the same value scale as the perspectives of Process and
Output, and was conducted by the teacher and the PBL tutors
together, the client of the solution being present in the
monitoring meetings (Status Report). Figure 7 shows the
results of the teams in this perspective. Looking at the graph in
Figure 7, we see a direct relationship with the teams’
performance in the perspective of Output.
It is worth mentioning, that the involvement of the real
customer in the evaluation process is crucial to the PBL
approach, given that the stakeholder who will benefit from the
solution cannot be left aside. This was one of the points that
the teacher most worked on after the kickoff of the project:
namely, the need to bring the customer to the center of the
project, keeping him/her continuously close to the processes
and validating each stage of the project with him. This
reinforcement led to greater performances throughout the
project in this perspective as shown in Figure 7.
Fig. 7. Evolution of the perspective of Client Satisfaction.
B. PBL Evaluation
To evaluate the degree of PBL maturity in the course, in
accordance with the proposal described in [9], the PBL-Test
was applied at the end of the 2nd module. The objective of this
application was to check for possible deviations from PBL and
fix them in time for the 3rd module. The PBL-Test was taken by
28 students, which was therefore a representative total of more
than 80% of the class. Table V shows the results of the
evaluations.
TABLE V. RESULTS OF THE PBL-TEST.
PBL Principles Evaluated
Average
1. Problem (s) is at the core of the educational proposal.
0.88
2. Learner is the owner of the problem.
0.79
3. Authenticity of the problem or task is evident.
0.93
4. Authenticity of the learning environment is evident.
0.77
5. Solving the problem drives the process.
0.91
6. Complexity of the problem or task reflects the reality of
the market.
0.93
7. Evaluation and analysis of how the problem was resolved
follows the methodology
0.79
8. Reflexion on the content learned and the learning
process is regularly and continuously made.
0.93
9. Collaborative and multidirectional learning takes place
continuously.
0.96
10. Continuous Assessment takes place.
0.98
Overall Average:
8.7
In accordance with the proposal of the PBL-Test, each
principle evaluated can receive a maximum value of 1.0, so the
total assessment can reach a maximum of 10 points. In [9], the
authors define a maturity scale for PBL: Level 0 - Insufficient
(overall average <7); Level 1 - Initial (<= 7 overall average
<8); Level 2 - Satisfactory (<= 8 overall average> 9); Level 3 -
Good (<= 9 overall average> 10); Level 4 - Excellent (overall
average = 10).
From the results of Table V, we found that the PBL
maturity reached level 2 in the overall average, representing the
satisfactory level. Note that Principles 3, 5, 6, 8, 9 and 10 were
very close to the maximum value. It is noteworthy that with
regard to the Principles of lower values, little could be
exploited by the teacher and tutors because these Principles
related to the students’ profiles and the limitations of the
academic environment. As adopted action, we draw special
attention to face-to-face feedback given by the teacher and
technical tutors at the end of each monitoring meeting. The
objective was to explore the reflection as to the solutions that
the teams had adopted.
C. Final Assessment of the course
Finally, at the end of the course, a final assessment with the
class was conducted in order to evaluate two aspects: the
teacher’s performance and the planning of the discipline. We
adopted the same value scale as that for the perspectives of
Process and Results. In this evaluation, 17 responses were
received, representing a little more than 50% of the class.
Regarding the aspect of the Course itself, the following
criteria and respective results were considered: Approach of the
topic (4.2); Bibliography provided (3.8); Presentation of the
Objectives (4.3); Achievement of Objectives (4.1);
Contribution to the curriculum (4.2); Contribution to the
training of the student (4.5); Development of critical capacity
(4.3); General evaluation of the course (4.5). As to the aspect
of Teacher, the following criteria and results were obtained:
Presentation of ideas (4.3); Coherence of content (4.4);
Mastery of Content (4.6); Relationship between theory and
practice (4.4); Incentive to students (4.4); Method of
Evaluation (4.0); Empathy and ethics (4.7); Teaching
methodology (4.4). As a strength, the practical approach of the
course which focused on real problems stood out. As points for
improvement, the following matters were raised: evaluation
centered only on content in the 1st module; extensive content
and lack of revision sessions before the tests on content. Both
in the evaluation of the course and of the teacher, the results of
this evaluation showed that the students considered the course
worthwhile, most results being between "Very Good" and
"Excellent".
V. CONCLUSIONS
The NEXT group has been working with PBL in teaching
software engineering for more than seven years. Given the
many challenges of the first PBL applications in a professional
master’s degree of the area [2], the group’s constant concern
was to systematize the approach in methodological elements
that improve how they are understood and adopted, while being
supported by management processes and techniques.
Throughout this period, and after the processes had undergone
several cycles of evolution, xPBL was applied in a complete
PBL cycle (planning, implementation, evaluation and
improvement) in the context of this article, unlike the case
study reported in [1], in which the xPBL is formally presented.
It is also pointed out that it is the evaluation procedures that
enabled us to prove our objectives were reached. Based on the
five perspectives of authentic assessment, the performance of
the students could be evaluated in different aspects in the PGP
course. The evaluations of Content enabled the concepts and
fundamentals which the students had greatest difficulties with
to be identified. The Perspectives of Process, Result and
Customer Satisfaction allowed the teams to be analyzed,
thereby identifying the groups with greatest maturity in the
process of planning and project management. The evaluation of
performance allowed an individual look under interpersonal
skills that, in general, are developed in PBL-based teaching
approaches and, to some extent, led to collaboration between
members of the same team being perceived. The evaluation of
the PBL-Test allowed us to check throughout if we were really
heading in the right direction. Finally, the final evaluation
showed the students were satisfied with this discipline.
REFERENCES
[1] Santos S. C., Furtado F., Lins W. “xPBL: a Methodology for Managing
PBL when Teaching Computing”, FIE, Madrid, Spain, 2014.
[2] Santos, S. C.; Batista, M. C.; Cavalcanti, A. P. C.; Albuquerque, J. ;
Meira, S. R. L (2009). Appling PBL in Software Engineering Education.
CSEET, Hyderabad, Índia. v. 1. p. 182-189.
[3] S. C. Santos; F. S. F. Soares. “Authentic Assessment in Software
Engineering Education Based on PBL Principles: A Case Study in the
Telecom Market”. ICSE 2013, California, EUA, 2013.
[4] S. C. Santos and A. Pinto, “Assessing PBL with Software Factory and
Agile Processes”, CATE, Naples, Italy, 2012.
[5] Figuêredo C. O., Santos S. C., Alexandre, G. H. S. and Borba P. H. M.
(2011), “Using PBL to develop Software Test Engineering”, CATE,
Cambridge, UK.
[6] Alessio H. (2004). “Student Perception about Performance in Problem
Based Learning”, Journal of Scholarship of Teaching and Learning, Vol.
4, N. 1, pp. 25 – 36.
[7] M. Laguna. “Business Process Modeling, Simulation and Design”,
Second Edition – April 25, 2013.
[8] Walton, M. Método Deming na Prática (Deming Method in Practice).
Rio de Janeiro: Campus, 1992.
[9] Santos S. C., Figuerêdo, C. O., Wanderley, F. “PBL-Test: a Model to
Evaluate the Maturity of Teaching Processes in a PBL Approach”, FIE,
Oklahoma, EUA, 2013.
[10] Barrows, H. S. & Tamblyn, R. M (1980). Problem Based Learning: An
Approach to Medical Education. Interdisciplinary Journal of Problem
Based Learning. New York: Springer.
[11] D. Anderson, “Kanban – Succesful Evolutionary Change for Your
Technology Business”. Blue Hole Press, Washington, 2010.
[12] J. P. Womak and D.Jones. “Lean Thinking: Banish Waste and Create
Wealth in Your Corporation – June 10, 2003.
[13] I. B. Myers, Gifts Differing: “Understanding Personality Type”.
Mountain View, CA: Davies-Black Publishing, 1980.
[14] R. Delisle. How to use problem-based learning in the classroom, 1997.
[15] N. Brede, and T. Dingsoyr, "Scrum and Team Effectiveness: Theory and
Practice", Lecture Notes in Business Information Processing, 2008,
Volume 9, Part 1, p. 11-20.
