Code camps and hackathons in education - literature review and lessons learned

Abstract

Motivation: Code camps and hackathons been used in education for almost two decades. These approaches are usually intensive and for most times quite practical events for solving some real-world problems with various educational objectives. The objectives and structures of these events differ depending on the role of the event in curricula. Problem statement: Both code camps and hackathons been implemented in various ways, with varying success levels. As expected the implementation of the event varies considerably depending on the objectives set for the event, but that then leads to the difficulty and problem setting to understand what organizing of these events actually mean. For educational context, curricula have also its role in defining the targeted skills and competencies the events has to consider too. Approach: We applied a systematic literature review (SLR) to look at the various definitions and modes of these events. Whether it is called "code camp", or "hackathon", or anything else with the same basic meaning, we want to find out what skills and competencies these events emphasize, how they are used in Computer Science (CS) and Software Engineering (SE) education and what are the general structures of the actual arranged events. Contribution: It is aim of this SLR to i) identify various possible ways of implementing these intensive events, and ii) reflect the results to the lessons we have learned of almost two decades of various intensive code camps and hackathons we have been organizing building and participating into. Based on the results, we claim that there is tremendous potential of using these events in education and in the curriculum than how it has been applied so far.

Full text

Code camps and hackathons in
education -literature review and
lessons learned
Jari Porras, Antti Knutas, Jouni Ikonen, Ari
Happonen, Jayden Khakurel, Antti Herala
Lappeenranta University of Technology, Finland

Outline
•Motivation
•Background
•This research and process
•Articles
•Collected data
•Results
•Discussion

Motivation
•Evolution of the needed knowledge, skills and
competencies of CS/SE graduates
•Content and teaching methods evolve
•Capstones have been used for teaching/testing the skills
needed in real work life
•Code camps and hackathons have emerged to challenge/
extend the capstones
•Not been discussed so much (though all have heard of Hacks)

Background (for this paper)
•Code camps (and hackathons) in LUT since 2003
•Hackathons in software engineering education – lessons
learned from a decade of events – ICSE SEEM 2018
•Based on own experiences
•Questions, Feedback, Critics about the literature and
understanding of what hackathons mean
=> Literature based study on code camps and hackathons

This research
•The main aim of this research is to find out what has been
published on code camps or hackathons.
•RQ1: How are the code camps and hackathons defined in the
literature?
•RQ2: What kind of educational structures have been used in code
camps and hackathons?
•RQ3: What skills and competencies have been emphasized in the
code camps and hackathons presented in literature?
•RQ4: How code camps and hackathons have been used in SE and CS
education?
•A systematic literature review (SLR) [Kitchenham B.] is used to
collect data and analyse the literature.

General process for literature reviews
a) Define the research questions based on the objectives of the research.
b) Define search queries based on the research questions. Finding proper search queries
(terms) might require an iterative process. Tools like NAILS and HAMMER can be used for
the first iterations.
c) Search articles on primary studies using search strings on scientific libraries and
databases.
d) Screen the initial set of articles by applying inclusion and exclusion criteria to determine
whether each potential article should be included or excluded from this study. Inclusion
and exclusion happen in multiple stages, starting from the screening of titles and abstracts
and ending to the analysis of the whole document. Secondary articles can be added by
manually browsing cited articles in the selected set of primary articles.
e) Extract the predefined set of data from the selected set of articles.
f) Analyse the extracted data to answer the research questions. Various tools exist for the
analysis, such as HAMMER , KHCoder or VOSviewer .
g) Present the acquired results.

Network Analysis Interface for Literature
Studies (NAILS) hammer.nailsproject.net
Knutas, A. et al.: Cloud-based
Bibliometric Analysis Service for
Systematic Mapping Studies.
Proceedings of the 16th International
Conference on Computer Systems and
Technologies, (2015).

Our process
•RQ1: How are the code camps and hackathons defined in the literature?
•RQ2: What kind of educational structures have been used in code camps and
hackathons?
•RQ3: What skills and competencies have been emphasized in the code camps
and hackathons presented in literature?
•RQ4: How code camps and hackathons have been used in SE and CS
education?
•IC: Published by May 2018
•EC: Articles not in English
•EC: Articles that are not fully available
•EC: Not relevant to the research questions
•Search for Web of Science database

Search queries and collected data
Search queries
Found
papers
Without
replicates
Included papers in the
final dataset
“Hackathon” OR
“Code camp” and
“Definition”
115 94
33 (EC2 - 10 papers,
EC3 - 51 papers)
“Hackathon” AND
“Education”
21 21
11 (EC1 - 1 paper, EC2 -
6 papers, EC3 - 3
papers)
“Code camp” 9 9 7 (2 of EC3))
Total number of
papers
145 124 51
•Basic information: ID, Author(s),
Year of Publication, Title,
Publication type (workshop,
conference, journal), Keywords,
Abstract;
•Specific information: Definition
(RQ1), Length of the event
(demographics), Participants
(demographics), Number of
participants (demographics),
Structure of the event (RQ2),
Targeted learning outcomes - skills
and competencies (RQ3), How the
event is tied to education (RQ4),
Event domain (demographics),
Other issues.

Some demographies
•Not too much activity in
this field
•Hackathon term has been
around since 2000
•Outside research or
education community ?
•Increase in activity from
2012

Defining code camps and hackathons
•Similar terms though hackathon used much more (44 vs. 7),
essentially the same meaning

Defining code camps and hackathons
•Code camp used more in CS / SE education (core technical skills) while
hackathon is used more widely in business context (general skills)
•Hackathon, which fosters the innovation potential of small focused teams while at
the same time not taxing the financial resources of the corporation
•Civic hackathon is a participatory event to prototype innovative services through
collaboration between citizens and engineers to address social issues
•Hackathons are alternative meeting formats emphasizing the full participation
of everyone
•Hackathon can be a breeding ground for brainstorming, innovation, networking,
and product development, and as such they can have multiple outcomes
including the sparking of new businesses and entrepreneurial activity
•Challenges, competitions, various stakeholders and domains,
engagement, …
•Datathons, semesterathons, summerathons , designathons

Evolution of hackathons
•KHCoder
•Focus
•NAILS (LDA)
•Clustering (4 categories)
•hackathon, participation,
team, and student
(education)
•data, open, user, and civic
•core, science, data,
sample, and network
•health, event, care,
hackathon, and medical

Hackathon structures
•Typically – pre-event,
event, post-event
•Innovation, tasks, team
building
•Themed events
•Innovation, collaborative
work, pitches
•(Very) Intensive
•How could these be used
in education ?
approach it represents (i.e. rather intensive way of
working) is well understood and applied in different
domains. Based on the literature the most common
domain (15/51 papers) for hackathons seems to be
somehow related to health (e.g. healthcare, medicine,
health monitoring and data mining related to
healthcare). Other common domains are related to
service development for local challenges (e.g. through
smart cities, IoT). The emphasis of hackathons is more
on either applying IT for other domains or completely
forgetting IT and staying in more contextual or
business model levels.
As a summary of these terms, we could conclude
that although both terms in general emphasize short
time interdisciplinary collaborative working it seems
that a) term code camp refers to more IT or software
oriented events, b) term hackathon has almost
completely replaced it although c) term hackathon does
not so clearly implicate the focus of the event.
3.2 Evolution of hackathons
In order to get an even better perception of the
evolution of hackathons based on the selected literature
the content of the hackathon related papers were
analyzed by KHCoder and Nails. KHCoder, among
other things, can reveal how the events were developed
during these years. Figure A.1 in Appendix shows the
yearly emphasis on hackathons based on the literature.
First years of the hackathons were focusing on
collaboration (2012), agile development (2013) and
software (2014), whereas latest years the have seen the
rise of data (2105-2016, 2018), healthcare as
application domain (2017-2018) and community type
of events (2016-2018).
Another perception to the same issues can be seen
from topic modeling. Figure A.2 in Appendix shows
the groups generated by LDA (general terms shown for
all topics). The topic modeling process found four
themes when guided by the semantic coherence
heuristic [26]. First theme was characterized by the
words hackathon, participation, team, and student.
Second theme was characterized by the words data,
open, user, and civic. Third theme was characterized
by the words core, science, data, sample, and network.
Fourth theme was characterized by the words health,
event, care, hackathon, and medical. First two themes
are close to each other but show clear distinction
between these groups of papers. Third and fourth
themes are further away from these first two and each
other. If one would be interested in education related
hackathons, one should look more closely to the papers
in the first theme of the model.
3.3 What kind of structures have been used
for code camps and hackathons
Code camps and hackathons have been proposed
for different lengths as presented in Table 2. Majority
of the presented events are two days (that seems to be
the de facto standard for these intensive events as 45%
of found publications refer to that) though shorter and
longer events have been proposed (22% for shorter and
around 20% for 3-5 days events). Extremely short (i.e.
few hours) or long (i.e. weeks) events have been
proposed rarely and it remains unclear what makes
these code camps or hackathons. In general, they can
follow the same general structure of these intensive
events but do they really emphasize the same learning
outcomes that code camps and hackathons represent?
How much do participants engage or interact in a 4h or
a semester-long hackathon?
Table 2. Lengths of the reported events.
The structure of the code camp and hackathon can
typically be divided in three parts; pre-event, event and
post-event [27]. Pre-event activities might consist of
some preparatory activities like reading of background
material or pre-lectures [28], idea generation [29] or
team building. However, quite often both idea
generation and team building are part of the actual
event, especially in a themed event in which all teams
innovate under the same theme or target setting. Idea
generation could happen individually or in small
groups that then are pitched to the others (other teams,
organizers, customers or other stakeholders) and after
that each person may select a team whose idea is
closest to own heart. The event itself contains at
minimum the collaborative working but often also
some themed presentations in the beginning and team
demonstrations in the end. Especially in competition-
based events the final demonstrations are evaluated
and used for deciding the “winners” (various winning
conditions or even categories can be used). Working
during the event may vary depending on the type of the
event. Many of the events emphasize interdisciplinary
collaborative working [30], [31], [32], [28]. In some
Length Number of papers
Share of
publications
24h or less
13 (out of which 4
less than 12h)
25%
Two days (48h) 23 45%
3-5 days 11 22%
More than 5 days 2 4%
Not mentioned 2 4%
Total 51 papers 100%

Skills and competencies
•Pre-defined wishes for the
outcomes
•Learning ?
•Emphasis on soft skills
•Not clearly described in
papers
•Still not seen as a viable
education approach ?
•”Team got experience from
the other team. Different
working style can be brought
from the other team if good
practices were found.”
events teams might work in completely different
spaces [33] that can strengthen intra-team
collaboration but decrease the inter-team collaboration
regardless of digital platforms (e.g. Wiki or Slack)
available. Of course, other programme can be used to
promote the inter-team collaboration. For example, and
interesting approach was proposed by [34] to use
breakout sessions to create discussions between teams.
As a summary of the structures used in code
camps and hackathons, one could emphasize three
elements that almost all papers emphasize;
collaborative idea generation, inter-disciplinary
working and intensive pitching. Collaborative idea
generation lets everybody to contribute to the work to
be done and it helps all the teams to achieve better end
results (by not wasting peoples time to “reinvent the
wheel”). Working should always happen in teams with
various talents and perceptions as that allows thinking
that is more heterogeneous and solutions. Various team
sizes (from 2 to close to 10) have been proposed and
depends heavily how multidisciplinary approach is
used. Finally yet importantly the active pitching of
ideas distribute them the best.
3.4 What skills and competencies have been
emphasized
The organizers of the hackathons defined some
sort of pre-defined wishes for the outcomes. These
outcomes can be divided into learning outcomes (for
more education-oriented events) and practical
outcomes (i.e. tangible results of working in the event).
Table 3 presents the most emphasized skills and
learning outcomes in the selected set of papers. The
four most emphasized learning outcomes for the
participants in these papers are a) teamwork or
collaboration skills, b) creativity or innovation skills,
and c) some context specific skill (e.g. Java
programming skills) and d) presentation (or
discussion) skills.
Table 3. Emphasized skills in the events.
Nevertheless only rarely these learning outcomes
are emphasized clearly in the papers [35], [36], [37]. It
seems that the focus is more on practical project
outcomes than in learning outcomes. Regardless of the
actual analysis of the outcomes, all papers of the
literature study have a very positive perception towards
hackathon. Even though, for many, the event has been
the first attempt to use hackathon type of an approach
and the results have been promising like presented by
the quotes from papers:
“...Plenty of collaboration was observed during the
events. Teams were actively encouraged to help each
other and as such no competitive behaviors were
observed during the events. As an example, the
following was observed: If a team had an issue with
PHP, for example, they might yell out "is anyone here
a PHP guru?" and someone from another team would
leave their project for a bit and help with the other
project...” [38].
”Team got experience from the other team. Different
working style can be brought from the other team if
good practices were found.” [36].
This is the same behavior we have noticed in the
events we have organized. Individuals help each other
across the team boundaries. On the other hand,
competitive type of event may not show this kind of
behavior. We have seen exceptions to this in our event
observations, especially when small startups do team
up to meet the event goals set by large enterprise.
As a summary of the analysed set of papers, one
could claim that most of the code camp events and
hackathons do not set clear educational learning
outcomes but use the hackathon more like a tool to
innovate new solutions in which the solution is more
important than the learning at the process of
developing the new solution. This is partly due to the
approach of using these events as an innovation
platform and not as a learning platform. One
explaining reason for this could be the perception from
literature, that the emphasis on hackathons is more to
the stakeholder collaboration side [32], [29], [39]. As
all papers point, the approach considered in these
events are appropriate for influencing and creating
innovations, there is definitely room to extend the
work towards the learning aspects. Producing good
innovations is not contradictory of learning at the same
time but the learning should be emphasized at the
event.
3.5 How code camps and hackathons have
been used in SE and CS education
Surprisingly few papers describe any kind of
relationship to education in general or even any special
Emphasized skill Number of papers
Share of
publications
Project, teamwork,
collaboration
22 43%
Presentation etc. 15 29%
Programmin, domain
knowledge
14 27%
Innovation, Creativity 12 24%
Business aspects 7 14%
Critical thinking,
problem solving
5 10%

How they have been used in education ?
•Surprisingly few papers describe any kind of relationship to
education in general or even any special course/module concepts.
Although many papers involve students of various levels (pre-
university, undergraduate, master level or even phd level it seems
that code camps and hackathons are just a new way of working
rather than an integral part of the education.
•Even though code camps and hackathons are widely used,
especially in connection with companies, their use as an integral
part of curriculum is immature or at least it has not been properly
documented in academic papers

Discussion & Conclusion
•(RQ1) “hackathon” is the term used mostly today but it does not
necessarily describe an educational event
•(RQ2) code camps and hackathons come in different forms and
lengths but it seems clear that both event types follow roughly the
same structures
•(RQ3) hackathons emphasize more of the innovation aspects and
as such the final solutions than actual learning (outcomes) that
would be of interest of educational events
•(RQ4) code camps and hackathons have not so much been
integrated into the curriculum