Content uploaded by Suharjito Suharjito
Author content
All content in this area was uploaded by Suharjito Suharjito on Dec 02, 2017
Content may be subject to copyright.
Software Development Evaluation Process Using
CMMI-Dev on Limited Resources Company
I Made Sugi Ardana, Suharjito
Computer Science Department,
BINUS Graduate Program – Master of Computer Science, Bina Nusantara University,
Jakarta, Indonesia
sugiardana@gmail.com, suharjito@binus.edu
Abstract—The quality of products is very important for a
software development industry. The product quality is strongly
influenced by the software development process. Software
development company with limited resources is hard to improve
the quality of the software produced. The method to be taken is
to improve the software development process to achieve maturity
level 2. In this study, CMMI-Dev (CMMI for Development)
version 1.3 with continuous representation is used as a reference.
SCAMPI-C with PST Tool was used to conduct the assessment of
the 5 (five) process areas of CMMI level 2. Data were collected by
interviews and observations of project documentation and the
process of software development. Unapplied practice from
assesment then be analyzed using Ishikawa Analysis to find root
of the problem refers to the SPF (Software Process Framework)
and subsequently use Pareto Analysis to determine the
improvements priority. The process which have already good are
in the process area PP (Project Planning), CM (Configuration
Management), and most of the processes in the area of PMC
(Project Monitoring and Control). Software development
processes which are need to be improved in the process area with
priority PPQA (Process and Product Quality Assurance), REQM
(Requirements Management), then PMC (Project Monitoring
and Control). The prior improvement from the SPF category
were in the order of procedure category and then process
category.
Keywords—Software Process Framework; CMMI-Dev;
SCAMPI; Ishikawa Analysis; Pareto Analysis
I. INTRODUCTION
As the time goes by, the software development will
continue to evolve as the complexity of feature additions,
change business processes, error recovery, and application of
new technologies. Thus, the high quality of the software
become a vital need [1]. To get a high-quality software, a good
process quality of making software is required. Software
Engineering Institute (SEI) said that the quality of a system or
product determined by the quality of the process and the
maintenance [2]. The most important thing that affect the
success or failure of software development is the process of
making the software [3].
Software development industry that has limited resources
continuously strives to improve the quality and delivery time of
software being made. This is done by improving software
development processes. But apparently there are still some
projects that are not in accordance with the expectation of the
company and the client. For example there is a delay of
implementation and there are many changes in the requirement
without being followed by increased prices resulting in
increased production costs and reduce the company's revenue.
There are some guidelines to help software developer results in
a process in accordance with needs. CMMI (Capability
Maturity Model Integration) is one of the guides that focus on
the software development process.
To survive in the competition of making software, software
development company creates software development strategies
by optimizing processes so that software development is not
too dependent on people, but rather on processes that have
been standardized. If related to the maturity level of CMMI,
then the level that wants to be achieved is a level 2 (Managed)
where at this level has been ascertained that the process is
planned and executed according to the policy (policy), the
project team consists of people who are competent so that the
results can be measured, involving relevant stakeholders, the
project is monitored, controlled, reviewed, and evaluated, as
well as job status can be determined at any time [2, 4].
In this study, CMMI will be used to evaluate the software
development process in limited resources company. From the
results of the evaluation can be suggested any
recommendations regarding the process that needs to be
enhanced and which processes are already good and should be
maintained.
II. RELATED WORKS
A. Software Development
Software development is a series of activities aimed to
produce a software product that suits the requirement. There
are four basic activities that are common to the development of
software, namely, software specification, software
development, software validation and software evolution [5].
For different type of systems require different development
processes. Increasing the hardware performance resulted the
need for software development to be increasingly large and
complex.
By using an informal approach to face an increasingly
complex development, the development of software gets
increasingly ineffective. This resulted in greater costs, more
time, as well as lower quality. To overcome the ineffectiveness,
of that sparked an idea to do software development with
engineering approach, resulting a more efficient, standardized
and scalable software development.
B. SDLC (System Development Life Cycle)
SDLC is a cyclical process of system development. The
entire process of system development is done through many
steps from the beginning until the formation of the product
through the analysis, design, implementation and maintenance.
Along with the development of software engineering, many
models have been developed to assist the software
development process, such as the waterfall, spiral, prototyping,
XP, and RAD. Although many models will evolve, but it
generally follows the basic stages of the SDLC in waterfall
development model, which consists of the analysis and
definition of requirements, system and software design,
implementation and unit testing, integration and system testing,
operation and maintenance [5] as in Fig. 1.
Fig. 1. Basic Stages of SDLC.
C. Software Development Process Framework
To obtain good results when developing software, the SEI
issued an operational framework that refers to the Capability
Maturity Model, called SPF (Software Process Framework).
The SPF stated that the framework forms the link between the
elements of the operation, namely policy, standard, process,
procedure, training, tools [6]. Framework that is formed will
be used to define and design the information, where it can be
used to help establish the necessary documents. Policy is used
as a rule directing, guiding, or limit the operation. Standard is
to provide operational definitions or criteria for acceptance of
the product or process. Process is to describe what is done in
the operation in accordance with the policy and standards.
Procedure describing instruction steps in implementing a
process. Training is to give individuals a knowledge and skills,
including training for the organization's policies, standards,
processes, procedures, and tools. Tool is to provide the
required assistance in supporting policy, standards, processes,
procedures, and training. The relation of these elements, in
establishing an operational framework is shown in Fig. 2.
.
Fig. 2. Operational Framework.
D. CMMI (Capability Maturity Model Integration)
CMMI published by the SEI (Software Engineering
Institute) of Carnegie Mellon University, is a model approach
in the assessment of the level of maturity and ability of an
organization [2]. CMMI is based on three concepts: Process
Area (PA), goal, and practice. There are two types of goal,
which are Specific Goals (SG) and the Generic Goal (GG).
There are two types of practice, which are Specific Practices
(SP) and the Generic Practice (GP). Specific Goals are a series
of activities or goals of the process area. Specific Practice is the
steps that are used to achieve goals [2]. Achievements and
fulfillment of the SG on the PA is very influential in the
maturity level achieved by the organization. Each PA will
consist of several SG and SG will consist of several SP [2].
E. CMMI-Dev (CMMI for Development)
CMMI-Dev is a model framework issued by the SEI
(Software Engineering Institute) of Carnegie Mellon University
in late 2001 and published in August 2006 to replace a similar
concept that is CMM that has been used for the assessment
process since the 1990s. CMMI-Dev was made to avoid the use
of various CMM models separately. Therefore, the model that
has been integrated in the CMMI for Development is the
codification of CMM models [2]. CMMI-Dev consists of 22
Process Areas which are: CAR (Causal Analysis and
Resolution), CM (Configuration Management), DAR (Decision
Analysis and Resolution), IPM (Integrated Project
Management), MA (Measurement and Analysis), OPD
(Organizational Process Definition), OPM (Organizational
Performance Management), OPF (Organizational Process
Focus), OPP (Organizational Process Performance), OT
(Organizational Training), PI (Product Integration), PMC
(Project Monitoring and Control), PP (Project Planning),
PPQA (Process and Product Quality Assurance), QPM
(Quantitative Project Management), RD (Requirements
Development), REQM (Requirements Management), RSKM
(Risk Management), SAM (Supplier Agreement Management),
TS (Technical Solution), VAL (Validation), VER
(Verification) [2].
There are four categories of process area which are: Process
Management, Project Management, Engineering, and Support.
The relationship between process areas, categories, and
maturity level can be seen in Table I [2]. In the organizations
with maturity level 2, the job status can be known by
management at certain points, for example, on major milestone
(major milestones) and on completion of a major task (major
tasks) [2].
TABLE I. PROCESSS AREA, CATEGORY, AND MATURITY LEVEL
Process
Area
Category
Maturity
Level
CAR
Support
5
CM
Support
2
DAR
Support
3
IPM
Project Management
3
MA
Support
2
OPD
Process Management
3
OPF
Process Management
3
OPM
Process Management
3
OPP
Process Management
5
OT
Process Management
4
PI
Process Management
3
PMC
Engineering
3
PP
Project Management
2
PPQA
Project Management
2
QPM
Support
2
RD
Project Management
4
REQM
Engineering
3
RSKM
Project Management
2
SAM
Project Management
3
TS
Project Management
2
VAL
Engineering
3
VER
Engineering
3
F. SCAMPI
SCAMPI stands for Standard CMMI Appraisal Method for
Process Improvement. SCAMPI is composed of three classes
namely class A, class B, and class C, which is distinguished by
the level of accuracy and the effort produced. SCAMPI-A is
the most rigorous method and the only method that can
generate value (score) and judgement necessary to lead a
certified appraisal. Usually costs incurred for scampi A pretty
big. SCAMPI-B is a method that is less formal than the
SCAMPI-A, the activities are fewer than SCAMPI-A. By
applying the SCAMPI-B, organization can predict the results
obtained when doing SCAMPI-A. This method does not need a
certified appraisal lead. SCAMPI-C is shorter, flexible, and less
expensive with SCAMPI SCAMPI-A and-B. Decision from
SCAMPI-C is usually used to measure the readiness of the
organization before applying CMMI. By doing SCAMPI
appraisal-C, it can easily get a gap of a process that has been
carried out by the organization in comparison with the best
practices of the CMMI. Scope of SCAMPI-C can also be
customized with the objective appraisal [7, 8, 9].
For SCAMPI class C, the need to do the appraisal reduces
the amount of effort and cost, but it reduces the accuracy and
reliability as well but the accuracy and reliability are also
lower. SCAMPI-C even allows reducing the number of
projects that will be investigated to zero and just do the
standard documented appraisal process, not the implementation
of the project [7]. The comparison between the three SCAMPI
classes can be seen in Table II.
TABLE II. COMPARISON OF SCAMPI CLASS
Class A
Class B
Class C
Evaluation team size
8 – 10
3 – 4
1 – 2
Evaluation time
10 days
3-4 days
1-2 days
Min data source
3
2
1
Reliability and success
High
Medium
Low
Effort and cost needed
High
Medium
Low
Direct interview
Yes
Yes
No
SCAMPI
A
B
C
III. METHODOLOGY
A. Research Object
Based on the comparison table class of SCAMPI, for
SCAMPI C, the minimum number of data sources is 1 (one)
[1]. In this study the data source will be taken from two (2)
software development project that has been completed.
To select the projects that will be evaluated, which
represents a typical software development project of the
enterprise, interviews will be conducted the Director of the
company.
B. Framework of Thinking
Based on a literature review, in order to increase the
success of the project development, we should improve the
process used. Increasing development process can refer to the
best practices of CMMI-Dev.
Operations of the software development refers to the
software development process framework. Framework is
already adapted to the implementation of the process areas of
the CMMI. To determine which parts need to be improved, we
need a method to measure it.
To perform these measurements, will be using SCAMPI
method. In this research, the measurement is only to determine
which parts need to be improved and not to measure the score
of maturity level. As explained respectively, then SCAMPI
class C to be choosen [7].
Based on the results of these measurements, it determined
the weaknesses and what improvements need to be done in
accordance with the reference CMMI-Dev and SPF. Pareto
analysis then will be use to make the priority of improvements
[10].
C. Research Stage
Stages of research in Fig. 3, started by defining the issues
that will be examined along with the boundary and scope. Then
determine the objective and benefits of the research. After that
doing study literature and taking case to be investigated.
Data collection is done after preparing the data collection
instruments. With reference to the CMMI-Dev level 2, there
are 7 Process Areas had to be evaluated. But in this study only
5 Process Area from collected data will be examined, which
relates directly to the software development process, namely
the PP (Project Planning) which consists of a total of 3 Specific
Goal and 14 Specific Practice, PMC (Project Monitoring and
Start
Research Problem
Research Objective
Study Literature
Setup Data
Collection
Instrument
CMMI Assessment
Data Collection
Assesment
Result
Assesment
Result
Ishikawa Analysis
Pareto Analysis Assesment
Result
Establish
Recommendation
End
Control) , consists of a total of 2 Specific Goal and 10 Specific
Practice, REQM (Requirement Management), consists of a
total of 1 Specific Goal and 5 Specific Practice, CM
(Configuration Management), consisting of a total of 3 Specific
Goal and 7 Specific Practice, PPQA (Process and Product
Quality Assurance), consists of a total of 24 Specific Specific
Goal and Practice.
After sufficient data has available, then conduct CMMI
assessment by performing data processing of the results of
these observations by using PST Tool. From this assessment
will be obtained CMMI process that appropriate and the
process that needs to be improved. For the processes that need
to be improved, will be analyzed using Ishikawa diagrams and
refer to the SPF to find the root of the problem. To determine
the priority of process improvements, will be used Pareto
diagram, with reference to the SPF. Finally, establish
recommendations based on the results of data processing as a
whole.
Fig. 3. Research Stage.
D. Data Collection Method
Data collection methods in this study consist of interviews,
observation of the software development process, and analysis
of the documents associated with software development
process at software development company with limited
resources.
E. Data Processing Method
Data obtained from interviews, observation and document
analysis will be used as a reference to perform data analysis.
The analysis will be done based on the procedures used to
carry out an appraisal of the process area by using the
SCAMPI-C method. By doing the assessment will be obtained
which practice is in a good state and practice that need to be
improved. For practice that needs to be improved will look for
the root of the problem by using the Ishikawa Analysis.
Furthermore, for the prior of improvements will be analyzed
using Pareto analysis. Finally will be made recommendations
for improvement.
IV. RESULT AND DISCUSSION
A. Data Collection
Interviews were conducted for the selection of projects that
will be evaluated. Based on the results of interviews regarding
the selection of projects to be evaluated, then taken software
development projects for a group insurance in the insurance
company ABC and insurance company XYZ. The data being
collected in the form of project documents and observations
results on the process of software development on these
projects.
1) Profile of Software Development Company
Software development company being used in this study
specializes in making software for insurance company. Initially
the software development process is highly dependent on the
employee so that when there are employees who resigned, the
company desperate. Likewise, the status of the project, known
only by employees who manage the project. Learning from this
experience, the company started making policies to planning,
reporting and monitoring of the project. Thus, the development
of a project can be seen at regular intervals, including when
there are issues or risks that must be escalated to management
level.
The management of the company hope the future of
software development is not too dependent on people, but
rather on processes that have been standardized. By referencing
to the maturity level CMMI, then the level that want to be
achieved is level 2 (Managed). At this level has been
ascertained that the process is planned and executed according
to the policy, the project team consists of people who are
competent so that results can be measured, involving relevant
stakeholders, the project is monitored, controlled, reviewed,
and evaluated, as well as job status can be determined at any
time.
Currently, an employee could have a different role
according to the phases of the project and its involvement in
the project. After the software project closing, an employee is
still engaged to support, because there are no support division,
which is a different team with software developers team. An
employee who became a systems analyst in a project could also
be a programmer in another project. Likewise, a system analyst
in a phase of defining requirements, then will be a programmer
when development phase and to provide guidance to the user
during implementation phase. This resulted in limited resources
working on a project.
2) Software Development Project for Insurance Group in
PT. ABC
PT. ABC previously had experienced failure in
implementing insurance software group from the previous
vendor. This causes a very high in overseeing the work of
vendors. Software being created integrating the 12 departments
that conduct insurance group operations.
Often each department which is the owner of each module
have diametrically opposed interests. Sometimes the
requirements that have been approved by a department could
be changed when discussion on other related modules from
other departments. Replacement of user in each department as
well be the next obstacle. If one user resigned and his
replacement had worked at another insurance company before,
usually brings flow and business processes from the previous
company.
3) Software Development Project for Insurance Group in
PT. XYZ
PT. XYZ also had a bad experience with a previous vendor,
where the previous software can not accommodate the needs of
the business of PT. XYZ. Besides, due to vendor’s dispersion,
a lack of adequate support arises in the event of a software
problem. Conflict of interests between departments are also
faced by the software development project in PT. XYZ
although not as hard as in PT. ABC, as previously it has been
using desktop software with integrated database. At the time of
software development at PT. XYZ, the company had an
integrated web-based software, so it does not create from the
scratch. Defining the software requirement has also become a
bit easier. It done by gap analysis between existing software
and user needs.
B. Results of Process Area Assessment using SCAMPI-C
To assess the process area using PST Tool from Dr.
Kneuper, artefacts on selected projects is used to replenish the
tools. In this study, the artefacts is available in the form of
project documents. For process area with its practice that no
documents as object evidence, the practice has not been
successfully applied to the project and will be found with
yellow background. If it is not yet implemented, will be found
with red background. All finding will be counted for each
process area. Number of finding for each process area can be
seen in Table III. The most finding is in the PPQA (11
findings) and the least finding in the CM (2 findings).
TABLE III. FINDING ON EACH PROCESS AREA
Process Area
Number of
Findings
Process and Product Quality Assurance (PPQA)
11
Requirement Management (REQM)
6
Project Monitoring and Control(PMC)
3
Project Planning (PP)
3
Configuration Management (CM)
2
C. Results of Ishikawa Analysis
Using Ishikawa Analysis for each finding that was
associated with the SPF, each finding is analyzed whether in
the category of SPF Procedures, Processes, Policies, or
Standards. All finding will be counted for each category.
The number of finding for each category are summarized in
Table IV. The most findings is in the procedures category (10
findings) and the least finding in the standards category (1
finding).
TABLE IV. FINDING ON EACH SPF CATEGORY
SPF Category
Number of Findings
Procedures
10
Processes
8
Policies
6
Standards
1
D. Results of Pareto Analysis
Based on the analysis Ishikawa for each category of SPF,
the number of finding for each category can be summarized to
be used for pareto analysis based on SPF category. The finding
is sorted from the most to the least finding, then each category
is calculated of count, cumulative and cumulative percentage
finding and can be seen as in Table V.
TABLE V. PARETO ANALYSIS BY CATEGORY SPF
SPF Category
Number of
Findings
Cumulative
Cumulative %
Procedures
10
10
40
Proocesses
8
18
72
Policies
6
24
96
Standards
1
25
100
Based on data in Table V then can be made Pareto diagram
as shown in Fig. 4.
Fig. 4. Pareto Diagram By SPF Category.
Based on the pareto diagram in Fig. 4, the priority of
process improvement by SPF category is procedures category
and then followed by process category.
Based on the the Ishikawa analysis for each process area,
the number of finding for each process area can be summarized
to be used for pareto analysis based on process area. The
finding is sorted from the most to the least finding, then each
process area is calculated of count, cumulative and cumulative
percentage finding and can be seen as in Table VI.
TABLE VI. PARETO ANALYSIS BY PROCESS AREA
Process Area
Number
of
Findings
Cumulative
Cumulative
%
Process and Product
Quality Assurance
(PPQA)
11
11
44
Requirement
Management (REQM)
6
17
68
Project Monitoring and
Control(PMC)
3
20
80
Project Planning (PP)
3
23
92
Configuration
Management (CM)
2
25
100
Based on data in Table VI then can be made pareto diagram
as shown in Fig. 5.
Fig. 5. Pareto Diagram By Process Area.
Based on the Pareto diagram in Fig. 4, the priority of
process improvement by process area sequentially are PPQA
(Process and Product Quality Assurance), REQM
(Requirement Management), and PMC (Project Monitoring
and Control).
E. Reccomendation
By combining pareto analysis by SPF category and pareto
analysis by process area, can be establish recommendation on
process improvements that can be done in the company.
The first recommendation is to establish procedures of
recording module testing, procedures of testing results signoff,
and procedure of testing planning. Then conduct training on
how to perform QA, using best practice of experience
accomplished. After that establish procedure of defining
software requirement, procedure of defining software
requirements planning, and procedure of definition software
requirements evaluation.
Next recommendation is to conduct training on software
requirements definition, bi-direction software requirements
tracking. The last recomendation is establish procedures of
project monitoring and controlling then conduct training on
project monitoring and controlling.
Conclusion
After evaluation of the software development process at the
company with limited resources using SCAMPI-C based on
CMMI-Dev version 1.3 with level 2 and analyzed with
ishikawa analysis and pareto analysis, it can be concluded that
some software development process already good and some
need to be improved. Software development processes which
already good were process contained in the area of PP (Project
Planning), CM (Configuration Management), and PMC
(Project Monitoring and Control).
Software development process which need to be improved
were processes contained in the area of Process and PPQA
(Product Quality Assurance), REQM (Requirements
Management), and a small portion in the area of the PMC
(Project Monitoring and Control).
Priority of software development process improvement is
recommended using the following order. Start with establish
procedures of recording module testing, procedures of testing
results sign off, procedure of testing planning. Then conduct
training on how to perform QA, using best practice of
experience accomplished. Then establish procedure of defining
software requirement, procedure of defining software
requirements planning, procedure of definition software
requirements evaluation. Then conduct training on software
requirements definition, bi-direction software requirements
tracking. The last is establish procedures of project monitoring
and controlling and conduct training on project monitoring and
controlling.
REFERENCES
[1] W. Widodo, "Evaluasi Proses Pengembangan Perangkat Lunak pada
Virtual Team Development Menggunakan CMMI Versi 1.3," Jurnal
Informatika, vol. 10, no. 1, pp. 1140-1148, Jan 2016.
[2] CMMI Product Team, "CMMI for Development Version 1.3," Carnegie
Mellon University, Hanscom, 2010.
[3] J.-C. Liou, "On Improving CMMI in an Immature World of Software
Development," Journal of Information Science and Engineering, 2011.
[4] Kautsarina, "Penilaian Tingkat Kematangan Tiga Proses Area Level 2
CMMI Versi 1.2 pada Small Independent Software Vendor di Indonesia
(Studi Kasus: Inovasia)," Widyariset, vol. 14, no. 3, pp. 665-674, 2011.
[5] I. Sommerville, Software Engineering Ninth Edition, Boston: Addison-
Wesley, 2011.
[6] T.G. Oslon et.al, "A Software Process Framework for the SEI Capability
Maturity Model: Repeatable Level," Software Engineering Institute,
Pittsburgh, 1993.
[7] R. Kneuper, CMMI Inproving Software and System Developmen
Processing Using Capability Maturity Model Integration (CMMI-Dev).,
Santa Barbara: Rockynook, 2008.
[8] W. Hayes, G. Miluk, L. Ming and M. Glover, Handbook for Conducting
Standard CMMI Appraisal Method for Process Improvement (SCAMPI)
B and C Appraisals, Version 1.1, Hanscom: Carnegie Mellon University,
2015.
[9] "PIID and SCAMPI Tool (PST)," 2016. [Online]. Available:
http://www.kneuper.de/English/PIID-SCAMPI-Tool/. [Accessed 12
May 2016]
[10] D. Haughey, "Pareto Analysis Step by Step," 2016. [Online]. Available:
https://www.projectsmart.co.uk/pdf/pareto-analysis-step-by-step.pdf.
[Accessed 01 August 2016].
dsdsd
