Property-based Testing for LG Home Appliances using Accelerated Software-in-the-Loop Simulation

Abstract

As LG home appliances promise more convenience features to end-users, the complexity of their control software is also increasing, creating a higher pressure for software verification. However, since the embedded software is tightly coupled with its hardware counterpart, the development and verification schedules are dependent upon hardware development and this hinders integration testing to be performed as thoroughly as it deserves. Furthermore, the manually-crafted test cases have had limitations, both in terms of the thoroughness of state-space exploration and the power of test oracles.

To overcome these problems and facilitate a more efficient software verification, we introduce a property-based testing framework using software-in-the-loop simulation (SILS). SILS allows the software to be integrated virtually and tested before the hardware is fully developed, and, further, it enables an acceleration in test executions of up to a few tens of thousand times. Property-based testing is achieved by translating the formalized properties to synchronous observers which can concurrently check the violation of the verification property during test executions. In the field application, we discovered two fault cases in real products under development using our framework. According to our analysis, these cases could not have been found using manual testing, but made possible by our testing framework. These cases could have cost the company tens of million dollars each, if they were not discovered until after sale.

Full text

Unpublished working draft.
Not for distribution.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
Property-based Testing for LG Home Appliances using
Accelerated Soware-in-the-Loop Simulation
Mingyu Park
LG Electronics
Changwon, South Korea
pqrk8805@gmail.com
Hoon Jang∗
Hyundai Motor Company
Hwasung, South Korea
fctermer@gmail.com
Taejoon Byun
University of Minnesota
Minneapolis, Minnesota, USA
taejoon@umn.edu
Yunja Choi
Kyungpook National University
Daegu, South Korea
yuchoi76@knu.ac.kr
ABSTRACT
As LG home appliances promise more convenience features to end-
users, the complexity of their control software is also increasing,
creating a higher pressure for software verication. However, since
the embedded software is tightly coupled with its hardware coun-
terpart, the development and verication schedules are dependent
upon hardware development and this hinders integration testing
to be performed as thoroughly as it deserves. Furthermore, the
manually-crafted test cases have had limitations, both in terms of
the thoroughness of state-space exploration and the power of test
oracles.
To overcome these problems and facilitate a more ecient soft-
ware verication, we introduce a property-based testing framework
using software-in-the-loop simulation (SILS). SILS allows the soft-
ware to be integrated virtually and tested before the hardware is
fully developed, and, further, it enables an acceleration in test execu-
tions of up to a few tens of thousand times. Property-based testing
is achieved by translating the formalized properties to synchronous
observers which can concurrently check the violation of the ver-
ication property during test executions. In the eld application,
we discovered two fault cases in real products under development
using our framework. According to our analysis, these cases could
not have been found using manual testing, but made possible by
our testing framework. These cases could have cost the company
tens of million dollars each, if they were not discovered until after
sale.
CCS CONCEPTS
•Computer systems organization →Embedded software
;
•
Software and its engineering →Software testing and debug-
ging;Extra-functional properties.
∗This research was conducted by the author while at LG Electronics
Unpublished working draft. Not for distribution.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
Conference’17, July 2017, Washington, DC, USA
©2020 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
KEYWORDS
Software-in-the-loop, embedded systems, test automation, integra-
tion testing, property checking
ACM Reference Format:
Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi. 2020. Property-
based Testing for LG Home Appliances using Accelerated Software-in-the-
Loop Simulation. In Proceedings of ACM Conference (Conference’17). ACM,
New York, NY, USA, 10 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1 INTRODUCTION
LG Home appliances are typical embedded systems which interact
with the physical world with their main logic controlled by software.
As they make use of hardware actuators, such as a heater and a
motor, a malfunction in the software can cause damage to both
the users’ lives and properties. What makes the situation worse
is that it is very dicult to change the software once the systems
are deployed, and, if they need to be changed, all the devices have
to be recalled, which comes with a tremendous cost. For example,
for one recall which was required in order to x a critical software
issue, LG technicians had to visit every consumer household one by
one to perform a manual software update. An incidence of such a
recall can cost from hundreds of thousand dollars to tens of million
dollars.
Although the high cost of post-deployment maintenance justies
more investment in the software verication process, the status
quo verication practice in LG Electronics (LGE), especially the
integration testing, had to be improved. One of the most critical
bottlenecks in integration testing was that the home appliance soft-
ware was developed and tested only after the prototype machine
was fully developed. This was because we were not equipped with
the domain-specic hardware simulation for home appliances. This
sequence—hardware, rst, then software—used to create an undesir-
able pressure to sacrice the thoroughness of software verication
because, when hardware development gets delayed, software veri-
cation tends to be rushed, as the mass-production schedule is xed
a priori and cannot be deferred without paying a tremendous cost.
The dependence of software verication on hardware creates
other limitations as well, especially in the comprehensiveness of
verication. Since it takes from an hour to several days for a single
test case to be executed on home appliances—such as washing
machines or refrigerators—there is a physical limit on the number
2020-01-09 10:37. Page 1 of 1–10.

Unpublished working draft.
Not for distribution.
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
Conference’17, July 2017, Washington, DC, USA Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
Figure 1: Product development process in LGE before and after applying SILS framework
of tests that can be executed in a given amount of time, typically a
few weeks. Even if a sucient time budget is assumed, there are
almost innite combinations of test inputs which can compose a
test scenario, making it impossible to do a comprehensive testing.
Also, the status quo integration testing relies on manually-crafted
test cases and this creates several drawbacks. First, it is expensive
to create test cases manually. Second, a test case encodes a verica-
tion property only as a pair of inputs and expected outputs, and,
therefore, each test case can check only one aspect of a property.
This output-based comparison is much less eective than having a
stronger oracle mechanism—such as assertions—which can check
the property for every test execution; it is less ecient as well.
We believe that these three issues regarding integration testing of
embedded system—software being dependent on hardware, slow
test execution, and limited oracle—are not unique to LGE and rather
quite prevalent in the embedded systems domain.
Figure 2: SILS-based integration testing framework
To resolve the above-mentioned problems, we developed a property-
based integration testing framework for embedded control software
based on the software-in-the-loop simulation (SILS) [
33
] technique.
First, we resolved the dependency of software development and
testing on prototype hardware by introducing SILS, so that the
software can be tested in the virtual environment. Second, we over-
came the slow test execution problem by using a simple foreground
ISR injection technique and achieved up to a 10,000x accelerated
testing. Third, we achieved property-based integration testing by
applying a well-studied idea of using synchronous observers [
18
]
for dynamic property verication; we formalized verication prop-
erties in linear temporal logic (LTL) and automatically synthesized
synchronous observers which can detect property violation during
test executions. Leveraging the accelerated simulation and more
powerful test oracles synthesized from LTL properties, we also ap-
plied combinatorial test generation to explore the state-space more
thoroughly. As a eld application, we present the result of verifying
33 safety and reliability properties which are derived from domain
knowledge and experience accumulated over decades.
The key achievements of this work are as follows:
(1)
We resolved the dependency of software on hardware devel-
opment by introducing SILS. Therefore, we could utilize the
time spent waiting for the prototype hardware, shortening
the software development schedule by 30%, and save 70% of
the cost on manufacturing prototype hardware—each device
costs tens of thousands of dollars.
(2)
We reduced the test execution time by 99.99% by using ac-
celerated SILS-based testing—a 300x speed-up for washing
machines and 10,000x for refrigerators.
(3)
We enhanced the thoroughness of testing by generating a
much higher number of test inputs—up to 280 times more—
automatically using a pair-wise method [
21
]. As a result, we
achieved up to 72% higher branch coverage.
(4)
We removed the human eort in integration testing by in-
troducing synchronous observers for checking outputs auto-
matically. With the observers, we found two serious defects
that are almost impossible to nd with manual test cases,
early on in the development phase. The types of the faults
we found—stage inversion in the washing machines and an
innite freezing loop in the refrigerators—could have cost
tens of million dollars each if the faults had to be xed after
sale through recalls.
2 PROJECT OVERVIEW
2.1 Overview of LGE Home Appliance software
LGE home appliances are typical embedded systems wherein the
software is control-intensive and reactive to the environment. An
embedded control software accepts user input and environmental
signals through peripherals, computes the next state, actuates the
output through physical plants, such as a motor, and repeats this
cycle during operation. The embedded software is tightly coupled
with the hardware and the surrounding physical environment such
that it is impossible to develop or test the software in isolation.
For example, the control software for an LG refrigerator drives
the compressor for cooling and gets a feedback signal in order to
conrm normal operation. The physical environment is aected as
the evaporator cools down the interior, and the thermometer sensor
delivers the signal back to the controller so that it can maintain the
desired temperature. Testing such software requires full integration
with the hardware and the environment or at least a simulation of
both.
2.2 Software development process in LGE
The status-quo software development process at LGE is depicted in
Figure 1-a. Once the system requirement is identied, the software
2020-01-09 10:37. Page 2 of 1–10.

Unpublished working draft.
Not for distribution.
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
Property-based Testing for LG Home Appliances using Accelerated Soware-in-the-Loop Simulation Conference’17, July 2017, Washington, DC, USA
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
and hardware control development teams decompose the system-
level requirement into software and hardware requirements. The
hardware control team develops the hardware based on the hard-
ware requirements and provides the prototype hardware to the
software development team. The software development team must
wait for a new hardware prototype to be given to it because, when
developed in isolation, software alone cannot be executed nor de-
bugged. When the software is fully developed, the quality assurance
team performs an acceptance test and determines whether the prod-
uct is ready to be mass-produced.
The software development process at LGE is similar to the pro-
cess suggested in the IEC 61508 standard [
7
]. The software is de-
signed in accordance with the requirements, developed according
to the design, and unit-tested by developers; a code review and a
static analysis are also performed at this stage. Once the develop-
ment and unit testing are complete, integration testing is performed
on prototype hardware systems to verify that the behavior of the
software meets the requirements. This phase can take up to two
months for refrigerators and two weeks for washing machines—a
single cycle in a refrigerator is much longer than that of a wash-
ing machine. When a defect is discovered, several more rounds of
testing are performed, typically from two to three times until a
decision is made for a mass-production.
As shown in Figure 1-a, the integration testing takes the largest
amount of time—and thus, also costs the most money—in the whole
software development process, and it is of utmost importance as it
is the nal verication step. However, the eciency of testing, and
the lack of thoroughness thereof, has to be improved.
3 ACCELERATED SOFTWARE-IN-THE-LOOP
SIMULATION
Figure 3: Structure of the software-in-the-loop simulator
Software-in-the-loop simulation (SILS) refers to testing an ac-
tual production software in simulation where hardware and the
physical environment are modeled mathematically [
33
]. One of
the biggest advantages of employing SILS is that it decouples the
software from the hardware, allowing the software to be developed
and tested in parallel. (see Figure 1-b). In our SILS framework, we
simulate the hardware—including sensors and actuators—and the
physical environment that the system concerns (Figure 3), so that
control software can be tested in a virtual environment with mini-
mal changes. Furthermore, we achieved a massive acceleration in
test execution up to 10,000 times the normal speed, using a simple
technique for abstracting interrupts. This abstraction technique
and the details of our SILS environment will be explained in the
following subsections.
3.1 Accelerated simulation through foreground
ISR injection
Most of the embedded systems are time-dependent as time is one
of the key features of the physical environment. As such, when
control software is to be tested on a target hardware, the amount
of time it takes for testing the software is equal to the amount of
time it is designed to operate for. This is a detrimental factor in
improving the eciency of testing, especially when a feature to
test is designed to operate over a long period of time.
en um ST AG E s ta ge = { WA SH , R IN SE , D EH YD , . .. }
T gl o ba l_ t im e , t im e _s t am p_ m ai n , t i me _ st a mp _ wa s h ;
IS R b a s e_ t i m er _ 5 m s () { gl o b a l_ t i me += 5 ; } // b as e t im e r
IS R w at e r_ t em p _1 0 ms ( ) { wa t er _ te m p = re ad ( W A TE R _T M P ); }
BO O L i s_ e la p se d ( T * si nc e , T e la ps e d , T U un it ) {
if ( gl o ba l_ ti m e >= * si nc e + ela ps ed * u ni t)
* si n ce = g l ob a l _t i me ; r e tu r n T RU E ;
re t ur n F AL S E ;
}
vo id ma i n_ ta s k () {
if ( i s _ e la p s ed ( & t i m e_ s t am p _ ma i n , 5, MS ) )
if ( s t a ge = = W AS H & & i s _e l a ps e d ( & t i me _ wa s h , 3 0 , M IN ) )
st a ge = R IN S E ;
.. .
}
vo id ma i n () {
wh i le ( T RU E ) { m ai n _t a sk ( ) ; } // t he s upe r lo op
}
Listing 1: A simplied code of the control software of a
laundry machine
vo id t ri gg er _ is rs ( ) {
st at i c it er = 0 ;
base_timer_5ms();
if ( i te r % 2 == 0 )
water_temp_10ms();
it er + = 1;
}
vo i d m ai n ( ) {
wh il e (1 ) {
ma i n _t a s k ( ) ;
trigger_isrs();
}}
Listing 2: SILS main
Figure 4: Interrupt mechanism
We worked to achieve accelerated testing through accelerated
SILS by resolving time-dependent components in the software un-
der test. In our home appliances, it turned out that all the time-
dependent features are implemented in interrupt service routines
(ISRs), mostly to execute short periodic tasks. Listing 1 shows a
simplied code of our typical control software for washing ma-
chines, with the super loop, a main task, and two ISRs. The two
ISRs—
base_timer_5ms
and
water_temp_10ms
—are triggered by
timer hardware in predened frequencies—5ms and 10ms, respec-
tively. The main logic written in
main_task
is executed only when
the
time
is elapsed by more than 5 milliseconds, while
time
is up-
dated by the base timer interrupt
base_timer_5ms
. Thus, the only
time-dependent aspect in this code is the two periodic interrupts,
and, once they are triggered from within the code, the program can
run independent of the physical time and as fast as the simulation
host machine can aord.
2020-01-09 10:37. Page 3 of 1–10.

Unpublished working draft.
Not for distribution.
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
Conference’17, July 2017, Washington, DC, USA Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
As a simple workaround for under-approximating the periodic
interrupts, we injected the triggers in the super loop and called
the ISRs just like procedures (
trigger_isrs
in Listing 2). We rst
identied the greatest common divisor of all the periodic interrupts—
5ms in this example—and synchronized the invocation of them with
the average period of the super loop [
23
]. We, then, injected the
trigger of the 5ms interrupt at the end of every iteration of the
super loop and the trigger of the 10ms interrupt in every second
iteration. As a consequence, each loop iteration takes 5ms of virtual
time in the simulation by triggering the base timer interrupt, and
all the other time-dependent tasks can be synchronized accordingly,
based on the global_time variable.
This foreground ISR injection is a very simple and straight-
forward method, yet its benet is tremendous. The overall acceler-
ations for our products were measured around 300 times to 10,000
times compared to normal executions. This under-approximation,
however, only preserves a representative and desirable behavior
while eliminating the cases in which preemptions occur. Other
more complicated approaches for modeling interrupts in the simu-
lation exist [
30
], but we opted for the foreground injection method
because checking concurrency problems is not of concern in our
SILS-based testing framework and also to minimize the run-time
overhead. LGE handles the concurrency problems separately; es-
pecially, the work by Park et al. [
29
] was adopted successfully to
verify the concurrency problems in our home appliance systems.
3.2 Hardware and environment simulation
The control software in the embedded system understands the
environment through input devices and aects the environment
through output devices. In the SILS simulation framework, we
modeled the behavior of the input devices, output devices, and the
portion of the environment that the system is concerned with so
that the control software can be tested as-is with minimal change.
This section briey explains how we modeled each component in
our SILS framework.
3.2.1 Input device model. Sensors are hardware devices that sense
environmental conditions and produce input data for the software.
For instance, a washing machine has a thermistor for measuring the
water temperature; in this case, the temperature is an environmental
condition and is read by a sensor through electrical resistance.
Thus, a sensor should be modeled to read the temperature from the
environment and produce an analogue signal of electrical resistance.
We used Simulink [26] for modeling such input devices.
3.2.2 Output device and environment model. Control software pro-
duces control signals to output devices such as actuators and LEDs.
These output devices may aect the environmental conditions ei-
ther directly or indirectly, forming a feed-back loop from output
devices to input devices. For output devices, we modeled both the
normal and abnormal operations, according to the hardware re-
quirement specications. For the environment, we modeled the
feedback loop—the mathematical relation between the environmen-
tal conditions that actuators aect and the conditions that input
devices read—according to the description of the environment in
the software requirement specications.
For example, a heater delivers heat to the water inside a washing
machine based on the command of the control software (see Fig-
ure 5). The rate at which the water temperature changes based on
the heater output is modeled in the simulated environment and pa-
rameterized by dependent variables, such as the room temperature
and the volume of the water.
Figure 5: The feedback loop between a heater and a water
temperature sensor
4 PROPERTY-BASED TESTING IN SILS
Figure 6: Property-based integration testing using SILS
In a typical testing workow, the property verication is del-
egated to manually-designed test cases which implicitly encode
properties of concern as expected outputs for given inputs. These
test cases, however, can only check a single aspect of a property per
a test case. For instance, one would wish to check if “the washing
machine enters dehydration stage only after the wash stage is com-
pleted”. A few concrete test cases can be created for checking this
property, but, since there can be numerous distinct scenarios that
enter the dehydration stage, the thoroughness of property-checking
is sacriced compared to having a strong test oracle [
36
]—such as
synchronous observers—which can check the properties by observ-
ing the internal states of the program for every test execution.
We achieved property-based testing by automatically synthesiz-
ing synchronous observers from verication properties (see Fig-
ure 6-b). To fully leverage the power of the improved test oracle
and achieve a further automation, we automated the test input
generation as well, by applying a combinatorial technique on our
custom-dened test scenario template (see Figure 6-a). The sys-
tem under test (SUT) is veried almost as-is virtually in the SILS
environment.
4.1 Synchronous observer
Synchronous observer is a program that monitors the state vari-
ables of the system under verication and raises a ag when some
2020-01-09 10:37. Page 4 of 1–10.

Unpublished working draft.
Not for distribution.
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
Property-based Testing for LG Home Appliances using Accelerated Soware-in-the-Loop Simulation Conference’17, July 2017, Washington, DC, USA
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
condition is satised [
18
,
19
,
31
]. Typically, an observer is created
per each verication property so that it can raise a ag for prop-
erty violation. For creating such an observer, one can manually
write a state machine which expresses a property, but it can also
be created automatically when the properties are formalized, using
a well-known technique for synthesizing Büchi automaton directly
from LTL [14].
Our verication properties are formalized from the existing body
of functional and safety properties that we maintain per each prod-
uct line of home appliances. The properties also include failure
cases, which are a history of defects that have been identied in the
development phase as well as critical failures that have occurred
in the eld. Although the properties were originally written in
plain Korean, 90% of them could be formalized in LTL [
6
]. As an
instance of such a property,
ϕ1
for an LG washing machine states:
“The washing machine shall always be executed in an order of stages
where wash precedes rinse and rinse precedes dehydration. Each stage
can be skipped, or interleaved with pauses, but the precedence shall
strictly be obeyed.” This property can check whether an abnormal
stage inversion occurs, such as the dehydration stage being exe-
cuted before the wash stage. In LTL, the safety property
ϕ1
can be
expressed as follows:
ϕ1:□(wash → ⃝(wash ∨rinse ∨dehyd∨pause ∨end))
∧□(rinse → ⃝(rinse ∨dehyd∨pause ∨end))
∧□(dehyd→ ⃝(dehyd∨pause ∨end)) (1)
where □denotes globally and ⃝denotes the next state.
For constructing executable state machines, the LTL properties
can be translated to Büchi automaton, which is dened as a tuple
A=(Q,Σ,δ,qo,F)
, where
Q
is a nite set of states,
Σ
is a nite
set of symbols called alphabet,
δ
:
Q×Σ→Q
is a transition
function,
q0⊂Q
is a set of initial states, and
F⊆Q
is a set of ac-
cepting states [
3
]. Among others,
Σ
consists of boolean expressions
over global variables of the system under test, such as
stage ==
wash
, and
F
can denote either compliant behavior or unacceptable
behavior, depending on the kind of property.
For a safety property, an automaton for bad behaviors is con-
structed from the negation of the property so that it can ag the
path prexes that lead to property violations. For a liveness prop-
erty, the accepting states denote compliant behaviors, and it has to
be visited innitely and often for the property to hold. Verifying
or refuting a liveness property is impossible in testing; we give the
user a warning when the acceptance state is never visited during
the test execution.
Figure 7: Büchi automaton translated from ϕ1
As an example, property
ϕ1
in Equation 1 can be converted
into a Büchi automaton, as shown in Figure 7, and it can then be
synthesized into C code, as shown in listing 3. This code is invoked
after each main loop execution so that the observers can inspect
the system state and ag any property violation.
In our implementation, we used Spot [
9
] for converting LTL
to Büchi automata; Spot is an open-sourced tool for LTL and
ω
-
automata manipulation which can convert an LTL formula to Büchi
automata, create an abstract syntax tree, and export it to various
formats. Since the conversion from automata to C code was lacking,
we implemented the C converter ourselves.
st a ti c i nt s ta t e = ac ce p t_ i ni t ;
bo o l de h yd ra t e , r in se , was h , en d , p au s e ;
.. . // T o ma p pi ng Boo le an e xp re s si on on h er e
sw i tc h ( s ta te ) {
case accept_init:
st a te = e rr o r ;
if ( ! d eh y dr a te && ! r i ns e && ! w as h )
state = accept_init;
if ( w as h )
state = accept_wash;
if ( r i n se )
st a te = a cc e pt _ ri n se ;
if ( d e h y dr a t e )
st a te = a cc e pt _ de h yd r at e ;
br e ak ;
.. .
}
Listing 3: Source code generated from Figure 7
4.2 Test input generation
As the synchronous observer solved the test oracle problem, we
could pursue test generation separately without worrying about
the oracle and apply automated techniques which can explore the
system behavior as thoroughly as possible based on the use-cases
of the system. Since our use cases are described as scenarios, such
as when a user starts a washing process and inputs a series of com-
mands, we rst designed a test scenario template which provided
the test engineers a structure to specify typical test scenarios. A
combinatorial test generation can be applied on the predened
templates to explore the state space eectively. Although a combi-
natorial method may yield a large amount of test inputs compared
to the manually constructed ones, the accelerated SILS environ-
ment can execute a much larger quantity of tests—up to 10,000
times more—in a given amount of time.
4.2.1 Specifying test scenarios with a template. To facilitate an
abstract and compact description of a test scenario, we dened a
test scenario template, which could later be instantiated with actual
parameters. A test scenario template is dened as an ordered list
of tuples
(α,v)
, where
v
is an action and
α
is one of the following
types of actions:
(1)
Input user command (
i
): deliver a user-controlled input com-
mand.
(2)
Wait (
w
): monitor a system state. When instantiated, this
action waits for a Boolean condition over the system state
variables to be satised.
(3)
Change an environmental condition (
c
): manipulate the en-
vironment by setting a environmental condition.
2020-01-09 10:37. Page 5 of 1–10.

Unpublished working draft.
Not for distribution.
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
Conference’17, July 2017, Washington, DC, USA Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
(4)
Change an actuator state (
a
): change an actuator state, typi-
cally for testing fail-safe features.
Table 1: A test scenario template for washing machine
seq. 1 2 3 4 5 6 7 8 9 10
type c i i i i i w i i i
act.
water
_lv cycle Opt. rinse
_cnt
spin
_rpm run stage pause cycle run
For example, the test scenario “a user initiates a laundry sequence
by changing the water level, setting a cycle, an option, the rinse count,
and the spin RPM. The user presses the pause button at a specic
stage, changes to another cycle, and presses the run button” can be
written as follows: (
c
,
water_level
), (
i
,
cycle
), (
i
,
option
), (
i
,
rinse_count
), (
i
,
spin_rpm
), (
i
,
run
) (
w
,
stage
), (
i
,
pause
), (
i
,
cycle
), (
i
,
run
). A scenario template like this can be created by a
test engineer through a table-like interface as shown in Table 1.
Table 2: Combinatorial test input generated from Table 1
Test
No
water
_lv cycle Opt. rinse
_cnt
spin
_rpm run stage pause cycle run
1 CLR cotton turbo 2 1000 wash cotton+
2 CLR wool delay 3 0 rinse sports
3 CLR mix soak 4 400 dehy. cotton
4 CLR cotton+ pre. 0 600 wash wool
5 CLR wool wind 1 800 rinse mix
... ... ... ... ... ... ... ... ... ... ...
From a scenario template, we instantiated a multitude of test
scenarios by changing the valuation of actions using a combina-
torial method, as shown in Table 2. Each entry is automatically
translated into a state machine which gets executed in parallel with
the software under test to provide inputs to it. Figure 8 shows a
state machine which translates the rst entry in Table 2.
Figure 8: Test scenario translated from Table 2-#1
4.2.2 Pair-wise test input generation. The combination of input
button clicks in the initial state of a washing machine is 5,400.
This is equivalent to 45 hours of execution time, even when each
execution is assumed to take 30 seconds with a 300x accelerated
SIL simulation. This is only for the input combinations in the initial
state and there are many more states where additional input can
be provided.
For generating a compact ecient test input set, we applied a
pair-wise method [
21
] for systematically deriving test cases from
the above-mentioned test input templates. The template-based ap-
proach made it more amenable to a combinatorial method. In our
test generator implementation, we used PICT [
8
] as the pairwise
test generation engine.
5 FIELD APPLICATION
We applied our SILS-based testing framework in several embedded
system product lines of LG Electronics. In this section, we present
the results of two eld applications—1) a top-loader washing ma-
chine and 2) a four-door kimchi refrigerator. We rst show the test
input template and verication properties for the respective sys-
tems (in Section 5.1), followed by a comparison of testing eciency
and ecacy to the status-quo integration testing method in terms
of the time and code coverage (in Section 5.2). Due to the improve-
ments in the testing ecacy, we caught two critical corner-case
failures during verication, which will be explained in Section 5.3.
5.1 Verication properties and test scenario
templates
5.1.1 Washing machine. For verifying washing machines, we used
the test scenario template in Table 1 together with the existing
integration test scenarios. We applied a 5-wise combination on this
template and generated 8,302 test cases. The verication properties
are derived from eld failure cases that include
ϕ1
and 20 out of 23
product liability cases.
5.1.2 Refrigerator. The key function of refrigerator control soft-
ware is to sense and control the temperature in individual chambers.
A refrigerator can have up to three to four individual chambers,
with each chamber providing the capability of controlling and main-
taining its individual temperature as determined by a user. During
a cooling cycle, the temperature in each chamber is controlled by
the operation mode, the room temperature wherein the refrigerator
operates, and the desired temperature. Each cooling cycle nishes
by defrosting the evaporator. Given these variables as parameters,
a test scenario template of a refrigerator is given in Table 3. Preex-
isting integration test scenarios are included in the tests as well.
Table 3: Test scenario template for a refrigerator
seq. 1 2 3 4 5 6 7
type i i i i i c w
act.
RM1
Menu
RM2
Menu
RM3
Menu
RM4
Menu
Opt.
Menu
change
RT Stage
Table 4: Pair-wise test input generated from Table 3
Test
No
RM1
Menu
RM2
Menu
RM3
Menu
RM4
Menu
Opt.
Menu
change
RT Stage
1 kimchi kimchi kimchi kimchi non 30°C defrost
2 freezer freezer kimchi fruits express 25°C defrost
3 freezer kimchi rice fruits ferment 10°C defrost
4 kimchi freezer kimchi off non 0°C defrost
5 freezer freezer fruits off express -10°C defrost
... ... ... ... ... ... ... ...
The verication properties for a refrigerator consists of 13 out
of 15 product liability cases which could be expressed in LTL. One
2020-01-09 10:37. Page 6 of 1–10.

Unpublished working draft.
Not for distribution.
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
Property-based Testing for LG Home Appliances using Accelerated Soware-in-the-Loop Simulation Conference’17, July 2017, Washington, DC, USA
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
of the representative (liveness) properties ϕ2looks as follows:
ϕ2:□(coolinд→ ⃝♢de f rost )(2)
where
♢
is “in the future” operator.
ϕ2
states that whenever the
cooling stage is entered, the defrost stage shall come next eventually.
As the defrost stage is the last essential stage in a cycle to remove
the frost on the evaporator using a heater,
ϕ2
checks whether this
cycle is executed innitely often.
5.2 Result of the SILS-based integration testing
Table 5: Test result of the washing machine
Test
Suite
# Test
Scenario
Exec.
Time
Function
Coverage
Branch
Coverage
Property
Pass
Manual
Normal130 1 week258%
(889/1532)
41%
(2402/5850)
100%
(20/20)
Manual
Hard1100 2 weeks264%
(991/1532)
56%
(3313/5850)
100%
(20/20)
All-pairs
3-wise3573 6H 17M
77%
(1184/1532)
70%
(4112/5850)
100%
(20/20)
All-pairs
4-wise32,612 33H 21M
83%
(1285/1532)
76%
(4464/5850)
95%
(19/20)
All-pairs
5-wise38,402 112H 7M
85%
(1313/1532)
77%
(4549/5850)
95%
(19/20)
5.2.1 Washing machine. Using a combinatorial method, we gener-
ated 280 times more test inputs compared to the manually-crafted
test cases we had used previously. The execution time was reduced
signicantly as well: with the previous approach, we could only run
0.625 test cases per hour in each prototype machine, but were able
to run 75 test cases per hour with accelerated SILS-based testing.
The resulting coverage had also increased signicantly, yielding
32% higher function coverage and 37% higher branch coverage,
with 5-wise test suite compared to the manual-hard suite.
We found a critical failure with 4-wise and 5-wise test suites,
captured by the observer synthesized from property
ϕ1
. Given the
steep increase in the cost from 4-wise to 5-wise and a relatively little
gain of coverage, we determined that 4-wise testing is an optimal
choice, given our empirical observations and our time budget.
Table 6: Test result of the refrigerator
Test
Suite
# Test
Scenario
Exec.
Time
Function
Coverage
Branch
Coverage
Property
Pass
Manual
Normal1120 2 weeks270%
(361/511)
44%
(1902/4296)
100%
(13/13)
Manual
Hard1500 2 months274%
(380/511)
47%
(2049/4296)
100%
(13/13)
All-pairs
3-wise3814 4H 27M
78%
(403/511)
71%
(3093/4296)
100%
(13/13)
All-pairs
4-wise33,394 20H 37M
80%
(413/511)
78%
(3356/4296)
92%
(12/13)
All-pairs
5-wise313,469 97H 15M
85%
(436/511)
81%
(3483/4296)
92%
(12/13)
1before automation—prototype-hardware-based integration testing
28 hour workday with two prototype machines
3after automation—SILS based integration test result
5.2.2 Refrigerator. With a combinatorial method, we could gen-
erate up to 122 times more test cases compared to our existing
ones. With acceleration, we could run 3,332 test cases per each
day even with the largest 5-wise suite, a signicant improvement
over 6.25 test cases per day in each prototype machine with the
manual-hard suite. The function coverage has increased by 15% and
branch coverage by 72%, compared to those of manual-hard suite.
While testing with the 4-wise and 5-wise suite, we found a crit-
ical failure which violates property
ϕ2
in Equation 2. Given the
decreasing gain from 4-wise to 5-wise testing, we determined with
the refrigerator, again, that 4-wise testing is an optimal choice.
5.2.3 Uncovered branches. We analyzed the branches that were
not covered by our pair-wise testing. Most of them turned out to be
either a fail-safe logic for unexpected hardware failures and unused
features that are implemented for exibly incorporating the needs
of similar models. The unused code cannot simply be removed,
because doing so would require dierent codes to be loaded per
each variant of similar appliance models. The uncovered fail-safe
logic is not handled in SILS testing since the fail-safe features are
better tested with the actual hardware.
Table 7: Acceleration achieved by SILS testing
Test
Suite
Test
Scenario
On physical
machine On SILS Acceleration
Washing
Machine cotton
course 1H 27M 17S 307x
Kimchi
Refrig.
kimchi
ferment 56H 20S 10,080x
5.2.4 Acceleration in SILS. Our accelerated SILS framework im-
proves the eciency of testing dramatically compared to manual
testing. To measure the gain, we executed representative test sce-
narios on physical machines and SILS environment for each type of
product. We compared the elapsed time on the two dierent setups
in Table 7. The SILS testing environment was operated on a typical
PC with an Intel i5-3570 CPU and a 8GB RAM. The result shows
that the gain was 307x for a washing machine and 10,080x for a
refrigerator. The degrees of dierences generalize to any other test
scenario, as our acceleration mechanism compresses the physical
time to the same degree regardless of the test scenario.
5.3 Cases of Failures
We applied our SILS-based testing for verifying our products under
development and found two real faults that, otherwise, would not
have been found with our existing technique.
5.3.1 Stage inversion in the washing machine. As
ϕ1
states, the
stages in a washing machine are always executed in the right order
while still providing the exibility of editing the stages during
operation. Internally, the stage scheduler was implemented with
a heap-based priority queue. During testing, however, a serious
stage inversion fault was discovered. The problem occurred in the
feature that allows the user to remove an arbitrary stage, which
was implemented by adding a non-standard operation of deleting
a non-root node from the heap and by attempting to re-balance
the tree. The incorrect re-balancing algorithm attempts to rebuild
the sub-tree under the node of deletion regardless of the priority of
2020-01-09 10:37. Page 7 of 1–10.

Unpublished working draft.
Not for distribution.
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
Conference’17, July 2017, Washington, DC, USA Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
that node, while a correct implementation should have rebuilt the
whole tree.
Figure 9: An incorrect implementation of the heap resulted
in a stage inversion in a washing machine where the rinse
stage is executed prior to the wash stage.
The exact error-revealing scenario is as follows. The rst subg-
ure in Figure 9-1) shows a min-heap instance in a “Cotton” course.
The nodes correspond to stages such as wait (4), wash (8), rinse (9),
dehydration (10), end (12), and o (13). A fault-revealing sequence
adds a soak (6), removes a dehydration (10), and adds the dehy-
dration again; when the machine resumes, the stages are executed
in an order of wait, soak, rinse, wash, and dehydration. The nal
outcome of the whole wash cycle would be well-dehydrated clothes
with the detergent still intact.
Although it may look like a simple enough fault to catch, it is
actually dicult to discover in the system-level testing. This fault-
revealing scenario requires, rst of all, a user to customize a se-
quence to fully populate the heap, which happens rarely. Then, the
dehydration stage has to be removed and added immediately after-
wards. Even when this condition is met and stage inversion happens,
the tree can be recovered in most cases while re-balancing the heap.
In the status-quo testing approach, given the large state space of a
heap and the limited number of representative requirement-based
test cases, it is very unlikely that this behavior would be exercised.
With the benet of the pairwise method to generate comprehen-
sive test suites and using the synchronous observer to encode the
stage precedence property as in
ϕ1
, we discovered this failure only
after 33 hours in the accelerated SILS environment.
5.3.2 Defrost failure in the refrigerator. We applied our technique
for verifying a four-chamber kimchi refrigerator in the integration
testing and found a serious fault which causes the defrost to fail.
In a refrigerator, the evaporator may collect frost due to the
humidity; the frost becomes a bigger ice block when not removed
in time, and this undermines the cooling eciency. The defrosting
is performed by a heater inside the evaporator, melting the ice into
water. Since this heater has the undesirable side eect of heating
up the chamber, a preparatory step is introduced to counterbalance
this side eect by increasing the cooling power beforehand. The
defrost does not start until this preparatory cooling is nished.
The defrost fault was introduced in a new refrigerator under
development while implementing a feature that optionally skipped
the preparatory cooling step (see the ow shown in Figure 10-b)).
When the skipping condition is met, a ag variable should be set for
indicating that the preparatory cooling is also done (see Figure 10-
a)), so that defrost can be executed; the ag variable, however, was
never set in the faulty implementation (see Figure 10-b)). As a result,
the defrost stage cannot be entered indenitely since the variable
PreStepState is never set to FINISHED.
Figure 10: The defrost fault in a kimchi refrigerator. The
ow highlighted in red indicates the erroneous sequence.
This fault may look quite trivial from the simplied illustration
in Figure 10, but it is dicult to catch for the following reasons.
First, a four-chamber refrigerator allows the temperature of each
chamber to be controlled individually, resulting in a combinatorially
larger state space. The advanced features that a kimchi refrigerator
provides also contribute to a large state space because there are
dierent modes for fermenting dierent types of kimchi, each im-
plemented by a dierent algorithm. This particular fault-revealing
case was, indeed, a corner case that only one out of 500 combina-
tions of algorithms can trigger. A unit-level testing cannot trigger
this fault, since the condition the fault depends on gets triggered
only when multiple modules interact. Integration testing should
be able to catch this fault, but it was missed given a limited time
budget because it takes at least 74 hours for the defrost to initiate
from an initial startup; as a result, a worst-case execution time is
36,000 hours or four years.
With the benet of the pairwise method which could generate
comprehensive test suites and using the synchronous observer
of
ϕ2
, we discovered this failure in only 20 hours in the 10,000x
accelerated SILS testing environment.
6 RELATED WORK
SILS and test automation are two pillars of our testing framework,
both of which are well-studied topics in the research community as
well as popular techniques in the industry. SILS have been actively
2020-01-09 10:37. Page 8 of 1–10.

Unpublished working draft.
Not for distribution.
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
Property-based Testing for LG Home Appliances using Accelerated Soware-in-the-Loop Simulation Conference’17, July 2017, Washington, DC, USA
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
studied in the automotive domain for constructing mathematical
models that resemble the actual physical environment [
2
,
32
,
38
].
SILS is also applied to construct hardware-independent software
simulation for AUTOSAR [
1
]-compliant automotive software, espe-
cially for virtualizing CAN communication and hardware peripher-
als [
17
,
35
,
37
]. Our work can be viewed as a variant of these studies
as we simulate peripheral hardware in a similar manner, but we
further propose an automated testing framework which includes
test generation, oracle synthesis, and accelerated simulation.
Another stream of similar studies is ECU virtualization [
11
,
27
,
28
]. These studies emulate the instruction set architecture of the
ECU such that the binaries of the automotive control software can
be tested in the PC; they also virtualize connected hardware, such
as RAM and ROM. There are clear advantages to the ECU/MCU
emulation from a verication perspective, such as being able to
catch MCU-dependent failures. However, we did not adopt this
approach because it is inexible in relation to changes in MCU, and
MCU is frequently changed for LG home appliances.
HILS-based verication is also popular as a research topic and
as a practical technique in the industry [
13
,
20
,
34
]. It is employed
in cases where the control logic to be veried is tightly coupled
with the hardware. In our case, however, the set-up time and the
operating cost of a HILS environment turned out to be higher than
that of operating prototype hardwares; thus it was not adopted
widely in LGE.
The foreground ISR injection method was inspired by the works
on verifying an ISR concurrency problem using ISR code injec-
tion [
10
,
29
] and a work on time synchronization in distributed
systems [
23
]. Our work employed the idea of a code injection, but
used it for the dierent purpose of accelerating the simulation.
In the same LG home appliance domain, Park et al. [
29
] pro-
posed an approach for test automation using concolic testing. Their
work focused on verifying the correctness of the embedded con-
trol software with interrupts, when the interrupts are modeled as
branches and solved using concolic testing approach. Their testing
framework focused on verifying the assertions in the code where
the timing of the interrupts is treated as test inputs. This study, by
contrast, focuses on testing temporal properties for scenario-based
test inputs. Park et al.’s approach for testing interrupts is comple-
mentary to our approach and used in LGE for verifying the absence
of concurrency errors.
This work used a pairwise method [
8
,
22
,
25
] for generating test
inputs from the requirement-driven test scenario templates. State-
machine based techniques [
4
,
5
,
12
] could similarly be employed
for generating tests from the requirements, by allowing a user to
specify the state machines and generating tests from them. However,
we found it dicult to employ such a technique at LGE because
expressing the requirements in such a form requires more eort
and is dicult to maintain when a change in requirements occurs.
The approach of synthesizing synchronous observers from LTL
has been studied extensively [
18
,
19
,
31
]. We found that LTL [
6
] is
suitable for expressing our safety requirements because LTL can
express both liveness and safety properties.
For integration testing, several commercial tools are available [
15
,
16
] and widely adopted for integration testing in industry. Although
they are eective in running test cases—pairs of inputs and expected
outputs—they do not allow users to run testing against temporal
property as we do. Also, hardware dependency in the code has to
be resolved separately.
7 LESSON LEARNED
We discuss the lessons we have learned through a successful adop-
tion of SILS-based testing framework.
7.1 The eect
The biggest benet from the successful adoption of our proposed
framework is that software can be developed and veried in parallel
with hardware. As a result, we could shorten the software develop-
ment schedule by 30% and save the cost of manufacturing prototype
machines by 70%. This benet is appreciated company-wide, and,
for facilitating a further adoption, the LG Home Appliance Division
has formed a designated team for application of SILS in all home
appliance product-lines.
Another valuable lesson is that the combinatorial testing can
create a powerful synergy when combined with stronger property-
based oracle and accelerated testing. The three techniques combined
free the human eort from manual test design, while allowing
more test cases to be executed and checked more thoroughly, all
in an automated fashion. This approach enabled us to nd the two
critical failure cases—stage inversion fault in the washing machines
and an innite freezing loop fault in the refrigerators—in the eld
application. The two cases turned out to be corner-case faults that
are almost impossible to be caught using a manual testing approach.
After the operation of the framework, we have collected feed-
back from the developers. The developers appreciated that the SILS
can be used during development and allow them to run testing in
the IDE without worrying about damaging the hardware. They also
liked that SILS removed any human intervention in the testing;
previously, they had to constantly monitor the device since an
erroneous control could cause damage to the machines and the
surrounding environment by creating a re, for example. SILS-based
testing can run 24 hours a day without any human intervention,
and it provides a log and a report at the end. This allows testers to
focus on more important tasks, such as designing test cases.
7.2 Behind the success
This work applies theories and practices that have been studied in
the Software Engineering community for decades and would not
have been possible without the experienced engineers who have
researched model checking [
24
] during their Master of Science
studies. This project was initiated as a side project by two of our
engineers and came to fruition because they invested their leisure
time in it over the last two years.
Our success should also be attributed to the relatively low com-
plexity of the environment for home appliances compared to more
complex ones, such as automotive control systems. The hardware
components were also easy to model as they were standardized and
used in multiple products, facilitating a wide adoption of SILS. An
improvement in the software architecture also contributed to fast
adoption, which was carefully designed to maximize the separation
of hardware dependent code from independent code.
Lastly, the success of the proposed approach depends largely
on the quality of the verication properties. In our case study, we
2020-01-09 10:37. Page 9 of 1–10.

Unpublished working draft.
Not for distribution.
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
Conference’17, July 2017, Washington, DC, USA Mingyu Park, Hoon Jang, Taejoon Byun, and Yunja Choi
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
utilized the historic record of the critical eld failure cases that had
been documented throughout decades. These bad behaviors could
be converted directly to the verication properties, with which we
were able to nd real failures during testing. The requirement alone
would not have been sucient for nding these failures, as the
requirement mostly states the shalls rather than shall nots.
8 SUMMARY
This study introduces a successful adoption of a SILS-based testing
framework in verifying LG home appliances. The simulation-based
testing framework resolved the dependence of software develop-
ment and verication schedules on hardware development, facili-
tating software verication from early on in the development life-
cycle. We also introduced an approach of synthesizing synchronous
observers automatically from the requirements so that property
verication is performed during the test executions, and a pairwise
method was employed to automate the test generation.
As a result, we achieved 300 to 10,000 more times acceleration
in test executions, enabling us to execute many more test cases in
a given timeframe and achieve, at its height, a 32% higher branch
coverage. After deployment of the proposed framework at LG Elec-
tronics, we caught two serious system-level faults before mass
production, a success we believe was made possible by the pro-
posed framework. In the future, we plan to develop techniques for
enhancing the coverage of the integration testing, especially on the
unused features and the untested fail-safe logic.
REFERENCES
[1] AU TOSAR. 2019. AUTOSAR. https://www.autosar.org.
[2]
Holger Brückmann, Jochen Strenkert, Uwe Keller, Benno Wiesner-Tittes, and
Andreas Junghanns. 2009. Model-based Development of a Dual-Clutch Transmis-
sion using Rapid Prototyping and SiL. In PVDI Congress Transmissions in Vehicles
20092.
[3]
Julius Richard Büchi. 1962. On a decision method in restricted second order
arithmetic. In In Proc. Internat. Congr. Logic, Method and Philos. Sci. Stanford
University Press, 1 – 12.
[4]
Yunja Choi and Taejoon Byun. 2017. Constraint-based test generation for au-
tomotive operating systems. Software & Systems Modeling 16, 1 (01 Feb 2017),
7–24.
[5]
Yunja Choi, Mingyu Park, Taejoon Byun, and Dongwoo Kim. 2015. Ecient
safety checking for automotive operating systems using property-based slicing
and constraint-based environment generation. Science of Computer Programming
103 (2015), 51 – 70.
[6]
Edmund M. Clarke, E. Allen Emerson., and A. Prasad Sistla. 1986. Automatic Ver-
ication of Finite-state Concurrent Systems Using Temporal Logic Specications.
ACM Trans. Program. Lang. Syst. 8, 2 (April 1986), 244–263.
[7]
International Electrotechnical Commission. 2019. Functional Safety of Electri-
cal/Electronic/Programmable Electronic Safety-related Systems. https://www.
iec.ch/functionalsafety/.
[8]
Jacek Czerwonka. 2008. Pairwise Testing in Real World Practical Extensions to
Test Case Generators. In Proceedings of 24th Pacic Northwest Software Quality
Conference.
[9]
Alexandre Duret-Lutz, Alexandre Lewkowicz, Amaury Fauchille, Thibaud
Michaud, Etienne Renault, and Laurent Xu. 2016. Spot 2.0 — a framework for LTL
and
ω
-automata manipulation. In Proceedings of the 14th International Sympo-
sium on Automated Technology for Verication and Analysis (ATVA’16). Springer,
122–129.
[10]
Orit Edelstein, Eitan Farchi, Yarden Nir, Gil Ratsaby, and Shmuel Ur. 2001. Multi-
threaded Java Program Test Generation. In Proceedings of the 2001 Joint ACM-
ISCOPE Conference on Java Grande (JGI ’01). ACM, New York, NY, USA, 181–.
[11]
Guillaume Francois. 2013. New Approaches in Virtualization of ECU Software
Development. In SAE 2013 World Congress & Exhibition. SAE International.
[12] Peter Fröhlich and Johannes Link. 2000. Automated Test Case Generation from
Dynamic Models. In ECOOP 2000 — Object-Oriented Programming, Elisa Bertino
(Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 472–491.
[13]
Nicholas Gans, W.E. Dixon, Rick Lind, and Andrew Kurdila. 2009. A hardware in
the loop simulation platform for vision-based control of unmanned air vehicles.
Mechatronics 19, 7 (2009), 1043 – 1056.
[14]
Paul Gastin and Denis Oddoux. 2001. Fast LTL to Büchi Automata Translation.
In Computer Aided Verication, Gérard Berry, Hubert Comon, and Alain Finkel
(Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 53–65.
[15]
Razorcat Development GmbH. 2019. Tessy. https://www.razorcat.com/en/
product-tessy.html.
[16] Vector Informatik GmbH. 2019. VectorCAST. https://www.vectorcast.com/.
[17]
Gwangmin Park, Daehyun Ku, Seonghun Lee, Woong-Jae Won, and Wooyoung
Jung. 2009. Test methods of the AUTOSAR application software components. In
2009 ICCAS-SICE. 2601–2606.
[18]
Nicolas Halbwachs, Fabienne Lagnier, and Pascal Raymond. 1994. Synchronous
Observers and the Verication of Reactive Systems. In Algebraic Methodology
and Software Technology (AMAST’93), Maurice Nivat, Charles Rattray, Teodor
Rus, and Giuseppe Scollo (Eds.). Springer London, London, 83–96.
[19]
Nicolas Halbwachs and Pascal Raymond. 1999. Validation of Synchronous Re-
active Systems: From Formal Verication to Automatic Testing. In Advances in
Computing Science — ASIAN’99, P. S. Thiagarajan and Roland Yap (Eds.). Springer
Berlin Heidelberg, Berlin, Heidelberg, 1–12.
[20]
Martin Hillenbrand and K. D. Muller-Glaser. 2009. An Approach to Supply
Simulations of the Functional Environment of ECUs for Hardware-in-the-Loop
Test Systems Based on EE-architectures Conform to AUTOSAR. In Proceedings
of the 2009 IEEE/IFIP International Symposium on Rapid System Prototyping (RSP
’09). IEEE Computer Society, Washington, DC, USA, 188–195.
[21]
D. Richard Kuhn, Dolores R. Wallace, and Albert M. Gallo. 2004. Software Fault
Interactions and Implications for Software Testing. IEEE Trans. Softw. Eng. 30, 6
(June 2004), 418–421.
[22]
Kuo-Chung Tai and Yu Lei. 2002. A test generation strategy for pairwise testing.
IEEE Transactions on Software Engineering 28, 1 (Jan 2002), 109–111.
[23]
Sunghee Lee, Bueng Il Hwang, Kang-Bok Seo, and Woo Jin Lee. 2016. Relative
Time Synchronization of Distributed Applications for Software-in-the-Loop Sim-
ulation. In 2016 IEEE Intl Conference on Computational Science and Engineering
(CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC)
and 15th Intl Symposium on Distributed Computing and Applications for Business
Engineering (DCABES). 753–756.
[24]
Edmund M. Clarke, Orna Grumberg, and Doron Peled. 2001. Model Checking.
MIT Press.
[25]
Robert Mandl. 1985. Orthogonal Latin Squares: An Application of Experiment
Design to Compiler Testing. Commun. ACM 28, 10 (Oct. 1985), 1054–1058.
[26] MathWorks. 2019. Simulink. https://mathworks.com/products/simulink.html.
[27]
Kenta Morishima, Shigeru Thomas Oho, and Satoshi Shimada. 2016. A Virtual
ECU and Its Application to Control System Analysis - Power Window System
Demonstration. In SAE 2016 World Congress and Exhibition. SAE International.
[28]
Kenta Morishima, Yasuo Sugure, and Yoshihiro Miyazaki. 2018. Evaluation of
Parallel Executions on Multiple Virtual ECU Systems. In WCX World Congress
Experience. SAE International.
[29]
Yongbae Park, Shin Hong, Moonzoo Kim, Dongju Lee, and Junhee Cho. 2015.
Systematic Testing of Reactive Software with Non-deterministic Events: A Case
Study on LG Electric Oven. In Proceedings of the 37th International Conference
on Software Engineering - Volume 2 (ICSE ’15). IEEE Press, Piscataway, NJ, USA,
29–38.
[30]
John Regehr and Nathan Cooprider. 2007. Interrupt Verication via Thread
Verication. Electronic Notes in Theoretical Computer Science 174, 9 (2007), 139 –
150. Proceedings of the Thread Verication Workshop.
[31]
John Rushby. 2014. The Versatile Synchronous Observer. Springer Berlin Heidel-
berg, Berlin, Heidelberg, 110–128.
[32]
Riccardo Russo, Mario Terzo, and Francesco Timpone. 2007. Software-in-the-loop
development and validation of a Cornering Brake Control logic. Vehicle System
Dynamics 45, 2 (2007), 149–163.
[33]
Hesham Shokry and Mike Hinchey. 2009. Model-Based Verication of Embedded
Software. Computer 42, 4 (April 2009), 53–59.
[34]
Amir Soltani and Francis Assadian. 2016. A Hardware-in-the-Loop Facility
for Integrated Vehicle Dynamics Control System Design and Validation. IFAC-
PapersOnLine 49, 21 (2016), 32 – 38. 7th IFAC Symposium on Mechatronic
Systems MECHATRONICS 2016.
[35]
Sooyong Jeong, Yongsub Kwak, and Woo Jin Lee. 2016. Software-in-the-Loop
simulation for early-stage testing of AUTOSAR software component. In 2016
Eighth International Conference on Ubiquitous and Future Networks (ICUFN). 59–
63.
[36]
Matt Staats, Michael W. Whalen, and Mats P.E. Heimdahl. 2011. Programs, Tests,
and Oracles: The Foundations of Testing Revisited. In Proceedings of the 33rd
International Conference on Software Engineering. 391–400.
[37]
Herman Van der Auweraer, Jan Anthonis, Stijn De Bruyne, and Jan Leuridan.
2013. Virtual engineering at work: the challenges for designing mechatronic
products. Engineering with Computers 29 (2013), 389–408.
[38]
Gabriele Vandi, Nicolò Cavina, Enrico Corti, Giorgio Mancini, Davide Moro,
Fabrizio Ponti, and Vittorio Ravaglioli. 2014. Development of a Software in the
Loop Environment for Automotive Powertrain Systems. Energy Procedia 45
(2014), 789 – 798.
2020-01-09 10:37. Page 10 of 1–10.