Tutorial sobre Máquinas de Vectores Soporte (SVM)

Abstract

Este tutorial presenta una introducción al mundo de las máquinas de vectores soporte (SVM, del inglés Support Vector Machine), aplicadas a resolver tareas tanto de clasifi cación como de regresión. En el primer tipo de tareas, la descripción se restringe al caso de clasi ficación binaria y, atendiendo al tipo de separabilidad de los ejemplos de entrada, se consideran distintas opciones. Así, en primer lugar, se aborda el caso ideal de ejemplos perfectamente separables linealmente para, seguidamente, abordar el caso más realista de ejemplos que, aunque afectados por ruido, se consideran linealmente cuasi-separables y,
finalmente, se considera el caso de ejemplos no separables linealmente, donde las SVM demuestran su gran potencialidad. La descripción de las SVMs aplicadas a la tarea de regresión corre también de forma paralela, abarcando los casos tanto de regresión lineal como no lineal. Finalmente, se presentan algunas herramientas software de uso libre que implementan este tipo de paradigma de aprendizaje y con las que el usuario puede empezar a experimentar.

Full text

min f(x),x∈Ω
gi(x)≤0, i = 1, . . . , n

x∗gi(x∗)≤0f(x∗)≤f(x)∀xgi(x)≤
0i= 1, ..., n f gi
f(x)
gi(x)
L(x,α) = f(x) +
n
X
i=1
αigi(x)
αi≥0
αi
gi(x)
max ϕ(α) = inf
x∈ΩL(x,α)
αi≥0, i = 1, . . . , n
x α
gi(x)≤0αi≥0i= 1, ..., n ϕ(α)≤f(x)
ϕ(α) = f(x)α x
|ϕ(α)−f(x)|
L(x,α)xL(x,α)
α
x∗
f:
Rd→Rgi:Rd→R, i = 1, . . . , n

αi≥0, i = 1, . . . , n
∂f (x∗)
∂xj
+
n
X
i=1
αi
∂gi(x∗)
∂xj
= 0 j= 1, . . . , d
αigi(x∗)=0 i= 1, . . . , n
x∗
ϕ(α)
ϕ(α∗) = f(x∗)
infx∈ΩL(x,α)
αi≥0
S={(x1, y1),...,(xn, yn)}xi∈Rdyi∈
{+1,−1}
D(x)=(w1x1+... +wdxd) + b=<w,x>+b

w∈Rdb∈R<x1,x2>
x1x2xi
<w,xi>+b≥0yi= +1
<w,xi>+b≤0yi=−1, i = 1, ..., n
yi(<w,xi>+b)≥0, i = 1, . . . , n
yiD(xi)≥0, i = 1, . . . , n
τ
+1
−1

Hiperplano óptimo
τ
τm´ax
+1
−1
−1
+1
D(x)
x0
Distancia D(x),x0=|D(x0)|
kwk
|·| k·k w
b D(x)
yiD(xi)
kwk≥τ, i = 1, . . . , n
yiD(xi)
kwk=τ, ∀i∈VS
VS

Hiperplano óptimo, D(x)=0
1 / ||w||
|D(xi)| / ||w||
xi
D(x)= +1
D(x)= -1
D(x)> +1
D(x)< -1
xi
|D(xi)|/kwk1/kwk
|D(x)|= 1
τ,
kwkwλ(<w,x>+b)λ∈R
τw
τkwk= 1
yiD(xi)≥1, i = 1, . . . , n
yi(<w,xi>+b)≥1, i = 1, . . . , n
wb f(w) = kwk
min 1
2kwk2≡1
2<w,w>
s.a. yi(<w,xi>+b)−1≥0, i = 1, . . . , n
f(w) = kwk1/2kwk2

L(w, b, α) = 1
2<w,w>−
n
X
i=1
αi[yi(<w,xi>+b)−1]
αi≥0
∂L (w, b, α)
∂w≡w−
n
X
i=1
αiyixi= 0
∂L (w, b, α)
∂b ≡
n
X
i=1
αiyi= 0
αi[1 −yi(<w,xi>+b)] = 0, i = 1, . . . , n
w
αi
w=
n
X
i=1
αiyixi
αi
n
X
i=1
αiyi= 0
b
αi
L(w, b, α) = 1
2<w,w>−
n
X
i=1
αiyi<w,xi>−b
n
X
i=1
αiyi+
n
X
i=1
αi
g(x)≥0g(x)≤0

L(α) = 1
2 n
X
i=1
αiyixi!

n
X
j=1
αjyjxj
− n
X
i=1
αiyixi!

n
X
j=1
αjyjxj
+
n
X
i=1
αi
L(α) = −1
2 n
X
i=1
αiyixi!

n
X
j=1
αjyjxj
+
n
X
i=1
αi
L(α) =
n
X
i=1
αi−1
2
n
X
i=1
αiαjyiyj<xi,xj>
α∗
αi≥0
max L(α) = Pn
i=1 αi−1
2Pn
i,j=1 αiαjyiyj<xi,xj>
Pn
i=1 αiyi= 0
αi≥0, i = 1, . . . , n
n d
α∗
D(x) =
n
X
i=1
α∗
iyi<x,xi>+b∗
b∗
αi≥0αi>
0
yi(<w∗,xi>+b∗)=1
(xi, yi)αi>0i
αi>0

αj= 0
b∗
b∗=yvs−<w∗,xvs>
(xvs, yvs)
αi
b∗
b∗=1
NVSX
j∈VS
(yj−<w∗,xj>)
VSNVS
b∗
α∗
ξi, i = 1,...n
yi(<w,xi>+b)≥1−ξi, ξi≥0, i = 1, . . . , n
(xi, yi)ξi
yi

D(x)=0
xi
D(x)= +1
D(x)= -1
D(x)> +1
D(x)< -1
ξk=1+|D(xk)|
ξj=1+|D(xj)|
ξi=1-|D(xi)|
xk
xj
xi
xixjxk
ξi, ξj, ξk>0xixjxk
Pn
i=1 ξi
f(w,ξ) = 1
2kwk2+C
n
X
i=1
ξi
C
C ξi
C→ ∞ ξi→0
C ξi
C→0

ξi→ ∞ C
wb
min 1
2<w,w>+CPn
i=1 ξi
s.a. yi(<w,xi>+b) + ξi−1≥0
ξi≥0, i = 1, . . . , n
L(w, b, ξ,α,β) = 1
2<w,w>+C
n
X
i=1
ξi−
n
X
i=1
αi[yi(<w,xi>+b) + ξi−1] −
n
X
i=1
βiξi
∂L
∂w≡w∗−
n
X
i=1
αiyixi= 0
∂L
∂b ≡
n
X
i=1
αiyi= 0
∂L
∂ξi≡C−αi−βi= 0 i= 1, . . . , n
αi[1 −yi(<w∗,xi>+b∗)−ξi] = 0, i = 1, . . . , n
βi·ξi= 0, i = 1, . . . , n
(w, b, ξ)
(α,β)
w∗=
n
X
i=1
αiyixi
n
X
i=1
αiyi= 0
αi≥0βi≥0
i= 1,...,n
g(x)≥0g(x)≤0

C=αi+βii= 1, . . . , n
L(α) =
n
X
i=1
αi−1
2
n
X
i,j=1
αiαjyiyj<xi,xj>
max Pn
i=1 αi−1
2Pn
i,j=1 αiαjyiyj<xi,xj>
Pn
i=1 αiyi= 0
0≤αi≤C, i = 1, . . . , n
α∗
D(x) =
n
X
i=1
α∗
iyi<x,xi>+b∗
b∗
αi= 0 C=βi
ξi= 0
xiαi
ξi= 0
xiξi>0
βi= 0
αi=C
xiαi=C ξi>0
αi6= 0
1−yi(<w∗,xi>+b∗)−ξi= 0
1−yiD(xi) = ξi
xi
yiD(xi)≥0ξi= 1 − |D(xi)|
0≤ξi≤1xi
yiD(xi)<0ξi= 1 + |D(xi)|ξi>1
0< αi< C
βi6= 0 ξi= 0
0< αi< C
ξi= 0
1−yi(<w∗,xi>+b∗)=0
αi≤C αi, βi≥0

xi
xi0< αi< C
b∗
b∗=yi−<w∗,xi>∀i0< αi< C
b∗
xiαi>0
0< αi< C
b∗
b∗=1
NV0
SX
j∈V0
S
(yj−<w∗,xj>)
V0
Sαi0< αi< C NV0
S
b∗
b∗=yi−
n
X
j=1
α∗
iyi<xj,xi>∀αi0< αi< C
α∗
i, i = 1, . . . , n
α∗
i6= 0 0 < α∗
i< C
α∗
i=C
C
F
x

x = (x1,x2) Φ(x) = [Φ1(x), Φ2(x)]
X1
X2
Φ1(x)
Φ2(x)
Espacio de entradas Espacio de características
χ F
Φ: X F
X1
X2
Espacio de entradas
Φ : F X
x = (x1,x2)
χ
-1
Φ : X→ F
xFΦ(x) = [φ1(x), . . . , φm(x)]
∃φi(x), i = 1, ..., m φi(x)
D(x) = (w1φ1(x) + ... +wmφm(x)) =<w,Φ(x)>
D(x) =
n
X
i=1
α∗
iyi<Φ(x),Φ(xi)>
<Φ(x),Φ(xi)>
K:X×X→R
X
F
K(x,x0) =<Φ(x),Φ(x0)>=φ1(x)φ1(x0) + ... +φm(x)φm(x0)
b
φ1(x) = 1
w w ∈Rd+1

Φ : X→ F
D(x) =
n
X
i=1
α∗
iyiK(x,xi)
Φ = {φ1(x), . . . , φm(x)}
α∗
i, i = 1,...n
max Pn
i=1 αi−1
2Pn
i,j=1 αiαjyiyjK(xi,xj)
Pn
i=1 αiyi= 0
0≤αi≤C, i = 1, . . . , n
αi, i = 1,...n
(xi, y), i = 1, . . . , n K
C
x=
(x1, x2)
φ1(x1, x2) = 1 φ2(x1, x2) = x1φ3(x1, x2) = x2
φ4(x1, x2) = x1x2φ5(x1, x2) = x2
1φ6(x1, x2) = x2
2
φ7(x1, x2) = x2
1x2φ8(x1, x2) = x1x2
2φ9(x1, x2) = x3
1
φ10 (x1, x2) = x3
2

K:X×X→R
Φ : X→ F
K(x,x0) =<Φ(x),Φ(x0)>∀x,x0∈X
Φ(x)=(φ1(x), . . . , φm(x))
K
K:X×X→R
•KL(x,x0) =<x,x0>
•p
KP(x,x0) = γ < x,x0>+τp, γ > 0
•
KG(x,x0) = exp −γ
x−x0

2≡exp −γ < x−x0,x−x0>, γ > 0
•KS(x,x0) = tanh(γ < x,x0>+τ)
γ τ p
K:X×X→RK(x,x0) = K(x0,x)∀x,x0∈X
K:X×X→RPn
i=1 Pn
i=1 cicjK(xi,xj)≥0
x1,...,xn∈Xc1,...cn∈Rn > 0

−2 −1 0 1 2
−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
x1
x2
(x1, x2)y
1 (+1,+1) +1
2 (−1,+1) −1
3 (−1,−1) +1
4 (+1,−1) −1
p= 2 γ= 1 τ= 1
KP(x,x0) = <x,x0>+12
α∗
i, i = 1,...n
max
4
X
i=1
αi−1
2
4
X
i,j=1
αiαjyiyjKP(x,xi)
s.a.
4
X
i=1
αiyi= 0,0≤αi≤C, i = 1,...,4
α∗
i= 0.125, i = 1, . . . 4
i α∗
i= 0
α∗
D(x) =
n
X
i=1
α∗
iyiK(x,xi)=0.125
4
X
i=1
yiKP(x,xi)=0.125
4
X
i=1
yi[hx,xii+ 1]2

K(x,x0) =<Φ(x),Φ(x0)>=<x,x0>+12=
<(x1, x2),x0
1, x0
2>+12=
x2
1x0
12+x2
2x0
22+ 2x1x2x0
1x0
2+ 2x1x0
1+ 2x2x0
2+ 1 =
<1,√2x1,√2x2,√2x1x2, x2
1, x2
2,1,√2x0
1,√2x0
2,√2x0
1x0
2,x0
12,x0
22>
Φ2={φ1(x), . . . , φ6(x)}
φ1(x1, x2) = 1 φ2(x1, x2) = √2x1φ3(x1, x2) = √2x2,
φ4(x1, x2) = √2x1x2φ5(x1, x2) = x2
1φ6(x1, x2) = x2
2
D(x) = 0.125 ·P4
i=1 yiKP(x,xi)=0.125 ·P4
i=1 yi<Φ2(x),Φ2(xi)>=
0.125 · {[φ1(x) + √2φ2(x) + √2φ3(x) + √2φ4(x) + φ5(x) + φ6(x)]+
[(−φ1(x)) + √2φ2(x)−√2φ3(x) + √2φ4(x)−φ5(x)−φ6(x)]+
[φ1(x)−√2φ2(x)−√2φ3(x) + √2φ4(x) + φ5(x) + φ6(x)]+
[(−φ1(x)) −√2φ2(x) + √2φ3(x) + √2φ4(x)−φ5(x)−φ6(x)]}=
0.125 4√2φ4(x)=1
√2·φ4(x)
φ4(x)
# (x1, x2) (φ1(x), φ2(x), . . . , φ6(x)) y
1 (+1,+1) 1,√2,√2,√2,1,1+1
2 (−1,+1) 1,−√2,√2,−√2,1,1−1
3 (−1,−1) 1,−√2,−√2,√2,1,1+1
4 (+1,−1) 1,√2,−√2,−√2,1,1−1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
φ4(x)
← D(x)=0
D(x)=+1 → ← D(x)= −1
τ=√2τ=√2
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
x1
x2
← D(x1,x2)= +1D(x1,x2)= −1 →
D(x1,x2)= +1 → ← D(x1,x2)= −1
← D(x1,x2)= 0
↑
φ4(x) = √2
y= +1 φ4(x) = −√2y=−1
D(x) = 0
1
√2φ4(x)=0 ⇒φ4(x)=0
D(x) = +1 D(x) = −1
φ4(x)=+√2
φ4(x) = −√2
τ=√2
kw∗k
w∗α∗
i, i = 1, . . . 4
w∗=
4
X
i=1
α∗
iyixi=0,0,0,1
√2,0,0
xi, i = 1, . . . 4
τ=1
kw∗k=√2

x
φ4(x)
D(x) = x1x2
D(x)=0
x1x2= 0 ⇒


x1= 0
x2= 0
D(x) = +1 D(x) = −1
x1x2= +1 ⇒x2= 1/x1
x1x2=−1⇒x2=−1/x1
S={(x1, y1),...,(xn, yn)}
xi∈Rdyi∈RyiS
w= (w1, . . . , wd)b
f(x)=(w1x1+... +wdxd) + b=<w,x>+b
 L
2
L(y, f (x)) = 


0|y−f(x)| ≤ 
|y−f(x)| − 
±
ξ+
iξ−
i
ξ+
i>0
f(xi)yi
f(xi)−yi>  ξ−
i>0

ξ−
i, ξ+
j
L
 yi−f(xi)> 
ξ+
i>0ξ−
i>0
ξ+
i·ξ−
i= 0
min 1
2<w,w>+CPn
i=1 ξ+
i+ξ−
i
(<w,xi>+b)−yi−−ξ+
i≤0
yi−(<w,xi>+b)−−ξ−
i≤0
ξ+
i, ξ−
i≥0, i = 1, . . . , n
Lw, b, ξ+,ξ−,α+,α−,β+,β−=1
2<w,w>+CPn
i=1 Pn
i=1 ξ+
i+ξ−
i+
Pn
i=1 α+
i(<w,xi>+b)−yi−−ξ+
i+
Pn
i=1 α−
iyi−(<w,xi>+b)−−ξ−
i−
Pn
i=1 β+
iξ+
i−Pn
i=1 β−
iξ−
i

∂L
∂w≡w+
n
X
i=1
α+
ixi−
n
X
i=1
α−
ixi= 0
∂L
∂b ≡
n
X
i=1
α+
i−
n
X
i=1
α−
i= 0
∂L
∂ξ+
i≡C−α+
i−β+
i= 0, i = 1, . . . , n
∂L
∂ξ−
i≡C−α−
i−β−
i= 0, i = 1, . . . , n
α+
i(<w∗,xi>+b∗)−yi−−ξ+
i= 0, i = 1, . . . , n
α−
iyi−(<w∗,xi>+b∗)−−ξ−
i= 0, i = 1, . . . , n
β+
iξ+
i= 0, i = 1, . . . , n
β−
iξ−
i= 0, i = 1, . . . , n
w, b, ξ+,ξ−
α+,α−,β+,β−
w=
n
X
i=1 α−
i−α+
ixi
n
X
i=1 α+
i−α−
i= 0
β+
i=C−α+
i, i = 1, . . . , n
β−
i=C−α−
i, i = 1, . . . , n
L(α+,α−) = Pn
i=1 α−
i−α+
iyi−Pn
i=1 α−
i+α+
i−
1
2Pn
i,j=1 α−
i−α+
iα−
j−α+
j<xi,xj>
max Pn
i=1 α−
i−α+
iyi−Pn
i=1 α−
i+α+
i−
1
2Pn
i,j=1 α−
i−α+
iα−
j−α+
j<xi,xj>
Pn
i=1 α+
i−α−
i= 0
0≤α+
i, α−
i≤C, i = 1, . . . , n
α+
i≤C α+
i, β+
i≥0α−
i≤C
α−
i, β−
i≥0

f(x) =
n
X
i=1 α∗−
i−α∗+
i<x,xi>+b∗
α∗+α∗−
b∗
C−α+
iξ+
i= 0
C−α−
iξ−
i= 0
(xi, yi)
ξ−
i= 0 ξ+
i= 0
α+
i= 0 α−
i= 0
α+
i>0
(<w∗,xi>+b∗)−yi−−ξ+
i= 0 α+
i>0
α+
i< C
ξ+
i= 0 α+
i< C
(<w∗,xi>+b∗)−yi−= 0 0 < α+
i< C
α−
i>0α−
i< C
yi−(<w∗,xi>+b∗)−= 0 0 < α−
i< C
b∗
b∗=yi+−<w∗,xi>0< α+
i< C
b∗=yi−−<w∗,xi>0< α−
i< C
b∗
0< α+
i< C
α−
i= 0
0< α−
i< C α+
i= 0

(xi, yi)
 ξ−
i= 0 ξ+
i>0ξ−
i>0ξ+
i= 0 α+
i=C
α−
i=C
α+
iα−
i
α+
i=C α−
i= 0 α−
i=C α+
i= 0
α+
i=α−
i= 0
0< α+
i< C α−
i= 0
0< α−
j< C α+
j= 0 α+
i=C α−
i= 0
α−
j=C α+
j= 0
f(x) =
n
X
i=1 α∗−
i−α∗+
iK(x,xi)
b∗
φ(x)=1 α∗+
iα∗+
i
max Pn
i=1 α−
i−α+
iyi−Pn
i=1 α−
i+α+
i−
1
2Pn
i,j=1 α−
i−α+
iα−
j−α+
jK(xi,xj)
Pn
i=1 α+
i−α−
i= 0
0≤α+
i, α−
i≤C, i = 1, . . . , n
 C



C
C

light
light

Struct