Two Explicit Characterizations of the General Nonnegative-Definite Covariance Matrix Structure for Equality of BLUEs, WLSEs, and LSEs

Abstract

We provide a new, concise derivation of necessary and sufficient conditions for the explicit characterization of the general nonnegative-definite covariance structure V of a general Gauss-Markov model with E(y) and Var(y) such that the best linear unbiased estimator, the weighted least squares estimator, and the least squares estimator of Xβ are identical. In addition, we derive a representation of the general nonnegative-definite covariance structure V defined above in terms of its Moore-Penrose pseudo-inverse.

Full text

http://ijsp.ccsenet.org International Journal of Statistics and Probability Vol. 9, No. 6; 2020
Two Explicit Characterizations of the General Nonnegative-Definite
Covariance Matrix Structure for Equality of BLUEs, WLSEs, and
LSEs
Phil D. Young1, Joshua D. Patrick2& Dean M. Young2
1Department of Information Systems and Business Analytics, Baylor University, Waco, TX
2Department of Statistical Science, Baylor University, Waco, TX
Correspondence: Phil D. Young, Department of Information Systems and Business Analytics, Baylor University, Waco,
TX 76798, USA. Tel: 1-254-710-7394. E-mail: philip young@baylor.edu
Received: June 30, 2020 Accepted: October 11, 2020 Online Published: October 21, 2020
doi:10.5539/ijsp.v9n6p108 URL: https://doi.org/10.5539/ijsp.v9n6p108
Abstract
We provide a new, concise derivation of necessary and sufficient conditions for the explicit characterization of the general
nonnegative-definite covariance structure Vof a general Gauss-Markov model with E(y) and Var(y) such that the best
linear unbiased estimator, the weighted least squares estimator, and the least squares estimator of Xβare identical. In
addition, we derive a representation of the general nonnegative-definite covariance structure Vdefined above in terms of
its Moore-Penrose pseudo-inverse.
Keywords: matrix equations, orthogonal-projection matrices, matrix column space, matrix rank, Moore-Penrose pseudo-
inverse
1. Introduction
We consider the general Gauss-Markov model
y=Xβ+,(1)
where yis an n×1 vector of observations, Xis an n×pknown fixed, non-null model (design) matrix such that rank(X)=p,
βis a p×1 vector of unknown model parameters, and is an n×1 vector of random perturbations such that E()=0n×1
and Var()=V, where Vis a known n×nnon-null, symmetric nonnegative-definite (n.n.d.) matrix. We denote the
Gauss-Markov model defined above by y,Xβ,V, and we assume y∈ C(X:V), where C(X:V) represents the column
space of the partitioned matrix (X:V).
Throughout the remainder of this paper, the notation Rm×nrepresents the vector space of all m×nmatrices over the real
field R,RS
ndenotes the set of n×nreal symmetric matrices, R≥
nrepresents the cone of all symmetric n.n.d. matrices in
Rn×n, and R>
ndenotes the interior of R≥
n, which is the set of all symmetric positive-definite (p.d.) matrices. We use the
notation K0to denote the transpose of the real matrix K∈Rm×n. Furthermore, we let K+∈Rn×mand K−∈Rn×mrepresent
the Moore-Penrose pseudo-inverse and a generalized inverse of K, respectively. Also, for K∈Rm×n, we use the notation
PKand P⊥
Kto denote the orthogonal projection matrix onto C(K) and C(K)⊥, respectively.
Given X, we define the ordinary least squares (LS) estimator of Xβas
Xˆ
βLS =XX0X−X0y.
Puntanen, Styan, and Isotalo (2011) have defined the best linear unbiased (BLU) estimator of Xβas
Xˆ
βBLU =XX0T−X−X0T−y,(2)
where T=V+XU0Xand U∈RS
nis any n×nmatrix such that C(T)=C(X:V). Puntanen et al. (2011) have defined the
weighted least squares (WLS) estimator as
Xˆ
βWLS =XX0V+X−X0V+y.
In this paper we give two characterizations of the general n.n.d. error covariance structure Vin the Gauss-Markov model
{y,Xβ,V}for which Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS where y∈ C (X:V). We define these covariance matrices to be BLU-WLS-
LS estimator-equivalent (e.e.) covariance matrices. Specifically, in the first characterization we give a derivation of the
108

http://ijsp.ccsenet.org International Journal of Statistics and Probability Vol. 9, No. 6; 2020
explicit general n.n.d. BLU-WLS-LS e.e. covariance structure that is considerably more concise and straight-forward than
the derivation given in Young, Odell and Hahn (2000). In the second characterization, we demonstrate that the Moore-
Penrose pseudo-inverse of the covariance matrices contained in the set of n.n.d. BLU-WLS-LS e.e. covariance structures
are themselves elements of the set.
A large majority of previous work has focused on implicitly and explicitly characterizing the general covariance matrix
Vsuch that the BLU and LS estimators are equal. Puntanen and Styan (1989), Alalouf and Styan (1984), Tian and Wiens
(2006), Proposition 10.1 in Puntanen et al. (2011), and numerous additional journal articles have presented many of these
implicit characterizations.
However, we have found fewer results on explicit n.n.d. WLS-LS e.e. covariance structure characterizations. Plackett
(1960), McElroy (1967), and Williams (1967) have derived sufficient (p.d.)WLS-LS e.e. covariance matrices. Additionally,
for certain model matrices X, Herzberg and Aleong (1985) have presented a sufficient p.d. WLS-LS e.e. covariance matrix,
and Zyskind and Martin (1969), Searle (1994), and Tian and Wiens (2006) have presented several implicit WLS-LS e.e.
covariance-structure characterizations. Results on both implicit and explicit characterizations of the general n.n.d. BLU-
WLS-LS e.e. covariance structure for the Gauss-Markov model {y,Xβ,V}appear to be more sparse. Herzberg and Aleong
(1985) have presented two sufficient WLS-BLU-LS e.e. covariance matrices. Moreover, Baksalary and Kala (1983) have
given an implicit characterization of the general n.n.d. e.e. covariance structure for V, and Young et al. (2000) have
explicitly characterized the general n.n.d. BLU-WLS-LS e.e. covariance structure.
We have organized the remainder of the paper as follows. In Section 2 we state two lemmas that we use to derive the first
of our two theorems. In Section 3 we present a new concise derivation our general n.n.d. BLU-WLS-LS e.e. dependency
structure characterization for V. We also demonstrate that the Moore-Penrose pseudo-inverse of elements contained in
the set of n.n.d. BLU-WLS-LS e.e. covariance structures are themselves elements of this set. Last, in Section 4 we briefly
summarize the two characterization results proven here.
2. Preliminary Lemmas
We next present two lemmas that we use in the proof of our first e.e.-covariance-structure characterization. The first
lemma gives conditions for Vsuch that Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS . A proof of part a) is in the lemma in Zyskind (1967),
a proof of b) is in Zyskind and Martin (1969), and a proof of part c) is in Theorem 2.2 of Baksalary and Kala (1983).
Lemma 1. For the Gauss-Markov model y,Xβ,V, we have
a) Xˆ
βBLU =Xˆ
βWLS if and only if C(X)⊂ C(V),
b) Xˆ
βBLU =Xˆ
βLS if and only if C(VX)⊂ C(X), and
c) Xˆ
βWLS =Xˆ
βLS if and only if Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS .
In the second lemma, we state the general symmetric n.n.d. solution matrix to a particular homogeneous matrix that
contains the column space of a specified matrix.
Lemma 2. Let A∈Rn×qsuch that rank(A)=k,where k≤q<n, and let U:=U∈R≥
n:C(A)⊂ C(U). Then, a
representation of the general n.n.d. solution to PAZ P ⊥
A=0such that C(A)⊂ C(Z) is
Z=PAU1PA+P⊥
AU2P⊥
A,
where U1∈Uand U2∈R≥
nis arbitrary.
Proof. The proof is similar to the proof of Lemma 6 in Young et al. (2000).
3. Main Results
We now present a concise proof of the explicit characterization of the general n.n.d. BLU-WLS-LS e.e. covariance
structure. The proof immediately below is considerably shorter and more direct than a previous proof given in Young et
al. (2000).
Theorem 1. For the general Gauss-Markov model y,Xβ,V, we have Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS if and only if V∈V,
where
V:=nV∈R≥
n:V=PXW1PX+P⊥
XW2P⊥
Xo(3)
with
W1∈nW∈R≥
n:C(X)⊂ C(Wo,(4)
and W2∈R≥
nis arbitrary.
109

http://ijsp.ccsenet.org International Journal of Statistics and Probability Vol. 9, No. 6; 2020
Proof. From Lemmas 1 and 2, we have that
Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS ⇐⇒ PXVX =V X and PVX=X
⇐⇒ PXVX −V X =0and PVX=X
⇐⇒ (PX−I)V PX=0and PVX=X
⇐⇒ P⊥
XV PX=0and VV+X=X
⇐⇒ V∈V, where Vis given in (3).2
Next, for the general Gauss-Markov model y,Xβ,V, we characterize the n.n.d. e.e. covariance matrices V∈V, defined
in (3), by showing that for V∈V, the Moore-Penrose inverse V+has a particular form.
Theorem 2. For the general Gauss-Markov model y,Xβ,V, consider the covariance matrices V∈Vdefined in (3).
Then, V∈Vif and only if V+∈V.
Proof. We first prove the necessity portion of Theorem 2. Let V∈Vbe defined as in (3). In addition, let
V∗=PXW+
1PX+P⊥
XW+
2P⊥
X.
Then, using the definition of a Moore-Penrose pseudo-inverse and the facts that for Wi∈R≥
n,PXP⊥
X=P⊥
XPX=0, and
PXWi=WiPX=Wi,i=1,2,we have
VV∗V=PXW1PX+P⊥
XW2P⊥
XPXW+
1PX+P⊥
XW+
2P⊥
XPXW1PX+P⊥
XW2P⊥
X
=PXW1PXPXW+
1PXPXW1PX+P⊥
XW2P⊥
XP⊥
XW+
2P⊥
XP⊥
XW2P⊥
X
=PXW1W+
1W1PX+P⊥
XW2W+
2W2P⊥
X
=PXW1PX+P⊥
XW2P⊥
X
=V.
Next, we have
V∗VV∗=PXW+
1PX+P⊥
XW+
2P⊥
XPXW1PX+P⊥
XW2P⊥
XPXW+
1PX+P⊥
XW+
2P⊥
X
=PXW+
1PXPXW1PXPXW+
1PX+P⊥
XW+
2P⊥
XP⊥
XW2P⊥
XP⊥
XW+
2P⊥
X
=PXW+
1W1W+
1PX+P⊥
XW+
2W2W+
2P⊥
X
=PXW1PX+P⊥
XW2P⊥
X
=V∗.
Third, let W1be defined as in (4). Then, using the fact that WiW+
i=big(WiW+
i0=W0+
iW0
i=W+
iWi,i=1,2,we have
VV∗0=hPXW1PX+P⊥
XW2P⊥
XPXW+
1PX+P⊥
XW+
2P⊥
Xi0
=PXW+
1PX+P⊥
XW+
2P⊥
X0PXW1PX+P⊥
XW2P⊥
X0
=PXW+
1PXPXW1PX+P⊥
XW2P⊥
XP⊥
XW+
2P⊥
X.
=PXW+
1W1PX+P⊥
XW+
2W2P⊥
X
=PXW1W+
1PX+P⊥
XW2W+
2P⊥
X
=PXW1PXPXW+
1PX+P⊥
XW2P⊥
XP⊥
XW+
2P⊥
X.
=PXW1PX+P⊥
XW2P⊥
XPXW1PX+P⊥
XW2P⊥
X.
=VV∗.
Last, again using the fact that WiW+
i=W+
iWi,i=1,2,we have that
V∗V0=hPXW+
1PX+P⊥
XW+
2P⊥
XPXW1PX+P⊥
XW2P⊥
Xi0
=PXW1PX+P⊥
XW2P⊥
X0PXW+
1PX+P⊥
XW+
2P⊥
X0
=PXW1PXPXW+
1PX+P⊥
XW2P⊥
XP⊥
XW+
2P⊥
X.
=PXW1W+
1PX+P⊥
XW2W+
2P⊥
X
=PXW+
1W1PX+P⊥
XW+
2W2P⊥
X
=PXW+
1PXPXW1PX+P⊥
XW+
2P⊥
XP⊥
XW2P⊥
X.
=PXW+
1PX+P⊥
XW2P⊥
XPXW1PX+P⊥
XW2P⊥
X.
=V∗V.
110

http://ijsp.ccsenet.org International Journal of Statistics and Probability Vol. 9, No. 6; 2020
Hence, V∗=V+. The sufficiency portion of the proof is similar to the necessity portion because of the facts that [V+]+=V
and [Wi+]+=Wi,i=1,2.2
The following corollary, which follows directly from Theorems 1 and 2, gives several implicit characterizations of the
general n.n.d. BLU-WLS-LS e.e. dependency matrix.
Corollary. Let Vbe defined as in (3). Then, V∈Vif and only if C(X)⊂ C(V), and
a) P⊥
XV P⊥
X=P⊥
XV
b) P⊥
XV+P⊥
X=P⊥
XV+
c) PXV PX=PXV
d) PXV+PX=PXV+
e) PXV=V PX
f) PXV+=V+PX
g) P⊥
XV=V P⊥
X
h) P⊥
XV+=V+P⊥
X.
4. Summary
We have derived two explicit characterizations of the general n.n.d. e.e. covariance structure such that Xˆ
βBLU =Xˆ
βWLS =
Xˆ
βLS . Theorem 1 provides a brief derivation of the explicit general n.n.d. BLU-WLS-LS e.e. dependency structure that
considerably shortens a proof given in Young et al. (2000). Theorem 2 presents a second characterization of the general
n.n.d. BLU-WLS-LS e.e. covariance matrix Vin which we prove that Vand V+have the same general structure. Last, we
give some implicit characterizations of the general n.n.d. e.e. covariance matrices such that Xˆ
βBLU =Xˆ
βWLS =Xˆ
βLS .
Acknowledgements
We wish to thank Joy L. Young for her help in the writing of this paper.
References
Alalouf, I. S., & Styan, G. P. (1984). Characterizations of the conditions for the ordinary least squares estimator to be
best linear unbiased. In Y. P. Chaubey, & T. D. Dwivedi (Eds.), Topics in Applied Statistics, Dept. of Mathematics,
Concordia Univ., Montreal, (pp. 331-344).
Baksalary, J., & Kala, R. (1983). On equalities between BLUEs, WLSEs, and SLSEs. Canadian Journal of Statistics, 11,
119-123. https://doi.org/10.2307/3314978
Herzberg, A. M., & Aleong, J. (1985). Further conditions on the equivalence of ordinary least squares and weighted least
squares estimators with examples. In J. Lanke & G. Lindgren (Eds.), In Contributions to Probability and Statistics
in Honour of Gunnar Blom, University of Lund, (pp. 127-142).
McElroy, F. W. (1967). A necessary and sufficient condition that ordinary least-squares estimators be best linear unbiased.
Journal of the American Statistical Association, 62, 1302-1304. https://doi.org/10.1080/01621459.1967.10500935
Plackett, R. L. (1960). Principles of Regression Analysis. Clarendon Press, Oxford.
Puntanen, S., & Styan, G. P. (1989). The equality of the ordinary least squares estimator and the best linear unbiased
estimator. American Statistician, 43, 153-161. https://doi.org/10.1080/00031305.1989.10475644
Puntanen, S., Styan, G. P., & Isotalo, J. (2011). Matrix Tricks for Linear Models. Springer, New York.
Searle, S. R. (1994). Extending some results and proofs for the singular linear model. Linear Algebra Appl., 210, 139-151.
https://doi.org/10.1016/0024-3795(94)90469-3
Tian, Y., & Wiens, D. P. (2006). On equality and proportionality of ordinary least squares, weighted least squares,
and best linear unbiased estimators in the general linear model. Statistics and Probability Letters, 76, 1265-1272.
https://doi.org/10.1016/j.spl.2006.01.005
Williams, J. S. (1967). The variance of weighted estimators. Journal of the American Statistical Association, 62, 1290-
1301. https://doi.org/10.1080/01621459.1967.10500934
111

http://ijsp.ccsenet.org International Journal of Statistics and Probability Vol. 9, No. 6; 2020
Young, D. M., Odell, P. L., & Hahn, W. (2000). Nonnegative-definite covariance structures for which the BLU, WLS, and
LS estimators are equal. Statistics and Probability Letters, 49, 271-276. https://doi.org/10.1016/S0167-7152(00)000-
572
Zyskind, G. (1967). On canonical forms, nonnegative covariance matrices and best and simple least squares linear estima-
tors in the linear model. Annals of Mathematical Statistics, 38, 1092-1109. https://doi.org/10.1214/aoms/1177698779
Zyskind, G., & Martin, F. B. (1969). On best linear estimation and a general Gauss-Markov theorem in linear model with
arbitrary nonnegative covariance structure. SIAM Journal on Applied Mathematics, 17, 1190-1202. https://doi.org/10-
.1137/0117110
Copyrights
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license
(http://creativecommons.org/licenses/by/4.0/).
112