No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Ridge regression in two-parameter solution: Research Articles
Abstract
We consider simultaneous minimization of the model errors, deviations from orthogonality between regressors and errors, and deviations from other desired properties of the solution. This approach corresponds to a regularized objective that produces a consistent solution not prone to multicollinearity. We obtain a generalization of the ridge regression to two-parameter model that always outperforms a regular one-parameter ridge by better approximation, and has good properties of orthogonality between residuals and predicted values of the dependent variable. The results are very convenient for the analysis and interpretation of the regression. Numerical runs prove that this technique works very well. The examples are considered for marketing research problems. Copyright © 2005 John Wiley & Sons, Ltd.
In our previous studies, although we had small data sets for our applications, the
uncertainty matrices for the input data had a huge size since vertices were too many to handle. Consequently, we had no enough computer capacity to solve our problems for those uncertainty matrices. To overcome this difficulty, we obtained different weak RMARS (WRMARS) models for all sample values (observations) applying a combinatorial approach and solved them by using MOSEK program. Indeed, we have a trade-off between tractability and robustification. In this presentation, we present a more robust model using cross-polytope and demonstrate its performance with the application of Natural Gas consumption prediction. Applying robustification in MARS, we aim to reduce the estimation variance.
A generalization of the one-parameter ridge regression to a two-parameter model and its asymptotic behavior, which has various better fitting characteristics, is considered. For the two-parameter model, the coefficients of regression, their t-statistics and net effects, the residual variance and coefficient of multiple determination, the characteristics of bias, efficiency, and generalized cross-validation are very stable and as the ridge parameter increases they eventually reach asymptotic levels. In particular, the beta coefficients of multiple two-parameter ridge regression models converge to a solution proportional to the coefficients of paired regressions. The suggested technique produces robust regression models not prone to multicollinearity, with interpretable coefficients and other characteristics convenient for analysis of the models.
With a simple transformation, the ordinary least squares objective can yield a family of modified ridge regressions which outperforms the regular ridge model. These models have more stable coefficients and a higher quality of fit with the growing profile parameter. With an additional adjustment based on minimization of the residual variance, all the characteristics become even better: the coefficients of these regressions do not shrink to zero when the ridge parameter increases, the coefficient of multiple determination stays high, while bias and generalized cross-validation are low. In contrast to regular ridge regression, the modified ridge models yield robust solutions with various values of the ridge parameter, encompass interpretable coefficients, and good quality characteristics.
Consider the ridge estimate (λ) for β in the model unknown, (λ) = (XX + nλI) Xy. We study the method of generalized cross-validation (GCV) for choosing a good value for λ from the data. The estimate is the minimizer of V(λ) given bywhere A(λ) = X(XX + nλI) X . This estimate is a rotation-invariant version of Allen's PRESS, or ordinary cross-validation. This estimate behaves like a risk improvement estimator, but does not require an estimate of σ, so can be used when n − p is small, or even if p ≥ 2 n in certain cases. The GCV method can also be used in subset selection and singular value truncation methods for regression, and even to choose from among mixtures of these methods.
Working with multiple regression analysis a researcher usually wants to know a comparative importance of predictors in the model. However, the analysis can be made difficult because of multicollinearity among regressors, which produces biased coefficients and negative inputs to multiple determination from presum ably useful regressors. To solve this problem we apply a tool from the co-operative games theory, the Shapley Value imputation. We demonstrate the theoretical and practical advantages of the Shapley Value and show that it provides consistent results in the presence of multicollinearity. Copyright © 2001 John Wiley & Sons, Ltd.
Multiple regression analysis is one of the most widely used statistical procedures for both scholarly and applied marketing research. Yet, correlated predictor variables—and potential collinearity effects—are a common concern in interpretation of regression estimates. Though the literature on ways of coping with collinearity is extensive, relatively little effort has been made to clarify the conditions under which collinearity affects estimates developed with multiple regression analysis—or how pronounced those effects are. The authors report research designed to address these issues. The results show, in many situations typical of published cross-sectional marketing research, that fears about the harmful effects of collinear predictors often are exaggerated. The authors demonstrate that collinearity cannot be viewed in isolation. Rather, the potential deleterious effect of a given level of collinearity should be viewed in conjunction with other factors known to affect estimation accuracy.
Master linear regression techniques with a new edition of a classic text Reviews of the Second Edition: "I found it enjoyable reading and so full of interesting material that even the well-informed reader will probably find something new . . . a necessity for all of those who do linear regression." -Technometrics, February 1987 "Overall, I feel that the book is a valuable addition to the now considerable list of texts on applied linear regression. It should be a strong contender as the leading text for a first serious course in regression analysis."-American Scientist, May-June 1987 Applied Linear Regression, Third Edition has been thoroughly updated to help students master the theory and applications of linear regression modeling. Focusing on model building, assessing fit and reliability, and drawing conclusions, the text demonstrates how to develop estimation, confidence, and testing procedures primarily through the use of least squares regression. To facilitate quick learning, the Third Edition stresses the use of graphical methods in an effort to find appropriate models and to better understand them. In that spirit, most analyses and homework problems use graphs for the discovery of structure as well as for the summarization of results. The Third Edition incorporates new material reflecting the latest advances, including: Use of smoothers to summarize a scatterplot Box-Cox and graphical methods for selecting transformations Use of the delta method for inference about complex combinations of parameters Computationally intensive methods and simulation, including the bootstrap method Expanded chapters on nonlinear and logistic regression Completely revised chapters on multiple regression, diagnostics, and generalizations of regression Readers will also find helpful pedagogical tools and learning aids, including: More than 100 exercises, most based on interesting real-world data Web primers demonstrating how to use standard statistical packages, including R, S-Plus, SPSS, SAS, and JMP, to work all the examples and exercises in the text A free online library for R and S-Plus that makes the methods discussed in the book easy to use With its focus on graphical methods and analysis, coupled with many practical examples and exercises, this is an excellent textbook for upper-level undergraduates and graduate students, who will quickly learn how to use linear regression analysis techniques to solve and gain insight into real-life problems.
Consider the standard linear model . Ridge regression, as viewed here, defines a class of estimators of indexed by a scalar parameter k. Two analytic methods of specifying k are proposed and evaluated in terms of mean square error by Monte Carlo simulations. With three explanatory variables and determined by the largest eigenvalue of the correlation matrix, least squares is dominated by these estimators in all cases investigated; however, mixed results are obtained with determined by the smallest eigenvalue. These estimators compare favorably with other ridge-type estimators evaluated elsewhere for two explanatory variables.
Relations between pairwise correlations and the coefficient of multiple determination in regression analysis are considered. The conditions for the occurrence of enhance-synergism and suppression effects when multiple determination becomes bigger than the total of squared correlations of the dependent variable with the regressors are discussed. It is shown that such effects can occur just for stochastic relations among the variables that have non-transitive signs of pairwise correlations. Consideration of these problems facilitates better understanding the properties of regression.
Smoothing splines are well known to provide nice curves which smooth discrete, noisy data. We obtain a practical, effective method for estimating the optimum amount of smoothing from the data. Derivatives can be estimated from the data by differentiating the resulting (nearly) optimally smoothed spline. We consider the model yi(ti)+εi, i=1, 2, ..., n, ti∈[0, 1], where g∈W2(m)={f:f, f′, ..., f(m-1) abs. cont., f(m)∈ℒ2[0,1]}, and the {εi} are random errors with Eεi=0, Eεiεj=σ2δij. The error variance σ2 may be unknown. As an estimate of g we take the solution gn, λ to the problem: Find f∈W2(m) to minimize {Mathematical expression}. The function gn, λ is a smoothing polynomial spline of degree 2 m-1. The parameter λ controls the tradeoff between the "roughness" of the solution, as measured by {Mathematical expression}, and the infidelity to the data as measured by {Mathematical expression}, and so governs the average square error R(λ; g)=R(λ) defined by {Mathematical expression}. We provide an estimate {Mathematical expression}, called the generalized cross-validation estimate, for the minimizer of R(λ). The estimate {Mathematical expression} is the minimizer of V(λ) defined by {Mathematical expression}, where y=(y1, ..., yn)t and A(λ) is the n×n matrix satisfying (gn, λ (t1), ..., gn, λ (tn))t=A (λ) y. We prove that there exist a sequence of minimizers {Mathematical expression} of EV(λ), such that as the (regular) mesh {ti}i=1n becomes finer, {Mathematical expression}. A Monte Carlo experiment with several smooth g's was tried with m=2, n=50 and several values of σ2, and typical values of {Mathematical expression} were found to be in the range 1.01-1.4. The derivative g′ of g can be estimated by {Mathematical expression}. In the Monte Carlo examples tried, the minimizer of {Mathematical expression} tended to be close to the minimizer of R(λ), so that {Mathematical expression} was also a good value of the smoothing parameter for estimating the derivative.
We apply generalized cross-validation (GCV) as a stopping rule for general linear stationary iterative methods for solving very large-scale, ill-conditioned problems. We present a new general formula for the influence operator for these methods and, using this formula and a Monte Carlo approach, we show how to compute the GCV function at a cheaper cost. Then we apply our approach to a well known iterative method (ART) with simulated data in positron emission tomography (PET).
We propose a dual- and triple-mode least squares for matrix approximation. This technique applied to the singular value decomposition produces the classical solution with a new interpretation. Applied to regression modelling, this approach corresponds to a regularized objective and yields a new solution with properties of a ridge regression. The results for regression are robust and suggest a convenient tool for the analysis and interpretation of the model coefficients. Numerical results are given for a marketing research data set. Copyright © 2003 John Wiley & Sons, Ltd.
In this work we develop a new multivariate technique to produce regressions with interpretable coefficients that are close to and of the same signs as the pairwise regression coefficients. Using a multiobjective approach to incorporate multiple and pairwise regressions into one objective we reduce this technique to an eigenproblem that represents a hybrid between regression and principal component analyses. We show that our approach corresponds to a specific scheme of ridge regression with a total matrix added to the matrix of correlations.Scope and purposeOne of the main goals of multiple regression modeling is to assess the importance of predictor variables in determining the prediction. However, in practical applications inference about the coefficients of regression can be difficult because real data is correlated and multicollinearity causes instability in the coefficients. In this paper we present a new technique to create a regression model that maintains the interpretability of the coefficients. We show with real data that it is possible to generate a model with coefficients that are similar to easily interpretable pairwise relations of predictors with the dependent variable, and this model is similar to the regular multiple regression model in predictive ability.
Regression data sets typically have many more cases than variables, but this is not always the case. Some current problems in chemometrics—for example fitting quantitative structure activity relationships—may involve fitting linear models to data sets in which the number of predictors far exceeds the number of cases. Ridge regression is an approach that has some theoretical foundation and has performed well in comparison with alternatives such as PLS and subset regression. Direct implementation of the regression formulation leads to a O(np2+p3) calculation, which is substantial if p is large. We show that ridge regression may be performed in a O(np2) computation—a potentially large saving when p is larger than n. The algorithm lends itself to the use of case weights, to robust bounded influence fitting, and cross-validation. The method is illustrated with a chemometric data set with 255 predictors, but only 18 cases, a ratio not unusual in QSAR problems.
In multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory, if not incorrect, if the prediction vectors are not orthogonal. Proposed is an estimation procedure based on adding small positive quantities to the diagonal of X′X. Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X′X to obtain biased estimates with smaller mean square error.
Managing multicollinearity
- A Grapentine
Grapentine, A. (1997), "Managing multicollinearity", Marketing Research, 9, 11-21.
base -length of wheelbase, in inches, as supplied by manufacturer
- Wheel
Wheel.base -length of wheelbase, in inches, as supplied by manufacturer;
-distance between the car's head-liner and the head of a 5 ft. 9 in. front seat passenger, in inches, as measured by CU
- Front
- Hd
Front.Hd. -distance between the car's head-liner and the head of a 5 ft. 9 in. front seat
passenger, in inches, as measured by CU (Consumers Union Reports);
Hd -distance between the car's head-liner and the head of a 5 ft 9 in. rear seat passenger, in inches, as measured by CU
- Rear
Rear.Hd -distance between the car's head-liner and the head of a 5 ft 9 in. rear seat
passenger, in inches, as measured by CU;
Room maximum front leg room, in inches, as measured by CU
- Frt
- Leg
Frt.Leg.Room maximum front leg room, in inches, as measured by CU;
Seating -rear fore-and-aft seating room, in inches, as measured by CU
- Rear
Rear.Seating -rear fore-and-aft seating room, in inches, as measured by CU;
Shld -front shoulder room, in inches, as measured by CU
- Frt
Frt.Shld -front shoulder room, in inches, as measured by CU;