{"id":289,"date":"2013-07-09T14:14:02","date_gmt":"2013-07-09T10:44:02","guid":{"rendered":"http:\/\/vua.nadiran.com\/?p=289"},"modified":"2013-07-09T15:12:45","modified_gmt":"2013-07-09T11:42:45","slug":"%d9%85%d8%ab%d8%a7%d9%84-%d9%87%d8%a7%db%8c-%d8%b1%da%af%d8%b1%d8%b3%db%8c%d9%88%d9%86-%d9%87%d8%a7%db%8c-lasso","status":"publish","type":"post","link":"https:\/\/vua.nadiran.com\/?p=289","title":{"rendered":"\u0645\u062b\u0627\u0644 \u0647\u0627\u06cc \u0631\u06af\u0631\u0633\u06cc\u0648\u0646 \u0647\u0627\u06cc Lasso"},"content":{"rendered":"

http:\/\/www.stanford.edu\/~boyd\/papers\/admm\/lasso\/lasso_example.html<\/a><\/p>\n

http:\/\/www-stat.stanford.edu\/~susan\/courses\/b494\/index\/node35.html<\/a><\/p>\n

http:\/\/www-stat.stanford.edu\/~susan\/courses\/s227\/node5.html<\/a><\/p>\n

http:\/\/www-stat.stanford.edu\/~susan\/courses\/b494\/index\/node29.html<\/a><\/p>\n

<\/a><\/section>\n
\n

lasso<\/h1>\n

<\/p>\n

\n

Regularized least-squares regression using lasso or elastic net algorithms<\/p>\n<\/div>\n<\/div>\n

Syntax<\/h2>\n
\n

B = lasso(X,Y)<\/p>\n

[B,FitInfo] = lasso(X,Y)<\/span><\/p>\n

[B,FitInfo] = lasso(X,Y,Name,Value)<\/span><\/p>\n<\/div>\n

<\/div>\n
<\/a><\/div>\n

Description<\/h2>\n

B<\/tt><\/a> = lasso(X<\/tt><\/a>,Y<\/tt><\/a>)<\/tt> returns fitted least-squares regression coefficients for a set of regularization coefficients Lambda<\/tt><\/a>.<\/p>\n

[B<\/tt><\/a>,FitInfo<\/tt><\/a>] = lasso(X<\/tt><\/a>,Y<\/tt><\/a>)<\/tt> returns a structure containing information about the fits.<\/p>\n

[B<\/tt><\/a>,FitInfo<\/tt><\/a>] = lasso(X<\/tt><\/a>,Y<\/tt><\/a>,Name,Value<\/tt><\/a>)<\/tt> fits regularized regressions with additional options specified by one or more Name,Value<\/tt> pair arguments.<\/p>\n

<\/a><\/p>\n

Input Arguments<\/h2>\n\n\n\n\n
<\/a>X<\/tt><\/td>\nNumeric matrix with n<\/tt> rows and p<\/tt> columns. Each row represents one observation, and each column represents one predictor (variable).<\/td>\n<\/tr>\n
<\/a>Y<\/tt><\/td>\nNumeric vector of length n<\/tt>, where n<\/tt> is the number of rows of X<\/tt>. Y(i)<\/tt> is the response to row i<\/tt> of X<\/tt>.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/a><\/p>\n

Name-Value Pair Arguments<\/h3>\n

<\/a><\/p>\n

\n

Specify optional comma-separated pairs of Name,Value<\/tt> arguments. Name<\/tt> is the argument name and Value<\/tt> is the corresponding value. Name<\/tt> must appear inside single quotes (' '<\/tt>). You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN<\/tt>.<\/p>\n<\/div>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/a>'Alpha'<\/tt><\/td>\nScalar value from 0<\/tt> to 1<\/tt> (excluding 0<\/tt>) representing the weight of lasso (L<\/i>1<\/sup>) versus ridge (L<\/i>2<\/sup>) optimization. Alpha\u00a0=\u00a0\u06f1<\/tt> represents lasso regression, Alpha<\/tt> close to 0<\/tt> approaches ridge regression, and other values represent elastic net optimization. See Definitions<\/a>.Default: <\/b>1<\/tt><\/td>\n<\/tr>\n
<\/a>'CV'<\/tt><\/td>\nMethod lasso<\/tt> uses to estimate mean squared error:<\/p>\n
<\/a>'DFmax'<\/tt><\/td>\nMaximum number of nonzero coefficients in the model. lasso<\/tt> returns results only for Lambda<\/tt> values that satisfy this criterion.Default: <\/b>Inf<\/tt><\/td>\n<\/tr>\n
<\/a>'Lambda'<\/tt><\/td>\nVector of nonnegative Lambda<\/tt> values. See Definitions<\/a>.<\/p>\n
    \n
  • If you do not supply Lambda<\/tt>, lasso<\/tt> calculates the largest value of Lambda<\/tt> that gives a nonnull model. In this case, LambdaRatio<\/tt> gives the ratio of the smallest to the largest value of the sequence, and NumLambda<\/tt> gives the length of the vector.<\/li>\n
  • If you supply Lambda<\/tt>, lasso<\/tt> ignores LambdaRatio<\/tt> and NumLambda<\/tt>.<\/li>\n<\/ul>\n

    Default: <\/b>Geometric sequence of NumLambda<\/tt> values, the largest just sufficient to produce B<\/tt>\u00a0=\u00a0\u06f0<\/tt><\/td>\n<\/tr>\n

<\/a>'LambdaRatio'<\/tt><\/td>\nPositive scalar, the ratio of the smallest to the largest Lambda<\/tt> value when you do not set Lambda<\/tt>.If you set LambdaRatio\u00a0=\u00a0\u06f0<\/tt>, lasso<\/tt> generates a default sequence of Lambda<\/tt> values, and replaces the smallest one with 0<\/tt>.Default: <\/b>1e-4<\/tt><\/td>\n<\/tr>\n
<\/a>'MCReps'<\/tt><\/td>\nPositive integer, the number of Monte Carlo repetitions for cross validation.<\/p>\n
    \n
  • If CV<\/tt> is 'resubstitution'<\/tt> or a cvpartition<\/tt> of type 'resubstitution'<\/tt>, MCReps<\/tt> must be 1<\/tt>.<\/li>\n
  • If CV<\/tt> is a cvpartition<\/tt> of type 'holdout'<\/tt>, MCReps<\/tt> must be greater than 1<\/tt>.<\/li>\n<\/ul>\n

    Default: <\/b>1<\/tt><\/td>\n<\/tr>\n

<\/a>'NumLambda'<\/tt><\/td>\nPositive integer, the number of Lambda<\/tt> values lasso<\/tt> uses when you do not set Lambda<\/tt>. lasso<\/tt> can return fewer than NumLambda<\/tt> fits if the if the residual error of the fits drops below a threshold fraction of the variance of Y<\/tt>.Default: <\/b>100<\/tt><\/td>\n<\/tr>\n
<\/a>'Options'<\/tt><\/td>\nStructure that specifies whether to cross validate in parallel, and specifies the random stream or streams. Create the Options<\/tt> structure with statset<\/tt><\/a>. Option fields:<\/p>\n
<\/a>'PredictorNames'<\/tt><\/td>\nCell array of strings representing names of the predictor variables, in the order in which they appear in X<\/tt>.Default: <\/b>{}<\/tt><\/td>\n<\/tr>\n
<\/a>'RelTol'<\/tt><\/td>\nConvergence threshold for the coordinate descent algorithm (see Friedman, Tibshirani, and Hastie [3]<\/a>). The algorithm terminates when successive estimates of the coefficient vector differ in the L<\/i>2<\/sup> norm by a relative amount less than RelTol<\/tt>.Default: <\/b>1e-4<\/tt><\/td>\n<\/tr>\n
<\/a>'Standardize'<\/tt><\/td>\nBoolean value specifying whether lasso<\/tt> scales X<\/tt> before fitting the models.Default: <\/b>true<\/tt><\/td>\n<\/tr>\n
<\/a>'Weights'<\/tt><\/td>\nObservation weights, a nonnegative vector of length n<\/tt>, where n<\/tt> is the number of rows of X<\/tt>. lasso<\/tt> scales Weights<\/tt> to sum to 1<\/tt>.Default: <\/b>1\/n * ones(n,1)<\/tt><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
<\/a><\/p>\n

Output Arguments<\/h2>\n\n\n\n\n
<\/a>B<\/tt><\/td>\nFitted coefficients, a p<\/tt>-by-L<\/tt> matrix, where p<\/tt> is the number of predictors (columns) in X<\/tt>, and L<\/tt> is the number of Lambda<\/tt> values.<\/td>\n<\/tr>\n
<\/a>FitInfo<\/tt><\/td>\nStructure containing information about the model fits.<\/p>\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n
Field in FitInfo<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
Intercept<\/tt><\/td>\nIntercept term \u03b2<\/i>\u06f0<\/sub> for each linear model, a 1<\/tt>-by-L<\/tt> vector<\/td>\n<\/tr>\n
Lambda<\/tt><\/td>\nLambda parameters in ascending order, a 1<\/tt>-by-L<\/tt> vector<\/td>\n<\/tr>\n
Alpha<\/tt><\/td>\nValue of Alpha<\/tt> parameter, a scalar<\/td>\n<\/tr>\n
DF<\/tt><\/td>\nNumber of nonzero coefficients in B<\/tt> for each value of Lambda<\/tt>, a 1<\/tt>-by-L<\/tt> vector<\/td>\n<\/tr>\n
MSE<\/tt><\/td>\nMean squared error (MSE), a 1<\/tt>-by-L<\/tt> vector<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

If you set the CV<\/tt> name-value pair to cross validate, the FitInfo<\/tt> structure contains additional fields.<\/p>\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n
Field in FitInfo<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
SE<\/tt><\/td>\nThe standard error of MSE for each Lambda<\/tt>, as calculated during cross validation, a 1<\/tt>-by-L<\/tt> vector<\/td>\n<\/tr>\n
LambdaMinMSE<\/tt><\/td>\nThe Lambda<\/tt> value with minimum MSE, a scalar<\/td>\n<\/tr>\n
Lambda1SE<\/tt><\/td>\nThe largest Lambda<\/tt> such that MSE is within one standard error of the minimum, a scalar<\/td>\n<\/tr>\n
IndexMinMSE<\/tt><\/td>\nThe index of Lambda<\/tt> with value LambdaMinMSE<\/tt>, a scalar<\/td>\n<\/tr>\n
Index1SE<\/tt><\/td>\nThe index of Lambda<\/tt> with value Lambda1SE<\/tt>, a scalar<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
<\/pre>\n
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<\/p>\n
%<\/p>\n
% File: Lasso.m<\/p>\n
% Author: Jinzhu Jia<\/p>\n
%<\/p>\n
% One simple (Gradiant Descent) implementation of the Lasso<\/p>\n
% The objective function is:<\/p>\n
% \u06f1\/\u06f2*sum((Y – X * beta -beta0).^2) + lambda * sum(abs(beta))<\/p>\n
% Comparison with CVX code is done<\/p>\n
% Reference: To be uploaded<\/p>\n
%<\/p>\n
% Version 4.23.2010<\/p>\n
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<\/p>\n
function [beta0,beta] = Lasso(X,Y,lambda)<\/p>\n
n = size(X,1);<\/p>\n
p = size(X,2);<\/p>\n
if(size(Y,1) ~= n)<\/p>\n
fprintf( ‘Error: dim of X and dim of Y are not match!!\\n’)<\/p>\n
quit cancel<\/p>\n
end<\/p>\n
beta0 = 0;<\/p>\n
beta= zeros(p,1);<\/p>\n
crit = 1;<\/p>\n
step = 0;<\/p>\n
while(crit > 1E-5)<\/p>\n
step = step + 1;<\/p>\n
obj_ini = 1\/2*sum((Y – X * beta -beta0).^2) + lambda * sum(abs(beta));<\/p>\n
beta0 = mean(Y – X*beta);<\/p>\n
for(j = 1:p)<\/p>\n
a = sum(X(:,j).^2);<\/p>\n
b = sum((Y – X*beta).* X(:,j)) + beta(j) * a;<\/p>\n
if lambda >= abs(b)<\/p>\n
beta(j) = 0;<\/p>\n
else<\/p>\n
if b>0<\/p>\n
beta(j) = (b – lambda) \/ a;<\/p>\n
else<\/p>\n
beta(j) = (b+lambda) \/a;<\/p>\n
end<\/p>\n
end<\/p>\n
end<\/p>\n
obj_now = 1\/2*sum((Y – X * beta -beta0).^2) + lambda * sum(abs(beta));<\/p>\n
crit = abs(obj_now – obj_ini) \/ abs(obj_ini);<\/p>\n
end<\/p>\n
————————————————————————-<\/p>\n
%% Lasso regularization example<\/p>\n
% Copyright (c) 2011, The MathWorks, Inc.<\/p>\n
%% Introduction to using LASSO<\/p>\n
using the lasso functionality introduced
\n% in R2011b. It is motivated by<\/p>\n
% This demo explains how to star<\/p>\n
tan example in Tibshiranis original paper<\/p>\n
% on the lasso.<\/pre>\n
via the lasso.
\n% J. Royal. Statist. Soc B., Vol. 58, No. 1, pages 267-288)<\/p>\n
% Tibshirani, R. (1996). Regression shrinkage and selection\r\n .\r\n\r\n% The data set that were working with in this demo is a wide<\/pre>\n
ferent
\n% variables and only 20 observations. 5 out of the 8 variables h<\/p>\n
% dataset with correlated variables. This data set includes 8 di\r\nfave\r\n% coefficients of zero. These variables have zero impact on the model. The<\/pre>\n
up workspace and set random seed<\/p>\n
clear all
\nclc<\/p>\n
% Set the random num<\/p>\n
% other three variables have non negative values and impact the model\r\n\r\n%% Clean\r\n ber stream\r\nrng(1981);\r\n\r\n%% Creating data set with specific characteristics\r\n\r\n% Create eight X variables<\/pre>\n
with one another
\n% The covariance matrix is spec<\/p>\n
% The mean of each variable will be equal to zero\r\nmu = [0 0 0 0 0 0 0 0];\r\n\r\n% The variable are correlate\r\ndified as\r\ni = 1:8;\r\nmatrix = abs(bsxfun(@minus,i',i));\r\ncovariance = repmat(.5,8,8).^matrix;<\/pre>\n
, covariance, 20);<\/p>\n
% Create a hyperplane that describes Y = f(X)
\nBeta = [3; 1<\/p>\n
% Use these parameters to generate a set of multivariate normal random numbers\r\nX = mvnrnd(m\r\nu.5; 0; 0; 2; 0; 0; 0];\r\nds = dataset(Beta);\r\n\r\n% Add in a noise vector\r\nY = X * Beta + 3 * randn(20,1);\r\n\r\n%% Use linear regression to fit the model \r\n\r\nb = regress(Y,X);<\/pre>\n
s, ‘PlotType’,<\/p>\n
ds.Linear = b;\r\n\r\n%% Use a lasso to fit the model\r\n\r\n[B Stats] = lasso(X,Y, 'CV', 5);\r\n\r\ndisp(B)\r\ndisp(Stats)\r\n\r\n%% Create a plot showing MSE versus lamba\r\n\r\nlassoPlot(B, Sta\r\nt 'CV')\r\n\r\n%% Identify a reasonable set of lasso coefficients\r\n\r\n% View the regression coefficients associated with Index1SE\r\n\r\nds.Lasso = B(:,Stats.Index1SE);\r\ndisp(ds)<\/pre>\n
(\u06f1\u06f0\u06f0,\u06f1);
\nCoeff_Num = zeros(100,1);
\nBetas = zeros(8,100);
\ncv<\/p>\n
%%  Create a plot showing coefficient values versus L1 norm\r\n\r\nlassoPlot(B, Stats)\r\n\r\n%%  Run a Simulation\r\n\r\n% Preallocate some variables\r\nMSE = zeros(100,1);\r\nmse = zero\r\ns_Reg_MSE = zeros(1,100);\r\n\r\nfor i = 1 : 100\r\n\r\n    X = mvnrnd(mu, covariance, 20);\r\n    Y = X * Beta + randn(20,1);\r\n\r\n    [B Stats] = lasso(X,Y, 'CV', 5);<\/pre>\n
E(i) = Stats.MSE(:, Shrink);<\/p>\n
regf = @(XTRAIN, ytrain, XTEST)(XTEST*<\/p>\n
    Shrink = Stats.Index1SE -  ceil((Stats.Index1SE - Stats.IndexMinMSE)\/2);\r\n    Betas(:,i) = B(:,Shrink) > 0;\r\n    Coeff_Num(i) = sum(B(:,Shrink) > 0);\r\n    M\r\nSregress(ytrain,XTRAIN));\r\n    cv_Reg_MSE(i) = crossval('mse',X,Y,'predfun',regf, 'kfold', 5);\r\n\r\nend\r\n\r\nNumber_Lasso_Coefficients = mean(Coeff_Num);\r\ndisp(Number_Lasso_Coefficients)\r\n\r\nMSE_Ratio = median(cv_Reg_MSE)\/median(MSE);<\/pre>\n
disp(MSE_Ratio)<\/p>\n","protected":false},"excerpt":{"rendered":"
http:\/\/www.stanford.edu\/~boyd\/papers\/admm\/lasso\/lasso_example.html http:\/\/www-stat.stanford.edu\/~susan\/courses\/b494\/index\/node35.html http:\/\/www-stat.stanford.edu\/~susan\/courses\/s227\/node5.html http:\/\/www-stat.stanford.edu\/~susan\/courses\/b494\/index\/node29.html lasso Regularized least-squares regression using lasso or elastic net algorithms Syntax B = lasso(X,Y) [B,FitInfo] = lasso(X,Y) [B,FitInfo] = lasso(X,Y,Name,Value) Description B = lasso(X,Y) returns fitted least-squares regression coefficients for a set of regularization coefficients Lambda. [B,FitInfo] = lasso(X,Y) returns a structure containing information about the fits. [B,FitInfo] = lasso(X,Y,Name,Value) fits […]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19],"tags":[],"class_list":["post-289","post","type-post","status-publish","format-standard","hentry","category-19","category-19-id","post-seq-1","post-parity-odd","meta-position-corners","fix"],"_links":{"self":[{"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/posts\/289"}],"collection":[{"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=289"}],"version-history":[{"count":5,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/posts\/289\/revisions"}],"predecessor-version":[{"id":291,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=\/wp\/v2\/posts\/289\/revisions\/291"}],"wp:attachment":[{"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=289"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=289"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vua.nadiran.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=289"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}