Just posted this and I was wondering if it's got any serious problems 
(other than lack of examples):

http://www.geocities.com/evilsnack/least_squares.pdf

In other news, OpenOffice's formula editor is rather nifty.

Regards,
John

Post a reply to this message

John VanSickle <evi### [at] hotmailcom> wrote:
> Just posted this and I was wondering if it's got any serious problems
> (other than lack of examples):
>
> http://www.geocities.com/evilsnack/least_squares.pdf
>
> In other news, OpenOffice's formula editor is rather nifty.

I can find no errors, and I read it over fairly carefully.  I don't know who
your audience is, but it couldn't hurt to emphasize that (2) is a sort of total
error to be minimized, and to see the resulting matrix equations put together at
the end.  Of course neither of these things are really necessary.

Since you had us proofread it though, it's my turn to rant about least squares.
Perhaps you know this already, but I'm just enjoying the review out loud.  This
is the derivation I had always seen, but last fall I was introduced to a neat
alternative version:

Think of a point lying off a line.  (Not a data point, but instead a point in
the space of the coefficients you're trying to determine.)  How close along the
line can we get to that point?  Well, 'squares' just means Euclidian distance,
so it's along a second line perpendicular to the first, of course.

So if it's a linear problem in the coefficients, the residual is just

r = b-Ax,

 where A is a matrix full of the basis functions evaluated at all points, x is
the unknown coefficients, and b is the data points.  A does not need to be
square!  In fact it will be m x n, where m is the number of data points and n
is the number of basis functions.

If the problem is solved, then the length of r has been minimized so that Ax is
as close to b as possible.  This corresponds to the situation above where we
can't actually get the point onto the line, so instead we have to wander off
perpendicular to the line until we hit a point we can use.  It turns out the
condition for orthogonality is

A* r = 0,

where * is just an adjoint (or transpose), and plugging in r,

A* A x = A* b

Solving for x,

x = (A* A)^(-1) A* b

Holy cow!  ((A* A)^(-1) A*) fits into the equation just like an A^(-1) would if
it weren't a least squares problem!  That's why it's called the pseudoinverse.
Of course, you should always avoid solving this equation (the Normal Equations)
directly for a robust implementation and should instead use a QR or SVD
factorization, both of which are much more stable.

Sorry.  Been a while, but I always that was a pretty neat idea.

 - Ricky

Post a reply to this message

I have no idea what any of it means but I recall using least squares 
regression for my research in grad school. It gives the best fit equation 
for a scatterplot. Is that even the same thing? I just let the software do 
all the work, as I was never really into statistics.

Mike


"John VanSickle" <evi### [at] hotmailcom> wrote in message 
news:49974249$1@news.povray.org...
> Just posted this and I was wondering if it's got any serious problems 
> (other than lack of examples):
>
> http://www.geocities.com/evilsnack/least_squares.pdf
>
> In other news, OpenOffice's formula editor is rather nifty.
>
> Regards,
> John

Post a reply to this message

triple_r wrote:

> I can find no errors, and I read it over fairly carefully.

Good.  Your point about (2) is noted.

 > I don't know who your audience is...

A guy on the JPatch forums who asked for resources on the method.

Here's the sad part.  At the moment, to make ends meet, I work the night 
shift at a local store.  I wrote the piece on a laptop during my lunch 
break.  While I was writing I thought, "Does anyone who understands this 
belong here?"

Regards,
John

Post a reply to this message

John VanSickle wrote:

> 
> Here's the sad part.  At the moment, to make ends meet, I work the night 
> shift at a local store.  I wrote the piece on a laptop during my lunch 
> break.  While I was writing I thought, "Does anyone who understands this 
> belong here?"

It *is* sad.

Read "Outliers" yet?

p.s. my daughter often needs math and chem notation for her papers, 
(she's in hs,) but she tends to shun OO.  I'm 'in her face' about it now!

Post a reply to this message

John VanSickle wrote:
> Here's the sad part.  At the moment, to make ends meet, I work the night 
> shift at a local store.  I wrote the piece on a laptop during my lunch 
> break.  While I was writing I thought, "Does anyone who understands this 
> belong here?"

I'm sorry to hear that.  Hopefully to work at the store is still 
enjoyable -- I know a few people with graduate degrees who've 
(voluntarily) taken time off from computers to work retail, and often 
they've had a decent time of it.  Best of luck in finding something more 
profitable quickly all the same.

I didn't notice any errors in your math, although I didn't read through 
in great detail, so I could have missed something.  I'm more used to 
seeing the method presented in pure matrix form as minimizing the L2 
norm of Ax-b, but your approach works well for the way you've formulated 
the problem.  Perhaps the title might be better phrased as "Least 
Squares Regression" rather than "Least Squares Method" to better 
highlight this difference?  Certainly a minor point at best though.

The main thing I can think of which you might (or might not) want to 
mention is that it's often not a bad idea to use a third-party solver, 
rather than constructing and inverting the A^T*A yourself, and that 
depending on the application a useful solution can still sometimes be 
obtained when the matrix inverse is zero, but that it's just not unique.

Overall I liked it, and thought it was a good, concise, and 
straightforward description.

Post a reply to this message

Jim Charter wrote:
> p.s. my daughter often needs math and chem notation for her papers, 
> (she's in hs,) but she tends to shun OO.  I'm 'in her face' about it now!

What does she use instead? I'll have to try out OO's, but the one in Word 
seems OK for pretty normal equations like in your paper. I don't know about 
chem stuff.  Certainly not TeX mind. :-)

It was an interesting paper, altho I expect it won't stick in my head for 
any length of time.

-- 
   Darren New, San Diego CA, USA (PST)
   "Ouch ouch ouch!"
   "What's wrong? Noodles too hot?"
   "No, I have Chopstick Tunnel Syndrome."

Post a reply to this message

Darren New wrote:
> Jim Charter wrote:
> 
>> p.s. my daughter often needs math and chem notation for her papers, 
>> (she's in hs,) but she tends to shun OO.  I'm 'in her face' about it now!
> 
> 
> What does she use instead? 

Microsoft Office.  That's what all her friends use :(  I can't even get 
her to use Works.  I am remiss to install the copies I have of MS Office 
because they are copies stolen from previous office jobs, and because I 
hate how things like Outlook Express get installed too.  But alas, a 
father cannot resist a daughter.

Post a reply to this message

Jim Charter wrote:
> Microsoft Office.  That's what all her friends use :( 

Well, as I said, Word's equation editor seems quite straightforward.

> hate how things like Outlook Express get installed too.

I'm pretty sure you can turn that off, if you take care with the 
installation procedure.  Of course, you have to have a license for it, yes.

-- 
   Darren New, San Diego CA, USA (PST)
   "Ouch ouch ouch!"
   "What's wrong? Noodles too hot?"
   "No, I have Chopstick Tunnel Syndrome."

Post a reply to this message

On 16-2-2009 1:38, Jim Charter wrote:
> Darren New wrote:
>> Jim Charter wrote:
>>
>>> p.s. my daughter often needs math and chem notation for her papers, 
>>> (she's in hs,) but she tends to shun OO.  I'm 'in her face' about it 
>>> now!
>>
>>
>> What does she use instead? 
> 
> Microsoft Office.  That's what all her friends use :(  I can't even get 
> her to use Works.
have you tried LaTeX?

;)

Post a reply to this message