![](/i/fill.gif) |
![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Just posted this and I was wondering if it's got any serious problems
(other than lack of examples):
http://www.geocities.com/evilsnack/least_squares.pdf
In other news, OpenOffice's formula editor is rather nifty.
Regards,
John
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
John VanSickle <evi### [at] hotmail com> wrote:
> Just posted this and I was wondering if it's got any serious problems
> (other than lack of examples):
>
> http://www.geocities.com/evilsnack/least_squares.pdf
>
> In other news, OpenOffice's formula editor is rather nifty.
I can find no errors, and I read it over fairly carefully. I don't know who
your audience is, but it couldn't hurt to emphasize that (2) is a sort of total
error to be minimized, and to see the resulting matrix equations put together at
the end. Of course neither of these things are really necessary.
Since you had us proofread it though, it's my turn to rant about least squares.
Perhaps you know this already, but I'm just enjoying the review out loud. This
is the derivation I had always seen, but last fall I was introduced to a neat
alternative version:
Think of a point lying off a line. (Not a data point, but instead a point in
the space of the coefficients you're trying to determine.) How close along the
line can we get to that point? Well, 'squares' just means Euclidian distance,
so it's along a second line perpendicular to the first, of course.
So if it's a linear problem in the coefficients, the residual is just
r = b-Ax,
where A is a matrix full of the basis functions evaluated at all points, x is
the unknown coefficients, and b is the data points. A does not need to be
square! In fact it will be m x n, where m is the number of data points and n
is the number of basis functions.
If the problem is solved, then the length of r has been minimized so that Ax is
as close to b as possible. This corresponds to the situation above where we
can't actually get the point onto the line, so instead we have to wander off
perpendicular to the line until we hit a point we can use. It turns out the
condition for orthogonality is
A* r = 0,
where * is just an adjoint (or transpose), and plugging in r,
A* A x = A* b
Solving for x,
x = (A* A)^(-1) A* b
Holy cow! ((A* A)^(-1) A*) fits into the equation just like an A^(-1) would if
it weren't a least squares problem! That's why it's called the pseudoinverse.
Of course, you should always avoid solving this equation (the Normal Equations)
directly for a robust implementation and should instead use a QR or SVD
factorization, both of which are much more stable.
Sorry. Been a while, but I always that was a pretty neat idea.
- Ricky
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
I have no idea what any of it means but I recall using least squares
regression for my research in grad school. It gives the best fit equation
for a scatterplot. Is that even the same thing? I just let the software do
all the work, as I was never really into statistics.
Mike
"John VanSickle" <evi### [at] hotmail com> wrote in message
news:49974249$1@news.povray.org...
> Just posted this and I was wondering if it's got any serious problems
> (other than lack of examples):
>
> http://www.geocities.com/evilsnack/least_squares.pdf
>
> In other news, OpenOffice's formula editor is rather nifty.
>
> Regards,
> John
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
triple_r wrote:
> I can find no errors, and I read it over fairly carefully.
Good. Your point about (2) is noted.
> I don't know who your audience is...
A guy on the JPatch forums who asked for resources on the method.
Here's the sad part. At the moment, to make ends meet, I work the night
shift at a local store. I wrote the piece on a laptop during my lunch
break. While I was writing I thought, "Does anyone who understands this
belong here?"
Regards,
John
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
John VanSickle wrote:
>
> Here's the sad part. At the moment, to make ends meet, I work the night
> shift at a local store. I wrote the piece on a laptop during my lunch
> break. While I was writing I thought, "Does anyone who understands this
> belong here?"
It *is* sad.
Read "Outliers" yet?
p.s. my daughter often needs math and chem notation for her papers,
(she's in hs,) but she tends to shun OO. I'm 'in her face' about it now!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
John VanSickle wrote:
> Here's the sad part. At the moment, to make ends meet, I work the night
> shift at a local store. I wrote the piece on a laptop during my lunch
> break. While I was writing I thought, "Does anyone who understands this
> belong here?"
I'm sorry to hear that. Hopefully to work at the store is still
enjoyable -- I know a few people with graduate degrees who've
(voluntarily) taken time off from computers to work retail, and often
they've had a decent time of it. Best of luck in finding something more
profitable quickly all the same.
I didn't notice any errors in your math, although I didn't read through
in great detail, so I could have missed something. I'm more used to
seeing the method presented in pure matrix form as minimizing the L2
norm of Ax-b, but your approach works well for the way you've formulated
the problem. Perhaps the title might be better phrased as "Least
Squares Regression" rather than "Least Squares Method" to better
highlight this difference? Certainly a minor point at best though.
The main thing I can think of which you might (or might not) want to
mention is that it's often not a bad idea to use a third-party solver,
rather than constructing and inverting the A^T*A yourself, and that
depending on the application a useful solution can still sometimes be
obtained when the matrix inverse is zero, but that it's just not unique.
Overall I liked it, and thought it was a good, concise, and
straightforward description.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Jim Charter wrote:
> p.s. my daughter often needs math and chem notation for her papers,
> (she's in hs,) but she tends to shun OO. I'm 'in her face' about it now!
What does she use instead? I'll have to try out OO's, but the one in Word
seems OK for pretty normal equations like in your paper. I don't know about
chem stuff. Certainly not TeX mind. :-)
It was an interesting paper, altho I expect it won't stick in my head for
any length of time.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> Jim Charter wrote:
>
>> p.s. my daughter often needs math and chem notation for her papers,
>> (she's in hs,) but she tends to shun OO. I'm 'in her face' about it now!
>
>
> What does she use instead?
Microsoft Office. That's what all her friends use :( I can't even get
her to use Works. I am remiss to install the copies I have of MS Office
because they are copies stolen from previous office jobs, and because I
hate how things like Outlook Express get installed too. But alas, a
father cannot resist a daughter.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Jim Charter wrote:
> Microsoft Office. That's what all her friends use :(
Well, as I said, Word's equation editor seems quite straightforward.
> hate how things like Outlook Express get installed too.
I'm pretty sure you can turn that off, if you take care with the
installation procedure. Of course, you have to have a license for it, yes.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
On 16-2-2009 1:38, Jim Charter wrote:
> Darren New wrote:
>> Jim Charter wrote:
>>
>>> p.s. my daughter often needs math and chem notation for her papers,
>>> (she's in hs,) but she tends to shun OO. I'm 'in her face' about it
>>> now!
>>
>>
>> What does she use instead?
>
> Microsoft Office. That's what all her friends use :( I can't even get
> her to use Works.
have you tried LaTeX?
;)
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |