Darren New wrote:
> Invisible wrote:
>> More to the point, there's masses of tricky parsing to wade through 
>> all the presentational HTML to extract the actual raw data I'm after.
> 
> That's because you're getting the results in HTML instead of "via an API".
> 
> That's what people are talking about when they talk about web APIs: 
> presenting data without making you parse it.
> 
> "REST" is a pattern for doing this in a way that *also* lets you use a 
> web browser to reverse-engineer the protocols by looking at it and which 
> is theoretically kind to intermediate proxies.
> 
> "SOAP" is a pattern for doing this in a way that lets you publish the 
> specification of the interface in a form that a tool can generate code 
> to decode it into whatever native data structures are available for your 
> language.

So you mean you get the web server to search an XML document so the 
client at the other end can parse it into something useful more easily?

> What you're doing is called "screen scraping", and yes, it breaks when 
> the format of the web page changes, which happens often when there's so 
> many people doing screen scraping that it starts to impact the actual 
> customers of the web site.

I doubt my one hit per month hurts PassMark. ;-)

Post a reply to this message

Invisible wrote:
> So you mean you get the web server to search an XML document so the 
> client at the other end can parse it into something useful more easily?

s/search/serve

Yes.

> I doubt my one hit per month hurts PassMark. ;-)

No, but when people distribute software that (for example) takes TV listings 
off Yahoo's web site without grabbing or showing any ads, and 100,000 people 
run that every day, Yahoo notices.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Invisible wrote:
> Uhuh. And how many PhDs work for Google?

Lots. So? It's not hard to make a web crawler, as long as you're not trying 
to understand data that *isn't* presented in HTML tags. How hard is it to 
find all the URLs in a HTML document?

>> But it's hard to get specific information out in an automated way. 
>> That's why people invented HTTP-based APIs like SOAP.
> 
> Why would you base a protocol on HTTP?

Because it's easy and the standards are there to say how to look things up 
and encode the bytes and do the logging in and encryption and things like 
that. If you have a request/response model, it's not a bad protocol. If you 
have something more complex than a request/response model, you should be 
using something like BEEP.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

>> So you mean you get the web server to search an XML document so the 
>> client at the other end can parse it into something useful more easily?
> 
> s/search/serve
> 
> Yes.

Hmm. Interesting.

Does anybody ever do this?

>> I doubt my one hit per month hurts PassMark. ;-)
> 
> No, but when people distribute software that (for example) takes TV 
> listings off Yahoo's web site without grabbing or showing any ads, and 
> 100,000 people run that every day, Yahoo notices.

It's news to me that Yahoo is still in business...

Post a reply to this message

Invisible wrote:
> Does anybody ever do this?

Um, yes, scads of people. For example, the whole Amazon "cloud computing" is 
based on this, Yahoo does it for their search engines, iTunes does it for 
music search, Google has APIs for the stuff they do, Amazon lets you search 
and buy and sell things that way, etc.

> It's news to me that Yahoo is still in business...

Do you do this on purpose? :-)  I mean, you don't even have to google 
anything to figure out if Yahoo is still running a web site.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

>> Does anybody ever do this?
> 
> Um, yes, scads of people. For example, the whole Amazon "cloud 
> computing" is based on this, Yahoo does it for their search engines, 
> iTunes does it for music search, Google has APIs for the stuff they do, 
> Amazon lets you search and buy and sell things that way, etc.

...wait, so you mean there's a way I can use this to find out how much 
money Amazon want for product X?

>> It's news to me that Yahoo is still in business...
> 
> Do you do this on purpose? :-)  I mean, you don't even have to google 
> anything to figure out if Yahoo is still running a web site.

I'm not disputing that Yahoo are still going - clearly they are - I'm 
just saying it's news to me. As in, nobody ever speaks about them any 
more...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> ...wait, so you mean there's a way I can use this to find out how much 
> money Amazon want for product X?

Yes.

Look at "amazon web services" on wikipedia for a list of all their APIs. The 
"associates" API is what you're looking for.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New wrote:
> Orchid XP v8 wrote:
>> ...wait, so you mean there's a way I can use this to find out how much 
>> money Amazon want for product X?
> 
> Yes.
> 
> Look at "amazon web services" on wikipedia for a list of all their APIs. 
> The "associates" API is what you're looking for.

Heh. Too bad Amazon don't sell CPUs. I might actually be able to write 
my price/performance autoplot tool after all. ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Invisible wrote:
> scott wrote:
>> How about RSS feeds, loads of programs written that access those over
>> the web?
> 
> I don't what what RSS is.

http://blog.orphi.me.uk/feed

Open it with a text editor. Isn't it easier to parse than all the
presentational HTML in your blog?

Post a reply to this message

Orchid XP v8 wrote:
> Heh. Too bad Amazon don't sell CPUs.

No, but they'll rent them to you. ;-)

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message