Hi,
in our country, we have a government-operated official website
which publishes opportunities in public tender.

sample URL of a resulting page :
http://www.ejustice.just.fgov.be/cgi_bul/bul_a_1.pl?DETAIL=DETAIL&caller=list&row_id=1&numero=1&rech=472&numac=2007051997&pd=2007-11-16&lg=F&pdf_file=%2Fhome%2Fmon1%2Fbul%2Fimage%2F2007%2F1116_1.pdf&trier=+order+by+numac+desc%2C+pd%3B&language=fr&choix1=ET&choix2=ET&fromtab=BUL&sql=objet+contains++%27architecture%27&objet=architecture

These people don't have an RSS feed availiable, or anything else that would
help us do otherwise than tedious manual checking with keywords every week
(hundreds of offers are published each day).

I'd like to be able to automate the process, so I can produce some kind of
digest from the offers we are likely interested into. The "examine a page
and determine if we are likely to be interested" will be easy.  I have a
problem with the first step : "automatically retrieve every page starting
from a given one".

After some observation and tests, I know how to get the "next offer" by tweaking
the URL string appropriately.  But I need to read the content of the resulting
page to determine when I need to change the date ('pd' in the query) so I can
continue incrementing the numbering ('numac' in query).  That's where it goes bad.

I tought 'well, just do some javascript, put the content of the url in an iframe,
read it, and act accordingly'.  Done that. It doesn't work. Why ?  The XMLHTTPRequest
function, which is used to put the content of the iframe in a string, is prohibited
(in any browser in existence) to work with content from another domain. Ouch !
I found some GreaseMonkey script which pretended to allow bypass of this "cross-domain
policy", but it didn't work.

So I'm still at the start of this seemingly simple project.  I'm currently thinking
of getting the pages with WGET, but can I pilot WGET from Javascript ? Or should
I try another language ?  Or a completely different path ?

Ideas ?

TIA,
Fabien.

Post a reply to this message

I would imagine you'll have endless problems trying to get round 
security issues if you try to script this from inside a web browser.

My suggestion would be to move to another programming language that has 
a HTTP library and try to do the stuff you want from there. It'll 
probably be much easier.

Obviously I recommend Haskell for this task - and, obviously, you're 
going to say no. ;-) That being the case, I'm pretty confident that Perl 
/ Python / Ruby / Tcl / any of those hackish scripting languages will 
have a library that makes this reasonably easy.

Post a reply to this message

Fa3ien a écrit :
> So I'm still at the start of this seemingly simple project.  I'm 
> currently thinking
> of getting the pages with WGET, but can I pilot WGET from Javascript ? 
> Or should
> I try another language ?  Or a completely different path ?
> 
> Ideas ?
> 

If I had to do this, I think I would write a script outside a web page, 
in whatever language able to download files on the web (I'm sure Andrew 
will come up with something in Haskell ;-) ). Then of course this script 
could generate an HTML page as its output.

Or you could probably do such things in PHP, if you want to have it 
driven by a web page consultation.

But I'm no expert in these matters... Never done that before.

-- 
Vincent

Post a reply to this message

Invisible a écrit :
> I would imagine you'll have endless problems trying to get round 
> security issues if you try to script this from inside a web browser.
> 
> My suggestion would be to move to another programming language that has 
> a HTTP library and try to do the stuff you want from there. It'll 
> probably be much easier.
> 
> Obviously I recommend Haskell for this task - and, obviously, you're 
> going to say no. ;-) 

How did you guess ?

> That being the case, I'm pretty confident that Perl 
> / Python / Ruby / Tcl / any of those hackish scripting languages will 
> have a library that makes this reasonably easy.

I'm tempted to get a hand on Ruby, for various reasons. Maybe I can
do it in Lisp... At first, I rejected the idea because it would
need AutoCAD, but, no, there might be some free LISP intepreter,
I should check.

In fact, it doesn't even need a http library, if I can shell to
Wget and read the downloaded file...

Fabien.

Post a reply to this message

"Fa3ien" <fab### [at] yourshoesskynetbe> a écrit dans le message de news: 
4744378d@news.povray.org...

> So I'm still at the start of this seemingly simple project.  I'm currently 
> thinking
> of getting the pages with WGET, but can I pilot WGET from Javascript ? Or 
> should
> I try another language ?  Or a completely different path ?

This could be done in PHP, something like this:

<?
$url = 'yoururl';
$needle = 'Architecture';
$haystack = file_get_contents($url);
if(strpos($haystack, $needle)!== false) {
echo 'found';
} else {
echo 'not found';
}
?>

You need to automate the url by passing the right keywords.

G.

Post a reply to this message

Fa3ien wrote:

>> Obviously I recommend Haskell for this task - and, obviously, you're 
>> going to say no. ;-) 
> 
> How did you guess ?

Mmm, because everybody hates Haskell? ;-)

>> That being the case, I'm pretty confident that Perl / Python / Ruby / 
>> Tcl / any of those hackish scripting languages will have a library 
>> that makes this reasonably easy.
> 
> I'm tempted to get a hand on Ruby, for various reasons. Maybe I can
> do it in Lisp... At first, I rejected the idea because it would
> need AutoCAD, but, no, there might be some free LISP intepreter,
> I should check.

I'm pretty sure I looked into this myself, and found that there are 
indeed free Common Lisp interpreters out there.

(And there's always emacs... bahahaha!)

> In fact, it doesn't even need a http library, if I can shell to
> Wget and read the downloaded file...

Yeah, that's true. Probably easier this way if there isn't a strong HTTP 
library already available. wget will already cover all the important 
edge cases...

Post a reply to this message

Invisible a écrit :
> Fa3ien wrote: 
> 
>>> Obviously I recommend Haskell for this task - and, obviously, you're 
>>> going to say no. ;-) 
>>
>> How did you guess ?
> 
> Mmm, because everybody hates Haskell? ;-)

Fears, not hate.  Personally, whenever you post Haskell code, I'm pretty
admirative of the powerfulness of what you say it does with such concise
code.  But I am also scared by the fact that I don't understand a bit
about how what it does relates to what the code looks like.

Since my first message (about 1:30 ago), I tried Ruby.  After 20 minutes
of an online hands-on tutorial (http://tryruby.hobix.com/), 5 minutes
checking for "ruby http library", and 5 minutes installing a Ruby
interpreter for Windows, I've been delighted to see that this line of code,
build with my thin newly acquired knowledge :

print Net::HTTP.get(URI.parse("http://www.google.be"))

produced exactly what I expected it to do ! (putting the content
of a web page in a string)

Ruby is a gem !

Fabien.

Post a reply to this message

I would do it with PHP (outside a webserver), because I did many 
scraping scripts that way. It's easy to parse HTML with PHP's DOM and 
loadHTML, handles all the bad syntax for you.

I wrote a 300-line PHP script that parsed some old HTML documentation 
into wiki formatting, submitted it to the wiki, then created a frameset 
page with the original HTML on one side and the new wiki page in edit 
mode on the other, and opened the page in Firefox (possible since the 
script was running locally).

And another I have running permanently on my system that scrapes the 
search page of that wiki searching for spam, and when finding something, 
gets the page history (to know the last revision ID) and deletes the 
last version (repeatedly for everything listed on the search results). 
Even has exponential backoff when finding nothing and a log.

I could give you the source if you want to learn from them :)

If you want to use JS, you can do it if you are on Windows: Windows 
Scripting Host. Run it by double-clicking the .js, or using 'wscript 
file.js' on the command line (or 'cscript' if you want to WScript.Echo 
stuff into the console instead of opening a popup). Remember to get the 
XMLHttpRequest using the ActiveX object, 'new XMLHttpRequest' doesn't 
work (well, maybe it will if you have IE7).

Post a reply to this message

Fa3ien wrote:

>> Mmm, because everybody hates Haskell? ;-)
> 
> Fears, not hate.

Ah, OK. I rephrase then: *most* people hate Haskell. The rest just ph33r 
it. ;-)

> Personally, whenever you post Haskell code, I'm pretty
> admirative of the powerfulness of what you say it does with such concise
> code.  But I am also scared by the fact that I don't understand a bit
> about how what it does relates to what the code looks like.

It seems Haskell has both the power to be completely transparent, and 
also entirely opaque. A bit like mathematical formulas, really!

> I've been delighted to see that this line of code,
> build with my thin newly acquired knowledge :
> 
> print Net::HTTP.get(URI.parse("http://www.google.be"))
> 
> produced exactly what I expected it to do ! (putting the content
> of a web page in a string)

Yeah. In Haskell you'd have to spend a few minutes installing GHC, a few 
more minutes downloading and compiling the 3rd party HTTP library, and 
then you'd have to write something like

   let uri = fromMaybe $ parseURI "http://www.google.be"
   maybePage <- httpGet uri
   let page = fromMaybe maybePage

(And replace those fromMaybe calls with some slightly larger construct 
if you actually want to do real error handling.)

> Ruby is a gem !

LOL! I bet you're not the first to think that one up...



Alternatively, if you feel ill, you might try to write the Haskell 
version as

   page <- (httpGet $ fromMaybe $ parseURI "http://www.google.be") >>= 
(return . fromMaybe)

Certainly I can see where the "scary" issue comes from...

Post a reply to this message

And lo on Wed, 21 Nov 2007 13:51:37 -0000, Fa3ien  
<fab### [at] yourshoesskynetbe> did spake, saying:

<snip>
> I tought 'well, just do some javascript, put the content of the url in  
> an iframe, read it, and act accordingly'.  Done that. It doesn't work.  
> Why ?  The XMLHTTPRequest function, which is used to put the content of  
> the iframe in a string, is prohibited (in any browser in existence) to  
> work with content from another domain. Ouch !

If it's any help I know IE6 didn't have this security restriction, but  
that 'hole' may have been plugged now.

> I found some GreaseMonkey script which pretended to allow bypass of this  
> "cross-domain policy", but it didn't work.
>
> So I'm still at the start of this seemingly simple project.  I'm  
> currently thinking of getting the pages with WGET, but can I pilot WGET  
> from Javascript ? Or should I try another language ?  Or a completely  
> different path ?

Depends what you've got to work with and how it's going to be applied. If  
you've a PHP server then as Gilles said that's your best bet, otherwise  
you're running a 'script' directly.

-- 
Phil Cook

--
I once tried to be apathetic, but I just couldn't be bothered
http://flipc.blogspot.com

Post a reply to this message

Nicolas Alvarez wrote:
> I would do it with PHP (outside a webserver), because I did many 
> scraping scripts that way. It's easy to parse HTML with PHP's DOM and 
> loadHTML, handles all the bad syntax for you.

As long as you start a new process for each page, you'll be OK. From 
what I can tell, PHP never, ever deallocates memory.  Try walking thru 
and processing a 600-megaline database table in CLI PHP, and you'll 
regret it.

You could write one that sucks up URLs (or runs wget), then iterates 
over the resulting files with one PHP script each or something.

Or use Tcl, which is what I did.

-- 
   Darren New / San Diego, CA, USA (PST)
     It's not feature creep if you put it
     at the end and adjust the release date.

Post a reply to this message

Darren New escribió:
> As long as you start a new process for each page, you'll be OK. From 
> what I can tell, PHP never, ever deallocates memory.  Try walking thru 
> and processing a 600-megaline database table in CLI PHP, and you'll 
> regret it.

I never had any problem with that, I let my spam-delete.php running for 
more than 24 hours and its memory usage didn't grow.

Are you calling mysql_free_result? :)

"mysql_free_result() only needs to be called if you are concerned about 
how much memory is being used for queries that return large result sets. 
All associated result memory is automatically freed *at the end of the 
script's execution*." (emphasis mine)

Post a reply to this message

Fa3ien wrote:
...
> So I'm still at the start of this seemingly simple project.  I'm
> currently thinking
> of getting the pages with WGET, but can I pilot WGET from Javascript ?

You may also want to look at cURL:

http://en.wikipedia.org/wiki/CURL


Other useful tools:

sed, awk, grep and others

http://tldp.org/LDP/abs/html/textproc.html


> Or should
> I try another language ?  Or a completely different path ?

bash, perl, tcl/tk ?

http://en.wikipedia.org/wiki/Bash
http://en.wikipedia.org/wiki/Perl
http://en.wikipedia.org/wiki/Tcl

-- 
Tor Olav
http://subcube.com

Post a reply to this message

Use a bash script to pull the pages down to a local webserver, manipulate 
and serve up from there instead?

Jim

Post a reply to this message

Jim Henderson <nos### [at] nospamcom> wrote:
> Use bash

  Btw, I always wonder why it seems that 100% of people out there who
use a shell use bash. I know it's because it's (for whatever reason)
the default shell in all linux distros, but I just can't understand
what's so great about bash.

  Personally I use zsh and there are so many handy features which I can't
find in bash that I really don't understand the popularity of the latter.
Bash isn't even 100% sh-compatible.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:

>   Btw, I always wonder why it seems that 100% of people out there who
> use a shell use bash. I know it's because it's (for whatever reason)
> the default shell in all linux distros, but I just can't understand
> what's so great about bash.
> 
>   Personally I use zsh and there are so many handy features which I can't
> find in bash that I really don't understand the popularity of the latter.
> Bash isn't even 100% sh-compatible.

I don't understand why 60% of the packages on the Haskell package site 
use bash scripts as part of their installation process. (As you can 
probably anticipate, this means that you can't install those on 
Windoze.) It's actually rather annoying... We develop a powerful 
cross-platform high-level programming langauge, and people insist on 
using bash scripts. Gah! >_<

Post a reply to this message

On Thu, 22 Nov 2007 08:45:37 -0500, Warp wrote:

>   Btw, I always wonder why it seems that 100% of people out there who
> use a shell use bash. I know it's because it's (for whatever reason) the
> default shell in all linux distros, but I just can't understand what's
> so great about bash.

Personally, I use tcsh, but I've often wondered this as well.

Jim

Post a reply to this message

Nicolas Alvarez wrote:
> Are you calling mysql_free_result? :)

Actually, I think the problem was that I was reading a CSV file and 
outputting SQL text that I'd later "mysql <xyz.sql" sort of thing.

> "mysql_free_result() only needs to be called if you are concerned about 
> how much memory is being used for queries that return large result sets. 
> All associated result memory is automatically freed *at the end of the 
> script's execution*." (emphasis mine)

I'm pretty sure the problem is that local variables weren't getting 
freed. Nothing to do with actual mysql accesses.  I'll see if I still 
have the original code and see what it's doing that might have been leaking.

-- 
   Darren New / San Diego, CA, USA (PST)
     It's not feature creep if you put it
     at the end and adjust the release date.

Post a reply to this message

"Invisible" <voi### [at] devnull> wrote in message
news:47444787$1@news.povray.org...
> Fa3ien wrote:
>
> >> Obviously I recommend Haskell for this task - and, obviously, you're
> >> going to say no. ;-)
> >
> > How did you guess ?
>
> Mmm, because everybody hates Haskell? ;-)
>
> >> That being the case, I'm pretty confident that Perl / Python / Ruby /
> >> Tcl / any of those hackish scripting languages will have a library
> >> that makes this reasonably easy.
> >
> > I'm tempted to get a hand on Ruby, for various reasons. Maybe I can
> > do it in Lisp... At first, I rejected the idea because it would
> > need AutoCAD, but, no, there might be some free LISP intepreter,
> > I should check.
>
> I'm pretty sure I looked into this myself, and found that there are
> indeed free Common Lisp interpreters out there.

Most are compilers these modern days, in an interactive environment. Anyway,
try out SBCL, CLisp, or OpenMCL depending on your platform.
http://gigamonkeys.com/book/lispbox/ for a nice package of several
distribution.

>
> (And there's always emacs... bahahaha!)

emacs has elisp, a viable lisp dialect. not sure where the humor is, emacs
is one of the most robust peices of end user software. guess what it's
written in...

It's so hard to ignore the troll.

Post a reply to this message

"Warp" <war### [at] tagpovrayorg> wrote in message
news:47458801@news.povray.org...
> Jim Henderson <nos### [at] nospamcom> wrote:
> > Use bash
>
>   Btw, I always wonder why it seems that 100% of people out there who
> use a shell use bash. I know it's because it's (for whatever reason)
> the default shell in all linux distros, but I just can't understand
> what's so great about bash.
>
>   Personally I use zsh and there are so many handy features which I can't
> find in bash that I really don't understand the popularity of the latter.
> Bash isn't even 100% sh-compatible.

I've never used zsh. What handy features does it have that I'm missing out
on?

Post a reply to this message

Ross <rli### [at] speakeasynet> wrote:
> I've never used zsh. What handy features does it have that I'm missing out
> on?

  You name it...

  I think it's quite descriptive that bash's man page is 4889 lines long
while zsh's is 20639 lines long. (It's so long, actually, that it has
been split into 16 different manpages by feature category, although you
can get the entire thing as one long manpage with 'man zshall'.)

  I must admit that I have not used bash so much as to be 100% sure that
many of the handy features I like about zsh cannot be found or turned on
(or installed as addons) in bash, but at least with the default bash
settings it seems to lack them.

  But to mention just one of the many, many handy features, zsh's
autocompletion is simply superb. It can very intelligently deduce
what you want to do from the context.
  For example, assume that in the current directory there's a bunch
of files, all of them with "-rw-r--r--" permissions except for one,
which has "-rw-------" permissions. To give it the same persmissions
as the other it's enough to write:

chmod og+r <tab>

  It will autocomplete to the only file for which the command would have
an effect. (In other words, it's not simply a dumb file completion feature.
It actually looks at what you are trying to execute, interprets it and
completes according to that. Since "chmod og+r" would be a no-op for all
the other files except one, it completes to the file for which it's not
a no-op.)
  Another of the many handy autocompletion features is that it knows the
command-line parameters of many programs. For example, you can write:

mplayer -q<tab>

and it will autocomplete that to:

mplayer -quiet

  It also completes filenames according to which program you are executing.
For example, if there are two files in the current directory, one named
"test.txt" and another named "test.avi" and you write:

mplayer t<tab>

it will autocomplete to

mplayer test.avi

  It knows that mplayer doesn't support txt files so it skips it. (Bash
does have this kind of feature too, but seems to be more limited that zsh's.
For instance, it doesn't seem to support mplayer.)

  There are many other handy autocompletion features as well.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Ross <rli### [at] speakeasynet> wrote:
>> I've never used zsh. What handy features does it have that I'm missing out
>> on?
> 
>   You name it...
> 
>   I think it's quite descriptive that bash's man page is 4889 lines long
> while zsh's is 20639 lines long. (It's so long, actually, that it has
> been split into 16 different manpages by feature category, although you
> can get the entire thing as one long manpage with 'man zshall'.)

	OK - Seems like it's worth a try. But what might break? Is there
anything that I do in BASH that may perhaps require different keystrokes?

-- 
Isn't it counterproductive to have incandescent bulbs in a fridge?


                    /\  /\               /\  /
                   /  \/  \ u e e n     /  \/  a w a z
                       >>>>>>mue### [at] nawazorg<<<<<<
                                   anl

Post a reply to this message

Ross wrote:

>> (And there's always emacs... bahahaha!)
> 
> emacs has elisp, a viable lisp dialect. not sure where the humor is

Many people mistakenly believe emacs to be... like... a "text editor". I 
think "hosted operating system" would probably be a more accurate 
description. :-P

Post a reply to this message

Mueen Nawaz <m.n### [at] ieeeorg> wrote:
>         OK - Seems like it's worth a try. But what might break? Is there
> anything that I do in BASH that may perhaps require different keystrokes?

  I don't know what you do in bash.

-- 
                                                          - Warp

Post a reply to this message

Thanks to anyone who cared to answer my question.

I'm currently falling in love with Ruby, and almost
finished (in Ruby) the analyse-web-offers-and-send-results-by-mail
app which I thought would be very complex (for me,
poor ever-beginner programmer) to achieve.  Instead,
it's quick and fun fun fun !

They say Ruby is designed with a "principle of least surprise"
in mind, I would also say it's "low annoyance"-compliant !

Wanna discover ?
http://tryruby.hobix.com/

Fabien.

Post a reply to this message

Fa3ien wrote:

> They say Ruby is designed with a "principle of least surprise"
> in mind, I would also say it's "low annoyance"-compliant !

I just round Ruby to be very ad-hoc and I-made-this-up-as-I-went-along. 
I prefer a language founded on a small set of basic axioms and 
everything derived from there. But hey, I'm told some folks quite like 
Ruby, so... ;-)

Post a reply to this message

Invisible wrote:
> I just round Ruby to be very ad-hoc and I-made-this-up-as-I-went-along. 

I agree with this.  It basically takes Smalltalk and reinvents it 
without any actual planning as to how things will work. I gave it up as 
soon as I realized they were changing defaults on what I'd think were 
fairly important library calls between v1.18 and v1.19, with no 
consideration for backwards compatibility and (afaict) no real good 
reason for it.

> I prefer a language founded on a small set of basic axioms and 
> everything derived from there. But hey, I'm told some folks quite like 
> Ruby, so... ;-)

I suspect it's mostly people who have worked with a very limited set of 
languages that quite like it. Sort of like living in Hawaii or San 
Diego, and then going to visit the great beaches of Europe and saying 
"Yah? So? What's the big deal with the Italian Riviera?"

-- 
   Darren New / San Diego, CA, USA (PST)
     It's not feature creep if you put it
     at the end and adjust the release date.

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> I just round Ruby to be very ad-hoc and I-made-this-up-as-I-went-along. 
> 
> I agree with this.

So does at least one other blog I read (by a computer science lecturer 
who has a liking for Haskell).

> I gave it up as 
> soon as I realized they were changing defaults on what I'd think were 
> fairly important library calls between v1.18 and v1.19, with no 
> consideration for backwards compatibility and (afaict) no real good 
> reason for it.

Mmm, sounds like *cough* Java. :-)

>> I prefer a language founded on a small set of basic axioms and 
>> everything derived from there. But hey, I'm told some folks quite like 
>> Ruby, so... ;-)
> 
> I suspect it's mostly people who have worked with a very limited set of 
> languages that quite like it. Sort of like living in Hawaii or San 
> Diego, and then going to visit the great beaches of Europe and saying 
> "Yah? So? What's the big deal with the Italian Riviera?"

Some day I'll figure that one out... heh.

Anyway, suffice it to say that Ruby doesn't seem to fit the kind of 
programs *I* write very well.

Post a reply to this message

Invisible wrote:
> Anyway, suffice it to say that Ruby doesn't seem to fit the kind of 
> programs *I* write very well.

Oh, and the idea that the best language manual is written by someone 
other than the one who designed the language, but rather someone who 
read the source code and tried to figure out how it worked...  Bleh.

-- 
   Darren New / San Diego, CA, USA (PST)
     It's not feature creep if you put it
     at the end and adjust the release date.

Post a reply to this message