 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
I am planning to write a C++ windows-based application to retrieve and
analyse some data. I imagine my data to be stored in some sort of database
table, with each row having perhaps 10 pieces of data (some text some
integer) plus a unique ID (this is generated for me). I expect absolute
maximum 1000 rows of data to be added per day, so after a year or so we're
talking a few hundred thousand rows in this "table", not more than a
million.
The type of analysis I will need to do will be fairly straightforward, like
taking averages and sums of each bit of data over all (or subsets of) rows,
and maybe even some simple filtering but nothing fancy.
My question is, should I be looking to use some external database engine to
do the backend work here for me, or can I get away with just using STL
containers like "set" or something, with a struct that holds my data?
Anything else I should think about before deciding which way to go?
If a database engine will be better, any recommendations of which one? An
easy to install/learn one would be better, and it must be free to distribute
with my program and of course easily accessible from C++. I know a *tiny*
bit of SQL but I think I could learn what I need quite quickly.
And if you think I can just do it in C++ without using an engine, is using
"set" from STL the best way? What is a practical limit to the size (in MB)
of a set I should be manipulating/loading/saving. I'm thinking even a
million rows times eg 256 bytes should be fine for an application nowadays
to load/save and work with in memory?
Any other thoughts?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott schrieb:
> The type of analysis I will need to do will be fairly straightforward,
> like taking averages and sums of each bit of data over all (or subsets
> of) rows, and maybe even some simple filtering but nothing fancy.
This can easily be done inside the database with SQL. There are so
called aggregate functions like sum(), avg(), etc.
> My question is, should I be looking to use some external database engine
> to do the backend work here for me, or can I get away with just using
> STL containers like "set" or something, with a struct that holds my data?
Doing as much as possible inside the database will be faster.
> If a database engine will be better, any recommendations of which one?
> An easy to install/learn one would be better, and it must be free to
> distribute with my program and of course easily accessible from C++.
The give PostgreSQL a try. It has an C++ API too:
http://www.postgresql.org/docs/6.5/static/libpqplusplus.htm
So long,
Bonsai
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> The give PostgreSQL a try. It has an C++ API too:
>
> http://www.postgresql.org/docs/6.5/static/libpqplusplus.htm
That looks quite nice, it seems that libpq++ has been superseded by libpqxx,
which doesn't seem to work very well on Windows. But anyway I used the
standard C API and got it working ok so far. It also looks like I should be
able to do a silent install of PostGres during installation of my app which
is cool.
I just need to revise some SQL now and first of all work out how to create
databases and tables from my code.
Thanks!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scott com> wrote:
> The type of analysis I will need to do will be fairly straightforward, like
> taking averages and sums of each bit of data over all (or subsets of) rows,
> and maybe even some simple filtering but nothing fancy.
While I know only little of SQL, that sounds to me like it could be
perfectly doable with a database engine and its SQL frontend directly.
SQL has been developed precisely to be an fast&easy way to retrieve and
filter data from a database, and perform diverse operations on that data.
"Stored procedure" is probably a term which will give you useful information
about this whole subject.
If you need to output the data in some special format, after it has been
processed, I really don't know what tools, if any, SQL offers for this.
However, if all you need is to simply retrieve some information (filtered
and processed by the SQL server itself) from the database and then write
it in some special format, perhaps a language which has direct support for
interacting with SQL servers could be easier than C++. (I'm thinking about
PHP, which is somewhat similar to C++ in syntax, and thus more easily
approachable by a C++ programmer than more exotic languages.)
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> I just need to revise some SQL now and first of all work out how to
> create databases and tables from my code.
You do it with SQL. Probably something along the lines of
CREATE DATABASE mystuff;
CREATE TABLE mytable (
somenumber INTEGER NOT NULL,
comestring VARCHAR(50) DEFAULT '',
primary key somenumber
);
Look up those keywords.
You'll also need to figure out how to connect in the first place. Grep the
manual for "connection string", which is usually how it's described. It
tells basically the address of the server (IP & port, usually) plus user
plus password.
Warp wrote:
> If you need to output the data in some special format, after it has been
> processed, I really don't know what tools, if any, SQL offers for this.
SQL doesn't offer too many tools as such, i.e., not in a portable way.
Particular engines might give you the option of generating XML or CSV or
some such. If you can generate XML, you could run it thru an XSLT script and
output pretty much whatever format you need without writing anything but
shell scripts, if you bashed it hard enough with a baseball bat (or with
bash). If your language has the equivalent of system() and you don't mind
being kludgey, you could put the queries on the command line and output to a
CSV and parse that up.
You could also use metakit (from equi4.com) if you want a nice embeddable
database that isn't SQL (it's not even normalized). Just a plug for a great
product. It's the sort of thing a lot of small wiki servers use for their
database back end. A good replacement for Access. Lots of interfaces from
lots of languages. Not bad if you don't need a lot of relational joins and
such.
You can also use "SQL Server Everywhere", which is Microsoft's SQL server in
a form that you link it into your program. There's another one called "star
db" or "db star" or "SQL*" or something, that's from Sun I think, that isn't
windows-specific. This will save you the hassle of installing SQL servers
everywhere you want this stuff to run, if that's a problem for you. One less
thing to go wrong.
Don't confuse "embedded SQL server" with "embedded SQL". The first is a SQL
server that runs in the same process and address space as the client. The
second is a way of writing SQL statements directly in the source file of
your program and having a preprocessor turn it into function calls to the
client library.
PHP is OK for doing database stuff, but it's kludgey, so if you expect the
system to grow, or other people to work on it, it's probably something to
stay away from unless you need it for some other reason anyway. It's a way
to quickly hack together web front ends, and its use as a desktop tool is poor.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> scott wrote:
>> I just need to revise some SQL now and first of all work out how to
>> create databases and tables from my code.
>
> You do it with SQL.
Incidentally, for history buffs, that was one of the three or four
revolutionary amazing-at-the-time things that the relational model
introduced: the metadata of the data is accessible as data. That hit the
data world the way von Neumann architecture hit the electrical engineering
world. :-)
(The others being a mathematically rigorous foundation, a lack of pointers,
and separation of modeling concerns from efficiency concerns, all of which
were astoundingly revolutionary at the time.)
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Incidentally, for history buffs, that was one of the three or four
> revolutionary amazing-at-the-time things that the relational model
> introduced: the metadata of the data is accessible as data. That hit the
> data world the way von Neumann architecture hit the electrical
> engineering world. :-)
>
> (The others being a mathematically rigorous foundation, a lack of
> pointers, and separation of modeling concerns from efficiency concerns,
> all of which were astoundingly revolutionary at the time.)
The way I heard it, before databases came along, if you had a dozen
programs that all accessed the same pot of data, they all had to
understand the same file format. And if you *changed* that file format,
all your programs broke - usually be silently producing gibberish
instead of real data. Then you'd have to go modify them all one by one
to fix them.
Story goes that one company changed their file format so that some of
their bills to customers contained gibberish which actually leaked
information about their supplier relationships. A competetor got hold of
this information, managed to figure out what the gibberish was, and
managed to gain a competetive advantage.
(Nice story, but sounds kinda made-up to prove a point to me...)
The thing about a database is... the DBMS knows how to read the data.
And if you change how the data is stored (e.g., add a new field, change
a table from heap-organised to index-organised, etc.), as long as the
DBMS still knows how to read it, the client applications don't need to
*care*, and they don't break.
(Hell, if you add new fields or change the type of existing ones, you
can even create a "view" for the old apps to work off. They'll never
know the difference!)
At least, that's the PoV I got. I wasn't there, so...
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 wrote:
> The way I heard it, before databases came along, if you had a dozen
> programs that all accessed the same pot of data, they all had to
> understand the same file format. And if you *changed* that file format,
> all your programs broke - usually be silently producing gibberish
> instead of real data. Then you'd have to go modify them all one by one
> to fix them.
Well, yes.
> The thing about a database is... the DBMS knows how to read the data.
> And if you change how the data is stored (e.g., add a new field, change
> a table from heap-organised to index-organised, etc.), as long as the
> DBMS still knows how to read it, the client applications don't need to
> *care*, and they don't break.
Well, no. That's true of RDBMs, which is why they were revolutionary. It
isn't true of database engines that were around before the relational model.
> (Hell, if you add new fields or change the type of existing ones, you
> can even create a "view" for the old apps to work off. They'll never
> know the difference!)
Yes. That was the amazingly cool thing about relational databases that you
didn't get out of CODASYL databases or hierarchical databases or whatever.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> You do it with SQL. Probably something along the lines of
>
> CREATE DATABASE mystuff;
>
> CREATE TABLE mytable (
> somenumber INTEGER NOT NULL,
> comestring VARCHAR(50) DEFAULT '',
> primary key somenumber
> );
>
> Look up those keywords.
OK thanks that's certainly a good start, I've never worked with creating
this stuff from scratch before.
> You'll also need to figure out how to connect in the first place. Grep the
> manual for "connection string", which is usually how it's described. It
> tells basically the address of the server (IP & port, usually) plus user
> plus password.
Got that all working now - at some later date I will need to figure out how
to install postgres silently with all the correct options so my app will run
without having to explain to the user how to install it. And then cope with
what happens when postgres is already installed...
> Warp wrote:
> > If you need to output the data in some special format, after it has
> > been
> > processed, I really don't know what tools, if any, SQL offers for this.
>
> SQL doesn't offer too many tools as such, i.e., not in a portable way.
Most of the "results" I will just be using to display tables or graphs
within my app.
> You could also use metakit (from equi4.com) if you want a nice embeddable
> database that isn't SQL (it's not even normalized). Just a plug for a
> great product.
Will take a look, thanks.
> You can also use "SQL Server Everywhere", which is Microsoft's SQL server
> in a form that you link it into your program.
Ditto.
> PHP is OK for doing database stuff, but it's kludgey, so if you expect the
> system to grow, or other people to work on it, it's probably something to
> stay away from unless you need it for some other reason anyway. It's a way
> to quickly hack together web front ends, and its use as a desktop tool is
> poor.
THere are certain other parts of my code that need to be run pretty quickly
(some simulation stuff) and as I already have a lot of that in C++ I think
my best bet is to find the best database backend to use from C++.
What I might do is write a wrapper class for the database parts of my code
that will initially just use simple STL structures to store/retrieve/filter
data and then later I can write code to use a db engine instead once I get
it working with small datasets.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> What I might do is write a wrapper class for the database parts of my
> code that will initially just use simple STL structures to
> store/retrieve/filter data and then later I can write code to use a db
> engine instead once I get it working with small datasets.
Heh. That's pretty much exactly what the LINQ stuff in the .NET packages
does. An XML file looks like a SQL server at the source code - you just
build a different db connection object at the beginning.
If you're going to be filtering your own values anyway, look into the
non-SQL databases like metakit. The primary advantage of the SQL databases
is you can write expressions in SQL that manipulate an entire table (like
finding the max, min, average and sum of a whole column) in one statement.
If you're going to do that yourself anyway, it's probably easier to have the
code linked into your app than it is to try to install postgress but only if
it's already not installed *or* if you have permission to use it, etc. :-)
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> The thing about a database is... the DBMS knows how to read the data.
>> And if you change how the data is stored (e.g., add a new field,
>> change a table from heap-organised to index-organised, etc.), as long
>> as the DBMS still knows how to read it, the client applications don't
>> need to *care*, and they don't break.
>
> Well, no. That's true of RDBMs, which is why they were revolutionary. It
> isn't true of database engines that were around before the relational
> model.
>
>> (Hell, if you add new fields or change the type of existing ones, you
>> can even create a "view" for the old apps to work off. They'll never
>> know the difference!)
>
> Yes. That was the amazingly cool thing about relational databases that
> you didn't get out of CODASYL databases or hierarchical databases or
> whatever.
Yeah, well, I have little concept of what other kinds of databases have
existed other than relational ones.
(We did hear some talk at uni about people working on object databases,
but nobody could really figure out precisely what such a term means...)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> You can also use "SQL Server Everywhere", which is Microsoft's SQL server
>> in a form that you link it into your program.
OK so that's changed name to SQL Server Compact Edition and seems to be the
pefect fit for what I need.
It's just a simple small download and install with no configuration needed,
and then from Visual Studio I can use a normal managed interface to create
and access a database. None of the messing about with installing server
services, service account, usernames passwords, horrible API interfaces etc,
it just works nicely out of the box and MS has lots of samples and tutorials
on how to use it :-)
Thanks, it has made my job a lot easier!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible wrote:
> Yeah, well, I have little concept of what other kinds of databases have
> existed other than relational ones.
Relational pretty much replaced *everything*, and then along came OODBs and
along came distributed databases.
> (We did hear some talk at uni about people working on object databases,
> but nobody could really figure out precisely what such a term means...)
It means it doesn't follow the relational model, but instead the OO model.
For example, say you're using an OO language. (I'll assume you know how OO
languages work.) At some point, you say "stick this object in the database."
It says "OK." Then you can exit. When you start up again, that object is
still around, as well as anything it had pointers to in member variables.
It's nice because you can stick big blobs of data in and efficiently fetch
and store it all at once. So something like "my facebook account" would be
one record, or "this wiki page with all the history, embedded images, etc",
or Amazon with "this product, all its pictures, descriptions, reviews,
keywords, tags, etc". When much of your work is with one large record, you
can fetch it, use it, and put it back with relatively little overhead
compared to the RDBM.
The hard part is when you want to do something like (say) add up the amount
of money in all outstanding shopping carts, and you have to figure out how
to iterate over the objects without sucking the entire user's account into
memory for each one. If you can fit one entire table into memory, this is
less of a problem, but obviously still not trivial.
The advantage of an OODB is that you don't have to write a lot of code to
deal with format changes between what you store on disk and what you store
in memory. The disadvantage is that the only easy way to get to it is
usually the way it's stored in memory.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Thanks, it has made my job a lot easier!
Glad to help! :-)
Note there are similar things on the other major OSes, but I don't recall
what they're called nowadays. Shouldn't be too hard to find if you ever wind
up working under Linux or something.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Yeah, well, I have little concept of what other kinds of databases
>> have existed other than relational ones.
>
> Relational pretty much replaced *everything*
Yeah, I get that part.
>> (We did hear some talk at uni about people working on object
>> databases, but nobody could really figure out precisely what such a
>> term means...)
>
> It means it doesn't follow the relational model, but instead the OO
> model. For example, say you're using an OO language. (I'll assume you
> know how OO languages work.) At some point, you say "stick this object
> in the database." It says "OK." Then you can exit. When you start up
> again, that object is still around, as well as anything it had pointers
> to in member variables.
>
> It's nice because you can stick big blobs of data in and efficiently
> fetch and store it all at once. So something like "my facebook account"
> would be one record, or "this wiki page with all the history, embedded
> images, etc", or Amazon with "this product, all its pictures,
> descriptions, reviews, keywords, tags, etc". When much of your work is
> with one large record, you can fetch it, use it, and put it back with
> relatively little overhead compared to the RDBM.
>
> The hard part is when you want to do something like (say) add up the
> amount of money in all outstanding shopping carts, and you have to
> figure out how to iterate over the objects without sucking the entire
> user's account into memory for each one. If you can fit one entire table
> into memory, this is less of a problem, but obviously still not trivial.
>
> The advantage of an OODB is that you don't have to write a lot of code
> to deal with format changes between what you store on disk and what you
> store in memory. The disadvantage is that the only easy way to get to it
> is usually the way it's stored in memory.
Now, see, from what I could tell, everybody has their own idea about
what an OODBMS actually "is". You say it follows the "object model", but
people don't seem to be able to agree on precisely what that is.
As it happens, during my degree we had to do a "research project". The
idea was to come up with a topic, "research it" (whatever the hell that
means) and then write something about it. (Needless to say, I failed
badly at this.) The topic I was given was "the object-relational
impedence mismatch".
It seems that basically it's possible to encode objects into the
relational model without too much difficulty. What an OODB gives you is
the ability to store stuff without all that pesky marshalling. On the
other hand, it appears you lose any ability to query the database.
In other words, you save a little bit of code, and lose almost
everything that makes databases useful. Which seems like a very, very
unfavourable compromise.
Unrelated, but... it seems, intuitively, that functional programs should
be a much better "fit" to the relational model.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 wrote:
> In other words, you save a little bit of code, and lose almost
> everything that makes databases useful.
You can gain performance in certain circumstances. For example, if you look
at "memcached", it's basically an OO database that serves as a front end to
relational databases, except you have to manage it manually. An OODB for the
right problem can be much more scalable *because* you lose the ability to
search.
> Unrelated, but... it seems, intuitively, that functional programs should
> be a much better "fit" to the relational model.
Yeah. I was thinking that. Imagine STM monoid values that don't disappear
when your program exits, for example. Sort of the same idea...
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> In other words, you save a little bit of code, and lose almost
>> everything that makes databases useful.
>
> An OODB for the right problem can be much more scalable
> *because* you lose the ability to search.
You mean like the way SQL is more scalable *because* you can't process
the next row depending on what the previous one was?
>> Unrelated, but... it seems, intuitively, that functional programs
>> should be a much better "fit" to the relational model.
>
> Yeah. I was thinking that. Imagine STM monoid values that don't
> disappear when your program exits, for example. Sort of the same idea...
Yeah, that could work...
Of course, trouble is, almost all Haskell libraries that involve C fail
to compile on Windoze. (I'm told it's because Windows lacks a "standard"
place to put header files, etc.)
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> scott wrote:
>> Thanks, it has made my job a lot easier!
>
> Glad to help! :-)
>
> Note there are similar things on the other major OSes, but I don't
> recall what they're called nowadays. Shouldn't be too hard to find if
> you ever wind up working under Linux or something.
>
SQLite maybe ?
http://www.sqlite.org/
--
Vincent
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 wrote:
> You mean like the way SQL is more scalable *because* you can't process
> the next row depending on what the previous one was?
Basically, yes. In order for SQL to maintain the consistency guarantees, it
has to have access to all the data. If you don't want duplicate primary
keys, at some point, your primary keys have to all be on the same computer.
If you have 5 TB of primary key information in one table, you have trouble.
You already use an OODB every day. You store your web pages on it. It's
called a file system. You can't do complex searches. You can't find all
your Word documents that refer to a particular piece of lab equipment, for
example, without actually reading all the Word documents. But it *does* let
you store both programs and documents in the same system, for example.
(OK, it's not quite OO, but it's not relational, which is my point.)
> Of course, trouble is, almost all Haskell libraries that involve C fail
> to compile on Windoze. (I'm told it's because Windows lacks a "standard"
> place to put header files, etc.)
Uh, well, it means there might be more than one compiler. :-) I thought the
idea was to use compiler switches for that. :-)
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Vincent Le Chevalier wrote:
> SQLite maybe ?
> http://www.sqlite.org/
That's one, yes. I think Sun had one, and at one time "mSQL" was similar
too, but I see that has gotten closed up since I last looked at it.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> You mean like the way SQL is more scalable *because* you can't process
>> the next row depending on what the previous one was?
>
> Basically, yes.
Mmm, OK.
> In order for SQL to maintain the consistency guarantees,
> it has to have access to all the data. If you don't want duplicate
> primary keys, at some point, your primary keys have to all be on the
> same computer.
Not necessarily. If you partition all the PKs into buckets and each
bucket resides in a single place, you can still check for duplicates.
> If you have 5 TB of primary key information in one table,
> you have trouble.
Yes. For a start, where do you find enough physical space to put that
many HDs to store it all?
> You already use an OODB every day. It's called a file system.
> You can't do complex searches.
That's why people use database engines when they want stuff to be
searchable. ;-)
Still, I guess even an OODB can be indexed... Google somehow manages to
find all the webpages in the world that contain the word "banana" in
less than 1000 ms, even though this is obviously impossible.
>> Of course, trouble is, almost all Haskell libraries that involve C
>> fail to compile on Windoze. (I'm told it's because Windows lacks a
>> "standard" place to put header files, etc.)
>
> Uh, well, it means there might be more than one compiler. :-) I thought
> the idea was to use compiler switches for that. :-)
It seems that on Linux, Haskell's package manager somehow automatically
detects if the C library is installed, and if so, where it can be found.
On Windoze, however, it tends to just spaz out.
OTOH, it's not unheard of for Haskell packages to use bash scripts as
part of their build process, so...
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 wrote:
> Not necessarily. If you partition all the PKs into buckets and each
> bucket resides in a single place, you can still check for duplicates.
And that's basically what the other systems do. An RDBM calls that
"sharding". It works until you need to do a join, or you have a FK
constraint. And it only works as long as the other data associated with that
record is in the same shard.
In other words, if you do this with customers, and you do this with orders,
and you do this with products, getting the list of customers who buy the
most different kinds of products goes to its knees.
> Yes. For a start, where do you find enough physical space to put that
> many HDs to store it all?
Err, I have two 3TB drives plugged into my machine right now. (Doing
backups.) A terabyte is no longer big. It's a standard size drive.
>> You already use an OODB every day. It's called a file system.
>> You can't do complex searches.
>
> That's why people use database engines when they want stuff to be
> searchable. ;-)
Yep. More specifically, they use something indexable. You can do this with a
file system - you just have to keep the index up to date.
And when you change a file, you will have some number of seconds between
closing the file and creating the index that points to that file for each
word, and in that time a search won't find that document. And *that* is
exactly the kind of inconsistency you don't want with a SQL database. See?
> Still, I guess even an OODB can be indexed... Google somehow manages to
> find all the webpages in the world that contain the word "banana" in
> less than 1000 ms, even though this is obviously impossible.
It's less hard when you just throw hardware at the problem. :-)
> OTOH, it's not unheard of for Haskell packages to use bash scripts as
> part of their build process, so...
That'll slow you down, yes.
--
Darren New, San Diego CA, USA (PST)
"Ouch ouch ouch!"
"What's wrong? Noodles too hot?"
"No, I have Chopstick Tunnel Syndrome."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> Glad to help! :-)
You won't be saying that when I come back with 100 SQL questions next week
:-) I think I will get a book though as my knowledge is a bit fragmented
and I'm probably doing things the wrong way - that's the worst, people who
think they know what they are doing but really they have no clue :-)
> Note there are similar things on the other major OSes, but I don't recall
> what they're called nowadays.
Nah it's ok, my program is to work alongside another data collection program
that is windows-only (actually it looks like it is for Windows 3.1 but works
under Vista, amazingly!).
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Got that all working now - at some later date I will need to figure out
> how to install postgres silently with all the correct options so my app
> will run
> without having to explain to the user how to install it. And then cope
> with what happens when postgres is already installed...
I think there is an embedded Postgres so you link it with your app, instead
of running it as a separate process and connecting to it.
Or you could use SQLite instead, which is specifically made for embedding :)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Orchid XP v8 wrote:
> Google somehow manages to
> find all the webpages in the world that contain the word "banana" in
> less than 1000 ms, even though this is obviously impossible.
You should stop saying "this is obviously impossible". Especially when you
have seen it work in the real world (so it's obviously not impossible).
</Warp-impersonation>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
