![](/i/fill.gif) |
![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Paul Fuller wrote:
> "Names are fickle, Identifiers are permanent"
The problem with tagging is that either a) it stores the data in an
external database that doesn't follow the file when you copy it to a new
medium or b) it alters the file, changing its MD5 hash and making it not
identical to the other copies of it on the internet (or wherever), so
it's harder to find duplicates in large sets of files or quickly
determine which images of an existing set are missing.
--
Tim Cook
http://empyrean.freesitespace.net
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Tim Cook wrote:
> medium or b) it alters the file, changing its MD5 hash and making it not
Store the pre-change MD5 as one of the attributes of the file. Or the name
of the file.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Tim Cook wrote:
> Paul Fuller wrote:
>> "Names are fickle, Identifiers are permanent"
>
> The problem with tagging is that either a) it stores the data in an
> external database that doesn't follow the file when you copy it to a new
> medium or b) it alters the file, changing its MD5 hash and making it not
> identical to the other copies of it on the internet (or wherever), so
> it's harder to find duplicates in large sets of files or quickly
> determine which images of an existing set are missing.
>
> --
> Tim Cook
> http://empyrean.freesitespace.net
Well if you rule out adding external data and using the file attributes
then you are stuck with hierarchy and that is just the least useful
thing for reasons already mentioned.
I agree with Darren's answer that you can always store the original MD5
as an attribute. Ditto for the original name, source website and any
other attributes that you are interested in.
I also should have said that you should of course normalise the data.
So for example don't tag the file with the Artist's DOB etc.
You can always generate whatever structure you want from the metadata.
Say if you want to export selected files for some purpose where the
database approach does not work.
The advantage is that you can generate whatever from the metadata.
Can't say the same for the structure approach.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Paul Fuller wrote:
> I agree with Darren's answer that you can always store the original MD5
> as an attribute. Ditto for the original name, source website and any
> other attributes that you are interested in.
If you use the original MD5 as the file name, then add the attributes to it,
you don't even have to look at the metadata to tell if you already have the
file. You don't have to hash it, or store the hash anywhere else.
> The advantage is that you can generate whatever from the metadata. Can't
> say the same for the structure approach.
You can. You're just storing the metadata in the file name. If you run thru
the full path, you can extract whatever metadata you need temporarily. Using
a file system as a database this way is a pretty common technique. (There
even used to be a pre-SQL database mechanism called "heirarchical database"
that worked just like that.)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
> If you use the original MD5 as the file name, then add the attributes to
> it, you don't even have to look at the metadata to tell if you already
> have the file. You don't have to hash it, or store the hash anywhere else.
Ehhh, well, the software I use generates and can perform operations on a
file's md5, but not (that I know how to do, at any rate) search for
similar images by comparing the filename to the md5. Also, when
casually browsing the drive from explorer, md5-as-filename isn't the
most user-friendly option. My original problem was whether or not to
include redundant data in a filename and its path...which I already do
for manga, so that question is more or less answered for the sake of
uniformity, as well as the fact that if I post a single image to a
bulletin board, it only refers to the filename, not its path.
--
Tim Cook
http://empyrean.freesitespace.net
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
>
> You can. You're just storing the metadata in the file name. If you run
> thru the full path, you can extract whatever metadata you need
> temporarily. Using a file system as a database this way is a pretty
> common technique. (There even used to be a pre-SQL database mechanism
> called "heirarchical database" that worked just like that.)
>
Well yes - not as easily and only up to some file system imposed limit.
Some scheme to store data within the file path/name is certainly
possible (I use it all the time). As the amount of metadata grows
though it becomes cumbersome and moving to a proper database makes sense.
Such a scheme is not assisted by the OS or most tools and utilities.
For example I could easily botch part of the path by swapping the 3rd
and 7th elements. Or format dates wrongly in the 4th element. Nothing
helps me to keep the scheme in place.
Actually I could store the content of a file in its filename (up to some
limit) and then just make the file have the EOF character in it. If it
is a binary file then just ASCII encode it! Extreme case.
Eg. C:\Shopping List\Groceries\20090524\Sunday\Tea, Biscuits (sweet), 4
Oranges, Fresh Corn on the cob (or canned if none available) and so on
until the OS won't let me continue.txt
Whereas if I put the metadata into database fields or file attributes
then some validation is enforced by the field type. There are well
supported concepts like sorting and searching.
Since you bring up hierarchical databases, I note that the relational
model has pretty much won that debate. Sure there are still examples of
H databases but I think you would agree that R is dominant today? And
for the same reasons that I advocated the approach to Tim.
Relational can't solve all problems but it is generally a closer
representation of the data, easier to manipulate, less redundancy,
better supported by tools ...
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Paul Fuller wrote:
> Well yes - not as easily and only up to some file system imposed limit.
Sure.
> Whereas if I put the metadata into database fields or file attributes
> then some validation is enforced by the field type. There are well
> supported concepts like sorting and searching.
Yep. And Windows at least does the indexing for you if you put the tags in
the metadata of the file, letting you do searches and editing of metadata
entirely within the tools that come with the system. (Which can be annoying
sometimes if that's not what you're trying to search. :-)
> Since you bring up hierarchical databases, I note that the relational
> model has pretty much won that debate. Sure there are still examples of
> H databases but I think you would agree that R is dominant today? And
> for the same reasons that I advocated the approach to Tim.
I wasn't trying to argue that he should use a non-relational database. I was
simply saying that (contrary to what some of the younger generation might
know) relational databases are at the end of the evolutionary chain, not the
start.
> Relational can't solve all problems but it is generally a closer
> representation of the data, easier to manipulate, less redundancy,
> better supported by tools ...
Yes to all that. I don't even know of any existing hierarchical databases
*other* than UNIX-style file systems.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Tim Cook wrote:
> Paul Fuller wrote:
>> "Names are fickle, Identifiers are permanent"
>
> The problem with tagging is that either a) it stores the data in an
> external database that doesn't follow the file when you copy it to a new
> medium or b) it alters the file, changing its MD5 hash and making it not
> identical to the other copies of it on the internet (or wherever), so
> it's harder to find duplicates in large sets of files or quickly
> determine which images of an existing set are missing.
for a) If you copy all the files to a new medium, copy the database along.
If you send just one file to someone, use a script that reads metadata from
the database and puts it on the filename before sending.
for b) Other people may also use tagging, or recompress a JPEG image, or
[...] which would already make your pictures have a different MD5 from
theirs. Use a real image comparison algorithm if you want to compare.
And in Windows, a lot of metadata may be stored in "alternate data streams"
instead of EXIF or something similar inside the file itself.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Darren New wrote:
>
> Yes to all that. I don't even know of any existing hierarchical
> databases *other* than UNIX-style file systems.
>
Well IBM IMS is still around. I don't have any stats on it but there
are still banks and possibly airlines with core systems that run happily
on it.
XML is also quite hierarchical in nature. Of course it can be used in
different ways but often it is just a tree structure with tags possibly
at different levels.
Not sure what to make of the likes of Hadoop and Big Table yet. I have
the impression that they are more distributed file systems that offer
some aspects of distributed DBMS - either hierarchical, relational or
whatever else you care to put on top of them.
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
Paul Fuller wrote:
> XML is also quite hierarchical in nature. Of course it can be used in
> different ways but often it is just a tree structure with tags possibly
> at different levels.
True. My take on that is that if you don't have arbitrary unstructured CDATA
in it, XML is the wrong tool. :-)
> Not sure what to make of the likes of Hadoop and Big Table yet. I have
> the impression that they are more distributed file systems that offer
> some aspects of distributed DBMS - either hierarchical, relational or
> whatever else you care to put on top of them.
GFS is just a big file system. Bigtable is a database shaped like a giant
spreadsheet (hence the name). And sure, there are all kinds of other
databases, like CouchDB, Amazon's various offerings, and you might consider
the whole web one giant distributed database in some sense. Lots of
interesting research going on out there.
But if you need a transactional ACID database, relational is pretty much the
only modern (non-legacy) contender.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |
| ![](/i/fill.gif) |
|
![](/i/fill.gif) |
|
![](/i/fill.gif) |
| ![](/i/fill.gif) |