C[omp]ute

From: andrew cooke <andrew@...>

Date: Sun, 20 May 2012 20:43:21 -0400

I'm currently working on some code that generates graphical representations of
hashes.  The idea is that you might use them to check downloaded files, much
like numerical hashes (in a perfect world you would use both, for details that
I won't go into here).

Now it's all fun + games developing an algorithm, but I am now wondering how
best to quantify the accuracy of the results.  In particular, how unique is
each image?

This is a hard problem - it involves psycho-optics (asusming I've not made
that word up) - but is simplified a little by the approach I have used.

First, the image is "quantised" as a mosaic (it is not a smooth image, but
built from squares of colour).  This removes issues about things like "feature
size".

Second, each mosaic is generated from a base colour and an array of float
values in the range [-1 1].  There is one float per tile in the mosaic, which
represents the "distance" from the base colour (this is translated into a
change in hue and lightness, which are correlated so that the results can be
distinguished even by colourblind users).

So to a first approximation we can ignore a lot of the hard parts and focus on
"how closely" arrays of float values match.  The rest of the email describes
how I will do this.

To find useful matches I will need (I hope!) quite a large data set.  So the
most expensive part of the processing is likely the generation of many hashes
(as you might expect, generating a hash involves quite complex calculations
since it relies on cryptographic primitives).

The first step in my analysis is, therefore, to generate a large set of data.
I will convert each [-1 1] range to a byte, and write the data to a file.
Since the value can be the "line" number, this could be a simple binary file -
that would support fast random access, although I am not sure I need it.

Next, two filters that operate on that data.  One selecting random (but fixed
per run) "pattern" of bytes and another reducing the byte by discarding least
significant bits.

And finally, a program that buckets the filtered data, looking for matches.

The idea is that the selected, reduced data form simple locality-sensitive
hashes, and that the sensitivity of the hashes can be tuned by hand (the
filters and buckets being fairly fast to re-run, and with easy-to-understand
parameters.

In this way I hope to be able to calculate how frequent collisions are for
different resolutions (bits per float).  Even if the bit resolution at which I
can detect collisions is so low that the "real" images look different I may be
able to extrapolate to higher resolutions.

Andrew