| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

A Verion That Redirects To Local HTTP Server; Spanish Accents For Idiots; Aluminium Cans; Advice on Spray Painting; Female View of Online Chat From a Male; UX Reading List; S4 Subgroups - Geometric Interpretation; Fucking Email; The SQM Affair For Idiots; Using Kolmogorov Complexity; Oblique Strategies in bash; Curses Tools; Markov Chain Monte Carlo Without all the Bullshit; Email Para Matias Godoy Mercado; The Penta Affair For Idiots; Example Code To Create numpy Array in C; Good Article on Bias in Graphic Design (NYTimes); Do You Backup github?; Data Mining Books; SimpleDateFormat should be synchronized; British Words; Chinese Govt Intercepts External Web To DDOS github; Numbering Permutations; Teenage Engineering - Low Price Synths; GCHQ Can Do Whatever It Wants; Dublinesque; A Cryptographic SAT Solver; Security Challenges; Word Lists for Crosswords; 3D Printing and Speaker Design; Searchable Snowden Archive; XCode Backdoored; Derived Apps Have Malware (CIA); Rowhammer - Hacking Software Via Hardware (DRAM) Bugs; Immutable SQL Database (Kinda); Tor GPS Tracker; That PyCon Dongle Mess...; ASCII Fluid Dynamics; Brandalism; Table of Shifter, Cassette and Derailleur Compatability; Lenovo Demonstrates How Bad HTTPS Is; Telegraph Owned by HSBC; Smaptop - Sunrise (Music); Equation Group (NSA); UK Torture in NI; And - A Natural Extension To Regexps; This Is The Future Of Religion; The Shazam (Music Matching) Algorithm; Tributes To Lesbian Community From AIDS Survivors; Nice Rust Summary; List of Good Fiction Books; Constructing JSON From Postgres (Part 2); Constructing JSON From Postgres (Part 1); Postgres in Docker; Why Poor Places Are More Diverse; Smart Writing on Graceland; Satire in France; Free Speech in France; MTB Cornering - Where Should We Point Our Thrusters?; Secure Secure Shell; Java Generics over Primitives; 2014 (Charlie Brooker); How I am 7; Neural Nets Applied to Go; Programming, Business, Social Contracts; Distributed Systems for Fun and Profit; XML and Scheme; Internet Radio Stations (Curated List); Solid Data About Placebos; Half of Americans Think Climate Change Is a Sign of the Apocalypse; Saturday Surf Sessions With Juvenile Delinquents; Ssh, tty, stdout and stderr; Feathers falling in a vacuum; Santiago 30m Bike Route; Mapa de Ciclovias en Santiago; How Unreliable is UDP?; SE Santiago 20m Bike Route; Cameron's Rap; Configuring libxml with Eclipse; Reducing Combinatorial Complexity With Occam - AI; Sentidos Comunes (Chilean Online Magazine); Hilary Mantel: The Assassination of Margaret Thatcher - August 6th 1983; NSA Interceptng Gmail During Delivery; General IIR Filters; What's happening with Scala?; Interesting (But Largely Illegible) Typeface; Retiring Essentialism; Poorest in UK, Poorest in N Europe; I Want To Be A Redneck!; Reverse Racism; The Lost Art Of Nomography; IBM Data Center (Photo); Interesting Account Of Gamma Hack; The Most Interesting Audiophile In The World; How did the first world war actually end?; Ky - Restaurant Santiago; The Black Dork Lives!; The UN Requires Unaninmous Decisions; LPIR - Steganography in Practice; How I Am 6; Clear Explanation of Verizon / Level 3 / Netflix; Teenage Girls

© 2006-2015 Andrew Cooke (site) / post authors (content).

Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Sun, 20 May 2012 20:43:21 -0400

I'm currently working on some code that generates graphical representations of
hashes.  The idea is that you might use them to check downloaded files, much
like numerical hashes (in a perfect world you would use both, for details that
I won't go into here).

Now it's all fun + games developing an algorithm, but I am now wondering how
best to quantify the accuracy of the results.  In particular, how unique is
each image?

This is a hard problem - it involves psycho-optics (asusming I've not made
that word up) - but is simplified a little by the approach I have used.

First, the image is "quantised" as a mosaic (it is not a smooth image, but
built from squares of colour).  This removes issues about things like "feature

Second, each mosaic is generated from a base colour and an array of float
values in the range [-1 1].  There is one float per tile in the mosaic, which
represents the "distance" from the base colour (this is translated into a
change in hue and lightness, which are correlated so that the results can be
distinguished even by colourblind users).

So to a first approximation we can ignore a lot of the hard parts and focus on
"how closely" arrays of float values match.  The rest of the email describes
how I will do this.

To find useful matches I will need (I hope!) quite a large data set.  So the
most expensive part of the processing is likely the generation of many hashes
(as you might expect, generating a hash involves quite complex calculations
since it relies on cryptographic primitives).

The first step in my analysis is, therefore, to generate a large set of data.
I will convert each [-1 1] range to a byte, and write the data to a file.
Since the value can be the "line" number, this could be a simple binary file -
that would support fast random access, although I am not sure I need it.

Next, two filters that operate on that data.  One selecting random (but fixed
per run) "pattern" of bytes and another reducing the byte by discarding least
significant bits.

And finally, a program that buckets the filtered data, looking for matches.

The idea is that the selected, reduced data form simple locality-sensitive
hashes, and that the sensitivity of the hashes can be tuned by hand (the
filters and buckets being fairly fast to re-run, and with easy-to-understand

In this way I hope to be able to calculate how frequent collisions are for
different resolutions (bits per float).  Even if the bit resolution at which I
can detect collisions is so low that the "real" images look different I may be
able to extrapolate to higher resolutions.


Re: Finding Matches in Graphical Hashes

From: Michiel Buddingh' <michiel@...>

Date: Mon, 21 May 2012 06:03:36 +0200

From what I've heard, one of the more current metrics to evaluate
image or video compression quality is the Structural Similarity Index

You might also want to incorporate something as described here:
http://stevehanov.ca/blog/index.php?id=62 , mapping the bit values to
a L*u*v colour space rather than to a RGB colour space, presuming you
don't already do this.

Good luck!  I've noticed that ssh-keygen had started outputting teeny
ascii-art visual fingerprints for newly minted keys, but in the
post-teletype age, what you describe makes a lot more sense.


Re: Finding Matches in Graphical Hashes

From: andrew cooke <andrew@...>

Date: Mon, 21 May 2012 20:24:52 -0400

Thanks for the pointers.  SSIM looks interesting - a lot simpler than I
expected.  I am in the middle of generating 10M hashes and will try that on

As for Luv - I am actually using HSL which is a clunky approximation that is
much easier to deal with but not as "physiological".  However, the work
described here is actually on an earlier form - just an array of float values
between -1 and 1 (the HSL is generated from those - basically they are used to
select a hue and lightness).


Comment on this post