| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

Last 100 entries

Headphone Test Recordings; Causal Consistency; The Quest for Randomness; Chat Wars; Real-life Financial Co Without ACID Database...; Flexible Muscle-Based Locomotion for Bipedal Creatures; SQL Performance Explained; The Little Manual of API Design; Multiple Word Sizes; CRC - Next Steps; FizzBuzz; Update on CRCs; Decent Links / Discussion Community; Automated Reasoning About LLVM Optimizations and Undefined Behavior; A Painless Guide To CRC Error Detection Algorithms; Tests in Julia; Dave Eggers: what's so funny about peace, love and Starship?; Cello - High Level C Programming; autoreconf needs tar; Will Self Goes To Heathrow; Top 5 BioInformatics Papers; Vasovagal Response; Good Food in Vina; Chilean Drug Criminals Use Subsitution Cipher; Adrenaline; Stiglitz on the Impact of Technology; Why Not; How I Am 5; Lenovo X240 OpenSuse 13.1; NSA and GCHQ - Psychological Trolls; Finite Fields in Julia (Defining Your Own Number Type); Julian Assange; Starting Qemu on OpenSuse; Noisy GAs/TMs; Venezuela; Reinstalling GRUB with EFI; Instructions For Disabling KDE Indexing; Evolving Speakers; Changing Salt Size in Simple Crypt 3.0.0; Logarithmic Map (Moved); More Info; Words Found in Voynich Manuscript; An Inventory Of 3D Space-Filling Curves; Foxes Using Magnetic Fields To Hunt; 5 Rounds RC5 No Rotation; JP Morgan and Madoff; Ori - Secure, Distributed File System; Physical Unclonable Functions (PUFs); Prejudice on Reddit; Recursion OK; Optimizing Julia Code; Cash Handouts in Brazil; Couple Nice Music Videos; It Also Works!; Adaptive Plaintext; It Works!; RC5 Without Rotation (2); 8 Years...; Attack Against Encrypted Linux Disks; Pushing Back On NSA At IETF; Summary of Experimental Ethics; Very Good Talk On Security, Snowden; Locusts are Grasshoppers!; Vagrant (OpenSuse and IDEs); Interesting Take On Mandela's Context; Haskell Cabal O(n^2) / O(n) Fix; How I Am 4; Chilean Charity Supporting Women; Doing SSH right; Festival of Urban Intervention; Neat Idea - Wormholes Provide Entanglement; And a Link....; Simple Encryption for Python 2.7; OpenSuse 13.1 Is Better!; Little Gain...; More Details on Technofull Damage; Palmrest Cracked Too....; Tecnofull (Lenovo Support) Is Fucking Useless; The Neuroscientist Who Discovered He Was a Psychopath; Interpolating Polynomials; Bottlehead Crack as Pre-amp; Ooops K702!; Bottlehead Crack, AKG K701; Breaking RC5 Without Rotation; Great post thank you; Big Balls of Mud; Phabricator - Tools for working together; Amazing Julia RC5 Code Parameterized By Word Size; Chi-Square Can Be Two-Sided; Why Do Brits Accept Surveillance?; Statistics Done Wrong; Mesas Trape from Bravo; European Report on Crypto Primitives and Protocols; Interesting Omissions; Oryx And Crake (Margaret Atwood); Music and Theory; My Arduino Programs; Elliptic Curve Crypto; Re: Licensing Interpreted Code; Licensing Interpreted Code; ASUS 1015E-DS03 OpenSuse 12.3 SSD

© 2006-2013 Andrew Cooke (site) / post authors (content).

(Not So) Random Walks on Graphs

From: andrew cooke <andrew@...>

Date: Tue, 27 Oct 2009 20:14:42 -0300

So, UKYFD - http://uykfd.googlecode.com/ - is working, sort of.  The
trouble is, that it seems to get stuck in a rather popular group of
artists.  It's quite possible this is a bug in the implementation, and
I should really add better logging and some kind of graphical
visualisation to understand better what's going on.

But it's cheaper to think.

And I think that what might be happening is that my rather jumbled and
ad-hoc exploration algorithm tends to guide the walk into this
cluster, then keep it there.  I don't have any proof, but this seems
quite a reasonable if that cluster has many highly-weighted links.

At this point I should record what the current algorithm is:

1 - Lastfm artist tags are used to construct "weights" between related
artists.  To guarantee a completely connected graph a very low weight
link is added between all artists (so the graph is not just connected,
but complete)

2 - From the weights, a table of "distances" for the complete graph is
calculated (these are just weights again, but take into account
"friend of a friend".

The above two steps are done once, the following steps are done for
each track in a playlist, starting with a track selected at random:

3 - Tracks for the K closest artists to the artist of the current
track are selected and given a score of 1.

4 - Tracks for artists directly connected (as 1 above, ignoring the
low weight link) have their score multiplied by 10.

5 - Tracks scores are modified according to distance from the previous
artist.  The idea here is that by penalising or rewarding distance we
can control the form of the walk (2nd order Markov process)

6 - Tracks that are from an artist played in the last L tracks have
scores lowered by a factor of 10.

7 Tracks that have been played in the last M tracks have scores
lowered by a factor of 10.

8 A track is chose at random, with likelihood proportional to the score.

And looking at that, I think it's clear that possibilities for the
kind of clustering described above exist.  So what would a better
process look like?

Steps 6-8 seem fine to me - they just push us towards a self-avoiding
walk, which is not a problem.  Any bias seems to come from the earlier
steps.  Here's a possible alternative:

A - Before generating a playlist (ie for each playlist) a new graph is
constructed over the artists in which all have the same number of
neighbours (some fairly small number like 5 or 10).  For some isolated
artists this will mean choosing at random from equi-distant "very
distant" artists (via the low weight tag added to keep the graph
connected).  This random choice should be different for each playlist
(hence the need to reconstruct this graph each time).

B - An artist is chosen, using a non-parametric (ie link count, not
weight) version of the second order Markov scoring.  Perhaps at first
this is simply random.  Perhaps avoidance of previously played artists
comes here.

C - Once the artist is chose, a track is chosen, avoiding repetition
if possible.

By keeping making all nodes the same order, and removing weights, I
hope the above addresses any clustering issues.  I don't think this is
certain though - it seems to me that there could still be
topological(?)/clustering effects.

Andrew

Comment on this post