Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next

C[omp]ute

Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

[Link, Computing] Useful gcc flags; [Link] Voynich Manuscript Decoded; [Bike] Notes on Servicing Suspension Forks; [Links, Computing] Snap, Flatpack, Appimage; [Link, Computing] Oracle is leaving Java (to die); [Link, Politics] Cubans + Ultrasonics; [Book, Link] Laurent Binet; VirtualBox; [Book, Link] No One's Ways; [Link] The Biggest Problem For Cyclists Is Bad Driving; [Computing] Doxygen, Sphinx, Breathe; [Admin] Brokw Recent Permalinks; [Bike, Chile] Buying Bearings in Santiago; [Computing, Opensuse] Upgrading to 42.3; [Link, Physics] First Support for a Physics Theory of Life; [Link, Bike] Peruvian Frame Maker; [Link] Awesome Game Theory Tit-For-Tat Thing; [Food, Review] La Fabbrica - Good Italian Food In Santiago; [Link, Programming] MySQL UTF8 Broken; [Link, Books] Latin American Authors; [Link, Computing] Optimizatin Puzzle; [Link, Books, Politics] Orwell Prize; [Link] What the Hell Is Happening With Qatar?; [Link] Deep Learning + Virtual Tensor Machines; [Link] Scaled Composites: Largest Wingspan Ever; [Link] SCP Foundation; [Bike] Lessons From 2 Leading 2 Trailing; [Link] Veg Restaurants in Santiago; [Link] List of Contemporary Latin American Authors; [Bike] FTHR; [Link] Whoa - NSA Reduces Collection (of US Residents); [Link] Red Bull's Breitbart; [Link] Linux Threads; [Link] Punycode; [Link] Bull / Girl Statues on Wall Street; [Link] Beautiful Chair Video; Update: Lower Pressures; [Link] Neat Python Exceptions; [Link] Fix for Windows 10 to Avoid Ads; [Link] Attacks on ZRTP; [Link] UK Jazz Invasion; [Review] Cuba; [Link] Aricle on Gender Reversal of US Presidential Debate; {OpenSuse] Fix for Network Offline in Updater Applet; [Link] Parkinson's Related to Gut Flora; Farellones Bike Park; [Meta] Tags; Update: Second Ride; Schwalbe Thunder Burt 2.1 v Continental X-King 2.4; Mountain Biking in Santiago; Books on Ethics; Security Fail from Command Driven Interface; Everything Old is New Again; Interesting Take on Trump's Lies; Chutney v6; References on Entropy; Amusing "Alexa.." broadcast; The Shame of Chile's Education System; Playing mp4 gifs in Firefox on Opensuses Leap 42.2; Concurrency at Microsoft; Globalisation: Uk -> Chile; OpenSuse 42.2 and Synaptics Touch-Pads; Even; Cherry Jam; Lebanese Writer Amin Maalouf; C++ - it's the language of the future; Learning From Trump; Chinese Writer Hu Fayun; And; Apricot Jam; Also; Excellent Article on USA Politics; Oh Metafilter; Prejudice Against The Rurals; Also, Zizek; Trump; Why Trump Won; Doxygen + Latex on CentOS 6; SMASH - Solve 5 Biggest Problems in Physics; Good article on racism, brexit, and social divides; Grandaddy are back!; Consciousness From Max Entropy; Democrats; Harvard Will Fix Black Poverty; Modelling Bicycle Wheels; Amusing Polling Outlier; If Labour keeps telling working class people...; Populism and Choice; Books on Defeat; Enrique Ferrari - Argentine Author; Transcript of German Scientists on Learning of Hiroshima; Calvert Journal; Owen Jones on Twitter; Possible Japanese Authors; Complex American Literature; Chutney v5; Weird Componentized Virus; Interesting Argentinian Author - Antonio Di Benedetto; Useful Thread on MetaPhysics; RAND on fighting online anarchy (2001); NSA Hacked

© 2006-2017 Andrew Cooke (site) / post authors (content).

(Not So) Random Walks on Graphs

From: andrew cooke <andrew@...>

Date: Tue, 27 Oct 2009 20:14:42 -0300

So, UKYFD - http://uykfd.googlecode.com/ - is working, sort of.  The
trouble is, that it seems to get stuck in a rather popular group of
artists.  It's quite possible this is a bug in the implementation, and
I should really add better logging and some kind of graphical
visualisation to understand better what's going on.

But it's cheaper to think.

And I think that what might be happening is that my rather jumbled and
ad-hoc exploration algorithm tends to guide the walk into this
cluster, then keep it there.  I don't have any proof, but this seems
quite a reasonable if that cluster has many highly-weighted links.

At this point I should record what the current algorithm is:

1 - Lastfm artist tags are used to construct "weights" between related
artists.  To guarantee a completely connected graph a very low weight
link is added between all artists (so the graph is not just connected,
but complete)

2 - From the weights, a table of "distances" for the complete graph is
calculated (these are just weights again, but take into account
"friend of a friend".

The above two steps are done once, the following steps are done for
each track in a playlist, starting with a track selected at random:

3 - Tracks for the K closest artists to the artist of the current
track are selected and given a score of 1.

4 - Tracks for artists directly connected (as 1 above, ignoring the
low weight link) have their score multiplied by 10.

5 - Tracks scores are modified according to distance from the previous
artist.  The idea here is that by penalising or rewarding distance we
can control the form of the walk (2nd order Markov process)

6 - Tracks that are from an artist played in the last L tracks have
scores lowered by a factor of 10.

7 Tracks that have been played in the last M tracks have scores
lowered by a factor of 10.

8 A track is chose at random, with likelihood proportional to the score.

And looking at that, I think it's clear that possibilities for the
kind of clustering described above exist.  So what would a better
process look like?

Steps 6-8 seem fine to me - they just push us towards a self-avoiding
walk, which is not a problem.  Any bias seems to come from the earlier
steps.  Here's a possible alternative:

A - Before generating a playlist (ie for each playlist) a new graph is
constructed over the artists in which all have the same number of
neighbours (some fairly small number like 5 or 10).  For some isolated
artists this will mean choosing at random from equi-distant "very
distant" artists (via the low weight tag added to keep the graph
connected).  This random choice should be different for each playlist
(hence the need to reconstruct this graph each time).

B - An artist is chosen, using a non-parametric (ie link count, not
weight) version of the second order Markov scoring.  Perhaps at first
this is simply random.  Perhaps avoidance of previously played artists
comes here.

C - Once the artist is chose, a track is chosen, avoiding repetition
if possible.

By keeping making all nodes the same order, and removing weights, I
hope the above addresses any clustering issues.  I don't think this is
certain though - it seems to me that there could still be
topological(?)/clustering effects.

Andrew

Comment on this post