| Andrew Cooke | Contents | Latest | RSS | Twitter | Previous | Next


Welcome to my blog, which was once a mailing list of the same name and is still generated by mail. Please reply via the "comment" links.

Always interested in offers/projects/new ideas. Eclectic experience in fields like: numerical computing; Python web; Java enterprise; functional languages; GPGPU; SQL databases; etc. Based in Santiago, Chile; telecommute worldwide. CV; email.

Personal Projects

Lepl parser for Python.

Colorless Green.

Photography around Santiago.

SVG experiment.

Professional Portfolio

Calibration of seismometers.

Data access via web services.

Cache rewrite.

Extending OpenSSH.

C-ORM: docs, API.

Last 100 entries

Consciousness From Max Entropy; Democrats; Harvard Will Fix Black Poverty; Modelling Bicycle Wheels; Amusing Polling Outlier; If Labour keeps telling working class people...; Populism and Choice; Books on Defeat; Enrique Ferrari - Argentine Author; Transcript of German Scientists on Learning of Hiroshima; Calvert Journal; Telephone System Quotes for Cat Soft LLC; Owen Jones on Twitter; Telephone System Quotes for Cat Soft LLC; Possible Japanese Authors; Complex American Literature; Chutney v5; Weird Componentized Virus; Interesting Argentinian Author - Antonio Di Benedetto; Useful Thread on MetaPhysics; RAND on fighting online anarchy (2001); Now Is Cat Soft LLC's Chance To Save Up To 32% On Mail; NSA Hacked; Call Center Services for Cat Soft LLC; Very Good LRB Article on Brexit; Nussbaum on Anger; Credit Card Processing for Cat Soft LLC; Discover new movies on demand in our online cinema; Tasting; Credit Card Processing for Cat Soft LLC; Apple + Kiwi Jam; Hit Me; Increase Efficiency with GPS Vehicle Tracking for Cat Soft LLC; Sudoku - CSP + Chaos; Recycling Electronics In Santiago; Vector Displays in OpenGL; Call Center Services for Cat Soft LLC; And Anti-Aliased; OpenGL - Render via Intermediate Texture; And Garmin Connect; Using Garmin Forerunner 230 With Linux; Payroll Service Quotes for Cat Soft LLC; (Beating Dead Horse) StackOverflow; Current State of Justice in China; Now Is Cat Soft LLC's Chance To Save Up To 32% On Mail; Axiom of Determinacy; Ewww; Fee Chaos Book; Course on Differential Geometry; Increase Efficiency with GPS Vehicle Tracking for Cat Soft LLC; Okay, but...; Sparse Matrices, Deep Learning; Sounds Bad; Applebaum Rape; Tomato Chutney v4; Have to add...; Culturally Liberal and Nothing More; Weird Finite / Infinite Result; Your diamond is a beaten up mess; Maths Books; Good Bike Route from Providencia / Las Condes to Panul\; Iain Pears (Author of Complex Plots); Plum Jam; Excellent; More Recently; For a moment I forgot StackOverflow sucked; A Few Weeks On...; Chilean Book Recommendations; How To Write Shared Libraries; Jenny Erpenbeck (Author); Dijkstra, Coins, Tables; Python libraries error on OpenSuse; Deserving Trump; And Smugness; McCloskey Economics Trilogy; cmocka - Mocks for C; Concept Creep (Americans); Futhark - OpenCL Language; Moved / Gone; Fan and USB issues; Burgers in Santiago; The Origin of Icosahedral Symmetry in Viruses; autoenum on PyPI; Jars Explains; Tomato Chutney v3; REST; US Elections and Gender: 24 Point Swing; PPPoE on OpenSuse Leap 42.1; SuperMicro X10SDV-TLN4F/F with Opensuse Leap 42.1; Big Data AI Could Be Very Bad Indeed....; Cornering; Postcapitalism (Paul Mason); Black Science Fiction; Git is not a CDN; Mining of Massive Data Sets; Rachel Kaadzi Ghansah; How great republics meet their end; Raspberry, Strawberry and Banana Jam; Interesting Dead Areas of Math; Later Taste; For Sale

© 2006-2015 Andrew Cooke (site) / post authors (content).

Erlang - Processes, Objects, Protocols

From: "andrew cooke" <andrew@...>

Date: Wed, 2 May 2007 22:13:06 -0400 (CLT)

I was going to write a long post about this code, but I really don't have
the time.  So it's just here in case anyone feels like reading it.  The
first section defines a protocol and supporting functions for chaining
data through different processes (the "unix pipe" idea).  The protocol is
minimal - just data or exit - but each process can have some persistent
state that is preserved across calls.

I'm particularly proud of the formatting here :o)  Took me some time to
work out what is (and isn't) possible with the syntax, scoping, etc.

forward(none, Message) -> Message;
forward(Next, Message) -> Next(Message).

new(Response, Filter, Name, Next) ->

    DoLog =
        fun(Template, Params) -> res_info(Response, Template, Params) end,

    % when the Filter is called with a {data, State, Data} tuple it
    % should return one of:
    % - {data, Data2} containing processed data
    % - {state, State2} containing new state
    % - {both, State2, Data2} containing new state and data
    % new data (Data2) are forwarded to the next filter in the chain,
    % if defined (it may be "none") even if identical to old values.
    % to avoid forwarding anything to the next filter, return state
    % alone.

    DoData =
        fun(Driver, _State, {both, State2, Data2}) ->
                forward(Next, {data, Data2}),
                Driver(Driver, State2);
           (Driver, State, {data, Data2}) ->
                forward(Next, {data, Data2}),
                Driver(Driver, State);
           (Driver, _State, {state, State2}) ->
                Driver(Driver, State2);
           (_Driver, _State, Other) ->
                DoLog("Unexpected result ~p in ~p", [Other, Name])

    % when the Filter is called with the {exit, State} tuple it should
    % return one of:
    % - {data, Data} containing final data to be forwarded
    % - exit
    % if data are returned they are passed to the next filter (if
    % defined), after which, "exit" is forwarded.

    DoExit =
        fun({data, Data}) ->
                forward(Next, {data, Data}),
           (exit) ->
           (Other) ->
                DoLog("Unexpected result ~p in ~p", [Other, Name])

    % driver loop.

    Driver =
        fun(Driver, State) ->
                    {data, Data} ->
                        DoData(Driver, State, Filter({data, State, Data}));
                    exit ->
                        DoExit(Filter({exit, State}));
                    Other ->
                        DoLog("Unexpected message ~p in ~p", [Other, Name])

    Pid = spawn(fun() -> Driver(Driver, Filter(start)) end),
    fun(Message) -> Pid ! Message end.

Next, some example processes.  First, a simple one that groups the data by
lines (ended by newline character).  Note the exit strategy handles a
missing final newline.

lines(start) -> [];
lines({data, Partial, Data}) ->
    {Partial2, Lines} = collect(Data, Partial, []),
    {both, Partial2, Lines};
lines({exit, Partial}) -> {data, [lists:reverse(Partial)]}.

collect([], Partial, Lines) -> {Partial, lists:reverse(Lines)};
collect([$\n|Data], Partial, Lines) ->
    Line = lists:reverse([$\n|Partial]),
    collect(Data, [], [Line|Lines]);
collect([C|Data], Partial, Lines) ->
    collect(Data, [C|Partial], Lines).

And second, a process that "gunzips" a stream.  As far as I can tell, the
Erlang zlib library doesn't (directly) support streaming gunzip, so this
accumulates the data beforehand.

unzip(start) -> [];
unzip({data, Acc, Data}) -> {state, [Data|Acc]};
unzip({exit, Acc}) ->
    Compressed = list_to_binary(lists:reverse(Acc)),
    Decompressed = binary_to_list(zlib:gunzip(Compressed)),
    {data, Decompressed}.

I was worried that this buffering would make subsequent chained processes
inefficient (processors waiting on data), so here's something that
fragments the input to a process (and shows that wrapping these processes
as functions pays off).

shape_before(Fun, ChunkLimit) ->
    fun({data, Data}) -> fragment(Fun, ChunkLimit, Data);
       (Message) -> Fun(Message)

fragment(_Fun, _Chunklimit, []) -> ok;
fragment(Fun, Chunklimit, Data) when length(Data) > Chunklimit ->
    {Chunk, Data2} = lists:split(Chunklimit, Data),
    Fun({data, Chunk}),
    fragment(Fun, Chunklimit, Data2);
fragment(Fun, _Chunklimit, Data) ->
    Fun({data, Data}).

Finally, this is construction of a chain of processes, with fragmentation
and optional compression:

close(Response) ->
    res_info(Response, "Closing"),
    Channel = response:channel(Response),
    socket:close(channel:browser(Channel), channel:proxy(Channel)).

build_chain(Response) ->
    Echo = chain:link(Response, chain:echo(Response), "Echo"),
    Links = chain:new(Response, fun chain_links:links/1, "Links", Echo),
    Lines = chain:new(Response, fun chain_lines:lines/1, "Lines", Links),
    res_info(Response, "Encoding ~p", [response:encoding(Response)]),
    case response:encoding(Response) of
        unknown -> Lines;
        "gzip" -> chain:new(Response, fun chain_unzip:unzip/1, "Gzip",
                            chain:shape_before(Lines, 2000));
        Encoding -> {error, Encoding}

The processes are in reverse order, so this (optionally) unzips, splits
into lines, searches for hypertext limits, and echoes the result to the
screen.  As you might have guessed, I'm processing HTTP data flows.


Re: Objects

From: "andrew cooke" <andrew@...>

Date: Wed, 2 May 2007 22:16:07 -0400 (CLT)

Ooops - see that I left "Objects" in the title from earlier.  I was
originally going to talk about persistent state and the similarities
between method calls and dispatching on tuple tags.  But I'm sure you can
see that anyway :o)


First bug

From: "andrew cooke" <andrew@...>

Date: Wed, 2 May 2007 22:22:29 -0400 (CLT)

Staring at that, I've already seen one bug.  No prizes!


Comment on this post