From:
andrew cooke <andrew@...>
Date:
Sun, 5 Feb 2012 11:23:10 -0300
These are a bunch of random notes I wrote for work when someone asked what was
"new". I realise some of this is quite old - that's partly me not knowing
stuff, and partly because I wasn't sure what was required. Anyway, feedback
is welcomed.
The original was a wiki page; I've had to reformat it here.
Cloud Computing
Various ideas / acronyms related to "the cloud" are worth understanding:
IAAS - Infrastructure As A Service - This is where we are heading with our
VPS. The idea is that infrastricture is something that you buy as a service
(obviously). VPS is still tied to the idea of individual machines; IAAS is
pushing that more towards have a network of machines that grows on demand.
AWS http://aws.amazon.com/ are dominant, but there are also a bunch of other
players like RackSpace http://www.rackspace.com/, etc.
PAAS - Platform As A Service - GAE http://code.google.com/appengine/ (Google
App Engine) is the prime example. They have layered software over IAAS to
make everything simpler to use. You develop programs according to the
platform's API and end up with a system that automatically scales with
demand.
SAAS - Software As A Service - Google docs https://docs.google.com
etc. Software that comes from the cloud and stores data there, using only a
"thin client".
Writing a web service for GAE in Python is a very productive way to get a
site running quickly that will handle loads well (you can use much of Django
- the only new thing to learn is that way that data storage works - see "big
data" elsewhere).
Big Data
Related to the above, and driven by companies like Facebook, Twitter and
Google, are various ideas related to "big data". The most visible of these
are a whole slew of "NoSQL" databases. These are just starting to die off -
many didn't do much that SQL can't do anyway, like Object and XML databases
before - but various niches remain:
Memcaced http://memcached.org/ and Redis http://redis.io/ are proving useful
for caching data in-memory;
HBase http://hbase.apache.org/ and Google's Big Table
http://research.google.com/archive/bigtable.html work on problems so large
that SQL really does fail (closely related are Hadoop
http://hadoop.apache.org/ and Map reduce
http://code.google.com/edu/parallel/mapreduce-tutorial.html, which allow
data stored in those databases to be processed and collected) (Update:
Amazon just released their database, Dynamo
http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html);
Neo4j http://neo4j.org/ is focussed on storing graphs.
Associated with NoSQL http://en.wikipedia.org/wiki/NoSQL are various spin-off
technologies - libraries for serialisation of data like Avro
http://avro.apache.org/ and Thrift http://thrift.apache.org/ - and the rising
importance of data visualisation.
ZeroMQ http://www.zeromq.org/ feels like it fits here somehow, if only
because it follows the trend of reinventing stuff that already worked - it's
the most fashionable messaging library at the moment (simple and easy to use,
but without the reliability guarantees of "enterprise" messaging - think of
it more like TCP on steroids).
Web
Web programming is changing rapidly. Web applications are becoming more and
more dynamic. This is supported on the client by increasing use of Javascript
(the latest library is Backbone.js http://documentcloud.github.com/backbone/
which helps assemble web pages on the client from fragments of data). This
requires the server, increasingly, to "push" data (a typical HTTP page used
to be "pulled" by the client; now the server sends data even when the client
doesn't "know" that an update is needed). Push used to imply Comet
http://en.wikipedia.org/wiki/Comet_(programming) (a technology hack using
stalled HTTP requests), but HTML 5 http://www.html5rocks.com/en/ (which is
the future of HTML - XHTML is a largely unused dead-end) is introducing web
sockets http://www.html5rocks.com/en/tutorials/websockets/basics/ that allow
bi-directional transfer (like TCP sockets).
Kind-of related to server push is the idea that servers should be
asynchronous and event driven. Instead of having a thread for each
connection, there's just one thread in the server, which reacts to incoming
client requests on demand. For this to work the thread cannot "block" (eg
while waiting for data to be read from the database) so everything is handled
via callbacks (eg so a database read is handled by a library function (which
is a separate thread or process); once the data have been read from the
database a callback is used to transfer the data to the main thread to
continue processing). That is very like the way events are handled in
Javascript on the client, which is where Node.js http://nodejs.org/ and
server-side Javascript come in (if you know how traditional GUIs work, with a
single render thread, this is very similar). Related: JSON
http://people.planetpostgresql.org/andrew/index.php?/archives/244-Under-the-wire.html
as a database type http://en.wikipedia.org/wiki/Document-oriented_database;
Javscript http://code.google.com/p/plv8js/wiki/PLV8 in the database.
A common theme here is communication; another aspect of that is SPDY
http://dev.chromium.org/spdy, which is intended to replace HTTP, offering
more efficient mutliplexing of sessions. See this article
http://www.igvita.com/2012/01/18/building-a-modern-web-stack-for-the-realtime-web/
for an overview of how this affects the technology stack.
Back on the client, new technologies include WebGL
http://www.chromeexperiments.com/webgl (this lets you do graphics on the GPU,
so you can have 3D effects etc), Canvas
http://www.html5rocks.com/en/tutorials/canvas/integrating/ and SVG
http://www.html5rocks.com/en/tutorials/raphael/intro/ (for displaying data),
liquid layouts
http://designfestival.com/resizing-fixed-fluid-or-responsive-layouts/ (very
fluid, handling everything from phones to tables to desktop brownsers in a
single page) layouts, web storage http://en.wikipedia.org/wiki/Web_Storage
(on the client), much better support for fonts
http://designmodo.com/custom-web-fonts-fontface, and CSS3 (or processors like
Sass http://sass-lang.com/) (Update: possibly relevant book
http://shop.oreilly.com/product/0636920018421.do).
A big thread running through much of current web work is the idea of social
cues and interaction - sharing, feedback, progressing, belonging, etc.
Small Computers
Many of the web developments are being driven by mobile phones. And phone
hardware technology is now starting to spread - we're starting to see the
"internet of things" (where famously, even your fridge and toaster have
internet connections). One example of this is the Nest Labs thermostat
http://www.nest.com/; it is "just" a house heating thermostat, but "done like
Apple". Another example is Raspberry PI http://www.raspberrypi.org/ - a
computer capable of running Linux for just $25. Small computers are going to
be everywhere, very soon.
Connected to this, there's the idea of a general "dumbing down" of consumer
computing. You can see this in the iPad, for example (one take is that it's
intended to help people consume, not create). The general idea is of
"computing appliances" (with restricted choice of software from "app stores")
replacing general computing. Another driver for this is the crackdown on
"pirating".
Programming Languages
Java is looking increasingly outdated, but there's still no obvious
replacement. The next big language is going to have to combine two sets of
ideas: functional programming and better types. Functional programming, when
it works, tends to lead to simpler, more compact code, that is easier to make
parallel across multiple cores. Better types means that people want
programming in Java to be as quick and easy as programming in Python, while
keeping Java's speed and ability to detect errors at compile time.
For a long time it looked like Scala http://www.scala-lang.org/ would be the
future, but increasingly the consensus seems to be that it's just too damn
complicated. Clojure http://clojure.org/ is a more recent contender, but it
probably doesn't have enough ideas from the "better types" to win out. More
obscure (and less ambitious) possibilities include Kotlin
http://blog.jetbrains.com/kotlin/, Xtend http://www.eclipse.org/Xtext/xtend/
and Rust http://www.rust-lang.org/.
For system level programming Go http://golang.org/ is actually pretty good. I
don't think it's going to be huge, but it really is a better C (and way, way
nicer than C++). It's curiously old-fashioned compared to some of the
languages above, but perhaps that's an advantage.
Meantime, in the background, Javascript is becoming more and more important,
even though people don't really like to use it (the Good Parts
http://shop.oreilly.com/product/9780596517748.do book is excellent BTW). It
is becoming more important on the client (see elsewhere) and is also starting
to play a role on the server when writing asynchronous servers (again, see
elsewhere). This has triggered various attempts to make it more friendly:
Coffeescript http://coffeescript.org/ is a pre-processor that makes it look
more like Python; Clojurescript https://github.com/clojure/clojurescript
allows programs written in Clojure to be "compiled" to Javascript; Closure
http://code.google.com/closure/ (completely unrelated to Clojure) is a
compiler from Javascript to Javascript (!).
Increasingly, languages are supporting distributed and parallel computing.
Erlang http://www.erlang.org/ was an early success in this space. Clojure, as
mentioned above, is also pushing new ideas (actors and software transactional
memory). It's also possible that we need a completely new paradigm -
non-deterministic computing
http://my-inner-voice.blogspot.com/2012/01/many-core-processors-everything-you.html.
3D Printing
Is as mindblowingly awesome as jet packs, IMHO.
Andrew