Beware of the Train

• I can't safely modify, or even understand, this code, because it has no tests.
• I can't test it without modifying it.

necaris

• CouchDB is, at its core, an HTTP server providing append-only access to B-trees of versioned JSON objects via a RESTful interface. Say what now? Well, you store your data as JavaScript-like objects (which allow you to nest arrays and hash tables freely); each object is indexed by a key; you access existing objects and insert new ones using the standard HTTP GET, PUT and DELETE methods, specifying and receiving data in JavaScript Object Notation; you can't update objects, only replace them with new objects with the same key and a higher version number; and it's cheap to request all the objects with keys in a given range.


• The JSON is not by default required to conform to any particular schema, but you can add validation functions to be called every time data is added to the database. These will reject improperly-formed data.


• CouchDB is at pains to be RESTful, to emit proper cache-invalidation data, and so on, and this is key to scaling it out: put a contiguous subset of (a consistent hash of) the keyspace on each machine, and build a tree of reverse HTTP proxies (possibly caching ones) in front of your database cluster.


• CouchDB's killer feature is probably master-to-master replication: if you want to do DB operations on a machine that's sometimes disconnected from the rest of the cluster (a mobile device, say), then you can do so, and sync changes up and down when you reconnect. Conflicts are flagged but not resolved by default; you can resolve them manually or automatically by recording a new version of the conflicted object. Replication is also used for load-balancing, failover and scaling out: you can maintain one or more machines that constantly replicate the master server for a section of keyspace, and you can replicate only a subset of keyspace onto a new database when you need to expand.


• CouchDB doesn't guarantee to preserve all the history of an object, and in particular replications only seem to send the most recent version; I think this precludes Git-style three-way merge from the conflicting versions' most recent common ancestor (and forget about Darcs-style full-history merging!).


• The cluster-management story isn't as good as for some other systems, but there are a couple of PaaS offerings.


• Queries/views and non-primary indexes are both handled using map/reduce. If you want to index on something other than the primary key - posts by date, say - then you write a map query which emits (date, post) pairs. These are put into another B-tree, which is stored on disk; clever things are done to mark subtrees invalid as new data comes in, and changes to the query result or index are calculated lazily. Since indices are stored as B-trees, it's cheap to get all the objects within a given range of secondary keys: all posts in February, for instance.


• CouchDB's reduce functions are crippled: attempting to calculate anything that isn't a scalar or a fixed-size object is considered Bad Form, and may cause your machine(s) to thrash. AFAICT you can't reduce results from different machines by this mechanism: CouchDB Lounge requires you to write extra merge functions in Twisted Python.


• Map, reduce and validation functions (and various others, see below) are by default written in JavaScript. But CouchDB invokes an external interpreter for them, so it's easy to extend CouchDB with a new query server. Several such have been written, and it's now possible to write your functions in many different languages.


• There's a very limited SQL view engine, but AFAICT nothing like Hive or Pig that can take a complex query and compile it down into a number of chained map/reduce jobs. The aforementioned restrictions on reduce functions mean that the strategy I've been taught for expressing joins as map/reduce jobs won't work; I don't know if this limitation is fundamental. But it's IME pretty rare to require general joins in applications: usually you want to do some filtering or summarisation on at least one side.


• CouchDB can't quite make up its mind whether it wants to be a database or a Web application framework. It comes by default with an administration web app called Futon; you can also use it to store and execute code for rendering objects as HTML, Atom, etc. Such code (along with views, validations etc) is stored in special JSON objects called "design documents": best practice is apparently to have one design document for each application that needs to access the underlying data. Since design documents are ordinary JSON objects, they are propagated between nodes by replications.


• However, various standard webapp-framework bits are missing, notably URL routing. But hey, you can always use mod_rewrite...


• There's a tool called Erica (and an older one called CouchApp) which allows you to sync design documents with more conventional source-code directories in your filesystem.


• CouchDB is written in Erlang, and the functional-programming influence shows up in other places: most types of user-defined function are required to be free of side-effects, for instance. Then there's the aforementioned uses of lazy evaluation and the append-only nature of the system as a whole. You can extend it with your own Erlang code or embed it into an Erlang application, bypassing the need for HTTP requests.

[ui]
username = Pozorvlak <pozorvlak@example.com>
merge = internal:merge
[pager]
pager = LESS='FSRX' less
[extensions]
rebase =
record =
histedit = ~/usr/etc/hg/hg_histedit.py
fetch =
shelve = ~/usr/etc/hg/hgshelve.py
pager =
mq =
color =

Inspired by Falsehoods Programmers Believe About Names, Falsehoods Programmers Believe About Time, and far, far too much time spent fighting autotools. Thanks to Aaron Crane, totherme and zeecat for their comments on earlier versions.

• Build graphs are trees.
• Build graphs are acyclic.
• Every build step updates at most one file.
• Every build step updates at least one file.
• Compilers will always modify the timestamps on every file they are expected to output.
• It's possible to tell the compiler which file to write its output to.
• It's possible to tell the compiler which directory to write its output to.
• It's possible to predict in advance which files the compiler will update.
• It's possible to narrow down the set of possibly-updated files to a small hand-enumerated set.
• It's possible to determine the dependencies of a target without building it.
• Targets do not depend on the rules used to build them.
• Targets depend on every rule in the whole build system.
• Detecting changes via file hashes is always the right thing.
• Detecting changes via file hashes is never the right thing.
• Nobody will ever want to rebuild a subset of the available dirty targets.
• People will only want to build software on Linux.
• People will only want to build software on a Unix derivative.
• Nobody will want to build software on Windows.
• People will only want to build software on Windows.
  (Thanks to David MacIver for spotting this omission.)
• Nobody will want to build on a system without strace or some equivalent.
• stat is slow on modern filesystems.
• Non-experts can reliably write portable shell script.
• Your build tool is a great opportunity to invent a whole new language.
• Said language does not need to be a full-featured programming language.
• In particular, said language does not need a module system more sophisticated than #include.
• Said language should be based on textual expansion.
• Adding an Nth layer of textual expansion will fix the problems of the preceding N-1 layers.
• Single-character magic variables are a good idea in a language that most programmers will rarely use.
• System libraries and globally-installed tools never change.
• Version numbers of system libraries and globally-installed tools only ever increase.
• It's totally OK to spend over four hours calculating how much of a 25-minute build you should do.
• All the code you will ever need to compile is written in precisely one language.
• Everything lives in a single repository.
• Files only ever get updated with timestamps by a single machine.
• Version control systems will always update the timestamp on a file.
• Version control systems will never update the timestamp on a file.
• Version control systems will never change the time to one earlier than the previous timestamp.
• Programmers don't want a system for writing build scripts; they want a system for writing systems that write build scripts.

[Exercise for the reader: which build tools make which assumptions, and which compilers violate them?]

1. Closures are great when you have several behaviours which can all vary independently; objects are great when you have several behaviours that must all vary in sync with one another. Put another way: if your problem is combinatorial explosion of objects, try using closures instead; if your problem is ensuring consistency among different behaviours, try grouping them into objects. Languages like Perl make it relatively easy to do either, so use the best tool for the job. The trick is to spot that you're using the wrong representation before you either create a kajillion Strategy classes or knock together a half-arsed object system out of hashes of closures.
2. In the correct quantity, housekeeping tasks can be your friends. By "housekeeping tasks" I mean tasks that don't directly contribute to solving the problem at hand, but which still add some value. An example from yesterday was converting a small class hierarchy to use the Moose object system. Other examples might be writing per-method documentation, cleaning up your version control history, and minor refactorings. If there's too much of this stuff to do, it can get dispiriting - you want to be solving the problem, not mucking about with trivia! But if there's not too much, it can be helpful: you have something to do while you're stuck on the main problem, and the housekeeping work is close enough to the main problem that it stays in your brain's L2 cache, where a background process can work away at it. If you're so stuck that you literally can't make any progress, your choices are (a) think very very hard and get depressed, (b) go and do something totally different, in the process forgetting lots of important details. I find both of these to be less productive.

#!/usr/bin/perl

print "> ";
while (<>) {
   s|-?\d+(\.\d+)?([eE][-+]?\d+)?|10**($&/10)|oeg;
   print 10*log(eval $_)/log(10)."\n> ";
}

> 1 + 1
4.01029995663982
> 10 * 10 
20
> cos(1)
-5.13088257108395

#!/usr/bin/python -u
import re, math, sys

def repl(match):
        num = float(match.group(0))
        return str(10**(num/10))

number = re.compile(r'-?\d+(\.\d+)?([eE][-+]?\d+)?')
while 1:
        line = sys.stdin.readline()
        if len(line) == 0:
                break
        line = re.sub(number, repl, line)
        print 10*math.log10(eval(line))

I disagree; I think it’s a lovely piece of code. It may not be a beautiful jewel of elegant abstractions for a complex data model, true. But it’s small, simple, trivial to write, works on anything with a Perl interpreter (of pretty much any vintage, and with no additional dependencies), and clearly correct once you’ve thought about the nature of the arithmetic you’re doing. While it’s not something you’d ship as part of a safety-critical system, for example, I can’t see any way it could be realistically improved as an internal tool, aimed at users who are aware of its potential limitations.

perl -e 'print join q[{,-,+}], 1..9'

perl -e 'print glob "1{,-,+}2"'

"Branch over unconditional jump" hack for arbitrary-length brcc.
    
 - brcc (branch and compare) instructions can have static branch-prediction
   hints, but can only jump a limited distance.
 - Calculating the distance of a jump at expand-time is hard.
 - So instead of emitting just a brcc instruction, we emit an unconditional
   jump to the same place and a branch over it.
 - I added a simple counter to emit/state to number the labels thus introduced.
 - With this commit, I forfeit all right to criticise the hackiness of anyone
   else's code.
 - Though I doubt that will stop me.

The simplest thing that could possibly work

pozorvlak@delight:~
0 $ ssh rydell ssh sentinel
Pseudo-terminal will not be allocated because stdin is not a terminal.

Automating the double-login with expect

#!/usr/bin/expect -f
set timeout 30
match_max 100000
spawn ssh rydell
send "\r"
expect "0 "             # prompt
send "ssh sentinel\r"
expect "0 "
send "cde\r"            # cd to work directory
interact

Using sshuttle

sshuttle -H -vvr rydell 0/0

ssh tunnelling

pozorvlak@delight:~
0 $ ssh -fN -L 9500:sentinel:22 rydell
pozorvlak@delight:~
0 $ ssh -p 9500 pvlak1@localhost
pvlak1@localhost's password: 
Last login: Thu May 19 14:31:32 2011 from rydell.my.employ.er
pvlak1@sentinel 14:34 ~
0 $ 

simont

Host sentinel
    User pvlak1
    ProxyCommand=ssh rydell nohup nc sentinel 22
    HostName sentinel.my.employ.er

14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.

1. In a generated-code scenario, do not assume that any particular snippet has been compiled successfully just because the whole codebase builds without errors.
2. Prefer languages with decent native abstraction mechanisms to code generators.
3. At least skim the subsequent error messages before dismissing them and working on the first bug: they may provide useful context.
4. Communication: if I'd enquired more carefully about the condition of the code to be merged I could have saved myself a lot of time.
5. Bear in mind the possibility that you might not be the guilty one.
6. Treat ESR's pronouncements with even greater caution in future. Same goes for Kenny Tilton, or any other Great Prognosticator.

7. If, despite (2), you find yourself writing a snippet-pasting code generator, give serious thought to providing "this snippet is unused" warnings.
8. Learn to spot when you're engaged in fruitless activity and need to step back and form a better plan. In my case, the time crawling through diffs was wasted, and I probably could have solved the problem much quicker if I'd rolled up my sleeves and tried to understand the actual code.

gareth_rees

test1 : 2 + 1 == 3
test1 = refl

test2 : 3 + 0 == 3
test2 = refl

aaroncrane