January 2018

S M T W T F S
  123456
78910111213
14151617181920
21222324252627
28293031   

Navigation

Page Summary

Style Credit

Expand Cut Tags

No cut tags

Fighting with make(1)

pozorvlak: (Default)
Saturday, July 2nd, 2011 06:37 pm
I'm currently running a lot of benchmarks in my day job, in the hope of perhaps collecting some useful data in time for an upcoming paper submission deadline - this is the "science" part of "computer science". Since getting a given benchmark suite built and running is often needlessly complex and tedious, one of my colleagues has written an abstraction layer in the form of a load of Makefiles. By issuing commands like "make build-eembc2", "make run-utdsp" or "make distclean-dspstone" you can issue the correct command (build/run/distclean) to whichever benchmark suite you care about. The lists of individual benchmarks are contained in .mk files, so you can strip out any particular benchmark you're not interested in.

I want to use benchmark runs as part of the fitness function for a genetic algorithm, so it's important that it run fast, and simulating another processor (as we're doing) is inherently a slow business. Fortunately, benchmark suites consist of lots of small programs, which can be run in parallel if you don't care about measuring wallclock seconds. And make already has support for parallel builds, using the -j option.

But it's always worth measuring these things, so I copied the benchmark code up onto our multi-core number crunching machine, and did two runs-from-clean with and without the -j flag. No speedup. Checking top, I found that only one copy of the simulator or compiler was ever running at a time. What the hell? Time to look at the code: 
TARGETS=build run collect clean distclean

%-eembc2: eembc-2.0
        @for dir in $(BMARKS_EEMBC2) ; do \
          if test -d eembc-2.0/$$dir ; then \
            ${MAKE} -C eembc-2.0/$$dir $* ; \
          fi; \
        done
Oh God. Dear colleague, you appear to have taken a DSL explicitly designed to provide parallel tracking of dependencies, and then deliberately thrown that parallelism away. What were you thinking?¹ But it turns out that Dominus' Razor applies here, because getting the desired effect without sacrificing parallelism is actually ( remarkably hard... )

( Doing it in redo instead )

Time to start teaching my colleagues about redo? I think it might be...

¹ He's also using recursive make, which means we're doing too much work if there's much code shared between different benchmarks. But since the time taken to run a benchmark is utterly dominated by simulator time, I'm not too worried about that.
Tags:

ssh problem followup

pozorvlak: (Default)
Saturday, May 21st, 2011 11:45 am
Thanks to all who helped out with my ssh problem the other day. A few different approaches worked: I'm writing them up here for easy reference.

To recap, the problem was as follows: I want to ssh from my home machine (delight) to my work machine (sentinel) without typing any passwords (and, while I'm at it, to various other work machines, such as the Subversion host). Unfortunately, sentinel isn't visible on the public Internet; first I need to ssh into a gateway machine (rydell) from which sentinel is visible. Oh, and neither sentinel nor rydell allow public-key ssh login; both require you to use the Kerberos authentication protocol, which is explained here.

Brute force and ignorance

My first attempt was to use ssh rydell ssh sentinel (which sshes into rydell, and invokes the command ssh sentinel thereon). This failed with the error "Pseudo-terminal will not be allocated because stdin is not a terminal". Marco Fontani pointed out that the -t switch to ssh allocates a pseudo-terminal, so ssh -t rydell ssh sentinel Does What I Want.

X11 Forwarding

Mat Brown pointed out that I was overthinking it: by using the -X and -C options to ssh (or adding the lines ForwardX11 yes and Compression yes to the relevant stanza of ~/.ssh/config) I could enable compressed forwarding of X11 connections; I could then create new terminal windows by sshing into sentinel once and creating new xterms on there. I already had ForwardX11 set, but didn't know about Compression, so I've enabled that; it seems to help.

Using ProxyCommand and AutoSSH

I added the lines
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r
ServerAliveInterval 60
ServerAliveCountMax 60
Host rydell
    User pvlak1
    HostName rydell.my.employ.er
Host sentinel
    User pvlak1
    ProxyCommand=ssh rydell nohup nc sentinel 22
    HostName sentinel.my.employ.er
Host svn.my.employ.er
    User pvlak1
    ProxyCommand=ssh rydell nohup nc svn 22
to ~/.ssh/config. Then, at the beginning of the day, I set things up with the commands 
kinit pvlak1
autossh -f -M 0 -N sentinel
autossh -f -M 0 -N svn.my.employ.er
I can now open a new ssh connection to sentinel in an eyeblink. I needed to install AutoSSH, but this was just an apt-get away.

Two things are going on here. The ProxyCommand lines (suggested by hatfinch, and debugged by [livejournal.com profile] simont) tell ssh how to reach sentinel: in this case, by sshing to rydell and using netcat to open a connection to port 22 on sentinel (on which sentinel's sshd is listening). The HostName line is necessary to stop Kerberos getting confused. The first four lines were suggested by Marco Fontani and Aaron Crane, and allow ssh to multiplex all its connections to sentinel (or any host, come to that) over one channel, eliminating the need for a cryptographic handshake on each new connection and leading to blazingly-fast startup times. To avoid various annoying problems with this setup, you'll need the AutoSSH invocations: Aaron explains why on his blog.

My original problem is solved - hurrah! Now, can anyone explain why resize events weren't being passed through my expect script, and what I could have done about it? :-)
Tags:

PeCoWriMo day 3

pozorvlak: (Default)
Wednesday, November 3rd, 2010 07:12 pm
Today I have
  • Submitted some more documentation patches for Idris.
  • Written a fragment of the Unix tool cat in Idris:
    eachLine : (String -> IO ()) -> File -> IO ();
eachLine f file = do {
	finished <- feof file;
	if finished then return II -- II is the sole element of the unit type ()
	else do {
		line <- fread file;
		f line;
		eachLine f file;
	};
};

main : IO ();
main = do {
	file <- fopen "fred" "r";
	eachLine putStrLn file;
};
    You'll notice that the file to catenate is hard-coded: I haven't yet worked out how to access your program's command-line arguments.
  • Started work on a SEKRIT WEB PROJECT for some friends; however, so far all my time has been spent mucking about with configuration files and installing dependencies rather than actually coding.
    Tags:

    Better living through shell scripting

    pozorvlak: (gasmask)
    Friday, March 28th, 2008 03:31 pm
    As I may have mentioned once or twice before, I'm writing up my thesis. This is an intensely slow and depressing process, and the temptation to slack off is overwhelming. Being a geek, I decided to write some code to help me. Writing code rather than writing thesis is still slacking off, obviously, but it's slacking off to a useful end. Writing blog posts about writing code that's meant to help me write my thesis is another matter...

    ( Read more... )
    Tags: