Kamil Choudhury

The Bangladesh Remittance Riddle

2023-01-03T00:00:00-05:00

An ongoing dump of stuff I've been thinking about with respect to Bangladesh's recent foreign exchange crisis and how the private sector could help.

Specifically:

The problem Bangladesh is facing.
Who's sending money home?
How much money is currently being sent home?
- How is this money being sent home?
How much money is still stuck outside Bangladesh?
- Is any of this money persuadable, i.e. willing to come home?
What might a modern platform to send money back to Bangladesh look like?
- How could such a platform be made financially sustainable?

We <3 Govtech

2022-07-27T00:00:00-05:00

Just a small placeholder to force the creation of the govtech tag on this blog.

More to follow.

Trade Model Thoughts

2022-01-01T00:00:00-05:00

This blog post started out as an internal memo, but nothing it reveals is terribly confidential, so I thought I'd post an extended version of it online and see what people have to say about it. I started working on it after posting this on Twitter:

The CDC now recommends financial firms implement technology to be able to mark all positions and all prices every day.
— Kamil Choudhury (@kchoudhu) December 30, 2021

The tweet jokingly cuts to the heart of what a modern financial institution needs from its technology stack: an accurate accounting of what is on its books, and what those positions are currently worth. Risk, analytics and trading systems are layered on top of this capability. Get this wrong function wrong, and your chances of having to do an expensive, productivity-killing technical retrofit approaches 1.

The question then becomes: how do you build such a tracking system?

Double Entry Book Keeping Rules Everything Around You

Counterintuitively, the first step in building a trade capture system is to stop thinking in terms of individual trades.

Instead, start thinking about position effects in double entry accounting. A quick refresher:

You have a ledger of accounts
A transaction that creates a credit or debit in one account must be countered by equal and opposite transaction notation in one (or more -- this becomes important later) other accounts
If everything doesn't tie out, there is a break.

Trades are by-products of recording the position effects of transactions between two accounts; going long a security creates a positive position effect in one account, and a negative one elsewhere.

Building Blocks

I think it's easier to think about these things with a couple of simple examples. But first we need some primitives though, so here are a few of them in Python. They should translate pretty easily to other languages.

Accounts are simple:

class Account(object):

    def add_position_effects(self, positions=[]):
        """
        Adds securities to account position store
        """
        pass

And here we have position effects:

class PositionEffect(object):

    """
    Container class to store positions in security:value
    format
    """

    def __init__(self, pos_effects):

        self._position_effects = pos_effects

    @property
    def inverse(self):
        """
        Return position effect object with quantities inverted

        Trust me.
        """
        return PositionEffects({
            sec:-amount
            for security, amount
            in self._position_effects.items()
        })

And what might a trade look like?

class Trade(object):

    def __init__(self, poseffects, portfolio1, portfolio2):

        portfolio1.add_position_effects(poseffects)
        portfolio2.add_position_effects(poseffects.inverse)

A Super Simple Example

Route53 From First Principles

2021-11-28T00:00:00-05:00

One of my favorite AWS services -- and I have a lot of them! -- has to be Route53: it makes programmatically maintaining DNS entries for dynamic services extremely simple.

But what if you don't have access to AWS?

Enter BIND

As it turns out, all of the tools to recreate Route53 are available via bind. Installation on FreeBSD is easy:

pkg install bind911

You can correct the horrible mistake FreeBSD made in removing nslookup from the base system by installing bind-tools; don't intall it if you can get by with drill:

pkg install bind-tool

Follow any the a million tutorials out there to get a basic DNS server up and running. In my case, I defined a zone in /usr/local/etc/namedb/dynamic/anserinae.net.db for anserinae.net:

$ORIGIN .
$TTL 604800     ; 1 week
anserinae.net           IN SOA  installation01.anserinae.net. administrator.anserinae.net. (
                                4          ; serial
                                604800     ; refresh (1 week)
                                86400      ; retry (1 day)
                                2419200    ; expire (4 weeks)
                                604800     ; minimum (1 week)
                                )
                        NS      ns1.
                        A       208.97.148.219 ; how to resolve a naked lookup for anserinae.net
                        MX      10 mxe.    ; mailserver
$ORIGIN anserinae.net.

Add the zone definition to the list of zones served by bind in /usr/local/etc/namedb/named.conf.local; the following configuration also allows hosts in 192.168.7.0/24 to update the zone definition via RPC calls:

acl "AllowUpdateIPs" {
    192.168.7.0/24;
};

zone "anserinae.net" {
    type master;
    file "/usr/local/etc/namedb/dynamic/anserinae.net.db";
    allow-update{ AllowUpdateIPs; };
};

RPC calls are made via the rndc mechanism, and are authenticated using a pre-shared key. Simply installing the bind should have generated a key in /usr/local/etc/namedb/rndc.key, but just in case it didn't you can create one as follows:

key "rndc-key" {
        algorithm hmac-md5;
        secret "<your super secret md5 key>";
};

Plug your zone definition and rndc key into the main /usr/local/etc/namedb/named.conf by adding the following to the bottom of the file:

include "/usr/local/etc/namedb/rndc.key";
include "/usr/local/etc/namedb/named.conf.local";

Finally, rig named to run at startup by editing /etc/rc.conf:

named_enable="YES"

At this point restart your machine or start named to get the service running:

service named start

Updating BIND Remotely

Now that we have a bind server that can be updated remotely, we use our favorite programming language (Python, doi) to update it. First, a few preliminaries:

pip-3.8 install dnspython

Supposing we want to create a new alias forgedcharity.anserinae.net that resolves to 192.168.0.150 on a server running at 192.168.0.1, we can execute the following:

from dns import query, update, tsigkeyring
from dns.tsig import HMAC_MD5

bindhost='192.168.0.1'
ip='192.168.0.150'

keyring = tsigkeyring.from_text({
    'rndc-key' : '<your super secrete md5 key>'
})

update = update.Update('anserinae.net.', keyring=keyring, keyalgorithm=HMAC_MD5)
update.replace('forgedcharity', 1, 'A', ip)

response = query.tcp(update, bindhost, timeout=10)

If we check the zone file at /usr/local/etc/namdb/dynamic/anserinae.net.db, we now see:

$ORIGIN .
$TTL 604800     ; 1 week
anserinae.net           IN SOA  installation01.anserinae.net. administrator.anserinae.net. (
                                4          ; serial
                                604800     ; refresh (1 week)
                                86400      ; retry (1 day)
                                2419200    ; expire (4 weeks)
                                604800     ; minimum (1 week)
                                )
                        NS      ns1.
                        A       208.97.148.219
                        MX      10 mxe.
$ORIGIN anserinae.net.
$TTL 1  ; 1 second
forgedcharity           A       192.168.0.150

At this point, carrying out an nslookup for forgedcharity.anserinae.net should resolve correctly to the appropriate IP. Hooray!

Should I Do This In Production?

Probably not. Getting a barebones nameserver up and running is easy, but justifying the cost and complexity of deploying something like this when Route53 is available is difficult. As always, production turns fun proofs of concept into monstrosities... why not just let someone else do the heavy lifting for you?

Trying Out Latex

2021-10-02T00:00:00-05:00

A simple inline math equation: $x^2$

A blocked equation:

$$x^2$$

And then an "I pretend I know stuff" equation:

$$\frac{\partial V}{\partial t} + \frac{1}{2}\sigma^2 S^2 \frac{\partial^2 V}{\partial S^2} + rS\frac{\partial V}{\partial S} - rV = 0$$

Book Talk - Fall; Or Dodge In Hell

2021-08-01T00:00:00-05:00

Every couple of years I dedicate a week to reading Neal Stephenson's newest book. It took me two years to get around to it this time round (young kids and COVID have not been good for reading...), but I finally found enough time during vacation to read Fall: Or Dodge In Hell.

I started recording my thoughts in this Twitter thread:

Alright: tonight’s book is FALL: DODGE IN HELL by Neal Stephenson.
— Kamil Choudhury (@kchoudhu) July 25, 2021

I won't discuss the first 200 or so pages of the novel: a detailed exploration of online disinformation and its effect on red America must have been immensely topical to Stephenson as he was writing the book (probably in the 2017-18 timeframe?), but it is ultimately vestigial to the rest of the novel, which tries to get to the root -- in typically voluminous fashion -- of what it means, really, to be alive.

The novel opens with Richard "Dodge" Forsyth dying during a routine medical procedure and having his consciousness uploaded to a virgin quantum computing cloud. Stephenson walks us in excruciating detail through Dodge (known in-cloud as Edgod) forging a world to live in out of quantum chaos, and then forming a new society in it alongside other newly uploaded souls.

The steady advance of upload technology in the real world results in power hierarchies organically forming in the cloud, and the fortuitous introduction of a Big Bad fills the back quarter of the novel leads to a moderately gripping Heroic Quest that ends with peace being restored to the (virtual) world, whose mythology now somehow explains our collective affinity for Greek mythology and our weirdly cross-culturally consistent creation myths. And oh yeah, as long as the cloud keeps running and expanding, humanity is effectivley immortal. Huzzah.

To Stephenson's credit, the 900 or so pages in the novel fairly fly by.

Unfortunately, accepting Dodge's virtual reality requires us to suspend belief about the 'real' world that ultimately hosts it. Stephenson tells us that creating and running a metaverse for departed souls is stupendously expensive, but hand waves the details away by invoking "financial bots". Really? There is the potential for a first rate financial drama here that is left completely unexplored: alpha decays, endowments fail... what happens when souls have to be terminated because of operating account shortfalls?

In the novel's universe, large swathes of middle America have been reduced to theocratic fundmentalism ("Ameristan") by the unrestricted flow of algorithmic disinformation ("Facebooking") on the internet. Left unexplored is how the denizens of this blighted subregion respond to the spiritual aftershock of humanity becoming immortal. And how does the American left respond to similar stimuli? Stephenson asks us to accept without question that the coastal elite will endure in innovative quantum utopia because They Believe In Science. Sure the book was written before Trump's COVID-fueled assault on good sense and science, but I would sooner believe that unicorns exist than accept that the left would be any better than the religious right at filtering insanity-inducing online disinformation.

Meanwhile, as our main characters happily create a virtual world for the souls of hundreds of millions of their dearly departed friends and family, Enoch Root, the seemingly immortal character from elsewhere in the Stephenson Extended Universe, drops by to tell us that his mission has ended now that humanity is able to simulate itself into immortality. With twenty pages to go, the reader is left with the unsettling prospect of worlds simulating worlds in infinite recursive futility. It's Rick and Morty for the literary crowd.

Until now, no one who has slogged through one of his novels could seriously claim that Stephenson lacks vision or ambition: just think back to the richly realized worlds of The Baroque Cycle or Anathem. Fall's flacid vision of immortality -- simulation without end in a computer playground, with no consciousness or link to what came before -- is profoundly unsatisfying and I cannot help but wonder why any of its characters would bend humanity's best efforts towards achieving it.

Near the end of the novel, the quest for cheap electricity and heat dissipation leads to the dense colonization of space and the proposed enclosure of the sun in a Dyson sphere. The novel's Humanity would rather use the energy of its only star to simulate itself in virtual eternity than explore the farthest reaches of its actual reality.

I would rather die.

Setting Up a Pkgsrc Repository

2020-07-20T00:00:00-05:00

I recently found myself needing to distribute software to developers' MacOS machines. The options were limited, and not great: I have had issues with MacPorts in the past, and Homebrew reminds me of Gentoo circa 2005 (compiling everything from scratch is great!), along with a hair-raising tendency to take over /usr/local.

Enter pkgsrc: developed for NetBSD, it is intended from the ground up to maintain a clean separation between itself and base, and is dead simple to write packages for to boot. While the NetBSD handbook explains how to write packages in exhaustive detail, it is less clear about how to distribute custom binary packages to a wider audience once you have compiled them. This blog post aims to fill that gap.

Bootstrap `pkgin`

Obviously, you are on a Mac. Start by installing developer tools:

xcode-select --install

Go get a cup of coffee, this is going to take a while.

Get your machine ready to install pkgsrc binary packages using the pkgin frontend:

BOOTSTRAP_TAR="bootstrap-macos14-trunk-x86_64-20200716.tar.gz"
BOOTSTRAP_SHA="395be93bf6b3ca5fbe8f0b248f1f33181b8225fe"

# Download the bootstrap kit to the current directory.
curl -O https://pkgsrc.joyent.com/packages/Darwin/bootstrap/${BOOTSTRAP_TAR}

# Verify the SHA1 checksum.
echo "${BOOTSTRAP_SHA}  ${BOOTSTRAP_TAR}" >check-shasum
shasum -c check-shasum

# Install bootstrap kit to /opt/pkg
sudo tar -zxpf ${BOOTSTRAP_TAR} -C /

# Reload PATH/MANPATH (pkgsrc installs /etc/paths.d/10-pkgsrc for new sessions)
eval $(/usr/libexec/path_helper)

This code is from the Joyent pkgsrc how-to, and it will change quarterly as they update their package sets, so do make sure to check out their website for fresh instructions.

Bootstrap the `pkgsrc` Tree

Next, install the pkgsrc tree on your machine. This is where you will develop your packages (or at least export them to after developing them somewhere else):

cd /opt

# Check to see if pkgsrc is already there. If it is, blow it away
if [[ -d ./pkgsrc ]]; then
        rm -rf ./pkgsrc
fi

# Pull in a compressed archive of the pkgsrc tree and unpack it
curl -O https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.xz
tar xzf pkgsrc.tar.xz

# Only root can install things
chown -R root:wheel ./pkgsrc

I put the tree in the /opt directory created by the pkgin bootstrap step. You could just as easily put it somewhere else.

Optional: Install Binary Versions of Build Tools

This step is optional in that pkgsrc will compile the build tools for you if your custom package needs them, but necessary in that building GCC or libtool from scratch will take hours that could be better spent doing other things.

pkgin install gcc7 cmake <etc>

Develop Your Package

I'm not going to get into this (go see the handbook!), but I recommend you become intimately familiar with the functioning of the url2pkg and createbuildlink packages, both which reduce the process of packaging external software into pkgsrc format to (basically) a point-and-click operation:

pkgin install url2pkg createbuildlink

Serve Your Packages

Once you have finished building your software, you should have a directory of distributable software in /opt/pkgsrc/packages/All. Serve it up using HTTP at https://pkgdist.testdomain.com; I use nginx, but it doesn't really matter what you use.

pkgin install nginx

A simple configuration would look this:

user   nginx  nginx;
worker_processes  1;

events {
    # After increasing this value You probably should increase limit
    # of file descriptors (for example in start_precmd in startup script)
    worker_connections  1024;
}

http {
    default_type  application/octet-stream;
    gzip          on;
    include       /opt/pkg/etc/nginx/mime.types;
    keepalive_timeout  65;
    sendfile      on;

    access_log    /var/log/nginx/access.log;
    error_log     /var/log/nginx/error.log;

    # Redirect all insecure connections to HTTPS
    server {
        listen 80;
        return 301 https://$host$request_uri;
    }

    server {
        listen 443 ssl;
        server_name pkgdist.testdomain.com;

        include common.conf;

        root /opt/pkgsrc/packages/;
        location / {
            autoindex on;
        }
    }
}

My SSL settings are in common.conf; you probably don't need to see them. The format of your configuration will vary slightly depending on what operating system you are serving from (I can't believe I have to say this, but yes, you can serve MacOS packages from Ubuntu. Or FreeBSD. Or anything at all, really).

Finally, index the package directory by using pkg_summary-utils to create a pkg_summary.gz:

pkgin install pkg_summary-utils
pkg_update_summary -r /opt/pkgsrc/packages/All/pkg_summary.gz \
    /opt/pkgsrc/packages/All 'gzip -dc' 'gzip -c'

Installing Your Binaries

Assuming you can go to https://pkgdist.testdomain.com/All/ and see a list of archives, you are now ready to install your software on your client Mac. Start by bootstrapping pkgin on your client machine just like you did on your build machine.

Next, add your new repository to the list of places the client pkgin looks for packages:

export REPO_ADDR=https://pkgdist.testdomain.com/All
export REPO_FILE=/opt/pkg/etc/pkgin/repositories.conf
grep -q ${REPO_ADDR} ${REPO_FILE} || echo ${REPO_ADDR} | tee -a ${REPO_FILE}

I cannot stress enough how important it is to not have a trailing slash at the end of REPO_ADDR; pkgin will not read your repository if you do.

And then install your newly developed software:

pkgin install <your package>

Make sure you add /opt/pkg/bin and /opt/pkg/sbin to your $PATH so that you can access your newly installed software.

Conclusion

I started this project looking for a way to serve MacOS software to developer machines and realized part way through that I could use the same infrastructure to serve software to our production machines running Linux: pkgsrc is, after all, fully cross-platform.

It's not often that a weekend afternoon of hacking results in actionable, complexity-reducing engineering insights, but here we are: it is good to have options.

Donate to NetBSD

Quick and Dirty Postgres Hacks

2020-05-12T00:00:00-05:00

I have a sieve-like memory, so I am going to start recording things about Postgres that I find myself googling every time I need to do them.

Resetting sequences

An attempt to insert a row into a table with a serial fails because the someone has manually inserted a value and failed to update the sequence.

SELECT setval('equities_id_seq', (SELECT MAX(id) from "equities"))

Where:

equities_id_seq is the Postgres sequence object
id is the serial being corrected, and
equities is the table

Adding sequence to an already existing table

The problem: you have a column with

What's holding locks in my database?

This query (or some variant of it) gives you all the queries currently holding locks on your database.

SELECT distinct
       a.query,
       a.pid,
       age(now(), a.query_start) AS "age",
       a.datname,
       l.transactionid,
       l.mode,
       l.GRANTED,
       a.usename,
       a.query,
       a.query_start
  FROM pg_stat_activity a
       INNER JOIN pg_locks l ON l.pid=a.pid
 WHERE a.query not like '%a.query%'
       ORDER BY a.pid, age;

F**k this lock in particular

Politely tell a troublesome query identified using the last command:

SELECT pg_cancel_backend(<pid>)

Go nuclear on said query:

SELECT pg_terminate_backend(<pid>)

Turn autovacuum off for a table

Don't do this lightly: autovacuum is needed for postgres to function well. But if you're doing periodic bulk loads on a table that has had surgery done it and autovacuum keeps interrupting the party...

ALTER TABLE table_name SET (autovacuum_enabled = false);

Finding how big a relation is

This one is pretty nifty:

SELECT pg_size_pretty( pg_total_relation_size('<table>') );

Querying how many rows are in a table without querying the table

This function is a roundabout way to get an estimate of how many rows are in a table without actually querying it:

DROP FUNCTION count_estimate(query text);
CREATE FUNCTION count_estimate(query text) RETURNS BIGINT AS
$func$
DECLARE
    rec   record;
    ROWS  BIGINT;
BEGIN
    FOR rec IN EXECUTE 'EXPLAIN ' || query LOOP
        ROWS := SUBSTRING(rec."QUERY PLAN" FROM ' rows=([[:digit:]]+)');
        EXIT WHEN ROWS IS NOT NULL;
    END LOOP;

    RETURN ROWS;
END
$func$ LANGUAGE plpgsql;

You can then obtain a reasonable estimate of a query return size by executing:

select count_estimate('select * from <table>')

The results returned by this function will get progressively worse as dead tuples accumulate; make sure autovacuum is turned on for the table!

Removing a sequence from a table

Sometimes you'll create a table with a sequence, add a bunch of data and then find that you don't need the sequence anymore.

First you need to find out what the sequence is called:

SELECT refobjid::regclass as table_name,
       objid::regclass as sequence
  FROM pg_depend
       WHERE refobjid = '<table_name>'::regclass
       AND deptype = 'a';

Then you:

Remove the sequence from the table:

ALTER SEQUENCE <sequence> OWNED BY NONE;

Alter the table to not assign from the sequence by default:

ALTER TABLE <table_name> ALTER COLUMN <sequence id column> DROP DEFAULT;

Convert the type of the id column back to int from serial:

ALTER TABLE expiration_rules ALTER COLUMN exp_row type int;

Post Metadata

Updated 5/15/2020: Added sequence assignment
Updated 6/10/2020: Added autovacuum shenanigans
Updated 6/15/2020: Added relation size and count estimate queries
Updated 4/13/2022: Added sequence removal

The Butter Chicken Recipe

2020-05-10T00:00:00-05:00

Ingredients

4 pounds of chicken. A mix of thigh and breast meat is the way to go.
8 tablespoons of olive oil.
2 teaspoons of red chili powder. I use the Kashmiri variant, but don't get hung up on it.
3 cans of crushed tomatoes.
6 ounces of cashew nuts.
16 ounces of whipped greek yogurt.
8-10 garlic cloves. If you want more, don't be shy about it.
2.5 large white onions.
4 tablespoons of butter.
3 teaspoons of sugar
2 teaspoons of grape vinegar.
Salt to taste

The Recipe

Mix chili powder and chicken, and brown the mixture 4 tablespoons of olive oil. Set aside. Do not discard the juices.

While the meat is browning, use a mixer to blend cashew nuts, tomatoes and yogurt until the cashew nuts have been reduced to fine grains. Set aside.

Dice two onions crudely, and blend them with the garlic cloves in the mixer.

Chop remaining half onion finely, and fry in the same pot used to brown meat.

Once the fried onions start browning, add the blended onion/garlic mixture and the tomato/cashew puree. Add in the butter and sugar, and a first helping of salt (I usually do about 1 teaspoon).

Turn the heat up and cover the pot; the sauce will turn a characteristic orange-red in about fifteen minutes. When it does, force the mixture through a strainer to remove its lumpiness: the makhani in this recipe refers not only to the butter used in the recipe, but its texture as well.

Add the chicken back into strained sauce along with the grape vinegar, and cook on high heat until the meat is tender. Continue cooking on lower heat until you're happy with the consistency of the sauce: I like it pretty thick, but this is your dinner.

Final Dispositions

This is a fast and loose recipe, and I tend to modify it as I cook. My only advice is to keep adding salt at the end until it tastes like butter chicken. Serve with hot white rice, and be prepared for the coronary which is certain to follow.

@SkeleCap has an interesting take on this recipe that involves a jar of Rao's pasta sauce. I'm not going to steal his thunder though -- make sure you make him tell you about it.

Book Talk - Pakistan: A Hard Country

2020-01-06T00:00:00-05:00

Some books merit more than a blurb on the annual culture review, and Pakistan: A Hard Country by Anatol Lieven is one of them. This entry serves as a dumping ground for my notes as I read the book, and will probably eventually be polished up into something more readable when I finish processing it.

I initially came across this list on my main man Tyler's annual reading list, and it stuck out to me because Anatol is actually my neighbor in Doha and I desperately need something light and airy to talk to him about when I have him over to brunch in a couple of weeks. So: Pakistan.

Dumb Things To Do With Money

2020-01-05T00:00:00-05:00

A couple of weeks ago I posted this on Twitter:

Bored and lacking excitement in my life, I am thinking of buying 3x leverage ETFs as part of my year end rebalance, send twete.
— Kamil Choudhury (@kchoudhu) December 28, 2019

Just to be absolutely clear, I believe in simple retirement portfolios: the bulk of my retirement funds are invested in a two mutual-fund portfolio with a 90/10 stock/bond split that has (predictably) done quite well over the last decade. If mutual fund prospectii are to be believed, I paid 0.04% of my portfolio value in expenses last year. As I get older, I'll cut the stock/bond ratio to something less risky.

But the experience has been boring. You take a bit of money every month, and stash it away in several designated accounts, and then forget about it. I miss the excitement -- remembered from several extremely poor decisions I made after joining the workforce out of college -- of having a substantial amount of money exposed to the depredations of markets.

Bottom line: boring portfolio planning has worked for my family and the lion's share of our money will remain invested that way. This year though, I'll put money to work doing something...spicier. I'm not sure what, but here are the basic parameters of the additional strategy:

The investment amount is going to be nominal in absolute portfolio terms, but substantial from a psychological point of view.
It is going to be a rational investment. I am not rolling dice. There will be due diligence involved.
I will post receipts.
At the conclusion of the strategy, 10% of any profit goes to a 501c3 organization TBD. Suggestions welcome.

Consider this an index page for the venture; I'll update it as time goes by.

Culture Report, 2020 Edition

2020-01-03T00:00:00-05:00

Books

What follows is a constantly updated list of books I read in 2020. At the very least it will save me the trouble of having to write up a humblebrag post at the end of the year.

Billion Dollar Whale: The Man Who Fooled Wall Street, Hollywood, and the World by Bradley Hope and Tom Wright: The book details the misadventures of Jho Low as he steals several billion dollars from the Malaysian government by exploiting holes in bank compliance departments across the US, Europe and South East Asia.
A Gentleman In Moscow by Amor Towles: Charting a Russian aristocrat's house arrest in the Metropol Hotel following the 1917 Revolution, it answers a very important question with unusual deftness: can meaning be found when one's circumstances are drastically reduced? The answer is perhaps not surprisingly in the affirmative, but is conveyed in language that somehow simultaneously evokes Wodehouse and Tolstoy. Best line in the book (and there are many, many candidates): "It is a gentleman's business to change with the times." Highly recommended.
Super Pumped: The Battle For Uber by Mike Isaac: Removed from Twitter rage, the book is a surprisingly dispassionate (and probably definitive) retelling of the story of Travis Kalanick's rise and fall at Uber. There aren't really any surprises for anyone who followed the saga as it unfolded, but is nonetheless a good reference summary to have on the bookshelf.
Sleeping Giants by Sylvain Neuvel: Shadowy unnamed operator/narrator assembles crew of Super Smart Authority Hating Misfits to rebuild a 200 foot gund^H^H^H^H robot left by aliens who are also actually the basis of all gods in Greek mythology. An attempt to consider the geopolitical implications of this discovery is undermined by the author's complete inability to write women: the Beautiful But Tough main character works out her emotional issues by having sex with her male coworkers who are, respectively, Nice Guy With Jealousy Issues and Nerd With Secret Tough Streak. Oof. On the positive side, the book is 300 pages long, and took only an afternoon to read; I will leave you to guess if I will be reading either of the two sequels.
Pakistan: A Hard Country by Anatol Lieven: Merits way more than a little blurb; my notes and review are here.
Foreign Devils on the Silk Road: The Search for the Lost Treasures of Central Asia by Peter Hopkirk: This retelling of Europe's steady exploration and theft of lost Silk Road civilizations in the early 20th century throws the characters of colonial explorers into bright relief: most were hucksters, and while serious academics did take part in the race for Central Asia, most let their thirst for scholarly fame overwhelm any sense of ethics they may have started out with. While much is made of Chinese laments about stolen cultural artifacts, it is difficult to take them too seriously in light of their contemporary efforts to erase said cultures by driving their Central Asians populations into concentration camps.
The Sympathizer by Viet Thanh Nguyen: Very close to being one of the best contemplations about the immigrant experience in America I have ever read. The book crystallized for me the fact that a part of me will never be American, and will instead always be lost abroad to a place that neither wants nor needs me. The book is also an excellent mini-history of the Fall of Saigon, and a serious reflection on the nature of American colonialism and how we see and forgive itself in the aftermath of the wars we start abroad.
The Future Is History: How Totalitarianism Reclaimed Russia by Masha Gessen: Gripping as a history of the revitalization of Russian totalitarianism since 1991, this book is far less convincing when it attempts to cast said revitalization as a manifestation of collective psychological trauma. A background in Hannah Arendt's work is almost mandatory to fully appreciate what Gessen is trying to drive at, but the argument that Russia can only exist in a stable equilibrium as a totalitarian state because of the trauma caused by Stalin almost a century ago never really landed for me.

Music

I stumble upon new music via Spotify these days, and don't really actively curate my music collection anymore. Instead, I thought I would just list the best stuff I find here.

Goldfrapp - Seventh Tree (2008): I fell off the Goldfrapp train after leaving college fifteen (jesus!) years ago. Listening to this album was an exercise in dissonance, because it doesn't sound anything like Supernature or Black Cherry. Much poppier than its predecessors (I thought I was listening to a Kylie Minogue album for the first few songs), Seventh Tree is still lushly produced and a regular fixture on my daily playlist.
Goldfrapp - Head First (2010): Having put out a highly competent pop rock album in the form of Seventh Tree, Goldfrapp reinvents itself yet again with peppy lyrics and a heavy dose of synthwave. It must have taken serious balls to release this album into the musical hellscape that was the early 2010s, and I can't help but think that it was a sign of things to come: just a few years later, everyone would be obsessed with Nightcall and scorpion-logo jackets.

The resurgence of synthwave in the last half of the 2010s was one of the few redeeming parts of an exceedingly terrible decade... I have a lot to say about it, but this Vice article does a much better job than I ever could.

The Reverse Proxy Test

2019-11-21T00:00:00-05:00

A quick blush to see if my reverse proxy is working correctly:

http://www.anserinae.net/the-reverse-proxy-test.html
https://www.anserinae.net/the-reverse-proxy-test.html
="https://www.anserinae.net/the-reverse-proxy-test.html"

And then this: ="https://www.anserinae.net/the-reverse-proxy-test.html"

Just saying

The Cuban Bean Recipe

2019-08-19T12:00:00-05:00

I'm working on updating my blogging workflow. Instead of making a "test post, please ignore" post, I thought I'd do a "delicious bean recipe, please enjoy" post.

Ingredients

Lots of olive oil
1 cup of diced purple onions
As much garlic as you can reasonably see yourself eating
1 shot glass of oregano
4 tablespoons of lime juice
3 cans of Goya black beans
2 red peppers, sliced into not too small pieces
3 packets of Goya "sazón"
2 tablespoons of cumin
3 teaspoons of salt

The Recipe

Sautee the onions and olive oil in a dutch oven until they are soft. Mix in the oregano and garlic, and sautee them some more. When it smells delicious (but before the onions start burning), add the remaining ingredients. Close the lid, and put the pot in the oven at 225F for four hours, stirring maybe once an hour to ensure that nothing is burning.

You can eat it the same day you cook it, but it's much, much better if you let it sit overnight in the fridge and then reheat it the next day.

Conclusion

This is usually the part of the recipe where I'm supposed to reminisce about how my family learned to heal and love after my brother came out as gay at my quadriplegic grandmother's funeral after she died of stage four cancer while whispering the recipe to me on her deathbed.

Unfortunately, this recipe has no more interesting backstory than "I cook it a lot, it's pretty good, and my children prefer it to chicken nuggets".

It's probably worth a shot.

One-Line Timing in Python

2019-08-19T00:00:00-05:00

I'm generally a huge proponent of profiling code to discern its performance characteristics, but sometimes dragging a full blown profiler into code is a pain. What if you just want to see how much wall-clock time a particular piece of code takes to run?

I use the following snippet so frequently for simple timing that I've moved it into a utility library that gets sucked into substantially all of my projects:

import time

class Timer:
    def __enter__(self):
        self.start = time.clock()
        return self

    def __exit__(self, *args):
        self.end = time.clock()
        self.interval = self.end - self.start
        print(self.interval)

When I need to time how long a piece of code takes to run, using it it's as simple as something like this:

with Timer():
    for i in range(100000):
        do_something(i)

Not much else to add here, other than the fact that I love context managers.

Caching Object Graphs Is Terrifying

2019-07-07T00:00:00-05:00

All systems start out with a webserver and a database. This arrangement promptly keels over at the first sign of any sustained load, at which point the bright-eyed engineer who really should know better says "let me just add some caching". Two weeks later, the engineer's daughter is asking why Daddy "has a lot of wizard hairs", and the engineer has a thousand mile stare.

Caching, as the old saying goes, is hard. Caching graphs of objects is even harder. Read on to find out why.

Example Object Chain

We will use the following trie of interdependent objects to guide our discussion; objects further up the trie depend on the ones lower down.

     F(E)   G(E)
       \    / \
        \  /   \           ⬆
       E(B,C)  D(C)     upstream
        /  \   /           ⬆
       /    \ /
     B(A)    C
      |
      |
      A

Assumptions

Our first assumption is that all downstream nodes already exist in the database before we attempt to persist the parent.

In the above chain, A would need to exist in the database before B(A) or C(A) could be created, and those would in turn have to exist before E(B,C) or D(B) could be constituted.

I like to call the second assumption in this system The Guarantee: when a cache item is created, updated or touched, all objects that depend on it are touched/created before it is finalized.

The reasoning behind this is driven by the LRU (least recently used) eviction algorithm used by memcache: since we guarantee that dependencies are going to be touched before their parents, the nightmare scenario of, say, E(B,C) being evicted and leaving no way for an invalidation signal to travel from A to F(E) or G(E) can never come to pass. At most, a leaf such as A can be evicted; this is okay, because it will be recreated in a consistent manner when it is next accessed (see Search below for details).

Caching Primitive

Keep it simple: for each node persisted, use a cryptographic hash (I use SHA256, feel free to choose your own poison) of the node as a key, and persist a pickled instance of this object as a value:

class CacheObject(object):

    # A direct copy of what's coming out of the database
    payload = None

    # *Upstream* Nodes that need to be notified of changes to this node
    dependencies = {}

Database Operations

Cache operations are married to database operations and transparent to the caller; cache actions for each possible database operation are listed below.

Create

If a database entry is not available, return an empty cache object. The client can figure out what to do with it.

Begin by creating an entry in cache.

If there are the node has downstream entries (e.g. E(B,C) or D(C)), DFS traverse the downstream nodes...

Creating cache entries for each and...
Adding the parent's key to each child's dependency array. This will form the basis of the cache update notification/invalidation chain later on.

Note that writes to cache for any given node happen post-traverse, meaning that The Guarantee is honored: each child is updated before its parent.

Update

Updates are the same for nodes with or without children. Once the node's database and cache entries are updated, upstream cache invalidation is carried out:

Using the dependency array on the cache object, recursively delete payload on all upstream cache entries, keeping in mind that...
Updates on cache items must be pre-traverse to preserve The Guarantee. If, for example we update B(A), the following nodes must also be updated, in order: E(B,C), F(E), G(E).
If retrieval of a key in any dependency array fails, it refers to an evicted cache item. This is nothing to worry about: just prune the key from the dependency array to prevent unnecessary future lookups.

Delete

Deletes have properties in common with creates and updates. The sequence of events that follows a delete is complicated enough that an example is illuminating.

Delete node B(A) in the database and in cache.
Delete all upstream cache items E(B,C), F(E), G(E). Yes, delete: with the deletion of B(A), the assumptions undergirding all upstream nodes are invalid and it doesn't make sense to maintain cache entries for them.
Notice that there are still stale references to B(A) and its deleted, upstream dependents in unrelated downstream nodes such as A (contains reference to B(A)) and D(B) (contains reference to G(E)). This is ok! Dependencies only matter for delete- and update-related invalidations. Remember: if a node cannot be found for invalidation, it is pruned from the dependency array during the invalidation process.

Search

When a request to retrieve information from the database is received, first check the cache store for it.

If a cache entry is available: Return it after checking it for validity. In this context, we define a valid parent node as one whose downstream children contain a reference to the parent in their dependency arrays.
- If the child node does not exist, it has been evicted. Create the child, adding the parent to its dependency list, and then execute an Update on the parent to maintain time-based ordering of the graph.
- If the child node exists but does not contain a reference to the parent, execute an Update on the child after adding the parent to the child's dependency array.
- If the child node exists and contains a reference to the parent, you're good to go.
If a cache entry is not available: Create a cache entry from the database (see Create above) and return it to the caller.

Conclusion

In this mind-numbing blog post, I have outlined a (probably) highly inefficient way of consistently caching the database representation of a graph of interdependent nodes using a caching store with memcache-like semantics.

It occurs to me that my eviction handling scheme breaks down completely once you start running multiple independent memcache servers (each with separate LRU queues) and consistently hashing graph nodes across them. If this is an issue for you, you are probably in much better operational (i.e. you're richer than me) and technical (i.e. you're smarter than me) shape to solve the issue than I am... my problems are tiny enough that simply upgrading the amount of RAM on my caching server is usually enough.

Anyhow: thanks for accompanying me on this terrifying journey. I'm going to go die some of these wizard hairs now, and possibly teach my four year old some better manners.

Containerize Everything: Food Edition

2019-04-20T00:00:00-05:00

Programming note

Every now and then, I am going to start condensing things that I rant about on Twitter here on my blog.

This is primarily so that I can index my thoughts in one place, but also because my blog is mine and not likely to disappear anytime soon. I'm not saying Twitter is going to disappear or anything, but I want to maintain the golden master of my words somewhere entirely under my control.

Onward!

In which The Guardian completely misses the point

Today's post starts with this tweet about Deliveroo running delivery operations out of containerized "dark kitchens" in the London Docklands:

Not one instance of the word "commissary". https://t.co/LwGfAtDXXA
— Kamil Choudhury (@kchoudhu) April 20, 2019

We should probably take a moment to reflect on how modern journalism can report things while completely missing the larger trends they are symptomatic of, but it's just easier to lay out what's actually going on.

Commissaries?

Commissary kitchens are industrial kitchens for rent. You buy slices of time in them to prep food for large catering orders, the day's product for a food truck, and increasingly commonly, orders coming off storefront apps like Seamless, Grubhub and Deliveroo.

Food for food delivery is no longer lovingly crafted in restaurant kitchens. Since commissary kitchens allow for the wholesale separation of the restaurant experience from the icky, irritating business of actually preparing food, the food industry, ever on the lookout for ways to improve its razor thin margins, has been falling over itself to move delivery prep to commissary kitchens.

It's not going to happen

Anyone who has ever opened a box of wilted fries and lukewarm hamburgers from a delivery driver (i.e. everyone) is scoffing right now: there is no way delivery could replace the experience of going to eat out at a restaurant.

Please reconsider.

The aggressively mediocre dining experience at most fast casual restaurants is making the idea of not getting in the car, not tipping 20% and not sitting in mediocre decor while eating reheated TV dinners seem increasingly attractive to a consumer base whose inability to reliably cook for itself is as dependable as its palate is jaded.

Fast casual dining is an $800 billion industry, and a good chunk of that is at play in the food delivery wars. The US is not the only place this particular war is going to play out: restaurants are at risk in any market with crappy existing restaurants, excessive tipping and weak labor laws.

What do labor laws have to do with any of this?

The entire commissary shift hinges on the availability of two pools of cheap labor: one that doesn't raise questions about cooking processes, and another that cheaply whisks food away to the customers.

Silicon Valley's experience with the first generation of labor law arbitrage apps (Uber, Postmates etc) has made it dangerously competitive to the existing restaurant industry (itself no slouch in the labor abuse department) at provisioning both pools of workers. Competition between the two teams is going to lead to a race to the bottom that guarantees cut corners and questionable adherence to food safety regulations. Big Tech will win this race because it has internalized that it is easier to ask forgiveness than get permission.

I'm not going to say much else about the food safety issue, other than to link to Sarah Taber, who is the internet's go-to person for thoughts on agriculture and food safety.

I don't I've quite articulated why I'm so critical of the tech industry.

Tech isn't just software anymore. They're coming for ag, food, & manufacturing- & they're bringing a negligent attitudes towards risk & safety that they learned in the cushy world of apps.
— Dr Sarah Taber (@SarahTaber_bww) March 11, 2019

She's right. Big Tech isn't going to change its ways just because it changed lanes.

Back to containers

Contrary to popular opinion, the comical mushrooming of food delivery startups isn't so much techbros trying to remove the last elements of friction in their over-privileged lives as it is Silicon Valley VCs with dollar signs in their eyes trying to parasitically muscle in on an entirely new field and replace the existing owners. The name of the game is containerization: take a business process, make it so that it doesn't matter where it's running, and then replicate the container as far and as fast as you can to drive down prices and (optionally) create shareholder value. Deliveroo couldn't have chosen a better implementation of the metaphor if it tried: I mean seriously, commissaries in containers in parking lots? Yikes.

"Docker for Food" has such a nice ring to it; I just wish it wasn't ushering in an era of VC-sponsored e-coli outbreaks.

Minimal devpi For Local Python Development

2019-03-20T00:00:00-05:00

Kronk Have Many Package

I am currently working on two internal two Python packages: an end-user application called openrelay, and its library dependency, openarc. In addition to this internal dependency chain, both packages have numerous external dependencies, and I would like everything to be installed nicely when I type pip install openrelay.

After struggling mightily with assorted hacks, I went searching for prior work. The common theme in all discussions was devpi, so I gave it a shot.

The experience was surprisingly intuitive, and I thought I'd write about it for future reference.

Setting up `devpi`

Pull the devpi suite in using pip:

pip-3.6 install devpi-server devpi-client --user

Initialize and start devpi-server. This will start a python package index on http://localhost:3141.

devpi-server --init --start

Check in on the status of the server:

ardentprayer% devpi-server --status
2019-03-20 07:03:53,744 INFO  NOCTX Loading node info from /home/kchoudhu/.devpi/server/.nodeinfo
2019-03-20 07:03:53,745 INFO  NOCTX wrote nodeinfo to: /home/kchoudhu/.devpi/server/.nodeinfo
server is running with pid 109

Out of the box, devpi creates a non-editable index root/pypi which simply proxies requests to external PyPI index and caches the result for future use:

pip-3.6 install Flask --index http://localhost:3141/root/pypi --user
Collecting Flask
  Downloading http://localhost:3141/root/pypi/+f/a08/0b744b7e345cc/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 24.4MB/s

Neat.

Create an internal-only index

A caching package proxy for external PyPI servers is cool and all, but our goal is to transparently distribute internal projects that are not available on external PyPI servers. To achieve this, we create a secondary index that inherits from root/pypi, and upload our development work to that.

Some admin work first. Create a user who will be able to upload to our secondary index:

devpi user -c distuser password=123

devpi login distuser --password=123

And create a new dev index:

devpi index -c dev bases=root/pypi volatile=True

The volatile keyword tells devpi that users will be able to upload to this index.

Upload projects to the internal-only index

Back to our initial problem: openrelay and openarc, and their external dependencies.

Both packages have coherent setup.py files, and I drop into their development directories and upload them to the devpi-server (for brevity's sake, only the output for openrelay is listed).

# cd ~/src/openrelay
# devpi upload
using workdir /tmp/devpi3
copied repo /usr/home/kchoudhu/src/openrelay/.git to /tmp/devpi3/upload/openrelay/.git
pre-build: cleaning /usr/home/kchoudhu/src/openrelay/dist
-->  /tmp/devpi3/upload/openrelay$ /usr/local/bin/python setup.py sdist --formats gztar
built: /usr/home/kchoudhu/src/openrelay/dist/openrelay-0.0.1.tar.gz [SDIST.TGZ] 15.535kb
register openrelay-0.0.1 to http://localhost:3141/distuser/dev/
file_upload of openrelay-0.0.1.tar.gz to http://localhost:3141/distuser/dev/

With both packages now uploaded to devpi, openrelay can be installed by issuing:

pip-3.6 install --index http://localhost:3141/distuser/dev openrelay --user

External dependencies are fetched and cached from PyPI, and openarc and openrelay were pulled in from devpi-server using one command.

Mission accomplished.

Configure `pip` to use `devpi`

Telling pip to use the new index with every invocation is tiresome, remove the need to do so by editing ~/.pip/pip.conf:

[global]
index-url = http://localhost:3141/distuser/dev

[search]
index = http://localhost:3141/distuser/dev

Do not put trailing slashes on the URLs; pip gets very testy indeed if you do.

Kronk Happy

I made developing multiple internal Python packages slightly little easier for myself, and heartily recommend that you follow in my footsteps. It is rare for a tool to improve productivity after, like, 6 commands, but here we are: devpi rules.

You should use it.

Scripting FreeBSD Updates

2019-03-06T00:00:00-05:00

A small set of nits that I stumbled upon while attempting to script freebsd-update.

Paging Dr. Obvious

The worst part, bar none, is that it demands interaction from the user. If you're doing manual upgrades, this is fine: you probably want to know what changed, when, and how. In a scripted environment where you are running freebsd-update on several thousand hosts and containers, this is...inefficient, especially when you are confident that the messages are informational only, and don't need any kind of user intervention [1].

It is not immediately obvious how to get past all the prompts. My usual approach of piping the output of yes to the program didn't work because, it turns out, freebsd-update displays its prompts using the system $PAGER. Obviously.

From the code:

# Set a pager if the user doesn't
if [ -z "$PAGER" ]; then
        PAGER=/usr/bin/more
fi

The fix here is to set, somewhere in the calling script, $PAGER to something that allows output, but doesn't wait for user input. I ended up using cat:

export PAGER=/bin/cat

Hooray: output, but no more prompts.

Go Away, It Worked For Me

The second sticking point: scripts seemed to work fine on my development machine, but then fail silently when deployed to the remote machine. Kind of a bummer if your stated goal is bulletproof upgrades.

I will save how debugging this problem revealed a desperate need for better logging in my systems for another day, but the root of the problem is that freebsd-update refuses to fetch updates if not running from a TTY:

if [ ! -t 0 -a $NOTTYOK -eq 0 ]; then
        echo -n "`basename $0` fetch should not "
        echo "be run non-interactively."
        echo "Run `basename $0` cron instead."
        exit 1
fi

This probably merits an explanation in fetch section of the freebsd-update man page, which currently reads:

 fetch     Based on the currently installed world and the configuration
           options set, fetch all available binary updates.

The solution (if not the explanation) to our problem turns out to be in the man page: call freebsd-update with the --not-running-from-cron option:

 --not-running-from-cron
                Force freebsd-update fetch to proceed when there is no
                controlling tty.  This is for use by automated scripts and
                orchestration tools.  Please do not run freebsd-update
                fetch from crontab or similar using this flag, see:
                freebsd-update cron

Using this option allowed fetch to proceed even when run remotely. Yay.

FreeBSD Updates Aren't Ready For Cluster Computing

Did you blink a couple of times at the --not-running-from-cron option?

I understand that freebsd-update wants to save the user from footgun scenarios -- but the default behavior is not consistent with how I suspect most people expect upgrades and updates to work on a modern operating system. We are well into the era of treating servers (where FreeBSD is supposed to excel) like cattle; having core functionality assume interactivity by default is weirdly anachronistic and actively hostile to cluster computing.

I am not sure what way forward is, but whatever path we choose surely should not involve shell hacks and inelegant command line options.

[1] Your mileage will definitely vary. In my case, systems are engineered to leave base intact, thus preventing merge conflicts and making automated updates a possibility.

Paying Less For Hotels

2019-02-26T00:00:00-05:00

While planning a recent trip, I didn't stick to The Plan and ended up paying full retail for a hotel stay.

I hate paying full retail.

This blog entry is a note to my future self: use this algorithm to get a reasonable percentage off a hotel stay, or to establish the baseline fair value of a hotel stay before attempting fancier savings strategies.

Once A Year

Load up on Hotels.com gift cards. Don't do anything fancy: once a year, go to Raise and buy as many gift certificates as you think you'll need in the coming year (you have a travel demand schedule right?). You should consistently be able to get them for 93-94 cents on the dollar.

Savings: 6.5%

Make sure you go to Raise through a cashback portal. Ebates has been my usual choice of late, but others can also work. Refer to Cashback Monitor for more possibilities; prefer payout reliability over an extra 0.5% in savings.

Savings: 1%

Use the right credit card for the gift cards. Your first port of call should always be the Bank of America Premium Rewards Cashback card; the way you have set it up, it gives you a flat 2.625% back on every dollar spent. It is possible that using another card will give you more back, but make sure the savings rate you are calculating includes the cash-out value of the points earned only. Avoid at all costs specious reasoning about the value of what you are saving.

Good Reasoning: These 4000 Ultimate Rewards points can be redeemed for a $40 statement credit.
Bad Reasoning: These 10K Membership Rewards points can be redeemed at 10 cents per mile flying first class on Singapore Airlines.

The baseline comparison point for this family's travel is always economy class.

Savings: 2.625%

Total savings, before going anywhere near a booking site: 10%

Administrative note: the gift cards are likely to come in small denominations of $50-200. Do yourself a favor and combine them all into one card here

Booking Individual Hotel Stays

The lead up to this section will have tipped you off to the fact that all stays are to be booked through Hotels.com. They have a meta-rewards program that gives you a credit towards a free night whenever you book 10 nights through them, the size of which is equal to the average value of the last 10 paid stays. This avoids the captive vendor syndrome that comes with using chain-specific loyalty programs, and also locks in your savings rate at around 10%.

To start with, go through a cashback portal.

Savings: 5% if booking without a free night credit, 2% if booking with one.

Pay attention to the nightly room rates. If the highest room rate for the stay is within ±5% of the free-night credit, use the credit: ultimately, you want to use as much of the credit as possible without sacrificing earning value on future credits. If you need to, roll extras you know you're going to need (breakfast, late checkout) in order to boost the room rate.

If you're feeling frisky, split tender between a credit card that offers trip protection (Chase Sapphire Reserve by choice) and the gift card when checking out in order to trigger trip protection. Credit card trip insurance isn't worth much to be honest though, so if you do anticipate an issue, go ahead and get actual trip insurance.

Upshot

Using this process it is possible to save, in aggregate, across 3-4 trips over the course of a year, about 22-25% off baseline hotel prices. That is not bad.

Memcache For My Brain

2018-12-09T00:00:00-05:00

I am going to keep updating this list because it is annoying to have to repeatedly spend hours googling things that are tantalizingly familiar but still forgotten by my increasingly decrepit brain.

That Movie where...

... a bunch of sailors on shore leave defraud a casino with their ship's computer

The Honeymoon Machine

... poor Chinese migrants get caught up in the noir-esque travails of a Vietnamese bar singer.

So Close To Paradise

That song...

... you listened to 800 times at 2AM while you were working graveyard shift for the 82nd time in 2009, but lost when your iPod went tits up:

Goo Goo Dolls - Black Balloon

Dashboad Confessional - Stolen

... that plays in Dogtown And The Z-Boys when the narrator is talking about Venice falling apart:

Sneaker Pimps - 6 Underground (Fila Brazilia Remix)

... that plays at the end of Dogtown and the Z Boys when they're showing what happened to all the Z Boys and is supposedly a remix of "Exchange" by Massive Attack:

Massive Attack - Exchange (Dogtown and Z Boys Version)

Yes, I wrote to Stacy Peralta to find out what it was and where I could get a copy.

That quote...

... by Kim Stanley Robinson in Sixty Days And Counting on what to tell oil companies when they resist changing their ways.

"None of the alternatives are competitive [to oil]."

Charlie’s pencil tip snapped. “Competitive for what?” he demanded. He had not spoken until that point, and now the edge in his voice stopped the discussion.

Everyone was staring at him. He stared back at the World Bank guys.

"Damage from carbon dioxide emission costs about $35 a ton, but in your model no one pays it. The carbon that British Petroleum burns per year, by sale and operation, runs up a damage bill of fifty billion dollars. BP reported a profit of twenty billion,so actually it’s thirty billion in the red, every year. Shell reported a profit of twenty-three billion, but if you added the damage cost it would be eight billion in the red. These companies should be bankrupt. You support their exteriorizing of costs, so your accounting is bullshit. You’re helping to bring on the biggest catastrophe in human history. If the oil companies burn the five hundred gigatons of carbon that you are describing as inevitable because of your financial shell games, then two-thirds of the species on the planet will be endangered, including humans. But you keep talking about fiscal discipline and competitive edges in profit differentials. It's the stupidest head-in-the-sand response possible."

... and why everyone has a problem with the Baby Boomers now:

Lots of [people] are locked into an Oedipal hatred of the baby-boomer generation. They hate the boomers for what we were given when we were young, the world gone for just the briefest moment out of its mind into a realm of sex, drugs and rock and roll, of revolution and war and history right there in our hands, a time of excess and joy, a feeling that things could still change -- a freedom that was so extreme no one who was there can even remember it properly, and no one who wasn't there can imagine it, because it was before AIDS and crack and meth and terrorism had returned everything to something like the weird and violent Victorian repression/transgression state of fear that we've all been living in these past years. So I see a fair bit of resentment. You old Vietnam vet, I see their eeys saying, you old hippie, you got lucky and were born in the right little window and got to grab all the surplus happiness that history ever produced, and you blew it, and you stood around and did nothing while the right Reaganed back into power and shut down all possibility of change for an entire generation. You blew it in a ten year party and staggered off stoned and complicit. You neither learned to do machine politics nor dismantled the machine. Not one of you imagined what had to be done. And so the backlash came down, the reactionary power structure stronger than ever, and now we're the ones who have to pay the price for that.

It was probably a follow on from his thoughts on change and those who make it possible in Years Of Rice And Salt:

In every group a Ka and a Ba, Ka always complaining with the kaw of a the crow, the cough of the cat, the cry of coyote, kaw, kaw, that fundamental protest; and then Ba always Ba, the banal baa of the water buffalo, the sound of the plow bound to the earth, the bleat of hope and fear, the bone inside. The one who missed the missing Ka. The world was changed by the Kungs, but the Baos had to try to hold it together.

Heady stuff.

Python Profiling for Babies

2018-10-05T00:00:00-05:00

If you have ever written a big stack of Python code and wondered why it is stupidly slow, you need a profiler. My go-to choice is pyinstrument, which samples your program while it is running and prints out pretty trees of where your program is spending time.

Installing it is pretty easy:

pip-3.6 install pyinstrument --user

And then you pop into your python code and have it sample the code you want to profile:

def big_terrible_piece_of_code()

    from pyinstrument import Profiler
    profiler = Profiler()
    profiler.start()

    # Do a whole of stuff

    profiler.stop()
    print(profiler.output_text(unicode=True, color=True))

That'll output a big old graph like this:

2.449 None  None
└─ 2.431 _handle_and_close_when_done  gevent/baseserver.py:24
   └─ 2.431 handle  gevent/pywsgi.py:1498
      └─ 2.431 handle  gevent/pywsgi.py:442
         └─ 2.431 handle_one_request  gevent/pywsgi.py:592
            └─ 2.431 handle_one_response  gevent/pywsgi.py:945
               └─ 2.431 run_application  gevent/pywsgi.py:907
                  └─ 2.431 __call__  flask/app.py:1995
                     └─ 2.431 wsgi_app  flask/app.py:1952
                        └─ 2.431 full_dispatch_request  flask/app.py:1600
                           └─ 2.431 dispatch_request  flask/app.py:1578
                              └─ 2.431 wrapfn  wwwd.py:214
                                 └─ 2.431 view_home_thread  wwwd.py:952
                                    └─ 1.452 <listcomp>  wwwd.py:995
                                       └─ 1.426 __next__  openarc/graph.py:404
                                          └─ 1.397 next  openarc/graph.py:169
                                             └─ 1.209 _set_attrs_from_cframe  openarc/_rdf.py:344
                                                └─ 0.818 add  openarc/_rdf.py:144

The most expensive code paths are near the top of the tree -- optimize those first if you can.

There are other tools in my optimization toolkit, but this is the one I find myself turning to most often. It is insanely easy to use, and in my book that trumps sophisticated tools any day of the week: sometimes a general, birdseye view is all you need to notice that your program is doing something stupid.

Previously: performance is not an accident

Recursive Blogging

2018-10-04T00:00:00-05:00

This is my submission for that most irritating genre of technology writing, the blog post about setting up a blog.

It's more a note to future me on how specifically to get this blog running on a new machine than it is a general guide on how to run a Pelican-based website... but if it helps anyone else, well, you do you.

Preliminaries:

sudo pkg install gmake python3 python36 py36-pip

Clone the blog respository:

git clone https://www.github.com/kchoudhu/src/blog

Setup has gotten easier recently, so hit up the setup target. This will install the flex theme and ensure that all plugins are available to pelican:

gmake setup

Start the devserver, which will serve on port 8000:

gmake html
gmake devserver

Work on a new article:

gmake newblog ASUM=a-dash-separated-slug

This will create a new branch which will be used to create the new article:

git branch
* a-dash-separated-slug
  master

If at any time you want to create a new article while working on the first, issue a new newblog command; this will commit any outstanding changes to the current article branch, force a switch to master and create a new blog branch.

Use rsync to publish everything once you've merged:

gmake rsync_upload ASUM=a-dash-separated-slug

If you want to simply push what's committed on master, leave out the ASUM parameter.

Congratulations, you have written stuff on the internet that no one is going to read. Shouldn't you be doing real work instead?

Stupid Systems Monitoring Tricks

2018-10-03T00:00:00-05:00

Yes, we are all supposed to have moved into the fourth age of Web 7.0 systems monitoring (I think the cool kids are calling it observability today?), but sometimes you need to know the state of a machine without a collection endpoint on it. Or you are offended by the thought of loading 68MB of HTML and Javascript when a simple shell command will do.

Print all sockets matching a certain PID:

~> while true; do sockstat -4 | egrep $PID; sleep 2; echo '-----'; done

Print number of file descriptors in use:

~> while true; do sysctl -a | egrep openfiles; sleep 2; echo '-----'; done

And as a personal reminder, don't forget to check ulimit to see what your session allows when you invariably run out of memory/filedescriptors/whatever:

~> ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          33554432
-s: stack size (kbytes)             524288
-c: core file size (blocks)         unlimited
-m: resident set size (kbytes)      unlimited
-l: locked-in-memory size (kbytes)  64
-u: processes                       8499
-n: file descriptors                116946
-b: socket buffer size (bytes)      unlimited
-v: virtual memory size (kbytes)    unlimited
-p: pseudo-terminals                unlimited
-w: swap size (kbytes)              unlimited
-k: kqueues                         unlimited
-N 14:                              unlimited

A New Cycle

2018-08-11T00:00:00-05:00

As the days melt into months and the months into years, we are on the move again. Two years, two children, so many miles; there are days when I wonder if we'll ever stop. And then we are on the road again: lock the door, turn left on Springfield, merge on to the 215, south to Los Angeles, get a on a flight to Doha and far, far beyond.

I see the way open in front of me and I cannot help walking down it.

Diagramming Python Object Graphs

2018-06-06T00:00:00-05:00

If you ever want to look at the dependency graph of a python object:

import objgraph
obj = objgraph.by_type('OAG_AutoNode1a')[0]
objgraph.show_backrefs([obj], filename='depgraph.png', max_depth=100)

This will dump the dependency graph for the first OAG_AutoNode1a object into a file depgraph.png.

Useful for finding memory leaks. Requires the objgraph python module and the graphviz (system) package:

pip-3.6 install objgraph --user
pkg install graphviz

Evolution on KDE

2018-06-01T00:00:00-05:00

Data loss shuffle

After a few too many instances of NTFS eating my work-in-progress on Windows, I recently decided to move my day-to-day workflow over to a ZFS-on-root FreeBSD system.

I installed KDE 5 (a story for another day), and started sifting through my options for mail management. A few short years ago, this would have been a showstopper as my company uses Microsoft Exchange. Now that Microsoft has inexplicably started shitting interoperability rainbows, I can simply install mail/evolution and be off to the races:

pkg install evolution evolution-ews

Evolution started up beautifully, although it did look a little out of place... it is a GNOME application running on KDE after all.

Irritant

While I did not have any problems setting up my account, whenever I refreshed my inbox, despite selecting the "store my credentials in the GNOME keyring" option, Evolution wanted me to re-enter my password

Oh wait, I'm on KDE.

The Fix

Give Evolution exactly what it wants:

pkg install gnome-keyring

Set up a little rc script for the newly installed package:

# nano /usr/local/etc/rc.d/gnome_keyring
#!/bin/sh

. /etc/rc.subr

name=gnome_keyring
bin=gnome-keyring-daemon
rcvar=gnome_keyring_enable

command="/usr/local/bin/${bin}"

load_rc_config $name
run_rc_command "$1"

Enable it in /etc/rc.conf:

gnome_keyring_enable="YES"

And start the keyring:

service gnome_keyring start

I haven't seen a password prompt in about two days now, so I'm assuming the case is closed.

Bring in the dancing lobsters.

Dynamically Altering Classes in Python

2018-05-26T00:00:00-05:00

The Very Basics

Everything in Python is an object. Don't believe me?

In [1]: type(42)
Out[1]: <class 'int'>

Suffice to say that more complicated things are also classes. Derive from object to declare your own classes, and then create an instance:

class Cat(object):
    # Add a property
    @property
    def legs(self):
        return 4
    # Add a method
    def meow(self):
        print('meow')

cat_1 = Cat()

cat_1.meow()
# meow
cat_1.legs
# 4

Running dir() on will show you what methods and properties are on the object:

In [2]: dir(cat_1)
Out[2]:
['legs', 'meow']

Adding methods dynamically

One of the best things about Python is its delightfully adult attitude towards runtime mutability: if you have ever found yourself asking "can I add a method to an object after initializing it?", you should know not only that the answer is "yes", but that Python also makes it easy (and relatively safe) to do so by using the MethodType method from the types module:

import types

def pounce(*args):
    print("pounce!")

setattr(cat_1, 'pounce', types.MethodType(pounce, cat_1))

In [3]: dir(cat_1)
Out[3]:
[ 'legs', 'meow', 'pounce']

It's important to note that the new method only applies to the current instance of the class:

# cat_2 is not affected by the changes made to cat_1
cat_2 = Cat()

In [4]: dir(cat_2)
Out[4]:
['legs', 'meow']

But what if you want to add a method dynamically to all instances of the class?

Enter metaprogramming

Remember how we said that everything in Python is an object? Take a look at this:

In [5]: type(Cat)
Out[5]: <class 'type'>

If you are like me, it'll will take a while to internalize the message being conveyed here, which is that the Cat class is an instance of the type metaclass. And as we saw in the last section, anything that is an instance can be manipulated at runtime:

setattr(Cat, 'pounce', pounce)

Checking in on cat_1, we see an unsurprising result -- we did after all previously bind the pounce method to it:

In [6]: dir(cat_1)
Out[6]:
[ 'legs', 'meow', 'pounce']

cat_2 on the other hand never had a pounce method attached to it, but now shows one inherited from the class definition:

In [7]: dir(cat_2)
Out[7]:
[ 'legs', 'meow', 'pounce']

In [8]: cat_2.pounce()
pounce!

Lovely.

Footgun Alert

A few warnings for those of you who, like me, are occasionally taken unawares by things that others perhaps wouldn't be surprised by.

Firstly, class implementations of methods can be overridden on a per-instance basis:

def alternate_pounce(*args):
    print('alternate pounce!')

setattr(cat_2, 'pounce', alternate_pounce)

In [9]: cat_2.pounce()
alternate_pounce!

In [10]: cat_1.pounce()
pounce!

Secondly, any changes made at the class level do not override local changes at the instance level:

def alternate_pounce_2(*args):
    print('alternate pounce 2!')

setattr(Cat, 'pounce', alternate_pounce_2)

# We overrode cat_2's 'pounce' method with alternate_pounce and
# changing the class definition of 'pounce' isn't visible from cat_2
In [11]: cat_2.pounce()
alternate pounce!

# A newly instantiated object shows the new 'pounce' output, however
cat_3 = Cat()

ln [12]: cat_3.pounce()
alternate pounce 2!

Finally, changes to the class last as long as the process is alive. This particular footgun can cause unintuitive behavior when using (and mutating) the class in unit tests that are not run in separate processes. You may think that the class is going to start out fresh, but really the definition of the class at the beginning of a test is what it was at the end of the last test that was run. Spare yourself hours of debugging test order and unittest internals, and reset the class in your test class' setUp and tearDown methods.

Why does this exist?

Every Python metaprogramming article seems to include this quote from Tim Peters:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why)

Under most circumstances, altering code while it is running increases its complexity and makes reasoning about it more difficult. There are, however, certain circumstances where it is warranted: while developing the database persistence portion of OpenARC, for example, it was necessary to map dynamically changing relationships between database objects. Metaprogramming fit the bill in that very limited use case, and I happily used it.

Knowing your options is good. Don't let black magic scare you.

On Marriage

2018-03-29T00:00:00-05:00

Being married in your 30s isn't easy, and adding young children to the mix makes things exponentially more difficult. As the trappings of courtship and early wedded life are slowly stripped away, your life is concentrated down to the few things matter most: mid-career effort, raising your children and the moments you steal from the first two in order to keep your marriage alive until Spring comes again. The ruthless calculus of effort and exhaustion demands that everything else be dropped.

Under these straightened circumstances, a successful marriage isn't necessarily an architectural marvel to behold from afar; rather, it is a particularly strong foundation that allows the other things in your life to flourish. And just as no one appreciates the strength of a foundation until they are caught in a building that is collapsing, a strong marriage is unobtrusive until it starts unravelling.

If you are lucky (and I am), you will receive gentle reminders of these truisms without being party to a disintegrating marriage. I was given one such reminder a short while ago: my wife got up from bed to visit the restroom after we fell asleep holding hands. My sleep-addled mind was struck by how cold my hand felt without her holding it -- so much so that two days later I still feel the need to commit my recollection of the experience to paper.

Anyhow... this code isn't going to write itself. I do love my wife though, and that makes the grind just that bit more tolerable.

Using Python Decorators for Authentication

2018-03-19T16:30:00-05:00

A Refresher

Python decorators are bits of syntactic sugar that allow you to execute complementary code every time a function is executed. Given this function...

def super_useful_decorator(fn):

    def wrapper():
        print "hello world"
        fn()

    return wrapper

...this:

def hello_world():
    return 42

super_useful_decorator(hello_world())()

...is exactly the same as this:

@super_useful_decorator
def hello_world():
    return 42

hello_world()

Who Cares?

At this point, most decorator tutorials wax lyrical about how python functions are first class and then spend a lot of time talking about closures. This is interesting, but not necessarily immediately useful. Here instead, is the cliffnotes version:

Function decorators let you do things automatically when you call a function.

Keeping this in mind, let's move on to an actual use case for decorators.

Authenticating API Calls

Say you're setting up a REST API which requires an authenticated token from the user:

def token_is_valid(token):
    # Replace with a call to your token validation service
    return token=='42'

def authenticate_access(token):
    return token_is_valid(token)

def rest_function_1(token, input_1, input_2)
    rv = {}

    if authenticate_access(token):
        rv['code'] = 200
    else:
        rv['code] = 401
        rv['payload'] = input_1+input_2

    return rv

def rest_function_2(token, input_1, input_2)
    rv = {}

    if authenticate_access(token):
        rv['code'] = 200
    else:
        rv['code'] = 401
        rv['payload'] = input_1*input_2

    return rv

This is perfectly workable code, but there are a couple of problems with it.

Boredom: Authentication code has to be written for each REST API call. This is boring for the programmer.
Brittleness: Changing the authentication code in any way means that every REST API function has to be altered.
Repetition: This follows from the last problem, but bears repetition (heh). Saying the same thing over and over in authentication code is dangerous: sooner or later, you're going to say it wrong and let a bunch of people use your service without authentication.

Python decorators are a pretty elegant solution to this problem:

def token_is_valid(token):
    # No seriously, replace this with your own validation code
    return token=='42'

def authenticate_access(fn):
    def wrapfn(*args, **kwargs):
        rv = {}
        token = args[0]
        if token_is_valid(token):
            rv['code'] = 200
            rv['payload'] = fn(*args, **kwargs)
        else:
            rv['code'] = 401
        return rv

    return wrapfn

@authenticate_access
def rest_function_1(token, input_1, input_2):
    return input_1+input_2

@authenticate_access
def rest_function_2(token, input_1, input_2):
    return input_1*input_2

Let's try executing this:

In [1]: rest_function_1('43', 2, 2)
Out[1]: {'code': 401}

In [2]: rest_function_1('42', 2, 2)
Out[2]: {'code': 200, 'payload': 4}

In [3]: rest_function_2('42', 32, 32)
Out[3]: {'code': 200, 'payload': 1024}

Neat.

As an additional bonus, there is now clear separation of concerns between authentication and whatever it is the REST API functions are supposed to be doing. The code just looks cleaner.

Conclusion

If you are using Python, you should be making heavy use of decorators.

If you want to see what other people have used decorators for, the Python wiki is kind of amazing.

Testing Done Right

2018-03-18T13:05:00-05:00

The SQLite project apparently has a 1:730 code-to-test ratio: How SQLite Is Tested.

Does your project need this kind of testing? Probably not. It sure is nice to see what the Platonic Ideal of testing looks like though.

Niche Sports Sunday

2018-03-18T00:00:00-05:00

Welcome, dear reader, to the first ever edition of Niche Sports Sunday, a blog thing that attempts to cover sports that I follow that no one else really cares about.

Princeton Crowned ECAC Champions

Congratulations to Princeton Hockey on extending their two month winning streak and capturing the Whitelaw Cup. Winning the postseason tournament is a great achievement, especially when you consider Princeton was ranked 7th in the league coming into it. That they pipped the top three teams in the league (Union, Cornell and Clarkson) on their way to the trophy makes the feat even more impressive.

The win also gives Princeton ECAC's only autobid to the NCAA tournament. Right now it looks like we'll be playing Notre Dame in the first round of the Mid Western Regionals. Ordinarily I would be nervous about playing the number 2 team in the country in a playoff, but we tied St. Cloud State (ranked first in the country) and beat Cornell (ranked second) in very recent memory. Princeton clearly has the firepower and defensive wherewithal to fight and win against the toughtest opponents; I expect great things from them in the tournament.

We've been waiting for this run since 2008: go Tigers!

Bangladesh and the Nidihas Trophy

Half a world away in Sri Lanka, Bangladesh won a heated do-or-die T20 against Sri Lanka to make the finals of the trilateral Nidihas Trophy. These kinds of tournaments are usually total snoozefests designed to meet FTP obligations, but this one has been insane because of poor player behavior on and off the field. Our 12th man got into a pushing match with a group of Sri Lankan players, Shakib Al Hasan caused all sorts of headaches by trying to call players off the field for debatably poor umpiring, and the team as a whole caused quite a stir by doing some serious damage to the their dressing room. The team's behavior was so attrocious that the Bangladesh board was then forced to issue a heretofore unimaginable apology to their hosts.

I have on occasion lamented the lack of assertiveness on the part of Bangladesh's cricketers (remember when one of our batsmen asked Steve Waugh for an autograph on the field?), but these kinds of shenanigans are a step too far. Our internet fans are already enough of an embarrassment without the national team also bringing our cricket establishment into disrepute.

Woof.

Cricket in Nepal

We're going to go even more niche now. Topic: Associate Cricket!

Associate Cricket is a second-league international cricket competition used to promote the next generation of test playing nations.

After a long period of inactivity (or was it incompetence?), the ICC overcame its anti-expansion tendencies and started promoting less storied cricket programs to the top tier via the World Cup Qualifier program. Ireland got Test status a few months back, and by qualifying for the World Cup, Nepal has earned One Day status for the next few years.

That's right: a tiny, mountainous country with virtually no cricket culture (and a currently suspended cricket board) has boostrapped itself to international ODI status.

To give an idea of what cricket in Nepal used to be like: in the late 90s I remember scouring all of Kathmandu to find a large enough and flat enough field in which to play pickup cricket with my neighbors. The only place we could find was a piece of undeveloped land owned by the Pakistan Embassy. We took a lot of hits to the head that summer because of the undeveloped nature of pitch; overcoming these kinds of infrastructure problems and producing an international grade cricket team is a massive achievement.

Congratulations to Nepal: I can't wait to see what they do with their hardearned ODI status.

What It All Means

I am generally down on the state of cricket. The ever-rising tide of T20 money has made most players lose sight of what makes cricket great (hint: it's Tests). While I can't find it in myself to be angry about this -- the players are only human -- the combination of money, incompetent administration and the lasting stench of corruption that has pervaded the game for the last decade means that I haven't enjoyed top tier cricket for a long time.

A lot of fans seem to have moved down the value chain to find good stories and meaningful competition. Cricket in places like Ireland, Nepal and Papua New Guinea has been a revelation: these teams play because they love the game. And while there may be riches down the road, for now it's all about their pure, unadulterated joy in the game.

Long live Associate Cricket.

A Child's Pain

2018-03-14T00:00:00-05:00

Last night, my eldest daughter slammed the heavy main door of our house on her tiny, two year old hand. The sound that came out of her was simultaneously the most piteous and terrifying thing I have ever heard in my life.

She's fine now, but yeah: go hug your kids.

What's Cooking, OpenARC Edition

2017-11-17T00:00:00-05:00

Dude, you're lazy

Cut me some slack: in the space of the last year, I had another kid, moved to Los Angeles, then Las Vegas, and then back to Doha. Now that the sleep deprivation is slowly wearing off, I have resumed work on OpenARC, and am happy to report that the project has gone from being a shitty ORM with lots of boilerplate overhead to a shitty ORM with slightly less boilerplate overhead.

Introducing Minimal Definition OAGs

With this commit, we can now define graph nodes (OAGs) like so:

class OAG_SubNode1(OAG_RootNode):
    ...

    @staticproperty
    def dbstreams(cls): return {
        'field_sn1'   : [ 'int', 0 ]
    }

class OAG_SubNode2(OAG_RootNode):
    ...

    @staticproperty
    def dbstreams(cls): return {
        'field_sn2'   : [ 'int', 0 ]
    }

class OAG_AutoNode(OAG_RootNode):
    ...

    @staticproperty
    def dbstreams(cls): return {
        'field2'   : [ 'int', 0 ],
        'field3'   : [ 'int', 0 ],
        'subnode1' : [ OAG_SubNode1 ],
        'subnode2' : [ OAG_SubNode2 ]
    }

    @oagprop
    def CalcDeriv(self):
        return self.subnode1.field_sn1 + self.subnode2.field_sn2

newOAG = OAG_AutoNode1()

The instantiation of OAG_AutoNode on the last line results in the cascading creation of tables in the database for OAG_AutoNode, OAG_SubNode1 and OAG_SubNode2, along with corresponding basic indicies and foreign key constraints among all the newly created tables.

In-code definition of database constructs and their automated lazy creation in the database should drastically lower the amount of database administration that needs to be done when creating new OAG types. On an aesthetic level, I find the new definition style much nicer than the previous state of affairs.

It is worth pointing out a few things about the new defintion framework:

It supports the addition of new dbstreams: Missing dbstreams are added to existing tables as they are detected.
Deleting or changing dbstreams is NOT supported: If you make a mistake and need to delete or rename dbstreams, manual database surgery is still necessary. The library detects missing and/or removed dbstreams and thows OAGraphIntegrityError exceptions, so at least you will know when you are screwed.
There is no cycle detection: OpenARC object graphs are intended to be directed acyclic graphs, but the framework does not yet do any kind of cycle detection. To make sure your current graphs are compatible with upcoming functionality, do not introduce cycles in your object graphs.
Old-style OAG definitions are still supported: A project with only 36 commits should not have legacy baggage, but here we are: OpenARC now has two ways to define OAGs. For now, there are no plans to deprecate or eliminate the old definition style in favor of the new one.

The Way Forward

OpenARC bills itself as a functional reactive graph database, but has so far only partially delivered on its functional reactive promises. The aim over the next few months is to flesh out the project's functional reactive story, with effort concentrated on the following projects.

Project 1: Event Propagation Up The Graph

Before jumping into how to extend OpenARC's FR model, a quick review of what it currently looks like is in order:

sn1 = OAG_SubNode2().create({ 'field_sn1' : 1 })
sn2 = OAG_SubNode2().create({ 'field_sn2' : 2 })

newOAG =\
    OAG_AutoNode1().create({
        'field2'   : 10,
        'field3'   : 11,
        'subnode1' : sn1,
        'subnode2' : sn2
    })

print newOAG.CalcDeriv
# output: 3

newOAG.sn1.field_sn1 = 2
newOAG.sn1.field_sn2 = 3
print newOAG.CalcDeriv
# output: 5

newOAG.clear()
print newOAG.CalcDeriv
# output: 3

Events on subnodes (in this case a change of value for several dbstreams) are "pulled" up the graph in order to calculate a desired derived value (the CalcDeriv oagprop on newOAG). While the clear() call gives us the ability to remove in-memory edits on the graph and carry out rapid "what if" experiments, applications looking for state changes are still limited to inefficiently polling calculated oagprops.

Allowing events to be bubbled up the DAG as dbstreams are edited mitigates this limitation and ensures that work is done only when needed. A sample session would look something like this:

# Define event handlers
class OAG_AutoNode(OAG_RootNode):
    ...

    @staticproperty
    def dbstreams(cls): return {
        'field2'   : ['int',         0    , none                  ],
        'field3'   : ['int',         0    , none                  ],
        'subnode1' : [ OAG_SubNode1, None , subnode_update_handler],
        'subnode2' : [ OAG_SubNode2, None , subnode_update_handler]
    }

    def subnode_update_handler(self):
        print "A subnode has been updated. CalcDeriv is now [%s]" % self.CalcDeriv

    @oagprop
    def CalcDeriv(self):
        return self.subnode1.field_sn1 + self.subnode2.field_sn2

sn1 = OAG_SubNode2().create({ 'field_sn1' : 1 })
sn2 = OAG_SubNode2().create({ 'field_sn2' : 2 })

newOAG =\
    OAG_AutoNode1().create({
        'field2'   : 10,
        'field3'   : 11,
        'subnode1' : sn1,
        'subnode2' : sn2
    })

print newOAG.calcDeriv
# output: 3

newOAG.subnode2.field_sn2 = 3

# aysnc output: A subnode has been updated. CalcDeriv is now 4.

On-graph event propagation will simplify any programming task that requires calculations carried out off the back of event changes, all for the marginal cost of defining an extra event handler.

Project 2: Remotely Accessible OAGs

Currently, an OAG is accessible only from the process that created it. After this change, OAGs will be globally addressable and accessible over the network.

A typical session would look something like this: on the first host, an OAG is instantiated and made remotely accessible:

exportOAG = OAG_SubNode1().create({'field_sn1' : 1024})

print exportOAG.oagurl
# Throws OAError("OAG is not accessible remotely")

print exportOAG.public
# output: False

exportOAG.public = True

print exportOAG.oagurl
# output: oag://ardentprayer.anserinae.net:7843/df0e896590484364ac1704893f5e4fb1

On the second host, exportOAG can be accessed and incorporated into local OAGs as if it is local, allowing for the same event propagation semantics as fully local graphs:

importOAG = OAG_SubNode1(oagurl="oag://ardentprayer.anserinae.net:7843/df0e896590484364ac1704893f5e4fb1")

print importOAG.field_sn1
# output: 1024

superOAG =\
    OAG_SubNode().create({
        'field2'   : 10,
        'field3'   : 11,
        'subnode1' : importOAG,
        'subnode2' : sn2
    })

The implications of this change are massive: changing dbstreams on one host can trigger events on another, making OpenARC an implicit RPC framework that supports distributed functional reactive programming.

Project 3: Defining the OAG Protocol

Centralizing and codifying the protocol used by OAGs to talk to each other and the underlying storage layer will make it possible to implement graph nodes in any language and use any storage backend. Codification will involve not only a written spec, but also a test suite against which any other OpenARC protocol implementation can be tested.

The motivation for this project is simple: OpenARC is currently implemented in Python, which does not offer parallelism, and we must plan for its eventual re-implementation in a language that does (Rust perhaps?) if we hope to support truly high-performance applications. Clarifying inter-OAG interfaces and providing a clear implementation test suite will make this (and any other) re-implementation more comprehensive and less error-prone.

Project 4: Documentation

This is self explanatory: high quality projects have high quality documentation.

The project aspires to documentation that is thorough and example-driven. OpenARC is already in use over at Levelcompute, and to the extent that it is possible I will be pulling actual production examples to show how the project can be used in production in a safe and robust manner.

Timelines

As you may have guessed, OpenARC is being actively developed as part of my work at Levelcompute. There is a pressing business need for its development, so you can expect a fairly rapid development cadence. We are hoping for feature-rich minor releases once a month, followed by an 1.0 release by mid-year 2018.

If you want to talk to me about the project, you can reach out to me on Twitter, Github or via e-mail: I'm always open to feedback!

Multiple SSIDs Using Hostapd and FreeBSD

2017-09-06T00:00:00-05:00

We All Run Crap On Our Networks

Part of the reason I run my own wireless infrastructure is because I have, over time, obtained a fair number of "smart" doodads that require wireless connectivity. Remember the 2016 Dyn attack? This stuff needs to be on a segregated network with no access to anything valuable on my home network. The IoT devices go on their own SSID, live on a separate subnet, and are firewalled from anything useful. In this post I am going to elide the PF shenanigans that make this possible (the FreeBSD handbook does a better job of this than I ever could), and instead concentrate on the business of running two wireless access points using only one wireless card.

Preliminaries

The idea here is that we have two SSIDs, one called Crystalnet-int and another called Crystalnet-iot. I leave it as an exercise to the reader to deduce which SSID my wayward IoT devices will be connecting to. As always, all commands are issued as root.

Our underlying wireless device shows up when we execute the following command:

# sysctl net.interfaces.wlan
# ath0

We then start adding things to rc.conf. First we define our new virtual wireless interfaces:

wlans_ath0="wlan0 wlan1"

We also tell the system that the virtual interfaces will be operating as access points:

create_args_wlan0="wlanmode hostap bssid"
create_args_wlan1="wlanmode hostap bssid"

The bssid argument above is critical: it ensures that the two new interfaces have different MAC addresses, and cannot be mistaken for each other by connecting clients.

Restart networking:

# service netif restart

Issuing ifconfig now shows two new interfaces: wlan0 and wlan1.

Nothing Is Ever Simple

Next we configure the builtin hostapd(8) module to manage our new virtual wireless interfaces. The underlying binary takes multiple configuration files before daemonizing itself, one each for every interface being managed. My configurations are simple, and taken directly from the handbook:

# vi /etc/hostapd-<iface>.conf
interface=<iface>
debug=1
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=<ssid name>
wpa=2
wpa_passphrase=<password>
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

Unfortunately, the existing hostapd rc script doesn't allow for multiple access points, so I had to do a bit of surgery. The results are here; paste the contents into /etc/rc.d/hostapd, and add the following to /etc/rc.conf:

hostapd_ifaces="wlan0 wlan1"
hostapd_enable="YES"

Upon starting the service...

service hostapd start

... we see two healthy SSIDs:

And They Lived Happily Ever After

After my pseudo rant last time round, you could be forgiven for thinking that I was being sarcastic with the title of this section. In this case, however, you would be wrong: shortly after finishing setting up my access point, I sent out an email asking if anyone wanted my multi-SSID modification, received confirmation that there was interest, and filed a PR with the patch. I hope to see the change make it into the base system sooner rather than later.

Yes, things are sometimes broken on FreeBSD, but it is worth taking a moment to celebrate how easy the Project makes it for even the most unmotivated among us¹ to help ourselves.

O HAI ↩

Adventures in WiFi, FreeBSD Edition

2017-08-23T00:00:00-05:00

** In Which I Make a Poor Purchase **

My recent relocation from Doha to Las Vegas sans computer gear has meant that I have had to build out my Las Vegas home's wifi infrastructure from scratch. I ordered a TP-LINK TL-WDN4800 off Newegg, and thought I would be off to the races.

# sysctl net.interfaces.wlan
#

Oh.

I'm not sure what the folks over at calomel.org (who bought the same card and are apparently running 10.3-RELEASE) did, but dmesg showed the following error for me on 11.1-RELEASE:

ar9300_attach: calling ar9300_hw_attach
ar9300_hw_attach: calling ar9300_eeprom_attach
ar9300_flash_map: unimplemented for now
Restoring Cal data from DRAM
Restoring Cal data from EEPROM
ar9300_hw_attach: ar9300_eeprom_attach returned 0
ath0: ath_getchannels: unable to collect channel list from hal, status 12
device_attach: ath0 attach returned 22

According to this comment on PR 194336, the failure to attach occurs due to the card reporting being in the Canadian regulatory domain, and support not being availble in the AR9300 driver for said regulatory domain.

** Once More Down the Rabbit Hole**

First of all, why is Newegg selling Canadian hardware in the US?

Second of all, shit.

Back to the PR, this comment indicates that a fix was committed to -CURRENT in 2016 and that it works, but that it had not yet made it into any production release. I'll have more to say on this in a moment, but first let's walk through actually fixing the problem at hand: we need to recompile our kernel with the fix.

I could track 11-stable, but I want to stay as close as possible to the current -RELEASE so that future security updates are as predictable as possible from a feature-set perspective. In light of this requirement, we check out the 11.1-RELEASE branch:

# svnlite checkout https://svn.freebsd.org/base/releng/11.1/ /usr/src

We then merge r306323 (which contains the wifi driver fix) into our local source tree:

# cd /usr/src/head/sys/dev/ath
# svnlite merge -r 305614:306323 https://svn.freebsd.org/base/head/sys/dev/ath/ .

r305614 is the last revision for directory sys/dev/ath in the 11-STABLE branch, from which the 11.1-RELEASE branch was originally cut. We would like our mergeset to include any changes to the directory between this initial revision and the fix eventually committed in r306323.

Create a custom build configuration and kick off a build:

# cd /usr/src/sys/amd64/conf
# cp GENERIC ARCBSD
# make buildkernel KERNCONF=ARCBSD

The last command will take a while to execute. Go get a cup of tea. If there are compile errors, you fucked up the SVN merge. Revert changes to the tree and try the merge again:

# cd /usr/src
# svnlite revert --recursive .

Once the build is done, reboot and check that the system is, in fact, using the new kernel:

# uname -a
FreeBSD arctemplate.anserinae.net 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0 r322740M: Mon Aug 21 08:19:33 UTC 2017     inara@arctemplate.anserinae.net:/usr/obj/usr/src/sys/ARCBSD  amd64

More importantly, check that the system can see the wifi card:

# sysctl net.wlan.devices
net.wlan.devices: ath0

We can now follow the FreeBSD handbook and use hostapd to set up a wireless access point.

And They Lived Happily Ever After

Except not really. Now that I am running a custom kernel, I can now longer use freebsd-update to update this machine. I am doomed to keep recompiling custom kernels with patches every time there is a security update to 11.1-RELEASE. This is work I wouldn't have to do if the FreeBSD developers merged what they built into extant production releases instead of saying "it's fixed in -CURRENT" and leaving users to flounder.

John Kozubic said this much more eloquently than I ever can in this measured, trenchant post on freebsd-hackers a few years ago. As time goes by and I deploy more and more FreeBSD servers to production, this line from the post rings so very true: FreeBSD is becoming an operating system by, and for, FreeBSD developers. The rest of us, trying to get work done, necessarily suffer from this lack of focus.

Enabling Nested Virtualization in libvirt Environments

2017-08-21T00:00:00-05:00

Friction

A good chunk of my day to day programming activities involve running the code I am developing in virtual machines. This led to a problem when I virtualized my desktop:

Curious: Virtualbox only allows 32 bit operating systems to be emulated.

One virtual machine over in my virtualized FreeBSD workstation, I received the following error when attempting to load the bhyve hypervisor:

# kldload vmm
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6

Curiouser and curiouser. The bhyve wiki suggests this is related to virtualization platform not supporting EPT, i.e. nested virtualization.

This is the kind of problem I hate: here I am, trying to get on with my day and write some code, and now I have to figure out how to enable nested virtualization on libvirt.

Half A Solution

For once, the documentation for this semi-obscure bit of Linux functionality isn't too terrible. First, make sure that nested virtualization is actually not allowed on the hypervisor:

# cat /sys/module/kvm_intel/parameters/nested
N

Having confirmed this, set kvm-intel.nested=1 for the kernel's boot arguments in /etc/default/grub:

GRUB_CMDLINE_LINUX="reboot=acpi kvm-intel.nested=1"

Don't forget to update grub:

# update-grub

After a system reboot, nested virtualization should be allowed:

# cat /sys/module/kvm_intel/parameters/nested
Y

Next, we tighten up our guest defintions. KVM offers degrees of fidelity when emulating CPUs, and unless the CPU is perfectly emulated, nested virtualization won't work. They way to fix this is to edit the virtual machine:

# virsh edit net.anserinae.sevenfoldgates

The CPU model is changed from:

<cpu mode='host-model'/>

...to:

<cpu mode='host-passthrough'/>

After another system reboot, 64-bit virtualization is now possible in Windows:

FreeBSD

Unfortunately, when the same changes are applied to my FreeBSD VM a new, equally inscrutable error is seen when attempting to active bhyve:

$ kldload vmm
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0xffffffff81a18592, 0) error 22

According to this PR, the error occurs because bhyve is a little too zealous in verifying CPU fidelity. This comment caught my eye though:

Ubuntu 17.04 (kernel 4.10) works without any changes to bhyve

Ubuntu 15.10 doesn't advertize INVVPID capability. While it would be possible to modify bhyve to work in this mode, I'm loathe to do it since it's such a niche case

Ubuntu 16.04 seems to have the capabilities that most folk have reported.

The quality of libvirt's host-passthrough implementation seems to be improving over time; could upgrading my hypervisor kernel to a sufficiently recent version do the trick? I followed this guide to upgrade my Debian Stretch installation from kernel version 4.9 to 4.11, and saw the following upon attempting to reload the vmm module in FreeBSD after a hypervisor reboot:

# kldload vmm
#

No news is good news, and I was subsequently able to run another virtual machine inside the FreeBSD VM.

Conclusion

bhyve's relatively short development history means that odd bugs are likely to be encountered in more exotic virtualization situations. In this case, good problem reporting and documentation meant that I was able to find a workaround without wasting too much time on upgrade hell.

Booting libvirt VMs from iSCSI Block Devices

2017-08-18T00:00:00-05:00

A little bit of background for my previous article: I am working on booting all of my libvirt virtual machines off ZVOLs exposed via iSCSI from a single, FreeBSD-based ZFS pool.

The FreeBSD handbook does a fine job of explaining how to set up iSCSI targets using ctld, so this article is going to focus on the inadequately documented business of mounting iSCSI block devices on linux, and installing operating systems on them using libvirt.

** Objective **

Install windows on virtual machine sevenfoldgates.anserinae.net whose primary disk is a 100G iSCSI LUN residing on highcharity.anserinae.net.

All commands are executed as root.

** Create a storage pool backed by iSCSI **

# virsh pool-define-as sevenfoldgates-1 iscsi \
        --source-host highcharity.anserinae.net \
        --source-dev iqn.2016-04.net.anserinae:sevenfoldgates-1 \
        --target /dev
Pool sevenfoldgates-1 defined

Build and confirm the pool:

# virsh pool-build sevenfoldgates-1
Pool sevenfoldgates-1 built
# virsh pool-start sevenfoldgates-1
# virsh pool-list --all
Name               State        Autostart
-----------------------------------------
sevenfoldgates-1   active       yes

sevenfoldgates-1 is a 100G zvol, and now shows as device /dev/sde on our system:

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  3.7T  0 disk
└─sda1   8:1    0  3.7T  0 part
sdb      8:16   0  3.7T  0 disk
└─sdb1   8:17   0  3.7T  0 part
sdc      8:32   0  3.7T  0 disk
sdd      8:48   0  3.7T  0 disk
└─sdd1   8:49   0  3.7T  0 part
*sde     8:64   1  100G  0 disk

Under normal circumstances, we would create volumes on our shiny new block device using virsh vol-create, and use the resulting volume to boot our virtual machine. Unfortunately, like so much of libvirt, iSCSI-based storage pools maddeningly deviate from what is expected:

# virsh vol-create-as sevenfoldgates-1 sevenfoldgates-1_1 100G
error: Failed to create vol sevenfoldgates-1_1
error: this function is not supported by the connection driver: storage pool does not support volume creation

It turns out that libvirt automatically maps individual LUNs to volumes, with no need for the intermediate volume creation step. Thanks, I guess?

# virsh vol-list sevenfoldgates-1
 Name                 Path
------------------------------------------------------------------------------
 unit:0:0:0           /dev/sde

** Install the VM **

It's almost all over now except for the screaming (yes, I still use Windows 7):

virt-install \
--name sevenfoldgates.anserinae.net \
--ram 32768 \
--disk vol=sevenfoldgates-1/unit:0:0:0 \
--vcpus 2 \
--network bridge=vmbr0 \
--graphics vnc,listen=0.0.0.0,port=5903 \
--console pty,target_type=serial \
--cdrom /var/lib/libvirt/images/Windows-7-SP1.iso

Note the structure of the --disk option: it tells virt-install to install the VM on volume unit:0:0:0 of our shiny new sevenfoldgates-1 storage pool.

VNC into the host on port 5903 and we see this, indicating that our virtual machine has successfully booted off our iSCSI LUN:

** Conclusion **

As always, I am amazed by how much effort goes into building complex functionality into Linux, and how little effort is expended on documenting said functionality. Most of the information on this post comes from this PDF, found on the Dell website via the 8th or 9th page of Google search results for 'libvirt iscsi boot'.

If you ever read this, thanks for the pointers Jose De La Rosa!

Find Recently Modified Files

2017-08-18T00:00:00-05:00

Here's a quick snippet to find files that have been recently modified on a computer:

find / -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort -r

Useful when you've made a fleet of changes to a machine and are trying to remember what you changed.

Terrible libvirt Network Performance

2017-08-17T00:00:00-05:00

I have recently started using Linux for virtualization on my desktop, and encountered terrible network performance between the host and (FreeBSD) guest.

Here are some representative numbers, obtained using iperf:

# Host IP: 192.168.3.254
# Guest IP: 192.168.3.1

# Host -> guest
# $ guest ~> iperf -s
# $ host  ~> iperf -c 192.168.3.1
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.7 sec  3.00 MBytes  2.14 Mbits/sec

# Guest to host
# $ host  ~> iperf -s
# $ guest ~> iperf -c 192.168.3.254
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.63 GBytes  1.14 Gbits/sec

This is a truly abysmal result. Remember, these numbers are for transfer rates between a host and a virtualized guest -- we are not even hitting my network.

What could the problem be?

    $ virsh domiflist net.anserinae.installation03
    Interface  Type       Source     Model       MAC
    -------------------------------------------------------
    vnet0      network    default    rtl8139     52:54:00:cd:58:29

Wait what? An rtl8139? From if_rl.c on FreeBSD:

 * The RealTek 8139 PCI NIC redefines the meaning of 'low end.' This is
 * probably the worst PCI ethernet controller ever made, with the possible
 * exception of the FEAST chip made by SMC. The 8139 supports bus-master
 * DMA, but it has a terrible interface that nullifies any performance
 * gains that bus-master DMA usually offers.

 [snip]

 * It's impossible given this rotten design to really achieve decent
 * performance at 100Mbps, unless you happen to have a 400Mhz PII or
 * some equally overmuscled CPU to drive it.

Let's use something slightly less terrible for our guest's NIC:

$ virsh edit net.anserinae.installation03

Change this...

<interface type='bridge'>
  [random output]
  <model type='rtl8139'/>
</inteface>

...to:

<interface type='bridge'>
  [random output]
  <model type='virtio'/>
</inteface>

Restart the virtual machine:

$ virsh reset net.anserinae.installation03

...and retry the performance tests again:

# Host -> guest
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.7 sec  52.9 MBytes  45.4 Gbits/sec

# Guest -> host
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  59.7 GBytes  51.3 Gbits/sec

Performance is, conservatively speaking, improved by three orders of magnitude.

I haven't tested rtl8139 performance on any other guest operating systems, but my gut feeling here is to say that the libvirt maintainers could have chosen a slightly less garbage NIC as the default option for newly created virtual machines.

Fun With SQL Constraints

2016-09-06T00:00:00-05:00

Say you are managing entities in SQL and each entity has multiple subentities, of which exactly one can be "core":

create table entities(
    _entity_id serial primary key,
    entity_name text,
    unique(entity_name)
);

create table subentities(
    _subentity_id serial primary key,
    _entity_id int not null references entities(_entity_id),
    subentity_name text not null,
    is_core boolean not null,
    unique(_entity_id, subentity_name)
);

Up until yesterday, I would have enforced the one-core-subentity-per-entity in the client code. And while it is fashionable to play fast and loose with data integrity in the name of performance these days, doing so would have bothered me. Relying on anything other than your database engine to enforce data correctness is ultimately a lot like smoking at a gas station: you'll get away with it for a while, but a stray spark will eventually result in the rapid, unplanned disassembly of your data.

As it turns out, Postgres provides us with a solution in the form of partial indices.

Here is what I ended up doing:

create unique index u_one_active_subent_per_ent on subentities(_entity_id)
    where is_core=TRUE;

This creates an index (duh) that filters subentities for all entities where is_core is set to TRUE. And since it's a unique index, there is an implicit constraint that there can only be one such entry per _entity_id. Here is the index in action:

--1. prime the entity table
insert into auth.entities(entity_name) values ('LEVELCOMPUTE');
--2. insert a subentity where is_core=True
insert into auth.subentities(_entity_id, subentity_name, is_core) values (1, 'LEVELCOMPUTE-1', True);
--3. try doing it again
insert into auth.subentities(_entity_id, subentity_name, is_core) values (1, 'LEVELCOMPUTE-2', True);
--4. resume regular service
insert into auth.subentities(_entity_id, subentity_name, is_core) values( 1, 'LEVELCOMPUTE-2', False);

While (4) succeeds, (3) results in an error as it introduces a second is_core=True for _entity_id=1:

ERROR:  duplicate key value violates unique constraint "u_one_active_se_per_e"
DETAIL:  Key (_entity_id)=(1) already exists.

Nice.

My only wish -- a minor one, natch -- is that all kinds of constraints (even the unusual ones like this) were exposed by the CONSTRAINT keyword.

Moving Abroad: A Financial Checklist

2016-09-05T00:00:00-05:00

I get a fair number of questions about how expats should set up their finances while living abroad. While the Choudhury family is still fairly new to this, I like to think that we have done a good job of building a resilient system that has worked well as we moved across North America, Europe and Asia over the last eight months.

Through it all, our guiding principle has been to build redundancy into our planning: fraud-related account freezes are a fact of life when traveling abroad, and we never wanted to be without a backup when they happen.

Without any further ado, and in no particular order, here is how we dealt with...

Mail: A US address is mandatory if you intend to keep dealing with US banks while abroad. We updated all of our banking addresses to my inlaws' place in the California, and made sure to sign up for electronic statements so they didn't get overwhelmed with our snail mail. If you don't have a relative you can rely on, services like Earth Class Mail or Virtual Post Mail can provide you with virtual addresses. Beware: some financial institutions have cottoned on to this trick, and now refuse to have you as a customer if you use addresses from these services. I'm looking you, American Express.

Communications: Again, a US phone number is essential if you want to deal with US financial institutions. We killed two birds with one stone (the other being our local communications problem) by buying a pair of Google FI phones before leaving for Qatar. Cumulatively, the plans cost $45 a month, and we can buy extra 4G data in 1GB increments for $10 apiece. The plans come with unlimited texting and free wifi calling back to the US ($0.20 if you use the local cell network). While we have local Qatari SIMs as backup, FI's versatility means that we haven't really had to use them.

Cash: Qatar is a cash-based society, so reliably cheap access to cash via ATM was a huge consideration for us. After some research, we settled on the Fidelity Cash Management account for our primary checking needs. The account doesn't charge ATM fees anywhere in the world (reimbursing you if someone does charge you one), and charges a flat 1% fee in the unlikely event you decide to use your debit card for a purchase. Schwab offers a similar product, which we use as a backup cash-stash in case Fidelity freezes our accounts (we like redundancy, remember?).

Credit Cards: My wife and I each have no-foreign-transaction-fee credit cards from four different banks (Amex, Barclays, Chase and Citi), for a total of 8 isolated lines of credit. This pretty much guarantees that one of them will always be available. Yes, we've been down three cards before one worked, so this paranoia isn't completely unjustified: getting caught without access to cash or credit while you are abroad is unpleasant.

Remittances: My wife's salary is deposited into our Fidelity checking account, but my situation was a little bit more complicated. When I was on GBP payroll with GS, I had my salary deposited in a UK HSBC account that I had lying around from my last posting to London, and used the Fidelity transfer service to get the cash back to the US. There is a markup, but I found it to be less than 0.5% off midmarket rates for 10K+ transfers (we usually saved up a little before transferring money back to the US). For the record, I found the effort of timing transfers to take advantage GBP/USD rate fluctuations to not be worth it as everything ended up averaging out over time.

Once I was moved over to Qatari payroll, I found myself in need of a local bank account. I signed up with QNB, and found their services to be perfectly respectable. They have international transfers to the US baked into their website, and I have found their exchange rates to be favorable and their execution fees reasonable (about $3 a pop; coming from the US where international wires start at $30, this was eye-opening).

Whatever method you use to transfer your money back to the US, start slow. Transfer small amounts a few times to begin with to get a feel for transfer times, fees and repeatability. Only transfer large amounts once you feel fully comfortable with a transfer service. Trust me on this: there is no worse feeling than the sinking feeling you get in the pit of your stomach after having $20,000 go astray in the international wire system and not knowing who to talk to about it.

Travel: As anticipated, travel is now a huge part of our life, and I have no intention for paying for my "fair" share of it if at all possible. Qatar Airways is the local flag carrier and is also conveniently a member of the Oneworld alliance, so we decided to concentrate our mileage churning/collection on Oneworld programs. We rotate our credit cards every year, and the sign up bonuses from the credit cards easily cover our annual trips back to the US, as well as a couple of trips around the neighborhood every year. If you aren't already subscribed to /r/churning, you should be!

Taxes: As Americans, we are (alas) taxed on our global income. We do get a hefty 100K per person exemption on our foreign income from the IRS though, which in conjunction with the 0% local income tax in Qatar greatly cuts down on our tax bill.

While our situation is simple, the interaction between pushier foreign tax authorities and the IRS can be much more complicated and it is generally simpler to pay an accountant to handle the mess for you. It is also worth checking to see if your employer provides international tax preparation: we were surprised to find that Amanda's employer contracts with PwC to provide such a service, and I was happy to let them deal with the unhappiness that is dealing with the IRS on my behalf.

Whether you get your own accountant or use the services provided by someone else, make sure your accountant provides guidance on navigating FATCA compliance (i.e. how to tell the government about your bank accounts abroad). The penalties for falling afoul of these regulations are truly horrific, and can negate whatever financial advantage you gain by moving abroad.

So there you have it: the basic outline of a plan to manage your money and investments while you live abroad. I am sure I have missed many things in this post, so I will keep updating it as they pop into my mind. Happy travels!

Last updated: 9/5/2016

OpenVPN In A Time Of Deep Packet Inspection

2016-08-25T00:00:00-05:00

I tunnel some (not all) traffic on my home network in Qatar through an OpenVPN server in the US. The reasons are completely un-nefarious: I have a couple of dev servers in a colo in New York that I didn't feel like uprooting and bringing to Qatar, so I now need to bridge my home network to the US so that I can still access my files. Fair dinkum.

Within a few moments of connecting to my US OpenVPN server, I noticed this in the server logs:

Aug 24 06:55:50 installation00 openvpn_tap0[92375]: TCP connection established with [AF_INET]ww.xx.yy.zz:61813
Aug 24 06:55:52 installation00 openvpn_tap0[92375]: ww.xx.yy.zz:61813 WARNING: Bad encapsulated packet length from peer (18245), which must be > 0 and <= 1579 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]
Aug 24 06:55:52 installation00 openvpn_tap0[92375]: ww.xx.yy.zz:61813 Connection reset, restarting [0]

The client configuration worked fine in the US, so the warning about an active attack is not a red herring: the local telco clearly recognizes OpenVPN connections and attempts to scupper them.

The next move in the standard playbook is to sheathe the VPN connection in something less suspicious. While tunnelling OpenVPN inside stunnel results in a viably stable connection, it is clear that the telco knows something fishy is going on. Here are the results of an iperf of the VPN's bandwidth on a connection that is nominally rated 30/5Mbps:

[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.4 sec 1.03 MBytes 144 Kbits/sec
[ 6] 0.0-60.6 sec 7.87 MBytes 1.09 Mbits/sec

Fortunately, git clone does not need much bandwidth, so I am happy living with the slow connection for now.

Rust: The Development Environment

2016-06-24T01:03:00-05:00

This is the first in a series of articles examining the basics of the Rust language; it is not intended to be comprehensive, and is more than likely to be a pale immitation of the Rust Book which I heartily encourage you to read before using Rust in anger.

** Installing Rust **

Download and install Rust directly using the following command:

curl -sSf https://static.rust-lang.org/rustup.sh | sh

This will install, among other things, the Rust compiler rustc, and the Rust package manager cargo.

** Hello World! **

Here is everyone's favorite introductory program in Rust:

fn main() {
    println!( "Hello world!"); 
}

Pop that into a file called main.rs and compile using the following command:

rustc main.rs

A list on your directory will show an executable "main" in your home directory:

edgeofumbra% ls -l
-rwxr-xr-x  1 inara  staff  345684 Jun 23 19:03 main
-rw-r--r--  1 inara  staff    1262 Jun 23 19:01 main.rs

Execute the binary:

edgeofumbra% ./main
Hello world!

Congratulations, you have run your first Rust program.

** Shipping With Cargo **

Instead of compiling your programs with rustc, it makes much more sense to use cargo to manipulate your project.

Set up a new application:

cargo new app-market --bin

This creates the following directory structure for you:

app-market
|____.gitignore
|____Cargo.toml
|____src
| |____main.rs

Nifty. Toss the "Hello World!" code from above into main.rs, and try this:

cargo run

You'll see the following happen:

edgeofumbra% cargo run
   Compiling app-market v0.1.0 (file:///Users/inara/src/tmp/app-market)
     Running `target/debug/app-market`
Hello, world!

This is way better than invoking rustc and running binaries manually.

There are a couple of other cargo commands that I find myself using frequently:

cargo build will build, but not run the contents of your project.
cargo test will build your project and run any tests you have defined for your project.

cargo is very ergonomic: everything works predictably and without surprises. It is totally worth digging around and getting familiar with it.

** Dependencies **

Cargo.toml is used by by cargo to manage project version information and dependencies. Go ahead and modify it to add a project dependency on the rand module:

[package]
name = "app-market"
version = "0.1.0"
authors = ["Kamil Choudhury <kamil.choudhury@anserinae.net>"]

[dependencies]
rand="*"

The * indicates that we want to pull in the latest version of the rand module from crates.io. I won't go into it here, but you can use semantic versioning to pin the app build to a specific version of the dependency.

Go ahead and build the program:

edgeofumbra% cargo build
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling libc v0.2.12
   Compiling rand v0.3.14
   Compiling app-market v0.1.0 (file:///Users/inara/src/tmp/app-market)

Nice. If you have spent any amount of time playing around with autoconf and its ilk, this is definitely a step up.

** Autocompletion **

If you are planning on using Rust for larger projects, the RACER utility is great for Sublime Text 3 autocompletion. First up, install RACER using cargo:

cargo install racer

Verify that racer was installed and can be found by your shell; you may need to add the .cargo directory to your PATH for this to work:

edgeofumbra% which racer
/Users/inara/.cargo/bin/racer

Install the rust source to a convenient location:

mkdir -p /path/to
git clone https://github.com/rust-lang/rust /path/to

Next, use ST3 Package Control to install the RustAutoComplete package, and edit the package preferences to point to the RACER binary and the newly installed Rust source. In my case, the preferences file ended up looking like this:

// Copy this and place into your Packages/User directory.
{
    // The full path to the racer binary. If racer is already
    // in your system path, then this default will be fine.
    "racer": "/Users/inara/.cargo/bin/racer",

    // A list of search paths. This should generally just
    // be the path to the rust compiler src/ directory.
    "search_paths": [
        "/path/to/rust/src"
    ]
}

At this point autocompletion options for the Rust standard library should show up when editing .rs files. You can add to the search_paths parameter to enable autocompletion for other, non-core libraries.

** Conclusion **

This post has shown you how to set up a basic Rust development environment. Next up: the Rust type system.

Brexit Thoughts

2016-06-22T07:38:00-05:00

The Choudhury family is happy to now be in Spain now after six weeks in London. Our visit was marked by a few trips to the doctor (highly recommended, we should get single player in the US!), unfamiliar weather (so much rain!) and truly terrible food (I can only assume Britain's empire building efforts were an effort to find better food).

There was also some talk about Britain leaving the EU while we were there. There really isn't much I can add to the debate, apart from observing that it is not necessarily Far Right racists driving the push to leave the Union. I had several surprising conversations with highly educated, otherwise liberal people who pointed out that as long as the immigration quota is taken up by low-skilled immigrants from less well-off parts of Europe, it makes it very difficult to make room for the more desirable (read: educated, skilled) kind of immigrant. Fair enough: self determination is a big part of being a democracy.

That said, Britain is (and has been for most of the last six hundred years) a trading nation, and leaving the EU would impose horrendous losses upon their society. I am reminded of Captain van Hoek's harrangue to his crew before they sailed from Manilla to Acapulco in Neal Stephenson's The Confusion:

Of all the enterprises to which a man can devote his energies [...] long-distance trade is the most profitable. It is what every Jew, Puritan, Dutchman, Huguenot, Armenian, and Banyan aspires to it is what built the Navies and palaces of Europe, the Court of the Great Mogul in Shahjahanabad, and many other prodigies besides. And yet in the world of trade, it is common knowledge that no circuit, not the slave trade of the Caribbean, not the spice trade of the Indies, exceeds the Manila-to-Acapulco run in sheer profit. The wealthiest Banyans in Surat and bankers in Genoa lay their perfumed heads on silken pillows at night, and dream of sending a few bales of cargo across the Pacific on the Manila Galleon. Even with all the dangers, and the swingeing duty that must be shelled out to the Viceroy, the profits never fall below four hundred percent. That city is founded upon such dreams, Jack. We are all going to go there now."

Instead of bales of silk and baskets of spices, Britain now deals in credit default swaps and cloud computing. But the basic message remains the same: the city on the hill -- London at close, Great Britain at large -- is founded on trade. Will the British really turn their backs on such fabulous wealth to keep out a few people from Europe looking to partake in the fountains of wealth created by British trading acumen?

I guess we will find out tomorrow.

Android SDK, Best Served Insecure

2016-02-25T17:30:00-05:00

The internet has spent several days excoriating Linux Mint for insecure software distribution practices. Fortunately, the potential damage is limited to the vanishingly small population of people who 1) use desktop Linux, 2) bothered to install Linux Mint, and 3) downloaded an ISO for Mint over a fairly small window of time.

The rage seems out of proportion to the offense; it certainly did not merit a line item on the Yahoo! home page:

It's also not as if everyone else is doing any better. On a recent excursion to the Android developer site, I was confronted by this:

Yes, Google (supposedly the best software shop in the world), developer of Android (the most popular operating system in the world) is serving executables over HTTP, validated by unsigned checksums also served over HTTP.

Honestly, if Google can't distribute sotware securely, what hope do the rest of us have?

Disputing UK Air Passenger Duty Charges

2016-02-15T06:15:00-05:00

This summer, the Choudhury household is summering in Spain, and I just used a good chunk of our American Airlines mile stash to book the LHR-SJU portion of our GRX-LHR-SJU-LAX itinerary.

Total cost: 135000 AAdvantage miles and $1532 in taxes and fees.

As airline awards go, this isn't a great redemption: it worked out to 1.9 cents per mile at prevailing economy fares. The flight is direct though, and with the arrival of our child, my wife and I have softened our stance on never paying BA's notorious fees and taxes.

That doesn't mean that we just ponied up the cash though. A careful look at the final receipt revealed that a huge chunk of the bill was made up of the Air Passenger Duty: a distance based tax charged on all departures from UK airports, magnified by the class of service flown.

On our itinerary, the APD came to a whopping $211 for each of our three tickets. Yikes.

Going back to the HRMC document:

Where the second of 2 flights (flight B) is an international flight (that is from the UK to an international destination), flight B is treated as connected if its booked time of departure falls within 24 hours of the scheduled time of arrival of the first flight (flight A).

This describes our situation perfectly: our GRX-LHR leg on Iberia (flight A) arrives in London at 6PM, 21 hours before my 3.25PM LHR-SJU leg (flight B) departs the next day. Legally, we had standing to ask for the removal of the APD from our ticket; the question now was whether AA's rate desk would concur.

After giving a very helpful American Airlines agent my Iberia ticket numbers starting in 075-, I asked if she couldn't get her rate desk to waive the APD on the award tickets. It took half an hour on hold, but the refund eventually came through.

Total savings: $633.

The redemption still isn't spectacular, but $633 is nothing to sneeze at, especially since all it took was the perusal of some very nicely formatted government documents and half an hour on the phone.

TLDR; Don't ever be afraid to call in to get a better deal.

Test Post, Please Ignore

2016-01-08T10:20:00-05:00

Well, it looks like we're back in the blogging business. Let's see if it sticks this time.

Kamil Choudhury

The Bangladesh Remittance Riddle

We <3 Govtech

Trade Model Thoughts

Double Entry Book Keeping Rules Everything Around You

Building Blocks

A Super Simple Example

Route53 From First Principles

Enter BIND

Updating BIND Remotely

Should I Do This In Production?

Trying Out Latex

Book Talk - Fall; Or Dodge In Hell

Setting Up a Pkgsrc Repository

Bootstrap pkgin

Bootstrap the pkgsrc Tree

Optional: Install Binary Versions of Build Tools

Develop Your Package

Serve Your Packages

Installing Your Binaries

Conclusion

Quick and Dirty Postgres Hacks

Resetting sequences

Adding sequence to an already existing table

What's holding locks in my database?

F**k this lock in particular

Turn autovacuum off for a table

Finding how big a relation is

Querying how many rows are in a table without querying the table

Removing a sequence from a table

Post Metadata

The Butter Chicken Recipe

Ingredients

The Recipe

Final Dispositions

Book Talk - Pakistan: A Hard Country

Dumb Things To Do With Money

Culture Report, 2020 Edition

Books

Music

The Reverse Proxy Test

The Cuban Bean Recipe

Ingredients

The Recipe

Conclusion

One-Line Timing in Python

Caching Object Graphs Is Terrifying

Example Object Chain

Assumptions

Caching Primitive

Database Operations

Create

Update

Delete

Search

Conclusion

Containerize Everything: Food Edition

Programming note

In which The Guardian completely misses the point

Commissaries?

It's not going to happen

What do labor laws have to do with any of this?

Back to containers

Minimal devpi For Local Python Development

Kronk Have Many Package

Setting up devpi

Create an internal-only index

Upload projects to the internal-only index

Configure pip to use devpi

Kronk Happy

Scripting FreeBSD Updates

Paging Dr. Obvious

Go Away, It Worked For Me

FreeBSD Updates Aren't Ready For Cluster Computing

Paying Less For Hotels

Once A Year

Booking Individual Hotel Stays

Upshot

Memcache For My Brain

That Movie where...

Bootstrap `pkgin`

Bootstrap the `pkgsrc` Tree

Setting up `devpi`

Configure `pip` to use `devpi`