Let's hope pkcs11 support is coming, so that SSH support improves and SSL keys can also be protected.

Parts of this howto are all but copied from YubiKey NEO and OpenPGP. I complete it with some details and the SSH parts.

GPG

GPG normally keeps your private key encrypted using your password. If your keyring is stolen someone can brute force your password and from there decrypt all your files. If someone steals your keyring you should revoke the key as soon as possible, but assuming this revokation gets to all interested parties this will only protect new messages from being encrypted to this key. Old encrypted files could be decrypted by the attacker. Signatures are a bit better off. Sure, you have to re-sign everything and redistributed the signatures, but not nearly as bad.

1. Install some dependencies

   sudo aptitude install gnupg gnupg-agent gpgsm

2. Install Yubikey personalizer

   Compile and install the Yubico library and the Yubikey personalization tool.

3. Switch Yubikey to HID/CCID (smartcard) mode

   $ ykpersonalize -m82
   Firmware version 3.0.1 Touch level 1536 Unconfigured
   
   The USB mode will be set to: 0x82
   
   Commit? (y/n) [n]: y

4. Set up USB device permissions

   While the Yubikey in normal keyboard mode will work without thinking about permissions, when used as a smartcard your user needs to have read/write access to the device. You have three options.
   • Set device owner or permissions manually: chmod 666 /dev/bus/usb/xxx/yyy
   • Set owner or permissions via udev in /etc/udev/rules.d/99-yubikeys.rules: SUBSYSTEMS=="usb", ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0111", OWNER="thomas"
   • Be in whatever group defaults to owning the device on your system

5. Create GPG key pair

   $ gpg --card-edit
   
   Application ID ...: D2760001240102000000000000010000
   Version ..........: 2.0
   Manufacturer .....: test card
   Serial number ....: 00000001
   Name of cardholder: [not set]
   Language prefs ...: [not set]
   Sex ..............: unspecified
   URL of public key : [not set]
   Login data .......: [not set]
   Signature PIN ....: forced
   Key attributes ...: 2048R 2048R 2048R
   Max. PIN lengths .: 127 127 127
   PIN retry counter : 3 3 3
   Signature counter : 0
   Signature key ....: [none]
   Encryption key....: [none]
   Authentication key: [none]
   General key info..: [none]
   
   gpg/card> admin
   Admin commands are allowed
   
   gpg/card> generate
   
   Please note that the factory settings of the PINs are
      PIN = `123456'     Admin PIN = `12345678'
   You should change them using the command --change-pin
   
   [enter 12345678 and 123456 when asked for admin PIN and PIN]
   
   [fill in standard GPG key info]
   

   GPG using the newly created key should now work. It shouldn't even look special, except it will ask you for the PIN when needed, and won't work when the Yubikey NEO is not connected.

SSH

Turns out gpg-agent can act as an ssh-agent too. The reason for doing this is so that you can use your GPG key as an SSH key. It doesn't appear to support SSH certificates though, so I'm not ready to use it as my default one.

1. Get public key in SSH format

   $ gpg-agent --enable-ssh-support --daemon ssh-add -l
   2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx cardno:000000000001 (RSA)
   $ gpg-agent --enable-ssh-support --daemon ssh-add -L
   ssh-add AAAA[...] cardno:000000000001

   Install this public key in ~/.ssh/authorized_keys of the remote machine.

2. Create SSH wrapper script for using gpg-agent

   #!/bin/sh
   exec gpg-agent --enable-ssh-support --daemon ssh "$@"

   You will be asked for your PIN code. Make sure you have a good PIN code, or no PIN code. Having the default of 123456 is intellectually dishonest.

   Using the key with SSH like this does not require setting up gnupg, or a keyring in ~/.gnupg or anything like that. Just install the required prerequisites, fix device permissions, and run the wrapper script.

Links

• Yubico blog describing setting up GPG with the NEO

For next time I've prepared a SheevaPlug computer I had laying around. I updated it to a modern Debian installation, added a USB network card, and set it up to provide always-on VPN. This could also be done using a raspberry pi, but I don't have one.

Always-on VPN is where you have NO network access unless your VPN is up, and then ALL traffic goes through the VPN. By setting up a plug computer as a VPN client you can just plug in an unprotected computer to the "inside" and you'll be protected against the attackers on your local network.

Here are my notes on setting this up. Some of it may not be useful to other people, but it was enough of a hassle looking this up the first time that I want to note it down in case I have to do it again.

WARNING: You may brick your device if you have a different device and/or don't know what you're doing.

1. Upgrade Uboot

   Download u-boot.kwb to your tftp server.

   Marvell>> version
   U-Boot 1.1.4 (Mar 19 2009 - 16:06:59) Marvell version: 3.4.16
   Marvell>> print ethaddr       # Write this down
   ethaddr=00:11:22:33:44:55
   
   setenv serverip 192.168.1.10 # IP of your TFTP server
   setenv ipaddr 192.168.1.20
   tftpboot 0x0800000 u-boot.kwb
   nand erase 0x0 0x60000
   nand write 0x0800000 0x0 0x60000
   reset
   setenv ethaddr 00:11:22:33:44:55
   saveenv
   reset
   

2. Install Debian

   Download the Debian installer uImage and uInitrd to the tftp server.

   setenv ipaddr 192.168.1.20
   setenv serverip 192.168.1.10
   tftpboot 0x00800000 uImage
   tftpboot 0x01100000 uInitrd
   setenv bootargs console=ttyS0,115200n8 base-installer/initramfs-tools/driver-policy=most
   bootm 0x00800000 0x01100000
   

   Then install normally.

3. Set up boot to load kernel and initrd from SD card

   setenv bootargs_console console=ttyS0,115200
   setenv bootcmd_mmc 'mmc init; ext2load mmc 0:1 0x00800000 /uImage; ext2load mmc 0:1 0x01100000 /uInitrd'
   setenv bootcmd 'setenv bootargs $(bootargs_console); run bootcmd_mmc; bootm 0x00800000 0x01100000'
   saveenv
   run bootcmd

4. Install some stuff

   mv /etc/motd{,.dist}
   apt-get install openssh-server screen mg openvpn git python-webpy sudo tcpdump tshark uptimed ntp ntpdate isc-dhcp-server arping pv gcc make kernel-package

5. Setup useful environment

   alias emacs=mg
   alias tl='sudo iptables -n -v --line -L'
   alias ls='ls --color'
   # set up .screenrc
   # visudo: thomas  ALL=NOPASSWD: ALL

6. Set up network interfaces

   /etc/network/interfaces

   iface eth0 inet static
       address 10.1.2.1
       netmask 255.255.255.0
   iface eth1 inet dhcp
       pre-up /.../bin/start
   iface eth2 inet dhcp
       pre-up /.../bin/start
   iface eth3 inet dhcp
       pre-up /.../bin/start

   (many interfaces since they may be swapped around, and the names are persistent by default)

7. Install OpenSSH user CA

   /etc/ssh/sshd_config:

   TrustedUserCAKeys /etc/ssh/user_ca.pub

   cat > /etc/ssh/user_ca.pub

8. Install dnetc

   Just for fun. Download.

9. Set up remote OpenVPN server

   Out of scope for this blog post.

10. Set up DHCP server

    /etc/dhcp/dhcpd.conf

    ddns-update-style none;
    option domain-name-servers 8.8.8.8, 8.8.4.4;
    default-lease-time 600;
    max-lease-time 7200;
    authoritative;
    log-facility local7;
    subnet 10.1.2.0 netmask 255.255.255.0 {
            range 10.1.2.11 10.1.2.250;
            option broadcast-address 10.1.2.255;
            option routers 10.1.2.1;
    }
    

11. Install firewalling scripts

    git clone git://github.com/ThomasHabets/profy.git

12. Set up OpenVPN

    Crate ovpn.conf config in cfg/ using ovpn.conf.template as a template. Symlink /etc/openvpn/ovpn.conf to that file.

13. Set noatime on all filesystems in /etc/fstab

14. Set up kernel build environment

    USB network drivers sucked in default kernel. I only got 2Mbps for the ones that even had drivers. 65% of CPU was interrupt handling, OpenVPN only about 5%.

    cd /usr/bin
    for i in ld objdump ar nm strip objcopy size; do
      ln -s {,arm-linux-gnueabi-}$i
    done

15. Build and install new kernel

    cp /boot/config* .config
    make menuconfig
    time fakeroot make-kpkg kernel_image
    dpkg -i ../kernel*.deb
    mkinitramfs -o /boot/initrd.img-3.7.6 3.7.6
    cd /boot
    mkimage -A ARM -O Linux -T Kernel -C none -a 0x00008000 -e 0x00008000 -n 3.7.6 -d vmlinuz-3.7.6  uImage-3.7.6
    mkimage -A ARM -O Linux -T RAMDisk -C gzip -a 0x00000000 -e 0x00000000 -n 3.7.6 -d initrd.img-3.7.6  uInitrd-3.7.6

16. Test the new kernel

    Copy new kernel to tftp server.

    setenv ipaddr 192.168.1.20
    setenv serverip 192.168.1.10
    tftpboot 0x00800000 uImage-3.7.6
    tftpboot 0x01100000 uInitrd-3.7.6
    bootm 0x00800000 0x01100000

17. If working, default to new kernel

    cd /boot
    mv uImage{,.dist}
    mv uInitrd{,.dist}
    ln -s uImage-3.7.6 uImage
    ln -s uInitrd-3.7.6 uInitrd

18. Profit

    I get about 14Mbps, with <50% CPU assigned to OpenVPN, and the rest curiously "idle". Keep in mind that the SheevaPlug is about 4 years old at this point.

Photo

Links

• Install Debian on SheevaPlug

That's fucked up.

Never EVER think that linking (at link time or runtime) is easy or obvious.

I link my current pet project with:

g++ -Wl,-z,now -Wl,-z,relro -pie -static-libstdc++

The binary then seems to work across the systems I currently want to run it on. Specifically it makes me able to run the binary compiled on Debian Testing on a Debian Stable installation.

Skipping that whole dynamic libraries thing is something Go got right.

Update: some clarification on why you'd want to compile statically

Let's start with the reason for wanting to compile static in the first place. While shared libraries are better in some aspects, "save RAM" is no longer a good reason for always compiling dynamically. There are reasons why you'd want dynamic linking still, but that aint it. The savings in RAM for the .text and .rodata sections (and unmodified .data, I guess) are negligible when compared to RAM footprint of interesting applications. There the interesting data is usually mmap()ed or simply read or generated. Shared libraries won't help you with that. KSM (Kernel Samepage Merging) will even help in doing reverse copy-on-write to take back even this loss.

Take /usr/bin/ssh for example.

$ ldd $(which ssh) | awk '{print $3}' | sed 1d | grep -v ^$ | xargs du -Dhcs | tail -1
4.6M total

No, the big benefit of shared libraries is the fact that you only have to upgrade in *one* place to have the code upgraded everywhere.

Where static linking wins is where you have control of "all" aspects of operations. That is, if there is a new version of openssl to fix a bug, it's feasible in your environment to recompile. You then also have the option to roll back, and roll back openssl to only one application in case there is an incompatibility. This won't, of course, help with dlopen()ed libraries.

Now back to what the options mean. "-static" is the option to ostensibly not require (assume existence and compatibility of) shared libraries on the target system. Turns out it's not that simple. "-static-libstdc++" is often (I dare almost say "usually") needed if the libstdc++ versions between build and target systems differ by a few years on Linux. Like I said I needed this to compile my C++11 stuff on Debian Testing (frozen to be stable, IIRC) and Debian Stable. One binary. Just scp it to the target and run. No dependencies.

As for libgcc it's mostly for cases where you can't count on access to *any* libraries (such as what you mentioned, as OS hacking), but isn't that why you gave "-static" in the first place? :-)

This post doesn't address other aspects of static linking, such as glibc being static-hostile, a whole topic in itself. Compiling static won't make everything "just work" either.

It said that unlike Arping 2.09, in Arping 2.11 the ARP cache was not updated after successful reply. I thought that was odd, since there's no code to touch the ARP cache, neither read nor write. Surely this behaviour hasn't changed?

I tried to reproduce the behaviour and sure enough, with Arping 2.09 the arp cache is updated, while with 2.11 it's not.

$ arp -na | grep 192.168.0.123
$ # --- First try Arping 2.11 ---
$ sudo ./arping-2.11 -c 1 192.168.0.123
ARPING 192.168.0.123
60 bytes from 00:22:33:44:55:66 (192.168.0.123): index=0 time=1.188 msec

--- 192.168.0.123 statistics ---
1 packets transmitted, 1 packets received,   0% unanswered (0 extra)
$ arp -na | grep 192.168.0.123
$ # --- Ok, that didn't change the ARP cache. Now try 2.09 ---
$ sudo ./arping-2.09 -c 1 192.168.0.123
ARPING 192.168.0.123
60 bytes from 00:22:33:44:55:66 (192.168.0.123): index=0 time=794.888 usec

--- 192.168.0.123 statistics ---
1 packets transmitted, 1 packets received,   0% unanswered (0 extra)
$ arp -na | grep 192.168.0.123
? (192.168.0.123) at 00:22:33:44:55:66 [ether] on wlan0

So what changed between 2.09 and 2.11?

$ git log --pretty=oneline arping-2.09..arping-2.11 | wc -l
43

• Switching to clock_gettime() (various patches). Read gettimeofday() should never be used to measure time for why.
• Switching to select() from poll()
• Adding support to use getifaddr() to find the correct output interface

Between Arping 2.09 and 2.10 I changed the interface finding code from an ugly hack of running /sbin/ip route get 1.1.1.1 to get the outgoing interface from the routing table. Since the output of the various "show me the routing table" commands are different in different OSs, I had to implement this subprocess (ugly) and parsing (ugly) several times. The new implementation uses getifaddrs() to traverse the interfaces programmatically.

The old code was still there as a fallback. It would never actually get used unless there's a Linux system out there that doesn't have getifaddrs(). It seems it was added to glibc 2.3 back in 2002. Anyway it was trivial to temporarily switch interface selection back to the old method. I confirmed that this was indeed what caused this change of behaviour.

Surely ip route get doesn't send an ARP request and populates the ARP cache when it gets the reply? No. So if ip route get 1.1.1.1 doesn't do it, and arping-2.11 1.1.1.1 doesn't do it, then surely ip route get 1.1.1.1 ; arping-2.11 1.1.1.1 doesn't do it?

Yes, yes it does. It seems ip route get 1.1.1.1 followed by arping-2.11 1.1.1.1 will cause 1.1.1.1 to show up in the ARP cache. And it doesn't matter if ip route get is run as an ordinary user or as root! (arping of course has to run as root or have NET_ADMIN capability). Only the exact address given to ip route get will be "open to be filled" by the second command, so it seems to be per address, and that ip route get will modify state in the kernel.

$ arp -na | grep 192.168.0.123
$ sudo ./arping-2.11 -i wlan0 -q -c 1 192.168.0.123
$ arp -na | grep 192.168.0.123
$ # --- Ok, still no entry in the ARP cache Now try running both commands ---
$ ip route get 192.168.0.123 ; sudo ./arping-2.11 -i wlan0 -q -c 1 192.168.0.123
192.168.0.123 dev wlan0  src 192.168.0.100 
    cache  mtu 1500 advmss 1460 hoplimit 64
$ arp -na | grep 192.168.0.123
? (192.168.0.123) at 00:22:33:44:55:66 [ether] on wlan0

I closed the bug since it's working as intended.

I have not dived into the kernel source to find the reason for this, but I may come back and update this post if and when I do.

Links

• Arping on github
• Arping home page
• Debian bug 667808: Arping 2.11 not update arp cache entry

Normally you shouldn't get in this situation. Only someone who hates their users makes a non backwards compatible change to a library ABI. You don't hate your users, do you?

(just kidding about hating your users.)

Disclaimer

I thought I'd dive into this problem as a weekend project. Don't rely on this article as a source of truth, but please correct me where I'm wrong. I'm not an expert in creating shared libraries, and it's much harder that it would first appear. The existence of libtool proves that.

Example project described can be found here.

Multiple versions of the same library

The lovely land of modern Unix will allow you to have multiple versions of the same library installed at the same time. That's not the problem. The problem is that when you load the libraries all the symbols are resolved inside the same namespace. There can't be two versions in use of the same function with the same name, even if they are in different libraries (well, it won't work the way you want it to). You can't choose which one to see from different code. (see breadth first search description here for details).

The scenario

In this scenario there are three separate libraries:

• libd1 - Calls functions in libd2 and libbase version 1. File name is simply libd1.so.
• libd2 - Calls functions in libd1 and libbase version 2. File name is simply libd2.so. (cross dependency between libd1 and libd2)
• libbase - This is the library that made non backwards compatible changes between version 1 and 2. Files are libbase.so.1.0 and libbase.so.2.0, with traditional symlinks from libbase.so.n. The important bit here is that the two versions are to be considered incompatible and must both be usable in the same process at the same time.

What we want to achieve is to have libd1 and libd2 be able to call functions in libbase, but to have them call different versions. Both libbase.so.1 and libbase.so.2 should be loaded and active (have their functions callable).

What normally happens

When you link an executable with -lfoo the linker finds libfoo.so and verifies that it (along with default libraries) contain all the symbols that are currently unresolved. Linking shared objects (DSO, .so files) is similar, except there's no requirement that all symbols are resolved. If an unresolved symbol exists in more than one library then only one is actually used. You can't therefore link two overlapping ABIs, even indirectly via an intermediate dependency. Well, you can, but are you comfortable with one of them being "hidden"? (which one is not important for this article)

Both versions of the library can therefore be loaded, but if they overlap then one will hide the other (in the overlap).

How to solve it

The hiding of one of the symbols in a name clash is done by the dynamic linker (ld.so) at load time (run time). Since symbols are referenced by name the only way to differentiate the two names is to force them to have different names.

This can be done in different ways; manually, or automatically. Both append some text to every symbol you want to differentiate.

Automatic (with --default-symver)

$ gcc -fPIC -Wall -pedantic   -c -o base1.o base1.c
$ gcc -shared \
		-Wl,--default-symver \
		-Wl,-soname,libbase.so.1 \
		-o libbase.so.1.0 base1.o
$ nm libbase.so.1.0
[...] 00000000000006f0 T base_print     <--- same name as without the special args
[...] 0000000000000000 A libbase.so.1   <--- this is new with --default-symver
[...]

Let's compile and link one of the libraries that uses libbase.so:

$ ldconfig -N -f ld.so.conf
$ ln -fs libbase.so.1 libbase.so  # library to link with
$ gcc -fPIC -Wall -pedantic   -c -o d1.o d1.c
$ gcc -Wl,-rpath=. -L. \
		-Wl,--default-symver \
		-Wl,-soname,libd1.so \
		-shared -o libd1.so d1.o -lbase
$ nm d1.o
[...]            U base_print
[...]            U d2_print
[...]
$ nm libd1.so
[...]            U base_print@@libbase.so.1
[...]            U d2_print
[...]
$ ldd libd1.so
[...]      libbase.so.1 => ./libbase.so.1
[...]

Also note that libd1.so knows that it's depending on libbase.so.1 (not plain libbase.so). This is the name given with the -soname option when linking libbase.so. So there's even more magic going on than I led on. And it's a good reason for having that option.

libd2.so is compiled similarly, except against version 2 of libbase.so. The program p is then linked to both libd1.so and libd2.so, which in turn pulls in both versions of libbase.so:

$ cc -Wl,-rpath=. -o p p.o -L. -Wl,-rpath=. -ld1 -ld2
$ ldd p
	linux-vdso.so.1 =>  (0x00007ffff41f8000)
	libd1.so => ./libd1.so (0x00007fe08b788000)
	libd2.so => ./libd2.so (0x00007fe08b586000)
	libc.so.6 => /lib/libc.so.6 (0x00007fe08b1e5000)
	libbase.so.1 => ./libbase.so.1 (0x00007fe08afe3000)
	libbase.so.2 => ./libbase.so.2 (0x00007fe08ade1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe08b98c000)
$ nm p | grep libd
                 U d1@@libd1.so
                 U d2@@libd2.so
$ ./p
base version 2> init
base version 1> init
d1> init
d2> init
d1()
 d2_print()
  base version 2> d1        <--- libd1 -> libd2 -> libbase version 2
d2()
 d1_print()
  base version 1> d2        <--- libd2 -> libd1 -> libbase version 1
d2> fini
d1> fini
base version 1> fini
base version 2> fini

Manual with --version-script

Instead of --default-syms one can use --version-script=base1.map and create the map file.

1 2 3 4

BASE1 { global: base_print; local: *; };

Manual from within the code

asm(".symver base_print_foo,base_print@@BASE1");

Notes

• "@@" in a name means "this version and the default". A single "@" means just "this version".

Summary

… yet Unix still has much less DLL hell than Windows.

I aimed to provide an accessible view into shared libraries by solving a specific problem. For more in-depth information from people who know more about it than me see the links below. I've barely scratched the surface.

The compile time linker (ld) and the dynamic linker (ld.so) do a lot of magic. More than you'd expect if you haven't thought about it before.

Links

• Example project
• How To Write Shared Libraries, by Ulrich Drepper
• Program Library HOWTO - 3. Shared Libraries
• ld.so GNU linker/loader
• Gold readiness obstacle #1: Berkeley DB. Also see other linker related posts on that blog.
• It's not all gold that shines — Why underlinking is a bad thing. Nice illustrations of the problem.
• Autotools Mythbuster - Chapter 3. Building All Kinds of Libraries — libtool
• Why there are shared object files and a bunch of symlinks
• Binutils ld manual - 3.9 VERSION Command

I thought I'd take a look at the hash_set/hash_map GNU C++ extension.

1
2
3
4
5
6
7
8

  inline size_t
  __stl_hash_string(const char* __s)
  {
    unsigned long __h = 0;
    for ( ; *__s; ++__s)
      __h = 5 * __h + *__s;
    return size_t(__h);
  }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#include<time.h>
#include<iostream>
#include<hash_set>

double getclock()
{
  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);
  return ts.tv_sec + ts.tv_nsec / 1e9;
}

_GLIBCXX_BEGIN_NAMESPACE(__gnu_cxx)
template<> struct hash< ::std::string> {
  size_t operator()(const ::std::string& x) const {
    return hash<const char*>()(x.c_str());
  }
};
_GLIBCXX_END_NAMESPACE

int main()
{
  __gnu_cxx::hash_set<std::string> the_set;
  double start = getclock();

  while (std::cin.good()) {
    std::string line;
    getline(std::cin, line);

    the_set.insert(line);
  }
  std::cout << "Size: " << the_set.size() << std::endl
            << "Time: " << getclock()-start << std::endl;
}

$ ./generateCollisions 5 > coll-5
$ ./generateCollisions 6 > coll-6
$ wc -l coll-5 coll-6
  13826 coll-5
 149696 coll-6
 163522 total
$ seq -f "%06.0f" 13826 > num-5
$ seq -f "%06.0f" 149696 > num-6
$ ./load < num-5
Size: 13827
Time: 0.0148379
$ ./load < num-6
Size: 149697
Time: 0.135555
$ ./load < coll-5
Size: 13827
Time: 1.44663
$ ./load < coll-6
Size: 149697
Time: 212.009

Collision generator

Summary

Links

• 28C3: Effective Denial of Service attacks against web application platforms

As you can plainly see from this graph, my TPM chip can do approximately 1.4 SSL handshakes per second. A handshake takes about 0.7 seconds of TPM time, so when two clients are connecting the average connect time is 1.4 seconds. This means probably not useful on server side, but should be good for some client side applications.

To replicate the test, start a server:

openssl s_server -keyform engine -engine tpm -accept 12345 -cert foo.crt -key foo.key -tls1 -CAfile foo.crt -verify 1 -status

for n in $(seq 100); do time openssl s_client -tls1 -connect localhost:12345 < /dev/null >/dev/null 2>/dev/null;done 2> timelog

plot [1:] [0:2] '2' using (2/$1) w l title '2 clients','1' using (1/$1) w l title '1 client'

Meaning even if you get hacked the attackers cannot impersonate you, if you manage to kick them off or just shut down the server. The secret key is safe. It has never been outside the TPM and never will be.

This can be used for both client and server certs.

Prerequisites

• A TPM chip. Duh. May need to be turned on in the BIOS. Could be called "security chip" or something. If you don't have a TPM chip but still want to follow along (maybe add TPM support to some program) then you can install a TPM emulator. See links at the end on how to install a TPM emulator.
• A working CA that will sign your CSR. I will assume you're running your own CA, but you can send the CSR to someone else to sign if you want cert creating step.
• Installed "stuff":

  $ aptitude install tpm-tools libengine-tpm-openssl

Initialize TPM

• Make sure you can communicate with the TPM chip:

  $ tpm_version
    TPM 1.2 Version Info:
    Chip Version:        1.2.4.1
    Spec Level:          2
    Errata Revision:     2
    TPM Vendor ID:       INTC
    Vendor Specific data: 00040001 0003040e
    TPM Version:         01010000
    Manufacturer Info:   494e5443
  

• Clear the chip:

  $ tpm_clear --force
  $ tpm_setenable --enable --force
  $ tpm_setactive --active

  These steps may require reboots. They will probably tell you if that's the case.
• Take ownership and create SRK:

  $ tpm_takeownership -u -y
  Enter SRK password: <just press enter>
  Confirm password: <just press enter>

  Yes really, empty password. This password protects TPM operations, so worst case is that someone with shell access can put some load on your TPM chip. They can't use it to crack keys or anything. This is not a password that protects your secrets, they're already secure. You may want to supply an owner password. If so, leave out the -y option.

Create SSL certificate

• Use script from libengine-tpm-openssl to create key: create_tpm_key foo.key
• Create CSR for key: openssl req -keyform engine -engine tpm -key foo.key -new -out foo.csr
• Sign the CSR with your CA: openssl x509 -req -days 365 -in foo.csr -CA ca.crt -CAkey ca.key -CAserial serial -out foo.crt

Test it

$ openssl s_server -accept 12345 -cert /path/to/server.crt -key /path/to/server.key -CAfile CA/ca.crt -Verify 1

$ openssl s_client -keyform engine -engine tpm -connect server-name-here:12345 -cert foo.crt -key foo.key

Use in your code

Instead of using SSL_CTX_load_verify_locations(), use ENGINE_*() and SSL_CTX_use_PrivateKey()

Without TPM:

1 2 3 4

void loadpriv(SSL_CTX* ctx, const char* file) { SSL_CTX_use_PrivateKey_file(ctx, file, SSL_FILETYPE_PEM); }

With TPM:

1 2 3 4 5 6 7 8 9 10

void loadpriv(SSL_CTX* ctx, const char* file) { ENGINE_load_builtin_engines(); ENGINE* engine = ENGINE_by_id("tpm"); ENGINE_init(engine); UI_METHOD* ui_method = UI_OpenSSL(); EVP_PKEY* pkey = ENGINE_load_private_key(engine, file, ui_method, NULL); SSL_CTX_use_PrivateKey(ctx, pkey); }

Error handling left as an exercise for the reader.

Use in the real world

Huh. I haven't actually seen this implemented in any Open Source software (besides the openssl command line interface) Well, actually curl and stunnel4 appear that they could support it, but it's not clear how. Here is someone else wondering how to get stunnel to do it. It's from 2006 with no replies.

The only thing I have that uses the TPM chip is my Chromebook. Client certs there are protected by it by default.

Please leave comments if you know of any TPM support in Open Source stuff.

tlssh

1
2
3

PrivkeyEngine tpm
TpmSrkPassword
PrivkeyPassword secretpasswordhere

If PrivkeyEngine tpm or -E tpm is supplied the secret key will be assumed to be TPM protected and should be in the form of a "BEGIN TSS KEY BLOB" section instead of a PEM format "BEGIN RSA PRIVATE KEY" section.

Performance

Oh, it's slow. Really slow. Luckily it's just used in the handshake. I haven't done proper benchmarks, but let's just say you won't be using it to power your webmail. I may do some benchmarking in a coming blog post.

Update: It can do about 1.4 connections per second. See Benchmarking TPM backed SSL for details.

Misc notes

• The .key file for the TPM-backed cert contains the secret key encrypted with something that's in the TPM. It's not usable without the exact same TPM chip.
• You can choose to create a key in the normal way and then "convert" it to a TPM-backed key. It's not as secure since you can't say the key was never stored outside the TPM chip, but use the -w switch to create_tpm_key if you want to do this.
• OpenSSL is probably the most horrible, disgusting, undocumented crap library I've ever seen. Some of this can be blamed on SSL, X509 and ASN.1, but far from all.
• Emulated TPM is not completely useless. Since the emulator runs as root and the user has no insight into it you can have users be able to use private keys that they can't read or copy to other machines. (caveat: I don't know if the emulator is actually secure in this mode, but it can be made to be)
• My TPM chip doesn't seem to like importing (create_tpm_key -w foo.key) 4096 bit keys. 2048 was fine though.

Links

• trusted platforms module (TPM), openssl and ecryptfs tutorial. Another nice use of TPM. Also installation instructions for TPM emulator
• Resources from a TPM workshop
• Use TPM on Linux

The obvious way to do this is with a proxy. The problem with that is that the traffic from the browser to the proxy is not encrypted. Even when you browse to secure SSL sites some traffic is being sent in the clear, such as the host name. That's not so bad, but I want to hide my HTTP traffic too.

Turns out that at least Chrome does support using SSL between the browser and the proxy, but you can't configure it directly. You have to use Proxy Auto Configuration to point to an HTTPS host:port.

Running OpenVPN is out in this case since I want it to work with everything, including Android and ChromeOS… and Windows (without installing "stuff"). Not that I've tried it with Windows yet, but it should work.

For added fun I wanted to authenticate to the proxy using a client certificate, so that I don't have to worry about remembering or losing passwords.

Before you set off to do the same thing I should warn you that either I haven't done it quite right or there is a bug in Chrome. If the network has a hickup so that the proxy or the .pac file is unavailable it will fall back to connecting directly. I want it to never fall back. That's kind of the point.

But all is not in vain. I also set up IPSec with these same certificates. There will be a follow-up post describing how I set this up to tunnel securely with both Chromebook and my phone.

This is what I did (caveat: some of this is from memory. If you find any errors please let me know):

1. Copy easyrsa 2.0 from openvpn sources

   This is a set of scripts to manage a CA. We'll be using this directory/set of scripts both to create a server and client certificates. But only the client certificates will be signed by our own CA.

2. Edit vars file

   I like to change KEY_SIZE to 2048, country and other stuff. These will be the defaults when creating certs.

3. Init easyrsa

   ./clean-all
   This is a one-time init. Don't run this script again.

4. Build the CA for the client certs

   ./build-ca

5. Copy CA cert to proxy server

   scp keys/ca.crt proxy:/etc/proxy-ssl/proxy-ca.crt

6. Build server key and signing request

   ./build-req proxy.foo.com

7. Get signed server cert

   Send keys/proxy.foo.com.csr to your real CA (e.g. cacert.org or Verisign). You'll get back proxy.foo.com.crt. If you do use cacert.org then make sure that your browser trusts their root by importing it.

8. Put proxy.foo.com cert and key on proxy server

   scp keys/proxy.foo.com.{crt,key} proxy:/etc/proxy-ssl/

9. Install squid proxy and stunnel on proxy server

   On Debian/Ubuntu: apt-get install squid3 stunnel

10. Configure & start stunnel

    1. set ENABLE=1 in /etc/defaults/stunnel.conf
    2. In /etc/stunnel/stunnel.conf

             cert=/etc/proxy-ssl/proxy.foo.com.crt
             key=/etc/proxy-ssl/proxy.foo.com.key
             verify=2
             CAfile=/etc/proxy-ssl/proxy-ca.crt
             [proxy]
             accept  = 12346
             connect = 3128

    3. /etc/init.d/stunnel4 start

11. Create .pac file somewhere on an https url

    function FindProxyForURL(url, host)
    {
          return "HTTPS proxy.foo.com:12346";
    }
    

    Make sure that the certificate of the https server is one that the browser will accept.

12. Set proxy autoconf to the url to this .pac file

    Spanner->Preferences->Under the Bonnet->Change proxy settings->Automatic Proxy Configuration
    Set the URL to where the file is, including "https://".

13. Try it now. It should fail with SSL errors

    It should fail because proxy doesn't accept your cert (you don't have a cert in the browser yet). Don't continue if you get some other error.

14. Create client cert signed by your own CA

    ./build-key my-proxy-key
    Don't set a password.

15. Convert the client key+cert to .p12, because that's what Android and Chrome wants

    openssl pkcs12 -export -in keys/my-proxy-key.crt -inkey keys/my-proxy-key.key -out my-proxy-key.p12
    It'll ask for a password that will only be used in the next step. No need to save it for later.

16. Import the client certificate into the browser

    On your Chrome (or chromebook) go Spanner->Preferences->Under the Bonnet->Certificate Manager->Your Certificates->Import (or "Import and Bind to Device").

17. Try to browse somewhere. Now it should work

    Make sure you're actually using the proxy. Go to www.whatismyip.com or something and make sure that it sees the IP of the proxy. It'll probably tell you that you're using a proxy. If it doesn't work then good luck. :-)

Problems

Like I said it seems that it falls back to connecting directly, even though the .pac file doesn't have a fallback mechanism configured. Stay tuned for the IPSec version.

This is pretty Linux-focused, but I'll add pointers for other OSs if I see them.

Short version

This will get a bit basic for some people, so here's the short version. Make sure you measure the effect of changing these settings. Don't just increase them and think "more is better".

ip route change default via x.x.x.x initrwnd 20

ip route change default via x.x.x.x initcwnd 20

ip route change default via x.x.x.x initcwnd 20 initrwnd 20

The path to enlightenment

In short, the number of bytes in flight in TCP is the lowest of the senders congestion window (cwnd) and the receiver window size. The window size is the receiver announcing how much it's ready to receive, and the sender will therefore not send any more than that. The cwnd is the sender trying to avoid causing congestion. (since TCP is two-way both sides have both, but think of it as one-way for now)

The window size is announced in every segment (TCP packets are called "segments") and is the receivers method of throttling. The cwnd is not sent across the network, but is a local sender-side throttling only.

Both of these default to "a few" multiples of the maximum size segments (MSS). The MSS is announced in the SYN and SYN|ACK packets and are adjusted so that a TCP segment fits in the path MTU (hopefully). Typical initial window size and cwnd is on the order of 4kB. Linux has an initial cwnd of 10*MSS nowadays, but that doesn't help if the receiver window is smaller that that. Both window size and cwnd will change as the connection progresses. How and why is a huge topic in itself. I'm focusing here on a specific problem of initial size and TCP slow start.

The problem I was having was that when I downloaded http://blog.habets.pp.se over a ~50ms RTT connection these values don't go up fast enough. During the ~300ms transaction (including everything) the sender stalls about 4-6 times waiting for the receiver to give the go-ahead. (I only have approximate numbers since the RTT between the servers I was testing was a bit unstable. Virtual servers and all that). The time between first and last byte was about 170ms. Really? ~50ms RTT and 17kB in 170ms? That's not good.

Ideally there should be two round trips involved. One where the client has sent SYN and is waiting for SYN|ACK, and the other when the client has sent the request and is waiting for the results. If this was a long-running connection (with HTTP keep-alive for example) this wouldn't be a problem, but since I'm looking at the first request in a new TCP connection it is.

Increase receiver window size

Since I couldn't find it in sysctl or /sys I looked at the source. net/ipv4/tcp_output.c has a function called tcp_select_initial_window(). The last parameter is __u32 init_rcv_win. Well, that was easy. Because I'm lazy I just compiled the kernel hardcoding it to 10 (it's in multiples of MSS). I started a lab virtual machine and sure enough the initial window is now a lot bigger. Still not seeing a reduction in round trip waits though, and it's as slow as before. At least now it's not the receivers fault. The window has lots of space left but the sender is quiet.

What is this dst_metric(dst, RTAX_INITRWND) that it was before I changed it to hardcoded 10 though? include/linux/rtnetlink.h which defines RTAX_INITRWND looks like mostly routing related stuff. Aha! Sure enough:

user@hostname$ ip route help
[...]
OPTIONS := FLAGS [ mtu NUMBER ] [ advmss NUMBER ]
       [ rtt TIME ] [ rttvar TIME ] [reordering NUMBER ]
       [ window NUMBER] [ cwnd NUMBER ] [ initcwnd NUMBER ]
       [ ssthresh NUMBER ] [ realms REALM ] [ src ADDRESS ]
       [ rto_min TIME ] [ hoplimit NUMBER ] [ initrwnd NUMBER ]
[...]  

Misc

• As far as I can see window size and cwnd can not be set per-socket by setsockopt() or similar. If they could this would cause people to write evil applications that don't play well with the rest of the Internet.
• Current cwnd, window size and other data can be seen with getsockopt(..., TCP_INFO, ...).
• An Argument for Increasing TCP’s Initial Congestion Window
• Google and Microsoft Cheat on Slow-Start. Should You?
• Linux kernel patch for adding setsockopt() for changing cwnd. Not needed. It was rejected for a reason.

Links

• Draft to increase TCP's initial window by default to 10 * MSS. This has been implemented in Linux.
• Increasing the TCP Initial Congestion Window on Windows 2008 Server R2