I thought I'd take a look at the hash_set/hash_map GNU C++ extension.

1
2
3
4
5
6
7
8

  inline size_t
  __stl_hash_string(const char* __s)
  {
    unsigned long __h = 0;
    for ( ; *__s; ++__s)
      __h = 5 * __h + *__s;
    return size_t(__h);
  }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#include<time.h>
#include<iostream>
#include<hash_set>

double getclock()
{
  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);
  return ts.tv_sec + ts.tv_nsec / 1e9;
}

_GLIBCXX_BEGIN_NAMESPACE(__gnu_cxx)
template<> struct hash< ::std::string> {
  size_t operator()(const ::std::string& x) const {
    return hash<const char*>()(x.c_str());
  }
};
_GLIBCXX_END_NAMESPACE

int main()
{
  __gnu_cxx::hash_set<std::string> the_set;
  double start = getclock();

  while (std::cin.good()) {
    std::string line;
    getline(std::cin, line);

    the_set.insert(line);
  }
  std::cout << "Size: " << the_set.size() << std::endl
            << "Time: " << getclock()-start << std::endl;
}

$ ./generateCollisions 5 > coll-5
$ ./generateCollisions 6 > coll-6
$ wc -l coll-5 coll-6
  13826 coll-5
 149696 coll-6
 163522 total
$ seq -f "%06.0f" 13826 > num-5
$ seq -f "%06.0f" 149696 > num-6
$ ./load < num-5
Size: 13827
Time: 0.0148379
$ ./load < num-6
Size: 149697
Time: 0.135555
$ ./load < coll-5
Size: 13827
Time: 1.44663
$ ./load < coll-6
Size: 149697
Time: 212.009

Collision generator

Summary

Links

• 28C3: Effective Denial of Service attacks against web application platforms

As you can plainly see from this graph, my TPM chip can do approximately 1.4 SSL handshakes per second. A handshake takes about 0.7 seconds of TPM time, so when two clients are connecting the average connect time is 1.4 seconds. This means probably not useful on server side, but should be good for some client side applications.

To replicate the test, start a server:

openssl s_server -keyform engine -engine tpm -accept 12345 -cert foo.crt -key foo.key -tls1 -CAfile foo.crt -verify 1 -status

for n in $(seq 100); do time openssl s_client -tls1 -connect localhost:12345 < /dev/null >/dev/null 2>/dev/null;done 2> timelog

plot [1:] [0:2] '2' using (2/$1) w l title '2 clients','1' using (1/$1) w l title '1 client'

Meaning even if you get hacked the attackers cannot impersonate you, if you manage to kick them off or just shut down the server. The secret key is safe. It has never been outside the TPM and never will be.

This can be used for both client and server certs.

Prerequisites

• A TPM chip. Duh. May need to be turned on in the BIOS. Could be called "security chip" or something. If you don't have a TPM chip but still want to follow along (maybe add TPM support to some program) then you can install a TPM emulator. See links at the end on how to install a TPM emulator.
• A working CA that will sign your CSR. I will assume you're running your own CA, but you can send the CSR to someone else to sign if you want cert creating step.
• Installed "stuff":

  $ aptitude install tpm-tools libengine-tpm-openssl

Initialize TPM

• Make sure you can communicate with the TPM chip:

  $ tpm_version
    TPM 1.2 Version Info:
    Chip Version:        1.2.4.1
    Spec Level:          2
    Errata Revision:     2
    TPM Vendor ID:       INTC
    Vendor Specific data: 00040001 0003040e
    TPM Version:         01010000
    Manufacturer Info:   494e5443
  

• Clear the chip:

  $ tpm_clear --force
  $ tpm_setenable --enable --force
  $ tpm_setactive --active

  These steps may require reboots. They will probably tell you if that's the case.
• Take ownership and create SRK:

  $ tpm_takeownership -u -y
  Enter SRK password: <just press enter>
  Confirm password: <just press enter>

  Yes really, empty password. This password protects TPM operations, so worst case is that someone with shell access can put some load on your TPM chip. They can't use it to crack keys or anything. This is not a password that protects your secrets, they're already secure. You may want to supply an owner password. If so, leave out the -y option.

Create SSL certificate

• Use script from libengine-tpm-openssl to create key: create_tpm_key foo.key
• Create CSR for key: openssl req -keyform engine -engine tpm -key foo.key -new -out foo.csr
• Sign the CSR with your CA: openssl x509 -req -days 365 -in foo.csr -CA ca.crt -CAkey ca.key -CAserial serial -out foo.crt

Test it

$ openssl s_server -accept 12345 -cert /path/to/server.crt -key /path/to/server.key -CAfile CA/ca.crt -Verify 1

$ openssl s_client -keyform engine -engine tpm -connect server-name-here:12345 -cert foo.crt -key foo.key

Use in your code

Instead of using SSL_CTX_load_verify_locations(), use ENGINE_*() and SSL_CTX_use_PrivateKey()

Without TPM:

1 2 3 4

void loadpriv(SSL_CTX* ctx, const char* file) { SSL_CTX_use_PrivateKey_file(ctx, file, SSL_FILETYPE_PEM); }

With TPM:

1 2 3 4 5 6 7 8 9

void loadpriv(SSL_CTX* ctx, const char* file) { ENGINE* engine = ENGINE_by_id("tpm"); ENGINE_init(engine); UI_METHOD* ui_method = UI_OpenSSL(); EVP_PKEY* pkey = ENGINE_load_private_key(engine, file, ui_method, NULL); SSL_CTX_use_PrivateKey(ctx, pkey); }

Error handling left as an exercise for the reader.

Use in the real world

Huh. I haven't actually seen this implemented in any Open Source software (besides the openssl command line interface) Well, actually curl and stunnel4 appear that they could support it, but it's not clear how. Here is someone else wondering how to get stunnel to do it. It's from 2006 with no replies.

The only thing I have that uses the TPM chip is my Chromebook. Client certs there are protected by it by default.

Please leave comments if you know of any TPM support in Open Source stuff.

tlssh

1
2
3

PrivkeyEngine tpm
TpmSrkPassword
PrivkeyPassword secretpasswordhere

If PrivkeyEngine tpm or -E tpm is supplied the secret key will be assumed to be TPM protected and should be in the form of a "BEGIN TSS KEY BLOB" section instead of a PEM format "BEGIN RSA PRIVATE KEY" section.

Performance

Oh, it's slow. Really slow. Luckily it's just used in the handshake. I haven't done proper benchmarks, but let's just say you won't be using it to power your webmail. I may do some benchmarking in a coming blog post.

Update: It can do about 1.4 connections per second. See Benchmarking TPM backed SSL for details.

Misc notes

• The .key file for the TPM-backed cert contains the secret key encrypted with something that's in the TPM. It's not usable without the exact same TPM chip.
• You can choose to create a key in the normal way and then "convert" it to a TPM-backed key. It's not as secure since you can't say the key was never stored outside the TPM chip, but use the -w switch to create_tpm_key if you want to do this.
• OpenSSL is probably the most horrible, disgusting, undocumented crap library I've ever seen. Some of this can be blamed on SSL, X509 and ASN.1, but far from all.
• Emulated TPM is not completely useless. Since the emulator runs as root and the user has no insight into it you can have users be able to use private keys that they can't read or copy to other machines. (caveat: I don't know if the emulator is actually secure in this mode, but it can be made to be)
• My TPM chip doesn't seem to like importing (create_tpm_key -w foo.key) 4096 bit keys. 2048 was fine though.

Links

• trusted platforms module (TPM), openssl and ecryptfs tutorial. Another nice use of TPM. Also installation instructions for TPM emulator
• Resources from a TPM workshop
• Use TPM on Linux

The obvious way to do this is with a proxy. The problem with that is that the traffic from the browser to the proxy is not encrypted. Even when you browse to secure SSL sites some traffic is being sent in the clear, such as the host name. That's not so bad, but I want to hide my HTTP traffic too.

Turns out that at least Chrome does support using SSL between the browser and the proxy, but you can't configure it directly. You have to use Proxy Auto Configuration to point to an HTTPS host:port.

Running OpenVPN is out in this case since I want it to work with everything, including Android and ChromeOS… and Windows (without installing "stuff"). Not that I've tried it with Windows yet, but it should work.

For added fun I wanted to authenticate to the proxy using a client certificate, so that I don't have to worry about remembering or losing passwords.

Before you set off to do the same thing I should warn you that either I haven't done it quite right or there is a bug in Chrome. If the network has a hickup so that the proxy or the .pac file is unavailable it will fall back to connecting directly. I want it to never fall back. That's kind of the point.

But all is not in vain. I also set up IPSec with these same certificates. There will be a follow-up post describing how I set this up to tunnel securely with both Chromebook and my phone.

This is what I did (caveat: some of this is from memory. If you find any errors please let me know):

1. Copy easyrsa 2.0 from openvpn sources

   This is a set of scripts to manage a CA. We'll be using this directory/set of scripts both to create a server and client certificates. But only the client certificates will be signed by our own CA.

2. Edit vars file

   I like to change KEY_SIZE to 2048, country and other stuff. These will be the defaults when creating certs.

3. Init easyrsa

   ./clean-all
   This is a one-time init. Don't run this script again.

4. Build the CA for the client certs

   ./build-ca

5. Copy CA cert to proxy server

   scp keys/ca.crt proxy:/etc/proxy-ssl/proxy-ca.crt

6. Build server key and signing request

   ./build-req proxy.foo.com

7. Get signed server cert

   Send keys/proxy.foo.com.csr to your real CA (e.g. cacert.org or Verisign). You'll get back proxy.foo.com.crt. If you do use cacert.org then make sure that your browser trusts their root by importing it.

8. Put proxy.foo.com cert and key on proxy server

   scp keys/proxy.foo.com.{crt,key} proxy:/etc/proxy-ssl/

9. Install squid proxy and stunnel on proxy server

   On Debian/Ubuntu: apt-get install squid3 stunnel

10. Configure & start stunnel

    1. set ENABLE=1 in /etc/defaults/stunnel.conf
    2. In /etc/stunnel/stunnel.conf

             cert=/etc/proxy-ssl/proxy.foo.com.crt
             key=/etc/proxy-ssl/proxy.foo.com.key
             verify=2
             CAfile=/etc/proxy-ssl/proxy-ca.crt
             [proxy]
             accept  = 12346
             connect = 3128

    3. /etc/init.d/stunnel4 start

11. Create .pac file somewhere on an https url

    function FindProxyForURL(url, host)
    {
          return "HTTPS proxy.foo.com:12346";
    }
    

    Make sure that the certificate of the https server is one that the browser will accept.

12. Set proxy autoconf to the url to this .pac file

    Spanner->Preferences->Under the Bonnet->Change proxy settings->Automatic Proxy Configuration
    Set the URL to where the file is, including "https://".

13. Try it now. It should fail with SSL errors

    It should fail because proxy doesn't accept your cert (you don't have a cert in the browser yet). Don't continue if you get some other error.

14. Create client cert signed by your own CA

    ./build-key my-proxy-key
    Don't set a password.

15. Convert the client key+cert to .p12, because that's what Android and Chrome wants

    openssl pkcs12 -export -in keys/my-proxy-key.crt -inkey keys/my-proxy-key.key -out my-proxy-key.p12
    It'll ask for a password that will only be used in the next step. No need to save it for later.

16. Import the client certificate into the browser

    On your Chrome (or chromebook) go Spanner->Preferences->Under the Bonnet->Certificate Manager->Your Certificates->Import (or "Import and Bind to Device").

17. Try to browse somewhere. Now it should work

    Make sure you're actually using the proxy. Go to www.whatismyip.com or something and make sure that it sees the IP of the proxy. It'll probably tell you that you're using a proxy. If it doesn't work then good luck. :-)

Problems

Like I said it seems that it falls back to connecting directly, even though the .pac file doesn't have a fallback mechanism configured. Stay tuned for the IPSec version.

This is pretty Linux-focused, but I'll add pointers for other OSs if I see them.

Short version

This will get a bit basic for some people, so here's the short version. Make sure you measure the effect of changing these settings. Don't just increase them and think "more is better".

ip route change default via x.x.x.x initrwnd 20

ip route change default via x.x.x.x initcwnd 20

ip route change default via x.x.x.x initcwnd 20 initrwnd 20

The path to enlightenment

In short, the number of bytes in flight in TCP is the lowest of the senders congestion window (cwnd) and the receiver window size. The window size is the receiver announcing how much it's ready to receive, and the sender will therefore not send any more than that. The cwnd is the sender trying to avoid causing congestion. (since TCP is two-way both sides have both, but think of it as one-way for now)

The window size is announced in every segment (TCP packets are called "segments") and is the receivers method of throttling. The cwnd is not sent across the network, but is a local sender-side throttling only.

Both of these default to "a few" multiples of the maximum size segments (MSS). The MSS is announced in the SYN and SYN|ACK packets and are adjusted so that a TCP segment fits in the path MTU (hopefully). Typical initial window size and cwnd is on the order of 4kB. Linux has an initial cwnd of 10*MSS nowadays, but that doesn't help if the receiver window is smaller that that. Both window size and cwnd will change as the connection progresses. How and why is a huge topic in itself. I'm focusing here on a specific problem of initial size and TCP slow start.

The problem I was having was that when I downloaded http://blog.habets.pp.se over a ~50ms RTT connection these values don't go up fast enough. During the ~300ms transaction (including everything) the sender stalls about 4-6 times waiting for the receiver to give the go-ahead. (I only have approximate numbers since the RTT between the servers I was testing was a bit unstable. Virtual servers and all that). The time between first and last byte was about 170ms. Really? ~50ms RTT and 17kB in 170ms? That's not good.

Ideally there should be two round trips involved. One where the client has sent SYN and is waiting for SYN|ACK, and the other when the client has sent the request and is waiting for the results. If this was a long-running connection (with HTTP keep-alive for example) this wouldn't be a problem, but since I'm looking at the first request in a new TCP connection it is.

Increase receiver window size

Since I couldn't find it in sysctl or /sys I looked at the source. net/ipv4/tcp_output.c has a function called tcp_select_initial_window(). The last parameter is __u32 init_rcv_win. Well, that was easy. Because I'm lazy I just compiled the kernel hardcoding it to 10 (it's in multiples of MSS). I started a lab virtual machine and sure enough the initial window is now a lot bigger. Still not seeing a reduction in round trip waits though, and it's as slow as before. At least now it's not the receivers fault. The window has lots of space left but the sender is quiet.

What is this dst_metric(dst, RTAX_INITRWND) that it was before I changed it to hardcoded 10 though? include/linux/rtnetlink.h which defines RTAX_INITRWND looks like mostly routing related stuff. Aha! Sure enough:

user@hostname$ ip route help
[...]
OPTIONS := FLAGS [ mtu NUMBER ] [ advmss NUMBER ]
       [ rtt TIME ] [ rttvar TIME ] [reordering NUMBER ]
       [ window NUMBER] [ cwnd NUMBER ] [ initcwnd NUMBER ]
       [ ssthresh NUMBER ] [ realms REALM ] [ src ADDRESS ]
       [ rto_min TIME ] [ hoplimit NUMBER ] [ initrwnd NUMBER ]
[...]  

Misc

• As far as I can see window size and cwnd can not be set per-socket by setsockopt() or similar. If they could this would cause people to write evil applications that don't play well with the rest of the Internet.
• Current cwnd, window size and other data can be seen with getsockopt(..., TCP_INFO, ...).
• An Argument for Increasing TCP’s Initial Congestion Window
• Google and Microsoft Cheat on Slow-Start. Should You?
• Linux kernel patch for adding setsockopt() for changing cwnd. Not needed. It was rejected for a reason.

Links

• Draft to increase TCP's initial window by default to 10 * MSS. This has been implemented in Linux.
• Increasing the TCP Initial Congestion Window on Windows 2008 Server R2

That pretty much sums up this blog post but I'm going to go on anyway. If you're thinking of introducing two-factor authentication to your company, or you're using something that's fundamentally broken (like RSA SecureID) you simply must at least take Yubikeys into consideration.

When I say that SecureID (and others) are fundamentally broken what I mean is that when (not if, as recent history has shown) RSA (the company) is broken into YOUR security is now compromised. When I first used SecureID and found out that you as a customer aren't in control of your own keys my first thought was "well that's just stupid". Why are you giving the keys to the kingdom to someone else?

Enter Yubikeys. They just beat SecureID in every way (almost). Benefits:

• Open specification.
• You can set your own keys (secrets) and don't have to show them to a third party who can lose them.
• They're cheap. 25 USD for one (yes you can buy just one if you want) and cheaper in bulk, of course. No server license.
• No server license. (this bears repeating). Yes there are products you can buy to get features and such, but with minimal coding you can actually get this up and running on your website / server logins for free!
• Looong OTP tokens. If you don't have some account locking feature on your servers you can brute-force 6 digits fairly quickly. OpenSSH by default allows 10 unauthenticated sessions with one attempt per second. You can script that to log in in less than a day.

• No clock in the device. While plugged in the token it generates will be a function of how long it's been powered up this session, but it doesn't have a battery-powered clock that you can sync with the server.
• No display on the device. Well, you wouldn't want to key in a really long token anyway, but if you can't insert USB into the terminal or want to use it to log in somewhere with your phone then you're screwed. I bet Yubico are working on some NFC to fix that though. (the RFID parts of some Yubikeys are something separate that I'm not including here)

So what can you do with a Yubikey?

Simple OTP generation

• The secret key (obviously)
• The time since this session started
• Number of tokens generated this session
• Number of sessions since device was programmed
• Key ID (user settable), so you know which secret key to use do decode the token.
• Some other less interesting fields. This list is intentionally not complete.

This waiting is not possible when using challenge-response mode (see below).

Symantec VIP

If you buy one for personal use I'd recommend buying the VIP key. It's the same price as the normal one but it comes pre-programmed in slot 1 with something you can use to log into your paypal account (once you associate the key with your account).

You can still use slot 2 for something else.

Static password

HOTP

Challenge-response mode

So what you need on your client is a program that can take the challenge from the server, send it to the Yubikey and then send the reply back to the server. If you are an online game developer this would just be something you put into your game. As long as the users have their Yubikey plugged into their machine they can log in and play.

I've created some scripts to use this for unix authentication that I'll be releasing in a week or two. You SSH to some server, the server says "Yubikey challenge: XXX[...]: ", you copy that challenge (just select it) and press a key combination that starts this script. The script takes the challenge from the clipboard, gets the reply from the Yubikey and "types" it. Challenge-responses aren't "typed" by the Yubikey so I'm doing that in the script. And presto, you get challenge-response Yubikey authentication with SSH and every other application without having to code it into your client.

YubiHSM

By the way, I don't care if you hash your passwords with unique salt for everyone. If you lose that database then you've lost the users' passwords.

YubiHSM is a USB-attached device that doesn't emulate a keyboard. It has several functions (including the ability to do the logic of a Yubikey authentication server) but to me the most interesting is protecting the password database.

In short, YubiHSM can give you an interface where you can use just two operations:

• Set password. This will not store anything on the YubiHSM but will give you back a blob that only the YubiHSM can decrypt.
• Check password. You supply a password and a blob and the YubiHSM says "yes" or "no". It does not give you the password.

Using yhsmpam I've used this to secure a unix machine. Think of it as moving from storing hashes in /etc/passwd to storing them in /etc/shadow, but this time moving from /etc/shadow to YubiHSM.

Unfortunately YubiHSM isn't for sale yet. Oooh! I just checked the website and it's going on sale on 18 August 2011. At 500 USD it's more expensive than the Yubikey, but then again you only need it for your authentication server, not for every user. 500 USD is NOT a lot for this kind of hardware. Before YubiHSM you'd have to cough up 15k+ USD to get this. PRE-ORDER NOW! Run, do not walk, to their online store.

Complaints

• No clock in the Yubikey
• Hard to use Yubikey with phone
• While all "normal" keyboard layouts work fine (including German), Dvorak will not work well with Yubikeys by default. Well, with my ChromeOS laptop it did work by default, but still.

Links

• Yubico
• pam_externalpass - PAM module I use to do more fancy auth.
• YHSMPAM - Backend for keeping UNIX passwords in YubiHSM
• YOracle - My own yubikey validator server.
• [Placeholder for when I release the challenge-response code]

They're not SSH PubkeyAuthentication

In other words if your .pub file doesn't end in -cert.pub and you haven't used ssh-keygen -s, then you aren't using certificates.

They're not SSL

They're not PEM, x509 ASN.1 or any other insane format

They're not easy to google for

What they do

Sign host keys

The authenticity of host 'xxx' can't be established.
RSA key fingerprint is xxx.
Are you sure you want to continue connecting (yes/no)? 

Sign user keys

Either or both of the above

Allow you to expire certs

Setting up host certificates

1. On the machine that you'll be storing your CA on: Create host CA key (host_ca & host_ca.pub):

   ssh-keygen -f host_ca

2. Sign existing host public key:

   ssh-keygen -s host_ca -I host_foo -h -n foo.bar.com -V +52w /etc/ssh/ssh_host_rsa_key.pub

   Copy /etc/ssh/ssh_host_rsa_key.pub from other servers and this command on those files too, and copy the resulting back to the server.
3. Configure server(s) to present certificate (/etc/ssh/sshd_config):

   HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

   and restart sshd.
4. Put host CA in clients known_hosts, such as ~/.ssh/known_hosts or a system-wide one. The line should look something like this:

   @cert-authority *.bar.com ssh-rsa AAAAB3[...]== Comment

5. Remove clients entry (including aliases such as its IP address) for the host itself in ~/.ssh/known_hosts (if any)
6. Logging in to the correct name should now work and you should not be asked about the host key:

   ssh foo.bar.com

7. Logging in to the IP address of the machine should present the message: "Certificate invalid: name is not a listed principal"

Do NOT forget the -n switch when creating host certificates. Otherwise anyone who cracks one machine will be able to impersonate any other machine in the domain to users who trust this CA.

You may need to add something like this to your client config (~/.ssh/config):

Host * 
  HostKeyAlgorithms ssh-dss-cert-v01@openssh.com,ssh-dss 

Setting up user certificates

1. On the machine you'll be storing your CA on: Create user CA key (user_ca & user_ca.pub):

   ssh-keygen -f user_ca

2. For every server that the certificates should work on:
   • Copy user_ca.pub from the CA machine to /etc/ssh/ on the server.
   • Add user_ca.pub to servers /etc/ssh/sshd_config:

     TrustedUserCAKeys /etc/ssh/user_ca.pub

   And restart sshd.
3. For each user: Sign existing user pubkey key:

   ssh-keygen -s user_ca -I user_thomas -n thomas,thomas2 -V +52w /path/to/id_rsa.pub

   Give the resulting *-cert.pub file back to the user.
4. Log in as user thomas or thomas2 on the server, even though the server has never seen your key or cert before, only user_ca.pub.

Some notes

• Never forget -n when creating host certs.
• If you put your user CA in ~/.ssh/authorized_keys instead of configuring the server you don't need -n for user certs. If you're doing system-wide (as described) it's mandatory.
• The certificate validity time is in UTC
• You can revoke keys and certs with RevokeKeys
• -I just specifies the key ID. It doesn't have to follow the pattern I used.
• Print certificate information with ssh-keygen -L:

  $ ssh-keygen -L -f user-thomas-cert.pub
  user-thomas-cert.pub:
          Type: ssh-rsa-cert-v00@openssh.com user certificate
          Public key: RSA-CERT-V00 a4:b3:6d:e2:bd:4a:39:01:31:c9:05:43:db:78:f6:c9
          Signing CA: RSA a5:e5:20:8e:ea:ea:15:7e:c3:31:60:2d:6b:93:a0:6b
          Key ID: "user_thomas"
          Valid: from 2011-07-07T15:37:00 to 2012-07-05T15:38:11
          Principals: 
                  thomas
                  thomas2
          Critical Options: 
                  permit-agent-forwarding
                  permit-port-forwarding
                  permit-pty
                  permit-user-rc
                  permit-X11-forwarding
  

Links

• Email thread: getting host certificates working
• Serverfault: How to revoke an SSH certificate

What's the problem?

gettimeofday() returns the current wall clock time and timezone. time() returns a subset of this info (only whole seconds, and not timezone).

Using these functions in order to measure the passage of time (how long an operation took) therefore seems like a no-brainer. After all, in real life you measure by checking your watch before and after the operation. The differences are:

1. Nobody sneaks in and changes your wristwatch when you're not looking

You usually aren't running NTP on your wristwatch, so it probably won't jump a second or two (or 15 minutes) in a random direction because it happened to sync up against a proper clock at that point.

Good NTP implementations try to not make the time jump like this. They instead make the clock go faster or slower so that it will drift to the correct time. But while it's drifting you either have a clock that's going too fast or too slow. It's not measuring the passage of time properly.

2. You're not stupid, the computer is

Doing something doesn't take less than 0 time. If you get <0 when you measure time, you'll realize something is wrong. A program will happily print that a ping reply came back before you sent it. Even if you check for time<0, the program that uses gettimeofday() still can't tell the difference between a 2 second delay and 3 seconds plus a time adjustment.

3. You are the limiting factor

In real life you are not expected to measure sub-second times. You can't measure the difference between 1.08 seconds and 1.03 seconds. This problem is mostly (but far from entirely) in the small scale.

What to use instead

The most portable way to measure time correctly seems to be clock_gettime(CLOCK_MONOTONIC, ...). It's not stable across reboots, but we don't care about that. We just want a timer that goes up by one second for each second that passes in the physical world.

So if you want to wait 10 seconds, then check the monotonic clock, add 10 seconds and wait until that time has come.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

#include <time.h> #include <stdio.h> /** * sleep for `sec' seconds, without relying on the wall clock of time(2) * or gettimeofday(2). * * under ideal conditions is accurate to one microsecond. To get nanosecond * accuracy, replace sleep()/usleep() with something with higher resolution * like nanosleep() or ppoll(). */ void true_sleep(int sec) { struct timespec ts_start; struct timespec ts_end; clock_gettime(CLOCK_MONOTONIC, &ts_start); ts_end = ts_start; ts_end.tv_sec += sec; for(;;) { struct timespec ts_current; struct timespec ts_remaining; clock_gettime(CLOCK_MONOTONIC, &ts_current); ts_remaining.tv_sec = ts_end.tv_sec - ts_current.tv_sec; ts_remaining.tv_nsec = ts_end.tv_nsec - ts_current.tv_nsec; while (ts_remaining.tv_nsec > 1000000000) { ts_remaining.tv_sec++; ts_remaining.tv_nsec -= 1000000000; } while (ts_remaining.tv_nsec < 0) { ts_remaining.tv_sec--; ts_remaining.tv_nsec += 1000000000; } if (ts_remaining.tv_sec < 0) { break; } if (ts_remaining.tv_sec > 0) { sleep(ts_remaining.tv_sec); } else { usleep(ts_remaining.tv_nsec / 1000); } } } int main() { true_sleep(10); }

The same method works if you want to schedule something in the future, or see how long something took. I've found that I very rarely actually need the wall clock time in programs. The only exceptions I can think of are when it should be store on disk (valid across reboot or with NFS across server) or when the time (not the time delta) should be shown to the user.

The case of "sleep" can actually be solved by clock_nanosleep(), but I wanted an example that could illustrate how to measure too.

If clock_gettime(CLOCK_MONOTONIC, ...) is not available on your system, then try to find a monotonic clock that is. Like gethrtime() or CLOCK_HIGHRES on Solaris. I have created a portable library for getting monotonic time.

The guilty ones

libpcap

The pcap_pkthdr struct (the "received packet" struct) contains a struct timeval ts that ruins our ability to measure the time it takes for the reply you get for some query you sent. They tell me the kernel supplies the timestamp, so it's not really libpcaps fault.

Calling clock_gettime() when libpcap gives you a packet has turned out to be useless, as the time difference between packet reception and the delivery to your program is too long and unstable. You're stuck with this wall-clock time until you fix all the kernels in the world and break binary compatibility with old libpcap programs.

ping

Try running a ping and setting the time in the past. Ping will freeze waiting until it thinks it's time for the next packet. Tried with iputils ping for Linux. A brief survey of the source code of other OpenSource pings show this:

• FreeBSD: Same as Linux
• NetBSD: Much code in common with FreeBSD. This too looks the same.
• Mac OSX: Same heritage, same fault
• OpenBSD: Seems OK because they (surprise surprise) ignore the C standard and do the work in a SIGALRM handler
• Solaris: Same as OpenBSD, except they don't support fractional second intervals

It seems that the only people who got the ping send scheduler right in this regard did it at the expense of not following the C standard, and nobody got the RTT calculation right. Solaris actually used a monotonic clock (gethrtime()) in other parts of the code, but not for RTT calculation.

I have sent a patch to fix this on Linux ping. It has not yet been applied to upstream.

Oracle database

If time was set backwards then Oracle would reboot the machine. Really. (bug ID seems to be 5015469)

When a leap second is inserted into the official time, such as 2008-12-31 23:59:60, all the other clocks are suddenly fast, and therefore adjust themselves backwards. On newyears day 2009 many people woke up to their Oracle servers having rebooted. Not simply shut down the oracle process, but actually rebooted the server.

F5

F5 BigIP load balancers seem to cut all active connections when time is set backwards. So if you use two NTP servers that don't have the same time, all your connections will be cut when your load balancer flips back and forth between the two upstream times.

Me

At least arping and gtping didn't handle this properly and could in some cases (not always) act like Linux ping. I have fixed both of them and it'll be in the next version. I had put off fixing them because I wanted a solution that solved packet timings as well as packet scheduling, but at least with arping that doesn't seem possible due to the libpcap issue mentioned above.

GTPing should now have this solved perfectly, and ARPing should only show incorrect times for replies when the wall clock changes or drifts, but the scheduler should still send one packet per interval.

Everyone else?

I'd be surprised to see any program handling this correctly. Please let me know if you find one.

Exception

If a wall-clock time is all you can get (such as from libpcap), then you're gonna have to use that. But you don't have to like it. And avoid wall time where at all possible.

Unrelated note

When I turn on the GPS I have in my car after it's been off for too long it asks me where I am and what time and date it is. Why would a GPS possibly want to ask me those things? Telling me where I am and what time it is is what it was made to do.

Update: As a couple of people have pointed out there is a "right" answer to this. It's not a problem for newer GPS units, but there it is.

Links

• Doug Coleman on Monotonic timers
• Portable library for getting monotonic time
• My feelings about GPSs
• Monotonic time in Python

Why? Because SSH has some drawbacks that sometimes annoy me. I also wanted an authentication scheme that's more similar to SSL/TLS than what SSH does.

With tlssh you don't specify username or password, you simply connect to the server using a client-side certificate to log in as the user specified in the certificate. No interaction until you reach the shell prompt on the server.

Of course you can log in using a public key with SSH, but it's only a public/private key pair, there's none of the PKI that SSL has.

Specifically, what I was missing in SSH was:

• Expiring keys, both login-keys and server certificates
• CRLs (Certificate Revocation Lists) - wouldn't it be nice to just revoke the all certificates that were on a compromised machine and they'll suddenly be unusable everywhere? (I will add OCSP too. Same thing but more "online")
• Pureness. Not all these channels and port forwarding features that complicate everything (and adds bugs).
• TCP MD5. I doubt SSH will ever get this. (temporarily disabled in tlssh)
• CA-signed server certificate. Use Verisign-signed certificates (if you trust them) and you'll have none of that "confirm server fingerprint" stuff.
• Will get some client-side features that I've missed in SSH. Like "take a file from the local filesystem as send it as if I typed it", to avoid having to scp 5 times because you have to transfer a file hop-by-hop. The OpenSSH folks rejected my patch for this.

It's built on OpenSSL, so as long as there aren't any bugs in the X.509 parsing in OpenSSL I should be fine. Unfortunately X.509 is a horrible mess of a format so this is not impossible. Activating TCP MD5 before SSL negotiation should lessen the risk, but still.

To log in to a server you create a certificate with CommonName bob.users.example.com, and get it signed by the CA the server trusts. The server will split this CommonName into the user part and the domain part. The domain part must match what the server is expecting. If everything is fine (and the certificate is not in the CRL, and the CRL is not too old), you are presented with a shell as the user mentioned in the user part of the certificate.

To log in as a different user you have to create a second certificate.

Another reason I'm writing tlssh is to learn OpenSSL. The OpenSSL API is horrible, as it turned out.

It's also fun to code system level stuff, creating terminals and such. You learn that OpenBSD doesn't seem to care about POSIX or other standards. When some library call is acting funny you can often google your way to some top OpenBSD-developer saying "who cares about POSIX? That was a stupid question" when someone notices the non-compliance. They still don't have wordexp() for example.

Drawbacks

• Not as portable as OpenSSH. Currently works on Linux, OpenBSD and Solaris.
• Not as audited (I wholeheartedly invite more eyes, and code too)
• Because it needs a CA to sign certificates, it's not initially as plug-and-play as SSH.
• Like I said X.509 is horrible. Hopefully OpenSSL parses it perfectly, but you never know.

Resources

• tlssh on Github
• git clone git://github.com/ThomasHabets/tlssh.git

Most networks don't do multicast routing, which means most network guys don't have much experience with it. Sure they know that it exists, and it's probably used on their layer 2, but they don't do multicast routing. These "rules" list some things that you should know when configuring or troubleshooting multicast.

The second rules of multicast is you do not forward packets coming from a non-RPF interface

If a multicast packet is received, and it's not the multicast RPF interface, it's dropped. The table can be viewed with "show ip mroute", and it will clearly show what interface is acceptable. Here's an example of group 239.0.0.1:

R1#show ip mroute 239.0.0.1
[...]
(*, 239.0.0.1), 00:00:08/stopped, RP 1.0.0.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    FastEthernet1/0, Forward/Sparse, 00:00:08/00:03:21

(1.0.0.2, 239.0.0.1), 00:00:01/00:02:58, flags: 
  Incoming interface: FastEthernet1/1, RPF nbr 1.1.12.2
  Outgoing interface list:
    FastEthernet1/0, Forward/Sparse, 00:00:01/00:03:28

To just show the RPF path to an IP address while not being multicast-specific, use "show ip rpf".

R1#show ip rpf 1.0.0.2
RPF information for ? (1.0.0.2)
  RPF interface: FastEthernet1/1
  RPF neighbor: ? (1.1.12.2)
  RPF route/mask: 1.0.0.2/32
  RPF type: unicast (ospf 1)
  RPF recursion count: 0
  Doing distance-preferred lookups across tables

Third rule: no show ip cef, no debug ip packet

You can't go low-level as in unicast and do "show ip cef <destination address>". In order to see what path a forwarded (or locally originating) multicast packet takes, you have to check "show ip mroute". Multicast packets do not show up when doing "debug ip packet". The command to use is "debug ip mpacket", and even then you must run "no ip mroute-cache" on interfaces where you want to see packets.

Fourth rule: Only one RP per group

If you use Auto-RP or PIMv2 BSR this won't be a problem. The protocol should take care to only create one RP per group. When you define your RPs statically (ip pim rp-address <rp-address>), however, you run the risk having two unrelated RPs. You can also get this if you only have one RP address IP, but you announce it from two or more routers in the network (anycast RP). If you have multiple RPs then they will have to talk to each other to make sure they all have a complete list of all senders in the network. Do this by setting up MSDP between them.

Rule 5: The unicast routing table is not to be trusted

"show ip route" is not used for multicast routing. Sure, the unicast topology is a source of information for the multicast one. But if you're troubleshooting it just might be (i.e.: probably is) because the process of creating the multicast routing table is not working properly. Your first source of information is and always should be "show ip mroute". You also have "show ip mroute <group-address> count", which shows counters for each source and a bit of information as to why packets (if any) are being dropped, and how many are forwarded.

Rule 6: For multicast VPNs, lo0 must have PIM turned on

The MDT speaks using the router id. Just make the router id the same for LDP, ISIS/OSPF and BGP and nobody will get hurt. lo0 is the simple choice for router id.

Rule 7: In order for IGMP snooping to work, you have to define the router port

Describing the problem and possible solutions would make this post way too big. So I'll let Cisco do that.

Short version of one of the solutions: Enable PIM on any router facing a switch, even if you don't expect to actually talk PIM with any other router on that interface. Also block PIM adjacencies from forming so that they aren't formed by mistake.

To confirm proper IGMP snooping, run "show mac-address-table multicast" on the switch.

Rule 8: Both sender and receiver side need PIM enabled (or not)

According to cisco, you need to turn on PIM on an interface in order for the router to listen to IGMP (receiver side) and in order to "perform IP multicast routing" (which I read as something you need on the sender side). My testing shows that sometimes you don't need PIM on the sender side though. But maybe you should enable it just in case. It probably should be turned on anyway due to rule 7.

But just as with rule 7, make sure that you don't accidentally set up a PIM adjacency with anyone unexpectedly (this can cause your multicast to fail, even if the other router isn't misconfigured). If you aren't using PIM to define the router port (rule 7) you can use "ip pim passive". Otherwise you should use "ip pim neighbor-filter".

Rule 9: MPLS TE networks need "mpls traffic-eng multicast-intact"

I have no idea why this isn't default, but you have to wave your magic wand inside your IGP section. "mpls traffic-eng multicast-intact". Otherwise you won't have "interoperability between PIM and MPLS-TE". What that means is that your RPF will be toast and traffic will be dropped.

Don't turn on PIM on your TE tunnel. It will not work. The short story is that a TE tunnel is unidirectional, so you won't be able to set up a PIM adjacency over it. But if you try you will only get your routers hopes up, but hoping for something that will never happen. You can see how hopeless it is with the command "show ip rpf <sender-address>".

If this is your first multicast network, you have to play around

show ip mroute, show ip rpf, debug ip mpacket, no ip mroute-cache. Play with them. Learn them. Learn them well. You can know all the unicast you want, but it won't impress multicast one bit.

Links

• Multicast Quick-Start Configuration Guide
• Multicast Does Not Work in the Same VLAN in Catalyst Switches
• mpls traffic-eng multicast-intact Command
• Enabling PIM on an Interface