pyOpenSSL and invalid certificates

I was trying to import some X509v3 certificates that were created with pyOpenSSL to a MikroTik router (RouterOS 6.1) but they were always being imported with an invalid validity period (not before 1970 and not after 1970).

Eventually I found out that this is because pyOpenSSL stores the validity field in an invalid format. Here's the story:

cert1.pem is the pyOpenSSL certificate and cert2.pem is a certificate created with openssl. Both have mostly the same information. Decoding the certificates with openssl shows that cert1.pem actually has an older notAfter date so it's not an issue of overflow.

[code language="shell" gutter="false"]
$ openssl x509 -noout -startdate -enddate < cert1.pem
notBefore=Jul 20 18:35:58 2013 GMT
notAfter=Jan  1 00:00:00 2032 GMT

$ openssl x509 -noout -startdate -enddate < cert2.pem
notBefore=Aug 31 21:44:54 2012 GMT
notAfter=Aug 26 21:44:54 2032 GMT
[/code]

I examined the certificates in python by decoding their DER structures and looking for the validity field (copy-paste the following in a python shell).

[code language="python"]
import Crypto.Util.asn1 as asn1
import OpenSSL.crypto as c

fn1="cert1.pem"
fn2="cert2.pem"

st1=open(fn1, 'r').read()
st2=open(fn2, 'r').read()

cert1=c.load_certificate(c.FILETYPE_PEM, st1)
cert2=c.load_certificate(c.FILETYPE_PEM, st2)

dump1=c.dump_certificate(c.FILETYPE_ASN1, cert1)
dump2=c.dump_certificate(c.FILETYPE_ASN1, cert2)

der1=asn1.DerSequence()
der2=asn1.DerSequence()

der1.decode(dump1)
der2.decode(dump2)

dcert1=der1[0]
dcert2=der2[0]

t1=asn1.DerSequence()
t2=asn1.DerSequence()

t1.decode(dcert1)
t2.decode(dcert2)

tt1=asn1.DerSequence()
tt2=asn1.DerSequence()

tt1.decode(t1[4])
tt2.decode(t2[4])
[/code]

at this point tt1 and tt2 are sequences of the validity field (notBefore, notafter) for the two certificates . Here's what they contain:

[code language="python" gutter="false"]
>>> tt1[0]
'\x18\x0f20130720183558Z'
>>> tt1[1]
'\x18\x0f20320101000000Z'

>>> tt2[0]
'\x17\r120831214454Z'
>>> tt2[1]
'\x17\r320826214454Z'
[/code]

SoaB! They differ!

Reading the X509 spec [1], section 4.1.2.5 indicates that there are two possible formats for the validity period: both notBefore and notAfter may be encoded as UTCTime or  GeneralizedTime.

• UTCTime is defined as YYMMDDHHMMSSZ


• GeneralizedTime is defined as YYYYMMDDHHMMSSZ

So pyOpenSSL uses GeneralizedTime while openssl uses UTCTime. So both are valid.

However the RFC also says:

CAs conforming to this profile MUST always encode certificate
validity dates through the year 2049 as UTCTime; certificate validity
dates in 2050 or later MUST be encoded as GeneralizedTime.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARG!

So it seems that pyOpenSSL uses GeneralizedTime unconditionally which is not RFC compliant and thus rejected by RouterOS.

A quick look at pyOpenSSL's code unfortunately proves that...

[1]   http://www.ietf.org/rfc/rfc3280.txt

X509v3 Authority Key Identifier pains (authorityKeyIdentifier)

"X509v3 Authority Key Identifier" or "authorityKeyIdentifier" is an X509v3 extension that's added to X509 certificates and identifies the CA that signed the Certificate. I suppose that this speeds up the certificate validation process by eliminating multiple checks.

Short version

Edit openssl.cnf and make sure that authorityKeyIdentifier does not include "issuer"

Long version

There's an issue when using the default OpenSSL configuration or when basing a config on that: the default OpenSSL configuration has the following:

authorityKeyIdentifier=keyid,issuer

In the section that lists options for user certificates (i.e. not the CA section). The above results in new certificates using the extension and include two identifiers for the signing CA:

• The Key ID of the CA's cert (because if "keyid")


• The subject and the serial number of the CA's cert (because of issuer)

For example:

X509v3 Authority Key Identifier: 
    keyid:7E:E5:82:FF:FF:FF:15:96:9B:40:FF:C9:5E:51:FF:69:67:4D:BF:FF
    DirName:/C=UK/O=V13/OU=V13/CN=V13 Certificate Authority
    serial:8E:FF:A2:1B:74:DD:54:FF

And this is where the pain and the suffering happens: If you ever decide that you want to re-create the CA's certificate using the same private key then you won't be able to do so because all certificates that are already signed dictate  the subject and the serial number of the old certificate as the CA certificate identifier. Thus your new CA certificate will not be able to verify the existing certificates.

Thus the only way to replace your certificate would be:

• To start from scratch recreating all certificates, or


• to create another CA certificate with the same subject and serial number (not tested)

Recreating a certificate with the same details (like serial number) will make it impossible to have both certificates available and will most probably cause a mess.

The best approach is to completely remove the "issuer" from authorityKeyIdentifier from the configuration file. Then only the Key ID will be used to identify the CA which should be more than enough.

So use the following and live a happy life:

authorityKeyIdentifier=keyid

IPsec, Racoon, setkey, Linux, Mikrotik, tunnel, transport and everything

It took me more than 6 months in order to sort all issues, so here are the experiences. Most of the trouble was because I didn't knew or I didn't had things clear in my mind.

I wanted to have IPsec communication between a bunch of servers and a home network. I believe that this includes almost all (if not all) the possible scenarios of IPsec so it's more complicated than it sounds. For obvious reasons I'm presenting a simplified version here omitting all duplicates (i.e. multiple hosts with the same characteristics).

The network

We have the following nodes:

• A network behind a DSL line (home network) (normal, home DSL line with non-static IP, with NAT)


• A server (srv1) somewhere on the Internet with a static public IP address without NAT.


• A server (srv2) in Amazon's EC2 which has an allocated public IP address but uses local IP addresses and thus has NAT. Also Amazon doesn't allow ESP and AH protocol to be carried by IP packets inside their network.

We also have the following systems:

• Home network: A bunch of Linux boxes on a private network plus a mikrotik router


• srv1 and srv2: Squeeze Debian Linux

The home network uses IP addresses from the network 10.1.0.0/16. A secondary prefix (10.5.0.0/16) is allocated for IPsec addressing only. All home nodes have addresses from the 10.1.0.0/16. Some nodes (including the servers) have addresses from 10.5.0.0/16.

Apart from the above there's a custom CA setup which publishes certificates for all nodes.

The problem

Setup IPsec so that:

• srv1 and srv2 can communicate with their public IP addresses with IPsec only


• boxes on the home network can communicate both with srv1 and srv2 using IPsec

The setup

Since there are more than one boxes on the home network, the home network needs to be connected with tunneled IPsec to srv1 and srv2. srv1 and srv2 need to be connected with transport mode between them in order to encrypt communication that uses their public IP addresses.

We have setup the DSL router to forward everything to the mikrotik box (routerboard). This is usually referred as DMZ. By doing that it's possible to avoid NAT in IPsec (i.e. UDP encapsulation).

The solution

Mikrotik

In short, Mikrotik's IPsec works quite well and is easy to setup assuming that everything is correct. It is however harder to debug than Racoon. Here's the setup:

• Add an IP address from 10.5.0.0/16


• Import the box's certificate to the certificate storage, both certificate and public key are needed


• Import CA's and other boxes' certificates to the certificate storage. Make sure you use sensible names to be able to look them up later.


• Create a new proposal as follows:
  
  
  
  • Name: short (or pick something else)
  
  
  • Lifetime: 00:10:00 - This is essential in older to allow quick recovery when the IP address changes or racoon is restarted.
  
  
  • Pick your favorite values for everything else
  
  
  
  


• Add two peers, one for each server:
  
  
  
  • srv1 (static public IP, no NAT):
    
    
    
    • Address: The public IP of srv1
    
    
    • Port: 500
    
    
    • Auth method: rsa signature
    
    
    • Certificate: Pick the local certificate (mikrotik's)
    
    
    • Remote certificate: Pick the certificate of srv1
    
    
    • Exchange Mode: main
    
    
    • Select: Send Initial Contact
    
    
    • Nat Traversal: No
    
    
    • My ID User FQDN: Leave empty - isn't needed
    
    
    • Proposal check: Claim (remember not to use similar or stricter on remote end)
    
    
    • Generate policy: No
    
    
    • Lifetime: 08:00:00
    
    
    • DPD Interval/Max failures: I use 10/3 but it doesn't make a difference. See notes bellow
    
    
    
    
  
  
  • srv2 (static IP, public IP, with NAT): Use the same settings as with srv1
    
    
    
    • I didn't use NAT but it may be worth testing it.
    
    
    
    
  
  
  
  


• You need to add two policies per peer. One for each local source IP address range (10.1.0.0/16 and 10.5.0.0/16). So you will end up with 4 policies:
  
  
  
  • Src Address: 10.1.0.0/16 or 10.5.0.0/16
  
  
  • Dst Address: srv1's or srv2's public IP address
  
  
  • Src/Dst Port: Empty
  
  
  • Protocol: all (255)
  
  
  • Action: Encrypt
  
  
  • Level: Unique - very important
  
  
  • IPsec protocols: ESP
  
  
  • Tunnel: Yes
  
  
  • SA Src address: 0.0.0.0
  
  
  • SA Dst address: srv1's or srv2's IPsec IP address (i.e. allocated addresses from the 10.5.0.0/16)
  
  
  • Proposal: short (or whatever name you picked for the proposal you created)
  
  
  
  


• Create a script named "ping-servers" (System -> Scripts) as follows:
  

  {
   :local servers
   :local locals
  
   :set servers {"10.5.1.11";"10.5.1.12"}
   :set locals {"10.1.1.1";"10.5.1.1"}
  
   foreach loc in=$locals do={
   foreach srv in=$servers do={
   put "ping $srv src-address=$loc count=1"
   ping $srv src-address=$loc count=1
   }
   }
  }

  
  servers is the list of server's addresses from the 10.5.0.0/16 network and locals are local addresses to the mikrotik box, one for each of the two networks.


• Schedule the script to be executed every minute (System -> Scheduler). This will keep the policies active and also reactivate them if they go down.

srv1 (static public IP, no NAT)

• Put the following in /etc/ipsec-tools.d/srv2.conf:
  

  spdadd srv1public srv2public[500] udp -P out none;
  spdadd srv2public srv1public[500] udp -P in none;
  spdadd srv1public srv2public[4500] udp -P out none;
  spdadd srv2public srv1public[4500] udp -P in none;
  spdadd srv1public srv2public 50 -P out none;
  spdadd srv2public srv1public 50 -P in none;
  spdadd srv1public srv2public 51 -P out none;
  spdadd srv2public srv1public 51 -P in none;
  
  spdadd srv1public srv2public any -P out ipsec
          esp/transport/srv1public[4500]-srv2public[4500]/require ;
  
  spdadd srv2public srv1public any -P in ipsec
          esp/transport/srv2public[4500]-srv1public[4500]/require ;

  
  


• Put the following in /etc/ipsec-tools.d/srv2-priv.conf. Somehow it is required in order to establish the IPsec connection when it's triggered by srv2:
  

  spdadd srv1public srv2private[500] udp -P out none;
  spdadd srv2private srv1public[500] udp -P in none;
  spdadd srv1public srv2private[4500] udp -P out none;
  spdadd srv2private srv1public[4500] udp -P in none;
  spdadd srv1public srv2private 50 -P out none;
  spdadd srv2private srv1public 50 -P in none;
  spdadd srv1public srv2private 51 -P out none;
  spdadd srv2private srv1public 51 -P in none;
  
  spdadd srv1public srv2private any -P out ipsec
          esp/transport/srv1public[4500]-srv2private[4500]/require ;
  
  spdadd srv2private srv1public any -P in ipsec
          esp/transport/srv2private[4500]-srv1public[4500]/require ;

  
  


• In the above, srv1public is the public static IP address of srv1, srv2public is the public static IP address of srv2 and srv2private is the private static IP address of srv2.


• Setup racoon.conf's section for srv2 and home as follows. Obviously you need to change to match your parameters:
  

  remote "srv2" {
   exchange_mode main,base;
   verify_identifier on;
   peers_identifier asn1dn "Common name of srv2's certificate";
   remote_address srv2public;
   verify_cert on;
   certificate_type x509 "srv1.crt" "srv1.key";
   ca_type x509 "cacert.pem";
   my_identifier asn1dn;
   lifetime time 24 hours;
   nat_traversal on;
   proposal {
    authentication_method rsasig;
    encryption_algorithm 3des;
    hash_algorithm md5;
    dh_group modp1024;
   }
   passive off;
   proposal_check obey;
   generate_policy off;
   dpd_delay 10;
   dpd_retry 10;
   dpd_maxfail 6;
   initial_contact on;
   ike_frag on;
  }

  
  


• Setup racoon.conf's section for the home network as follows:
  

  remote "home" {
   exchange_mode main,base;
   verify_identifier on;
   peers_identifier asn1dn "Common name of mikrotik's certificate ";
   verify_cert on;
   certificate_type x509 "srv1.crt" "srv1.key";
   ca_type x509 "cacert.pem";
   my_identifier asn1dn;
   nat_traversal off;
   proposal {
    authentication_method rsasig;
    encryption_algorithm 3des;
    hash_algorithm md5;
    dh_group modp1024;      # Group 2
   }
   passive on;
   proposal_check obey;
   generate_policy unique;
   dpd_delay 10;
   dpd_retry 10;
   dpd_maxfail 6;
   initial_contact on;
   ike_frag on;
  }

  
  


• Notice the differences: passive should be on  for the home network since it's not possible to trigger that without remote address.


• Notice the generate_policy. It must be "unique" and not "on". Otherwise only one policy per remote endpoint will be generated and will also cause problems when an SA becomes bad.


• Setup the additional address to a loopback interface and not to a physical interface.


• Add static routes for the two networks using the normal gateway and specifying the source IP address. Otherwise you will be using the tunnel with addresses that are not routed via the tunnel and are not protected by IPsec. Obviously this will prevent anything from working on top of IPsec. Surprisingly, this will work occasionally when the traffic is initiated by the remote end just because of the route cache. Your config can be added to the loopback interface as follows:
  

  auto lo:1
  iface lo:1 inet static
    address     10.5.1.12
    netmask     255.255.255.255
    up          ip route add 10.5.0.0/16 via <gw> src 10.5.1.12 || true
    up          ip route add 10.1.0.0/16 via <gw> src 10.5.1.12 || true
    down        ip route del 10.1.0.0/16 via <gw> src 10.5.1.12 || true
    down        ip route del 10.5.0.0/16 via <gw> src 10.5.1.12 || true

  
  where 10.5.1.12 is the address from the 10.5.0.0/16 network for srv1 and gw is the normal gateway of the server.

srv2 (static private IP, static public IP, NAT)

• Setup the /etc/ipsec-tools.d/*.conf files in a similar way to the srv1's. You will need an entry for both the private and the public address.


• Setup racoon like srv1's except from nat. You will have to set nat_traversal to on for srv1 and the home network.

The Hints / Lessons learned

• Either test DPD (Dead Peer Detection) or don't use it at all. It didn't work for me at all.


• You need to activate the policies from the home network's side proactively for both the IPsec networks (10.1.0.0/16 and 10.5.0.0/16). Otherwise it will be impossible for the remote ends to connect to local hosts. This is easily done by setting up a ping to run every minute. You need one ping per source IP address using -I.


• You need to exclude ISAKMP traffic (UDP ports 500 and 4500) from static IPsec policies or otherwise you will have problems since outgoing traffic will be encrypted and incoming traffic will be dropped if not encrypted, which causes huge issues when one end goes down and requires the IPsec SA to expire from both ends (or flushed) before working again.


• If you have firewall rules make sure that you allow ISAKMP traffic and IPsec traffic (protocols 50 (esp) and 51 (ah))


• If you get errors that say that a policy is not available then it is not available! I can't stress this enough. While trying to make IPsec to work your brain will enter a bad state and it will start making mistakes. It's extremely easy to confuse static IPsec rules. I've done all sorts of mistakes including (but not limited to): using the wrong direction (in/out), using the address of another server, using tunnel instead of transport (and vice versa), not including the port numbers for esp-udp (UDP encapsulation) mode, not using the .conf extensions for files under /etc/ipsec-tools.d/, etc. Here's an example of that:
  

  Sep 27 15:02:04 srvX racoon: ERROR: no policy found: A.B.C.D/32[0] E.F.G.H/32[0] proto=any dir=in
  Sep 27 15:02:04 srvX racoon: ERROR: failed to get proposal for responder.
  Sep 27 15:02:04 srvX racoon: [I.J.K.L] ERROR: failed to pre-process ph2 packet (side: 1, status: 1).

  
  


• When testing a connection from host A that has both the 10.1.1.1 and 10.5.1.1 addresses to host B with address 10.5.1.2 then you may not be able to ping from B to one of the A's addresses. That's because only one of the IPsec policies is activated. To activate both of them use -I parameter for ping:
  

  v13@hostA$ ping -I 10.1.1.1 10.5.1.2
  v13@hostA$ ping -I 10.5.1.1 10.5.1.2

  
  


• Pay attention to routing. You need to use the proper source IP addresses.

pyzor problem after debian squeeze upgrade

After upgrading some servers to Debian squeeze, the following log was filling the logs:
[code]
Feb 18 12:49:38 aetos check[982]: pyzor: [19952] error: TERMINATED, signal 15 (000f)
[/code]

The problem was caused by wrong pyzor servers. Unfortunately, pyzor keeps a servers list in each home directory in file ~/.pyzor/servers. This is what this file used to have:
[code]
82.94.255.100:24441
[/code]

This file is created automatically (with a proper value) so it is safe to remove it. That's what it should have (for now):
[code]
public.pyzor.org:24441
[/code]

In order to get rid of the error message all users' files should be deleted:
[code]
find /home -name servers | grep pyzor/servers > /tmp/lst
# examine /tmp/lst by hand to verify that nothing bad is there
cd /home
cat /tmp/lst | xargs rm
[/code]

That's it. There should be no more "TERMINATED" messages.

And this month's medal of stup^H^H^H^Hcleverness goes to...

Internet Explorer for this.

Really, if MS was not MS but a company that was payed to write applications, making this information public should be enough reason not to hire them.

Let me rephrase the explanation: "If you give me a letter and say that this letter is only handed once, I will drop it because... well... because... no reason..."

Following the KB's logic, if a document expires after 1 second and an application takes more than a second to launch then it should not be able to open the file...

... and it gets better: If the document expires after 10 seconds, you should have 10 seconds to read it. Else the document will be closed...

... and finally: if you save this document to a file, the file should be automatically deleted after it has expired.

Those are not true, but according to MS's logic, they could be. Perhaps in IE10 :-)

VMWare Server 2 RC2 fails to boot VMs

For a couple of days now, VMWare Server 2 RC2 was refusing to start virtual machines. It kept waiting at 95%.

It turns out that VMWare cannot boot virtual machines (or upgrade them) when kernel modules kvm_intel, kvm etc are loaded.

Here is the error message from hostd.log:

Question info: The virtualization capability of your processor is already in use. Disable any other running hypervisors before running VMware Server.

Of course the obvious solution is to rmmod the kvm related modules:

# lsmod |grep kvm
# rmmod kvm_intel
# rmmod kvm_amd
# rmmod kvm

The modules are most probably auto-loaded by /etc/init.d/kvm (but there may be other reasons too).

Trouble uploading files to wordpress

I had problems uploading files to wordpress hosted at the local machine. Conditions are:

• Firefox 3.0


• Flash 9.0


• Local squid proxy 2.7.STABLE3


• Firefox setup for using local proxy except from accessing local machine. This makes no difference at all.


• Environment variable http_proxy set to http://127.0.0.1:8080/

The result of trying to upload images to wordpress was "HTTP Error". A sample wireshark capture of the failed request looks like this:

Request:

POST http://localhost/wordpress/wp-admin/async-upload.php HTTP/1.1 
Host: localhost 
Pragma: no-cache 
Accept: */* 
Proxy-Connection: Keep-Alive 
User-Agent: Shockwave Flash
Connection: Keep-Alive
Cache-Control: no-cache 
Content-Length: 40861 
Expect: 100-continue 
Content-Type: multipart/form-data; boundary=----------------------------5ce16f6dfa7a

Response:

HTTP/1.0 417 Expectation failed 
Server: squid/2.7.STABLE3 
Date: Sun, 27 Jul 2008 09:31:50 GMT 
Content-Type: text/html 
Content-Length: 1507 
Expires: Sun, 27 Jul 2008 09:31:50 GMT 
X-Squid-Error: ERR_INVALID_REQ 0 
X-Cache: MISS from XXXXX.XXXX.XXXX 
X-Cache-Lookup: NONE from XXXXX.XXXX.XXXX:3128 
Via: 1.0 XXXXX.XXXX.XXXX:3128 (squid/2.7.STABLE3) 
Connection: close 

[...]
A page saying:
ERROR: The requested URL could not be retrieved
[...]

It is obvious that flash plugin is used to perform the upload and it is using the proxy. It does a request with an "Expect:" header and for an (unknown to me) reason the expectation fails because of the squid proxy.

Disabling proxy in firefox, restarting it and bypassing cache doesn't help. The problem lies in that the flash plugin uses the http_proxy environment variable to determine the proxy.

Solution:
Unsetting http_proxy in a terminal and starting firefox from there solves the problem.