Verify that a private key matches a certificate using PyOpenSSL and PyCrypto:
[code language="python"]
import OpenSSL.crypto
from Crypto.Util import asn1
c=OpenSSL.crypto
# The certificate - an X509 object
cert=...
# The private key - a PKey object
priv=...
pub=cert.get_pubkey()
# Only works for RSA (I think)
if pub.type()!=c.TYPE_RSA or priv.type()!=c.TYPE_RSA:
raise Exception('Can only handle RSA keys')
# This seems to work with public as well
pub_asn1=c.dump_privatekey(c.FILETYPE_ASN1, pub)
priv_asn1=c.dump_privatekey(c.FILETYPE_ASN1, priv)
# Decode DER
pub_der=asn1.DerSequence()
pub_der.decode(pub_asn1)
priv_der=asn1.DerSequence()
priv_der.decode(priv_asn1)
# Get the modulus
pub_modulus=pub_der[1]
priv_modulus=priv_der[1]
if pub_modulus==priv_modulus:
print('Match')
else:
print('Oops')
[/code]
The idea is to get the modulus from the two DER structures and compare them. They should be the same.
Note: You can use the above under the MIT license. If it doesn’t fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.
Friday, 12 April 2013
Thursday, 11 April 2013
Verifying an SSL certificate with python
This one took me a considerable amount of time and had to figure some parts from scratch.
Unfortunately there doesn't seem to exist an easy (out-of-the-box) way for checking whether a certificate is signed by another certificate in python.
After days of searching and despair, here is a solution without using M2Crypto:
[code language="python"]
import OpenSSL
from Crypto.Util import asn1
c=OpenSSL.crypto
# This is the certificate to validate
# an OpenSSL.crypto.X509 object
cert=...
# This is the CA certificate to use for validation
# again an OpenSSL.crypto.X509 object
cacert=...
# Get the signing algorithm
algo=cert.get_signature_algorithm()
# Get the ASN1 format of the certificate
cert_asn1=c.dump_certificate(c.FILETYPE_ASN1, cert)
# Decode the certificate
der=asn1.DerSequence()
der.decode(cert_asn1)
# The certificate has three parts:
# - certificate
# - signature algorithm
# - signature
# http://usefulfor.com/nothing/2009/06/10/x509-certificate-basics/
der_cert=der[0]
der_algo=der[1]
der_sig=der[2]
# The signature is a BIT STRING (Type 3)
# Decode that as well
der_sig_in=asn1.DerObject()
der_sig_in.decode(der_sig)
# Get the payload
sig0=der_sig_in.payload
# Do the following to see a validation error for tests
# der_cert=der_cert[:20]+'1'+der_cert[21:]
# First byte is the number of unused bits. This should be 0
# http://msdn.microsoft.com/en-us/library/windows/desktop/bb540792(v=vs.85).aspx
if sig0[0]!='\x00':
raise Exception('Number of unused bits is strange')
# Now get the signature itself
sig=sig0[1:]
# And verify the certificate
try:
c.verify(cacert, sig, der_cert, algo)
print "Certificate looks good"
except OpenSSL.crypto.Error, e:
print "Sorry. Nope."
[/code]
Note: You can use the above under the MIT license. If it doesn’t fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.
Unfortunately there doesn't seem to exist an easy (out-of-the-box) way for checking whether a certificate is signed by another certificate in python.
After days of searching and despair, here is a solution without using M2Crypto:
[code language="python"]
import OpenSSL
from Crypto.Util import asn1
c=OpenSSL.crypto
# This is the certificate to validate
# an OpenSSL.crypto.X509 object
cert=...
# This is the CA certificate to use for validation
# again an OpenSSL.crypto.X509 object
cacert=...
# Get the signing algorithm
algo=cert.get_signature_algorithm()
# Get the ASN1 format of the certificate
cert_asn1=c.dump_certificate(c.FILETYPE_ASN1, cert)
# Decode the certificate
der=asn1.DerSequence()
der.decode(cert_asn1)
# The certificate has three parts:
# - certificate
# - signature algorithm
# - signature
# http://usefulfor.com/nothing/2009/06/10/x509-certificate-basics/
der_cert=der[0]
der_algo=der[1]
der_sig=der[2]
# The signature is a BIT STRING (Type 3)
# Decode that as well
der_sig_in=asn1.DerObject()
der_sig_in.decode(der_sig)
# Get the payload
sig0=der_sig_in.payload
# Do the following to see a validation error for tests
# der_cert=der_cert[:20]+'1'+der_cert[21:]
# First byte is the number of unused bits. This should be 0
# http://msdn.microsoft.com/en-us/library/windows/desktop/bb540792(v=vs.85).aspx
if sig0[0]!='\x00':
raise Exception('Number of unused bits is strange')
# Now get the signature itself
sig=sig0[1:]
# And verify the certificate
try:
c.verify(cacert, sig, der_cert, algo)
print "Certificate looks good"
except OpenSSL.crypto.Error, e:
print "Sorry. Nope."
[/code]
Note: You can use the above under the MIT license. If it doesn’t fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.
X509v3 Authority Key Identifier pains (authorityKeyIdentifier)
"X509v3 Authority Key Identifier" or "authorityKeyIdentifier" is an X509v3 extension that's added to X509 certificates and identifies the CA that signed the Certificate. I suppose that this speeds up the certificate validation process by eliminating multiple checks.
Edit openssl.cnf and make sure that authorityKeyIdentifier does not include "issuer"
There's an issue when using the default OpenSSL configuration or when basing a config on that: the default OpenSSL configuration has the following:
In the section that lists options for user certificates (i.e. not the CA section). The above results in new certificates using the extension and include two identifiers for the signing CA:
For example:
And this is where the pain and the suffering happens: If you ever decide that you want to re-create the CA's certificate using the same private key then you won't be able to do so because all certificates that are already signed dictate the subject and the serial number of the old certificate as the CA certificate identifier. Thus your new CA certificate will not be able to verify the existing certificates.
Thus the only way to replace your certificate would be:
Recreating a certificate with the same details (like serial number) will make it impossible to have both certificates available and will most probably cause a mess.
The best approach is to completely remove the "issuer" from authorityKeyIdentifier from the configuration file. Then only the Key ID will be used to identify the CA which should be more than enough.
So use the following and live a happy life:
Short version
Edit openssl.cnf and make sure that authorityKeyIdentifier does not include "issuer"
Long version
There's an issue when using the default OpenSSL configuration or when basing a config on that: the default OpenSSL configuration has the following:
authorityKeyIdentifier=keyid,issuer
In the section that lists options for user certificates (i.e. not the CA section). The above results in new certificates using the extension and include two identifiers for the signing CA:
- The Key ID of the CA's cert (because if "keyid")
- The subject and the serial number of the CA's cert (because of issuer)
For example:
X509v3 Authority Key Identifier:
keyid:7E:E5:82:FF:FF:FF:15:96:9B:40:FF:C9:5E:51:FF:69:67:4D:BF:FF
DirName:/C=UK/O=V13/OU=V13/CN=V13 Certificate Authority
serial:8E:FF:A2:1B:74:DD:54:FF
And this is where the pain and the suffering happens: If you ever decide that you want to re-create the CA's certificate using the same private key then you won't be able to do so because all certificates that are already signed dictate the subject and the serial number of the old certificate as the CA certificate identifier. Thus your new CA certificate will not be able to verify the existing certificates.
Thus the only way to replace your certificate would be:
- To start from scratch recreating all certificates, or
- to create another CA certificate with the same subject and serial number (not tested)
Recreating a certificate with the same details (like serial number) will make it impossible to have both certificates available and will most probably cause a mess.
The best approach is to completely remove the "issuer" from authorityKeyIdentifier from the configuration file. Then only the Key ID will be used to identify the CA which should be more than enough.
So use the following and live a happy life:
authorityKeyIdentifier=keyid
Saturday, 23 March 2013
Raspberry Pi under QEMU
What: Run raspberry pi system under QEMU
Why: I wanted to have builder environment for Raspberry PI. Emulating it it much faster that running something on it
Disclaimer: I am not a qemu/raspberry-pi expert - Some things may be wrong here
Running qemu-arm is slightly different than running qemu for x86. There is no BIOS there and it needs to boot with a kernel image directly. The tricky part is that this image is not the standard Raspberry Pi (from now on mentioned as "raspi"). You can either download one from online or create your own.
I ended up creating my own qemu arm image
Get the official raspberry pi disk image from here. Place it somewhere and uncompress it.
By default the image does not have much free space. A trick to change that is to extend it with something like the following:
This will write 1MB after the first 4000 MBytes on the image. This will cause the image to become approximately 4GB but it will leave a hole in the file if the underlying filesystem supports it.
If you want to compile your own kernel image for qemu then follow first the instructions here. Follow just the instructions on cloning git and patching the kernel.
Instead of configuring it yourself get this configuration (will speed your life) which has some additional stuff likebuilt-in sound, vfat support, IPv6, nice fonts, etc. Put this under the cloned kernel directory with name .config and then run:
You will need the arm compilation toolchain. If you're running Debian then follow the instructions in http://www.emdebian.org/
The package you are looking for is gcc-4.4-arm-linux-gnueabi
Note: Raspberry Pi supports hardfloat while the above compile is not using hardfloat. There should be no problems with that since the kernel is not using floating point math itself and it won't make a difference. The attached kernel configuration however has hardfloat support and so does qemu.
I am using vde networking with qemu so your setup may be slightly different. In any case, this is a nice setup to get you started:
Use the proper path for the disk image and the kernel.
Now, here's the catch: The latest (as of 2013-03-23) raspberry pi image will have some issues booting with the above setup.
Change the qemu -append parameter to: "root=/dev/sda2 panic=0 ro single" and boot the image.
This should leave you with a nice prompt. Then:
Next, edit fstab and change mmcblk0p1 and mmcblk0p2 to sda1 and sda2 respectively
Then create the file /etc/udev/rules.d/90-qemu.rules with the following contents:
This should ensure that /dev/mmcblk0p1/2 exist and will make other stuff work.
Also edit /etc/ld.so.preload (if it exists) and remove or comment everything in it.
Now shutdown the qemu image, restore the -append parameter and run it again. If raspi-config does not start by itself, run it yourself after logging in as user "pi". You can now select "expand_rootfs" and do another reboot to get more space on the root partition.
Why: I wanted to have builder environment for Raspberry PI. Emulating it it much faster that running something on it
Disclaimer: I am not a qemu/raspberry-pi expert - Some things may be wrong here
Introduction
Running qemu-arm is slightly different than running qemu for x86. There is no BIOS there and it needs to boot with a kernel image directly. The tricky part is that this image is not the standard Raspberry Pi (from now on mentioned as "raspi"). You can either download one from online or create your own.
I ended up creating my own qemu arm image
Disk image
Get the official raspberry pi disk image from here. Place it somewhere and uncompress it.
By default the image does not have much free space. A trick to change that is to extend it with something like the following:
# dd if=/dev/zero of=2013-02-09-wheezy-raspbian.img \
bs=1024000 conv=notrunc \
seek=4000 count=1
This will write 1MB after the first 4000 MBytes on the image. This will cause the image to become approximately 4GB but it will leave a hole in the file if the underlying filesystem supports it.
Kernel Image (optional)
If you want to compile your own kernel image for qemu then follow first the instructions here. Follow just the instructions on cloning git and patching the kernel.
Instead of configuring it yourself get this configuration (will speed your life) which has some additional stuff like
# make ARCH=arm oldconfig
# nice make ARCH=arm -j 4 # Adjust for your number of cores
# cp arch/arm/boot/zImage /path/to/raspi/directory
You will need the arm compilation toolchain. If you're running Debian then follow the instructions in http://www.emdebian.org/
The package you are looking for is gcc-4.4-arm-linux-gnueabi
Note: Raspberry Pi supports hardfloat while the above compile is not using hardfloat. There should be no problems with that since the kernel is not using floating point math itself and it won't make a difference. The attached kernel configuration however has hardfloat support and so does qemu.
Booting
I am using vde networking with qemu so your setup may be slightly different. In any case, this is a nice setup to get you started:
# qemu-system-arm \
-cpu arm1176 \
-kernel kernel-v13-qemu-130323 \
-name 'builder-raspi' \
-hda 2013-02-09-wheezy-raspbian.img \
-m 256 \
-M versatilepb \
-no-reboot \
-serial stdio \
-net vde,sock=/var/run/vde2/tap0.ctl \
-net nic \
-append "root=/dev/sda2 panic=0 ro
Use the proper path for the disk image and the kernel.
Now, here's the catch: The latest (as of 2013-03-23) raspberry pi image will have some issues booting with the above setup.
Fix latest filesystem image
Change the qemu -append parameter to: "root=/dev/sda2 panic=0 ro single" and boot the image.
This should leave you with a nice prompt. Then:
# mount / -o remount,rw
Next, edit fstab and change mmcblk0p1 and mmcblk0p2 to sda1 and sda2 respectively
Then create the file /etc/udev/rules.d/90-qemu.rules with the following contents:
KERNEL=="sda", SYMLINK+="mmcblk0"
KERNEL=="sda?", SYMLINK+="mmcblk0p%n",
This should ensure that /dev/mmcblk0p1/2 exist and will make other stuff work.
Also edit /etc/ld.so.preload (if it exists) and remove or comment everything in it.
Now shutdown the qemu image, restore the -append parameter and run it again. If raspi-config does not start by itself, run it yourself after logging in as user "pi". You can now select "expand_rootfs" and do another reboot to get more space on the root partition.
Sunday, 30 September 2012
IPsec, Racoon, setkey, Linux, Mikrotik, tunnel, transport and everything
It took me more than 6 months in order to sort all issues, so here are the experiences. Most of the trouble was because I didn't knew or I didn't had things clear in my mind.
I wanted to have IPsec communication between a bunch of servers and a home network. I believe that this includes almost all (if not all) the possible scenarios of IPsec so it's more complicated than it sounds. For obvious reasons I'm presenting a simplified version here omitting all duplicates (i.e. multiple hosts with the same characteristics).
We have the following nodes:
We also have the following systems:
The home network uses IP addresses from the network 10.1.0.0/16. A secondary prefix (10.5.0.0/16) is allocated for IPsec addressing only. All home nodes have addresses from the 10.1.0.0/16. Some nodes (including the servers) have addresses from 10.5.0.0/16.
Apart from the above there's a custom CA setup which publishes certificates for all nodes.
Setup IPsec so that:
Since there are more than one boxes on the home network, the home network needs to be connected with tunneled IPsec to srv1 and srv2. srv1 and srv2 need to be connected with transport mode between them in order to encrypt communication that uses their public IP addresses.
We have setup the DSL router to forward everything to the mikrotik box (routerboard). This is usually referred as DMZ. By doing that it's possible to avoid NAT in IPsec (i.e. UDP encapsulation).
In short, Mikrotik's IPsec works quite well and is easy to setup assuming that everything is correct. It is however harder to debug than Racoon. Here's the setup:
I wanted to have IPsec communication between a bunch of servers and a home network. I believe that this includes almost all (if not all) the possible scenarios of IPsec so it's more complicated than it sounds. For obvious reasons I'm presenting a simplified version here omitting all duplicates (i.e. multiple hosts with the same characteristics).
The network
We have the following nodes:
- A network behind a DSL line (home network) (normal, home DSL line with non-static IP, with NAT)
- A server (srv1) somewhere on the Internet with a static public IP address without NAT.
- A server (srv2) in Amazon's EC2 which has an allocated public IP address but uses local IP addresses and thus has NAT. Also Amazon doesn't allow ESP and AH protocol to be carried by IP packets inside their network.
We also have the following systems:
- Home network: A bunch of Linux boxes on a private network plus a mikrotik router
- srv1 and srv2: Squeeze Debian Linux
The home network uses IP addresses from the network 10.1.0.0/16. A secondary prefix (10.5.0.0/16) is allocated for IPsec addressing only. All home nodes have addresses from the 10.1.0.0/16. Some nodes (including the servers) have addresses from 10.5.0.0/16.
Apart from the above there's a custom CA setup which publishes certificates for all nodes.
The problem
Setup IPsec so that:
- srv1 and srv2 can communicate with their public IP addresses with IPsec only
- boxes on the home network can communicate both with srv1 and srv2 using IPsec
The setup
Since there are more than one boxes on the home network, the home network needs to be connected with tunneled IPsec to srv1 and srv2. srv1 and srv2 need to be connected with transport mode between them in order to encrypt communication that uses their public IP addresses.
We have setup the DSL router to forward everything to the mikrotik box (routerboard). This is usually referred as DMZ. By doing that it's possible to avoid NAT in IPsec (i.e. UDP encapsulation).
The solution
Mikrotik
In short, Mikrotik's IPsec works quite well and is easy to setup assuming that everything is correct. It is however harder to debug than Racoon. Here's the setup:
- Add an IP address from 10.5.0.0/16
- Import the box's certificate to the certificate storage, both certificate and public key are needed
- Import CA's and other boxes' certificates to the certificate storage. Make sure you use sensible names to be able to look them up later.
- Create a new proposal as follows:
- Name: short (or pick something else)
- Lifetime: 00:10:00 - This is essential in older to allow quick recovery when the IP address changes or racoon is restarted.
- Pick your favorite values for everything else
- Add two peers, one for each server:
- srv1 (static public IP, no NAT):
- Address: The public IP of srv1
- Port: 500
- Auth method: rsa signature
- Certificate: Pick the local certificate (mikrotik's)
- Remote certificate: Pick the certificate of srv1
- Exchange Mode: main
- Select: Send Initial Contact
- Nat Traversal: No
- My ID User FQDN: Leave empty - isn't needed
- Proposal check: Claim (remember not to use similar or stricter on remote end)
- Generate policy: No
- Lifetime: 08:00:00
- DPD Interval/Max failures: I use 10/3 but it doesn't make a difference. See notes bellow
- srv2 (static IP, public IP, with NAT): Use the same settings as with srv1
- I didn't use NAT but it may be worth testing it.
- srv1 (static public IP, no NAT):
- You need to add two policies per peer. One for each local source IP address range (10.1.0.0/16 and 10.5.0.0/16). So you will end up with 4 policies:
- Src Address: 10.1.0.0/16 or 10.5.0.0/16
- Dst Address: srv1's or srv2's public IP address
- Src/Dst Port: Empty
- Protocol: all (255)
- Action: Encrypt
- Level: Unique - very important
- IPsec protocols: ESP
- Tunnel: Yes
- SA Src address: 0.0.0.0
- SA Dst address: srv1's or srv2's IPsec IP address (i.e. allocated addresses from the 10.5.0.0/16)
- Proposal: short (or whatever name you picked for the proposal you created)
- Create a script named "ping-servers" (System -> Scripts) as follows:
{
:local servers
:local locals
:set servers {"10.5.1.11";"10.5.1.12"}
:set locals {"10.1.1.1";"10.5.1.1"}
foreach loc in=$locals do={
foreach srv in=$servers do={
put "ping $srv src-address=$loc count=1"
ping $srv src-address=$loc count=1
}
}
}
servers is the list of server's addresses from the 10.5.0.0/16 network and locals are local addresses to the mikrotik box, one for each of the two networks. - Schedule the script to be executed every minute (System -> Scheduler). This will keep the policies active and also reactivate them if they go down.
srv1 (static public IP, no NAT)
- Put the following in /etc/ipsec-tools.d/srv2.conf:
spdadd srv1public srv2public[500] udp -P out none;
spdadd srv2public srv1public[500] udp -P in none;
spdadd srv1public srv2public[4500] udp -P out none;
spdadd srv2public srv1public[4500] udp -P in none;
spdadd srv1public srv2public 50 -P out none;
spdadd srv2public srv1public 50 -P in none;
spdadd srv1public srv2public 51 -P out none;
spdadd srv2public srv1public 51 -P in none;
spdadd srv1public srv2public any -P out ipsec
esp/transport/srv1public[4500]-srv2public[4500]/require ;
spdadd srv2public srv1public any -P in ipsec
esp/transport/srv2public[4500]-srv1public[4500]/require ; - Put the following in /etc/ipsec-tools.d/srv2-priv.conf. Somehow it is required in order to establish the IPsec connection when it's triggered by srv2:
spdadd srv1public srv2private[500] udp -P out none;
spdadd srv2private srv1public[500] udp -P in none;
spdadd srv1public srv2private[4500] udp -P out none;
spdadd srv2private srv1public[4500] udp -P in none;
spdadd srv1public srv2private 50 -P out none;
spdadd srv2private srv1public 50 -P in none;
spdadd srv1public srv2private 51 -P out none;
spdadd srv2private srv1public 51 -P in none;
spdadd srv1public srv2private any -P out ipsec
esp/transport/srv1public[4500]-srv2private[4500]/require ;
spdadd srv2private srv1public any -P in ipsec
esp/transport/srv2private[4500]-srv1public[4500]/require ; - In the above, srv1public is the public static IP address of srv1, srv2public is the public static IP address of srv2 and srv2private is the private static IP address of srv2.
- Setup racoon.conf's section for srv2 and home as follows. Obviously you need to change to match your parameters:
remote "srv2" {
exchange_mode main,base;
verify_identifier on;
peers_identifier asn1dn "Common name of srv2's certificate";
remote_address srv2public;
verify_cert on;
certificate_type x509 "srv1.crt" "srv1.key";
ca_type x509 "cacert.pem";
my_identifier asn1dn;
lifetime time 24 hours;
nat_traversal on;
proposal {
authentication_method rsasig;
encryption_algorithm 3des;
hash_algorithm md5;
dh_group modp1024;
}
passive off;
proposal_check obey;
generate_policy off;
dpd_delay 10;
dpd_retry 10;
dpd_maxfail 6;
initial_contact on;
ike_frag on;
} - Setup racoon.conf's section for the home network as follows:
remote "home" {
exchange_mode main,base;
verify_identifier on;
peers_identifier asn1dn "Common name of mikrotik's certificate ";
verify_cert on;
certificate_type x509 "srv1.crt" "srv1.key";
ca_type x509 "cacert.pem";
my_identifier asn1dn;
nat_traversal off;
proposal {
authentication_method rsasig;
encryption_algorithm 3des;
hash_algorithm md5;
dh_group modp1024;Â Â Â Â Â # Group 2
}
passive on;
proposal_check obey;
generate_policy unique;
dpd_delay 10;
dpd_retry 10;
dpd_maxfail 6;
initial_contact on;
ike_frag on;
} - Notice the differences: passive should be on for the home network since it's not possible to trigger that without remote address.
- Notice the generate_policy. It must be "unique" and not "on". Otherwise only one policy per remote endpoint will be generated and will also cause problems when an SA becomes bad.
- Setup the additional address to a loopback interface and not to a physical interface.
- Add static routes for the two networks using the normal gateway and specifying the source IP address. Otherwise you will be using the tunnel with addresses that are not routed via the tunnel and are not protected by IPsec. Obviously this will prevent anything from working on top of IPsec. Surprisingly, this will work occasionally when the traffic is initiated by the remote end just because of the route cache. Your config can be added to the loopback interface as follows:
auto lo:1
iface lo:1 inet static
address   10.5.1.12
netmask   255.255.255.255
up ip route add 10.5.0.0/16 via <gw> src 10.5.1.12 || true
up ip route add 10.1.0.0/16 via <gw> src 10.5.1.12 || true
down ip route del 10.1.0.0/16 via <gw> src 10.5.1.12 || true
down ip route del 10.5.0.0/16 via <gw> src 10.5.1.12 || true
where 10.5.1.12 is the address from the 10.5.0.0/16 network for srv1 and gw is the normal gateway of the server.
srv2 (static private IP, static public IP, NAT)
- Setup the /etc/ipsec-tools.d/*.conf files in a similar way to the srv1's. You will need an entry for both the private and the public address.
- Setup racoon like srv1's except from nat. You will have to set nat_traversal to on for srv1 and the home network.
The Hints / Lessons learned
- Either test DPD (Dead Peer Detection) or don't use it at all. It didn't work for me at all.
- You need to activate the policies from the home network's side proactively for both the IPsec networks (10.1.0.0/16 and 10.5.0.0/16). Otherwise it will be impossible for the remote ends to connect to local hosts. This is easily done by setting up a ping to run every minute. You need one ping per source IP address using -I.
- You need to exclude ISAKMP traffic (UDP ports 500 and 4500) from static IPsec policies or otherwise you will have problems since outgoing traffic will be encrypted and incoming traffic will be dropped if not encrypted, which causes huge issues when one end goes down and requires the IPsec SA to expire from both ends (or flushed) before working again.
- If you have firewall rules make sure that you allow ISAKMP traffic and IPsec traffic (protocols 50 (esp) and 51 (ah))
- If you get errors that say that a policy is not available then it is not available! I can't stress this enough. While trying to make IPsec to work your brain will enter a bad state and it will start making mistakes. It's extremely easy to confuse static IPsec rules. I've done all sorts of mistakes including (but not limited to): using the wrong direction (in/out), using the address of another server, using tunnel instead of transport (and vice versa), not including the port numbers for esp-udp (UDP encapsulation) mode, not using the .conf extensions for files under /etc/ipsec-tools.d/, etc. Here's an example of that:
Sep 27 15:02:04 srvX racoon: ERROR: no policy found: A.B.C.D/32[0] E.F.G.H/32[0] proto=any dir=in
Sep 27 15:02:04 srvX racoon: ERROR: failed to get proposal for responder.
Sep 27 15:02:04 srvX racoon: [I.J.K.L] ERROR: failed to pre-process ph2 packet (side: 1, status: 1). - When testing a connection from host A that has both the 10.1.1.1 and 10.5.1.1 addresses to host B with address 10.5.1.2 then you may not be able to ping from B to one of the A's addresses. That's because only one of the IPsec policies is activated. To activate both of them use -I parameter for ping:
v13@hostA$ ping -I 10.1.1.1 10.5.1.2
v13@hostA$ ping -I 10.5.1.1 10.5.1.2 - Pay attention to routing. You need to use the proper source IP addresses.
Friday, 17 August 2012
DNSSEC key tag (keyid) and DS signature calculation in python
This one took me a considerable amount of hours to figure out so here it is.
While trying to automate DNS zone generation I had to calculate some of the values programmatically. Two of the auto-generated values had to do with DNSSEC entries: The key tag (or keyid) and the DS record's signatures.
The required details on how these are calculated are found in the following places:
For the calculations you need to provide the following:
I used python for this but the approach is the same for other languages since the algorithms are the same.
So here it is:
[code lang="python"]
import struct
import hashlib
import base64
def calc_keyid(flags, protocol, algorithm, st):
"""
@param owner The corresponding domain
@param flags The flags of the entry (256 or 257)
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return The key tag
"""
# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)
# Calculate the tag
cnt=0
for idx in xrange(len(st2)):
s=struct.unpack('B', st2[idx])[0]
if (idx % 2) == 0:
cnt+=s<<8
else:
cnt+=s
ret=((cnt & 0xFFFF) + (cnt>>16)) & 0xFFFF
return(ret)
def calc_ds(owner, flags, protocol, algorithm, st):
"""
@param flags Usually it is 257 or something that indicates a KSK.
It can be 256 though.
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return A dictionary of hashes where the key is the hashing algorithm.
"""
# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)
# Ensure a trailing dot
if owner[-1]=='.':
owner2=owner
else:
owner2=owner+'.'
# Create the name wire format
owner3=''
for i in owner2.split('.'):
owner3+=struct.pack('B', len(i))+i
# Calculate the hashes
st3=owner3+st2
ret={
'sha1': hashlib.sha1(st3).hexdigest().upper(),
'sha256': hashlib.sha256(st3).hexdigest().upper(),
}
return(ret)
[/code]
The following were created by bind's dnssec tools:
[code light="true"]
$ cat Ktest.hell.gr.+005+33630.key
; This is a zone-signing key, keyid 33630, for test.hell.gr.
; Created: 20101007114826 (Thu Oct 7 14:48:26 2010)
; Publish: 20101007114826 (Thu Oct 7 14:48:26 2010)
; Activate: 20101007114826 (Thu Oct 7 14:48:26 2010)
test.hell.gr. IN DNSKEY 256 3 5 AwEAAb+lTDjZCfq7D5N9cNd1ug30wLrbCXB9mVJJQGlQQHpiHHlMaLGG
sV2/j5+eojHp+WQUzNpOzrULF6msbEvUuV2gSEnpbueRV4twO8muGE+x
eUuseSoHh/aTpA8Z9SPubb01mduqqaUEN5Juz2Q4hF0dSUSJYlJPKhp6
NrOgoeyj
$ cat dsset-test.hell.gr.
test.hell.gr.      IN DS 33630 5 1 A2AD2648B353365631EBC9C70EDA1E0C04563FCC
test.hell.gr.      IN DS 33630 5 2 4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2 439BB5C1
[/code]
[code lang="python" light="true"]
>>> domain='test.hell.gr'
>>> flags=256
>>> protocol=3
>>> algorithm=5
>>> key='AwEAAb+...goeyj' # Truncated
>>> calc_keyid(flags, protocol, algorithm, key)
33630
>>> r=calc_ds(domain, flags, protocol, algorithm, key)
>>> r['sha1']
'A2AD2648B353365631EBC9C70EDA1E0C04563FCC'
>>> r['sha256']
'4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2439BB5C1'
[/code]
You can use the above under the MIT license. If it doesn't fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.
While trying to automate DNS zone generation I had to calculate some of the values programmatically. Two of the auto-generated values had to do with DNSSEC entries: The key tag (or keyid) and the DS record's signatures.
The required details on how these are calculated are found in the following places:
- Key tag: RFC4034 - Appendix B
- DNSKEY RDATA wire format: RFC4034 - Section 2.1
- DS' digest field: RFC4034 - Section 5.1.4
- Wire format of names: RFC1035 - Section 3.1
For the calculations you need to provide the following:
- For the key tag: flags, protocol, algorithm, public key
- For the DS signatures: owner (the domain name), flags, protocol, algorithm, public key
Code
I used python for this but the approach is the same for other languages since the algorithms are the same.
So here it is:
[code lang="python"]
import struct
import hashlib
import base64
def calc_keyid(flags, protocol, algorithm, st):
"""
@param owner The corresponding domain
@param flags The flags of the entry (256 or 257)
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return The key tag
"""
# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)
# Calculate the tag
cnt=0
for idx in xrange(len(st2)):
s=struct.unpack('B', st2[idx])[0]
if (idx % 2) == 0:
cnt+=s<<8
else:
cnt+=s
ret=((cnt & 0xFFFF) + (cnt>>16)) & 0xFFFF
return(ret)
def calc_ds(owner, flags, protocol, algorithm, st):
"""
@param flags Usually it is 257 or something that indicates a KSK.
It can be 256 though.
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return A dictionary of hashes where the key is the hashing algorithm.
"""
# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)
# Ensure a trailing dot
if owner[-1]=='.':
owner2=owner
else:
owner2=owner+'.'
# Create the name wire format
owner3=''
for i in owner2.split('.'):
owner3+=struct.pack('B', len(i))+i
# Calculate the hashes
st3=owner3+st2
ret={
'sha1': hashlib.sha1(st3).hexdigest().upper(),
'sha256': hashlib.sha256(st3).hexdigest().upper(),
}
return(ret)
[/code]
Data
The following were created by bind's dnssec tools:
[code light="true"]
$ cat Ktest.hell.gr.+005+33630.key
; This is a zone-signing key, keyid 33630, for test.hell.gr.
; Created: 20101007114826 (Thu Oct 7 14:48:26 2010)
; Publish: 20101007114826 (Thu Oct 7 14:48:26 2010)
; Activate: 20101007114826 (Thu Oct 7 14:48:26 2010)
test.hell.gr. IN DNSKEY 256 3 5 AwEAAb+lTDjZCfq7D5N9cNd1ug30wLrbCXB9mVJJQGlQQHpiHHlMaLGG
sV2/j5+eojHp+WQUzNpOzrULF6msbEvUuV2gSEnpbueRV4twO8muGE+x
eUuseSoHh/aTpA8Z9SPubb01mduqqaUEN5Juz2Q4hF0dSUSJYlJPKhp6
NrOgoeyj
$ cat dsset-test.hell.gr.
test.hell.gr.      IN DS 33630 5 1 A2AD2648B353365631EBC9C70EDA1E0C04563FCC
test.hell.gr.      IN DS 33630 5 2 4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2 439BB5C1
[/code]
Result
[code lang="python" light="true"]
>>> domain='test.hell.gr'
>>> flags=256
>>> protocol=3
>>> algorithm=5
>>> key='AwEAAb+...goeyj' # Truncated
>>> calc_keyid(flags, protocol, algorithm, key)
33630
>>> r=calc_ds(domain, flags, protocol, algorithm, key)
>>> r['sha1']
'A2AD2648B353365631EBC9C70EDA1E0C04563FCC'
>>> r['sha256']
'4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2439BB5C1'
[/code]
Legal
You can use the above under the MIT license. If it doesn't fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.
Tuesday, 24 July 2012
rsync as root with rrsync and sudo
Here's how to rsync something to a remote host as root without allowing root logins and with directory restriction. I did that because I wanted to sync /srv across servers.
In general it will use rsync over ssh, sudo, rrsync and a remote non-root user. I assume that rsync will run from srv1 to srv2.
First you will need the rrsync (or rrsync.pl) script ad the server side that's part of the rsync package. In Debian you can find it at /usr/share/doc/rsync/scripts/rrsync.gz. This script acts as the server side and will restrict the destination directory (a'la chroot).
In short the server side will run "rrsync /srv". Then the client side will do something like this:
[code light="true"]
# rsync /srv remote:/
[/code]
and / will be relative to /srv that was defined as a parameter to rrsync.
You can put rrsync under /usr/local/bin.
At the destination server we will need a user that will be used for the ssh session. So go and create a user named 'syncer' on srv2. I'd avoid a username of 'rsync' as it may be used for other reasons at some point.
The user on srv2 should be able to run rrsync with sudo and with the -E parameter. -E is required in order to pass the checks of the rrsync script which checks for SSH_ORIGINAL_COMMAND in the environment. Feel free to make this even more strict to allow only this environment variable if you like.
Sample sudoers entry (e.g. to be put in /etc/sudoers.d/syncer):
[code light="true"]
syncer   ALL=SETENV:NOPASSWD:/usr/local/bin/rrsync /srv
[/code]
Obviously we need the user to be able to run this without requiring a password. SETENV will allow for the -E parameter to sudo.
Next step is to allow root@srv1 to ssh as syncer@srv2 using public key. If you don't have a key pair generated for root@srv1 then go ahead and create it:
[code light="true"]
# ssh-keygen
[/code]
Then copy the contents of /root/.ssh/id_rsa.pub and paste them in syncer@srv2's authorized_keys file which is most probably at /home/srv2/syncer/.ssh/authorized_keys. Create the directory and the file if they don't exist.
To make rrsync work and make things safer you need to use the command=".." parameter and you should use the from=".." parameter. So your authorized_keys file will look something like this:
[code light="true" wraplines="true"]
from="srv1",command="sudo -E /usr/local/bin/rrsync /srv" ssh-rsa AAAA......siW root@srv1
[/code]
Don't forget to ssh at least once from srv1 to srv2 by hand in order to accept srv2's key and let ssh have it in in known_hosts.
Finally you are done and you can do the rsync:
[code light="true"]
# rsync --rsh=ssh -a --delete /srv syncer@srv2:/
[/code]
In general it will use rsync over ssh, sudo, rrsync and a remote non-root user. I assume that rsync will run from srv1 to srv2.
rrsync
First you will need the rrsync (or rrsync.pl) script ad the server side that's part of the rsync package. In Debian you can find it at /usr/share/doc/rsync/scripts/rrsync.gz. This script acts as the server side and will restrict the destination directory (a'la chroot).
In short the server side will run "rrsync /srv". Then the client side will do something like this:
[code light="true"]
# rsync /srv remote:/
[/code]
and / will be relative to /srv that was defined as a parameter to rrsync.
You can put rrsync under /usr/local/bin.
User on srv2
At the destination server we will need a user that will be used for the ssh session. So go and create a user named 'syncer' on srv2. I'd avoid a username of 'rsync' as it may be used for other reasons at some point.
sudo on srv2
The user on srv2 should be able to run rrsync with sudo and with the -E parameter. -E is required in order to pass the checks of the rrsync script which checks for SSH_ORIGINAL_COMMAND in the environment. Feel free to make this even more strict to allow only this environment variable if you like.
Sample sudoers entry (e.g. to be put in /etc/sudoers.d/syncer):
[code light="true"]
syncer   ALL=SETENV:NOPASSWD:/usr/local/bin/rrsync /srv
[/code]
Obviously we need the user to be able to run this without requiring a password. SETENV will allow for the -E parameter to sudo.
SSH config
Next step is to allow root@srv1 to ssh as syncer@srv2 using public key. If you don't have a key pair generated for root@srv1 then go ahead and create it:
[code light="true"]
# ssh-keygen
[/code]
Then copy the contents of /root/.ssh/id_rsa.pub and paste them in syncer@srv2's authorized_keys file which is most probably at /home/srv2/syncer/.ssh/authorized_keys. Create the directory and the file if they don't exist.
To make rrsync work and make things safer you need to use the command=".." parameter and you should use the from=".." parameter. So your authorized_keys file will look something like this:
[code light="true" wraplines="true"]
from="srv1",command="sudo -E /usr/local/bin/rrsync /srv" ssh-rsa AAAA......siW root@srv1
[/code]
Don't forget to ssh at least once from srv1 to srv2 by hand in order to accept srv2's key and let ssh have it in in known_hosts.
Try it
Finally you are done and you can do the rsync:
[code light="true"]
# rsync --rsh=ssh -a --delete /srv syncer@srv2:/
[/code]
Subscribe to:
Posts (Atom)