Using TCP-LP with pycurl in Python

Intro

TCP-LP (low priority) is a TCP congestion control algorithm that is meant to be used by TCP connections that don't want to compete with other connections for bandwidth. Its goal is to use the idle bandwidth for file transfers. The details of TCP-LP are here.

With Linux' plugable congestion control algorithms, it is possible to change both the default algorithm for the whole system and the one used per connection. For the latter, one needs to be root.

Note: Changing the CC algorithm will only affect the transmissions. You cannot alter the remote end's behavior. This means that the below only make sense when you are going to upload data.

PyCurl

Changing the CC algorithm is a matter of using setsockopt on a socket. Doing this with pycurl can be a bit tricky. Even though pycurl supports the SOCKOPTFUNCTION, this is only for newer pycurl versions. For older, one can exploit pycurl's OPENSOCKETFUNCTION instead.

The trick is done with this piece of code:

[code language="python"]
import pycurl
import socket

def _getsock(family, socktype, protocol, addr):
    s=socket.socket(family, socktype, protocol)
    s.setsockopt(socket.IPPROTO_TCP, 13, 'lp' )
    return s

c = pycurl.Curl()
c.setopt(c.OPENSOCKETFUNCTION, _getsock)
c.setopt(c.URL, 'http://127.0.0.1/')
c.perform()
c.close()
[/code]

In the above, pycurl will call _getsock and expect it to return a socket. The function creates a new socket, then calls setsockopt with IPPPROTO_TCP and 13 (which is TCP_CONGESTION - see /usr/include/linux/tcp.h,  /usr/include/netinet/tcp.h). It then attempts to set the algorithm to "lp" which is the TCP-LP congestion control algorithm.

You most probably want to wrap the setsockopt around a try/except clause as it may fail if "lp" is not available (needs the module tcp_lp loaded) or if the program doesn't run as root.

The _getsock function also depends on the pycurl version, as its arguments have changed over time. Consult the docs for the fine details.

Results

Example of uploading two 500MB files in parallel on an already busy production network. One is with TCP-LP and the other with the default (TCP Cubic):

TCP-Cubic: 9.38 seconds
TCP-LP: 23.08 seconds

Same test, for 100MB files, again in parallel, on the same network:

TCP-Cubic: 3.14 seconds
TCP-LP: 5.38 seconds

Note: The above are random runs, presented to give an idea of the impact. For actual experimental results we would need to have  multiple runs and also monitor the background traffic.

Debian packaging for python2 and python3 at the same time

The problem

The scenario was like this:

• Python code that provides a library and a binary


• The code is compatible with both Python v2 and v3

The requirements were:

• Generate a package with the library part for python v2


• Generate a package with the library part for python v3


• Generate a binary package with the executable for python v3

I.e, from one source package (vdns) I wanted to create python-vdns (python2 lib), python3-vdns (python3 lib) and vdns (executable).

The approach

After trying other methods, I ended up using debhelper 9 and pybuild. Before that I tried using CDBS but had no luck there.

With DH9 it's easy to package a python library for multiple python versions as it handles everything itself. The only catch was the package that contained the binaries and how to make it use the v3 version instead of the v2

The solution

The solution was this rules file:

[code lang="bash" gutter="0"]
#!/usr/bin/make -f

# see EXAMPLES in dpkg-buildflags(1) and read /usr/share/dpkg/*
DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

# Don't set this or else the .install files will fail
#export PYBUILD_NAME = vdns
#export PYBUILD_SYSTEM = custom

# Otherwise the usr/bin/vdns.py file is from python2
export PYBUILD_INSTALL_ARGS_python2 = --install-scripts=/dev/null

# main packaging script based on dh7 syntax
%:
dh $@ --with python3,python2 --buildsystem=pybuild
[/code]

The control file is like this:

Source: vdns
Section: unknown
Priority: optional
Maintainer: Stefanos Harhalakis <v13@v13.gr>;
Build-Depends: debhelper (>= 9), dh-python,
 python-all (>=2.6.6-3~), python-setuptools,
 python3-all (>=3.2), python3-setuptools
Standards-Version: 3.9.5
X-Python-Version: >= 2.7
X-Python3-Version: >= 3.2
XS-Python-Version: >= 2.7
XS-Python3-Version: >= 3.2

Package: python-vdns
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}, python-psycopg2
Description: vdns python2 libraries
 These libraries allow the reading and the creation of bind zone files

Package: python3-vdns
Architecture: all
Depends: ${python3:Depends}, ${misc:Depends}, python3-psycopg2
Description: vdns python3 libraries
 These libraries allow the reading and the creation of bind zone files

Package: vdns
Architecture: all
Depends: ${python:Depends}, ${misc:Depends}, python3-vdns, python(>=3.2)
Description: Database-based DNS management
 vdns is a database-based DNS management tool. It gets data from its database
 and generates bind zone files. It supports A, AAAA, MX, NS, DS, PTR, CNAME,
 TXT, DNSSEC, SSHFP, DKIM and SRV records
 .
 vdns uses a PostgreSQL database to store the data
 .
 vdns is not a 1-1 mapping of DB<->zone files. Instead the databsae
 is meant to describe the data that are later generated. E.g:
 * DKIM data are used to generate TXT records
 * A, AAAA and PTR entries are generated from the same data
 * NS glue records for sub-zones are auto-created

And the .install files are like this:

python-vdns.install:
usr/lib/python2*/dist-packages/vdns/*.py*
usr/lib/python2*/dist-packages/vdns/src

python3-vdns.install:
usr/lib/python3*/dist-packages/vdns/*.py*
usr/lib/python3*/dist-packages/vdns/src

Explanation

What the above does is to use DH9 with pybuild. Pybuild takes care of multiple versions by "compiling" the binaries  twice under build/scripts-2.x and build/scripts/3.x directories. After that it copies them  to debian/tmp and finally splits the contents of debian/tmp based on the .install files to debian/python-vdns, debian/python3-vdns and debian/vdns.

The biggest problem were the files that were meant for the vdns package as those were present in both build/scripts-2.x and build/scripts/3.x, each one prepared for the appropriate debian version:

-rwxr-xr-x 1 v13 v13 1197 Jul 19 21:12 build/scripts-2.7/vdns.py
-rwxr-xr-x 1 v13 v13 1198 Jul 19 21:12 build/scripts-3.4/vdns.py

The following line in rules takes care of the conflict by skipping a version:

export PYBUILD_INSTALL_ARGS_python2 = --install-scripts=/dev/null

This way, the python2 version never gets installed and thus only the python3 version is available to be copied to debian/tmp. Otherwise the behavior was random (it was picking the first one it was finding).

Other attempts

I also tried using autoconf with CDBS but that proved to be even more difficult.

Acknowledgements

The above was accomplished only because of the help of folks in #debian-python @ OFTC; namely: p1otr, mapreri and jcristau.

pyOpenSSL and invalid certificates

I was trying to import some X509v3 certificates that were created with pyOpenSSL to a MikroTik router (RouterOS 6.1) but they were always being imported with an invalid validity period (not before 1970 and not after 1970).

Eventually I found out that this is because pyOpenSSL stores the validity field in an invalid format. Here's the story:

cert1.pem is the pyOpenSSL certificate and cert2.pem is a certificate created with openssl. Both have mostly the same information. Decoding the certificates with openssl shows that cert1.pem actually has an older notAfter date so it's not an issue of overflow.

[code language="shell" gutter="false"]
$ openssl x509 -noout -startdate -enddate < cert1.pem
notBefore=Jul 20 18:35:58 2013 GMT
notAfter=Jan  1 00:00:00 2032 GMT

$ openssl x509 -noout -startdate -enddate < cert2.pem
notBefore=Aug 31 21:44:54 2012 GMT
notAfter=Aug 26 21:44:54 2032 GMT
[/code]

I examined the certificates in python by decoding their DER structures and looking for the validity field (copy-paste the following in a python shell).

[code language="python"]
import Crypto.Util.asn1 as asn1
import OpenSSL.crypto as c

fn1="cert1.pem"
fn2="cert2.pem"

st1=open(fn1, 'r').read()
st2=open(fn2, 'r').read()

cert1=c.load_certificate(c.FILETYPE_PEM, st1)
cert2=c.load_certificate(c.FILETYPE_PEM, st2)

dump1=c.dump_certificate(c.FILETYPE_ASN1, cert1)
dump2=c.dump_certificate(c.FILETYPE_ASN1, cert2)

der1=asn1.DerSequence()
der2=asn1.DerSequence()

der1.decode(dump1)
der2.decode(dump2)

dcert1=der1[0]
dcert2=der2[0]

t1=asn1.DerSequence()
t2=asn1.DerSequence()

t1.decode(dcert1)
t2.decode(dcert2)

tt1=asn1.DerSequence()
tt2=asn1.DerSequence()

tt1.decode(t1[4])
tt2.decode(t2[4])
[/code]

at this point tt1 and tt2 are sequences of the validity field (notBefore, notafter) for the two certificates . Here's what they contain:

[code language="python" gutter="false"]
>>> tt1[0]
'\x18\x0f20130720183558Z'
>>> tt1[1]
'\x18\x0f20320101000000Z'

>>> tt2[0]
'\x17\r120831214454Z'
>>> tt2[1]
'\x17\r320826214454Z'
[/code]

SoaB! They differ!

Reading the X509 spec [1], section 4.1.2.5 indicates that there are two possible formats for the validity period: both notBefore and notAfter may be encoded as UTCTime or  GeneralizedTime.

• UTCTime is defined as YYMMDDHHMMSSZ


• GeneralizedTime is defined as YYYYMMDDHHMMSSZ

So pyOpenSSL uses GeneralizedTime while openssl uses UTCTime. So both are valid.

However the RFC also says:

CAs conforming to this profile MUST always encode certificate
validity dates through the year 2049 as UTCTime; certificate validity
dates in 2050 or later MUST be encoded as GeneralizedTime.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARG!

So it seems that pyOpenSSL uses GeneralizedTime unconditionally which is not RFC compliant and thus rejected by RouterOS.

A quick look at pyOpenSSL's code unfortunately proves that...

[1]   http://www.ietf.org/rfc/rfc3280.txt

Verify that a private key matches a certificate with PyOpenSSL

Verify that a private key matches a certificate using PyOpenSSL and PyCrypto:

[code language="python"]
import OpenSSL.crypto
from Crypto.Util import asn1

c=OpenSSL.crypto

# The certificate - an X509 object
cert=...

# The private key - a PKey object
priv=...

pub=cert.get_pubkey()

# Only works for RSA (I think)
if pub.type()!=c.TYPE_RSA or priv.type()!=c.TYPE_RSA:
raise Exception('Can only handle RSA keys')

# This seems to work with public as well
pub_asn1=c.dump_privatekey(c.FILETYPE_ASN1, pub)
priv_asn1=c.dump_privatekey(c.FILETYPE_ASN1, priv)

# Decode DER
pub_der=asn1.DerSequence()
pub_der.decode(pub_asn1)
priv_der=asn1.DerSequence()
priv_der.decode(priv_asn1)

# Get the modulus
pub_modulus=pub_der[1]
priv_modulus=priv_der[1]

if pub_modulus==priv_modulus:
print('Match')
else:
print('Oops')
[/code]

The idea is to get the modulus from the two DER structures and compare them. They should be the same.

Note: You can use the above under the MIT license. If it doesn’t fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.

Verifying an SSL certificate with python

This one took me a considerable amount of time and had to figure some parts from scratch.

Unfortunately there doesn't seem to exist an easy (out-of-the-box) way for checking whether a certificate is signed by another certificate in python.

After days of searching and despair, here is a solution without using M2Crypto:

[code language="python"]
import OpenSSL
from Crypto.Util import asn1

c=OpenSSL.crypto

# This is the certificate to validate
# an OpenSSL.crypto.X509 object
cert=...

# This is the CA certificate to use for validation
# again an OpenSSL.crypto.X509 object
cacert=...

# Get the signing algorithm
algo=cert.get_signature_algorithm()

# Get the ASN1 format of the certificate
cert_asn1=c.dump_certificate(c.FILETYPE_ASN1, cert)

# Decode the certificate
der=asn1.DerSequence()
der.decode(cert_asn1)

# The certificate has three parts:
# - certificate
# - signature algorithm
# - signature
# http://usefulfor.com/nothing/2009/06/10/x509-certificate-basics/
der_cert=der[0]
der_algo=der[1]
der_sig=der[2]

# The signature is a BIT STRING (Type 3)
# Decode that as well
der_sig_in=asn1.DerObject()
der_sig_in.decode(der_sig)

# Get the payload
sig0=der_sig_in.payload

# Do the following to see a validation error for tests
# der_cert=der_cert[:20]+'1'+der_cert[21:]

# First byte is the number of unused bits. This should be 0
# http://msdn.microsoft.com/en-us/library/windows/desktop/bb540792(v=vs.85).aspx
if sig0[0]!='\x00':
raise Exception('Number of unused bits is strange')

# Now get the signature itself
sig=sig0[1:]

# And verify the certificate
try:
c.verify(cacert, sig, der_cert, algo)
print "Certificate looks good"
except OpenSSL.crypto.Error, e:
print "Sorry. Nope."
[/code]

Note: You can use the above under the MIT license. If it doesn’t fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.

DNSSEC key tag (keyid) and DS signature calculation in python

This one took me a considerable amount of hours to figure out so here it is.

While trying to automate DNS zone generation I had to calculate some of the values programmatically. Two of the auto-generated values had to do with DNSSEC entries: The key tag (or keyid) and the DS record's signatures.

The required details on how these are calculated are found in the following places:

• Key tag: RFC4034 - Appendix B


• DNSKEY RDATA wire format: RFC4034 - Section 2.1


• DS' digest field: RFC4034 - Section 5.1.4


• Wire format of names: RFC1035 - Section 3.1

For the calculations you need to provide the following:

• For the key tag: flags, protocol, algorithm, public key


• For the DS signatures: owner (the domain name), flags, protocol, algorithm, public key

Code

I used python for this but the approach is the same for other languages since the algorithms are the same.

So here it is:

[code lang="python"]
import struct
import hashlib
import base64

def calc_keyid(flags, protocol, algorithm, st):
"""
@param owner The corresponding domain
@param flags The flags of the entry (256 or 257)
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return The key tag
"""
# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)

# Calculate the tag
cnt=0
for idx in xrange(len(st2)):
s=struct.unpack('B', st2[idx])[0]
if (idx % 2) == 0:
cnt+=s<<8
else:
cnt+=s

ret=((cnt & 0xFFFF) + (cnt>>16)) & 0xFFFF

return(ret)

def calc_ds(owner, flags, protocol, algorithm, st):
"""
@param flags Usually it is 257 or something that indicates a KSK.
It can be 256 though.
@param protocol Should always be 3
@param algorithm Should always be 5
@param st The public key as listed in the DNSKEY record.
Spaces are removed.
@return A dictionary of hashes where the key is the hashing algorithm.
"""

# Remove spaces and create the wire format
st0=st.replace(' ', '')
st2=struct.pack('!HBB', int(flags), int(protocol), int(algorithm))
st2+=base64.b64decode(st0)

# Ensure a trailing dot
if owner[-1]=='.':
owner2=owner
else:
owner2=owner+'.'

# Create the name wire format
owner3=''
for i in owner2.split('.'):
owner3+=struct.pack('B', len(i))+i

# Calculate the hashes
st3=owner3+st2
ret={
'sha1': hashlib.sha1(st3).hexdigest().upper(),
'sha256': hashlib.sha256(st3).hexdigest().upper(),
}

return(ret)
[/code]

Data

The following were created by bind's dnssec tools:

[code light="true"]
$ cat Ktest.hell.gr.+005+33630.key
; This is a zone-signing key, keyid 33630, for test.hell.gr.
; Created: 20101007114826 (Thu Oct  7 14:48:26 2010)
; Publish: 20101007114826 (Thu Oct  7 14:48:26 2010)
; Activate: 20101007114826 (Thu Oct  7 14:48:26 2010)
test.hell.gr. IN DNSKEY 256 3 5 AwEAAb+lTDjZCfq7D5N9cNd1ug30wLrbCXB9mVJJQGlQQHpiHHlMaLGG
sV2/j5+eojHp+WQUzNpOzrULF6msbEvUuV2gSEnpbueRV4twO8muGE+x
eUuseSoHh/aTpA8Z9SPubb01mduqqaUEN5Juz2Q4hF0dSUSJYlJPKhp6
NrOgoeyj

$ cat dsset-test.hell.gr.
test.hell.gr.        IN DS 33630 5 1 A2AD2648B353365631EBC9C70EDA1E0C04563FCC
test.hell.gr.        IN DS 33630 5 2 4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2 439BB5C1
[/code]

Result

[code lang="python" light="true"]
>>> domain='test.hell.gr'
>>> flags=256
>>> protocol=3
>>> algorithm=5
>>> key='AwEAAb+...goeyj' # Truncated
>>> calc_keyid(flags, protocol, algorithm, key)
33630
>>> r=calc_ds(domain, flags, protocol, algorithm, key)
>>> r['sha1']
'A2AD2648B353365631EBC9C70EDA1E0C04563FCC'
>>> r['sha256']
'4177EAEC09A37178357871EBE3FB361CABB2861F12A1D51DDE18CBA2439BB5C1'
[/code]

Legal

You can use the above under the MIT license. If it doesn't fit your needs let me know. My intention is to make this usable by anyone for any kind of use with no obligation.

A fast interpreted language

How can one claim that an interpreted language is fast?

Easily. If you implement the language using itself and the resulting interpreter is faster than the original then this means that the interpreted language manages to imprint the prorgammer's intentions to the C language (or assembly) better than the programmer herself.

For example, I may create an interpreted language named V using C and try to make the implementation as fast as I can. Of course this means that I have to implement a number of data structures and handle them. Also I have to optimize code path etc etc.

Then I re-implement the V language using the V language itself. If the re-implementation if faster than the original implementation then this means that the interpreter's logic produces faster code than me.

Well... here it is. The Python implementation in Python is faster than the Python implementation using C!

Read: It is very probable to end up with a faster program if you write it using python instead of C because the language will do better than you will in optimizing your data structures and your code. Just like it is better to write C than assembly and let your compiler to produce the optimized assembly code.

World's smallest IPv6 compatible web browser program

For a long time now I've become a fan of the python+Qt combination. It is great to have an easy to learn, easy to use language with great portability and minimalistic syntax.

The following program has become my favorite showcase for python. To my knowledge, it is the smallest, easier to understand, portable, IPv6 compatible web browser in the world.

If you have IPv6 connectivity just lunch python and copy-paste it save it to a file and run it (it may crash python otherwise):

[python]
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4.QtWebKit import *

app=QApplication([])
win=QMainWindow()
w=QWebView(win)
win.setCentralWidget(w)
w.setUrl(QUrl("http://ipv6.google.com/"))
win.show()
app.exec_()
[/python]

If you don't then you can try the IPv4 version:

[python]
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4.QtWebKit import *

app=QApplication([])
win=QMainWindow()
w=QWebView(win)
win.setCentralWidget(w)
w.setUrl(QUrl("http://www.google.com/"))
win.show()
app.exec_()
[/python]

In order to be able to run it you'll need the Qt4 library and the python Qt bindings. If you're under debian just run:

[bash light="true"]
# apt-get install python-qt4
[/bash]

This should work at least under Linux, Windows, Maemo and perhaps Symbian.

KAutostart and python

This one took me many hours to debug:

For KDE 4 and python there can be a hard to debug problem with KAutostart usage.

Suppose you have a window class

class MyWindow(KDialig, Ui_Dialog):
  def __init__(self):
    self.kas = KAutostart('myapp')
    self.kas.setAutostarts(True)

You may find that the setAutostarts() doesn't actually work. This is because MyWindow references kas and kas references MyWindow (as the parent object) (or the application - I'm not sure). The problem is that KAutostart() stores the information when it is deleted (in it's destructor) and since it is never deleted it won't store anything.

You need to understand that the destructors won't be called even when the program exits, because of the circular reference.

To solve the problem call:

self.kas=None

at some point.

Have a look here for more on python and its destructors.