Showing posts with label Linux. Show all posts

Sunday, 30 April 2017

Static IPv6 subnetting at home with dynamic prefix delegation

The problem

How to have IPv6 subnetting on a non-flat network at home when you are receiving a dynamic IPv6 prefix via DHCP6-PD (prefix delegation) via an ISP.

This is about a setup where you have (e.g.) a DSL line and a router that receives a prefix (e.g. a /56 prefix) via DHCP6-PD. Since the prefix is dynamic you cannot assign static IPv6 addresses, which means that you cannot possibly subnet this, which in turn means that you cannot have any kind of non-flat home network.

With IPv4 the problem doesn't exist as you can have a non-flat home network with some routing involved and rely on NAT/Masquerading. But Masquerading doesn't exists for IPv6 and most probably your home IPv6 router doesn't do IPv6 NAT

The initial setup

Suppose you have something like this:

Internet <--> DSL Router (R1) <--> Another router (R2) <--> Some IPv6 subnets

The new setup

I solved this with a raspberry pi. You can use whatever you like as long as it has two Ethernet interfaces.

I used a Raspberry Pi and a USB ethernet adapter plugged to it. Then changed the home setup to something like this:

Internet <---> DLS Router (R1) <---> Raspberry Pi (Pi) <---> Another Router (R2) <--> Home IPv6 Subnets

For simplicity I'll use the above setup, even though my actual setup is a triangle between R1, Pi and R2 so that IPv4 traffic goes directly from R1 to R2 while IPv6 traffic goes through the Pi.

What needs to be done

Your whole home network will be exiting with a single /64 IPv6 network with random address mappings via NAT.

In order to achieve this you need to do the following:

Have your DSL router provide addresses via SLAAC to the interface that connects to the Pi.

Setup your IPv6 home network behind the Pi with static IPs. Use a static prefix that is allocated to you somehow (e.g. from sixxs) or ULA or something sane. This is not the dynamic prefix you get from your ISP.

Have the Pi get a dynamic IPv6 address on the external interface.

Have the Pi NAT your internal IPv6 addresses to external IPv6 addresses from the same subnet it belongs to.

Have the Pi respond to ND requests for the NATed addresses.

Have a script that reconfigured the NAT if it changed to accomodate for the dynamic IPv6 prefix you are assigned.

How to do it

First setup your DLS router to do SLAAC (stateless) IPv6 address assignments. I.e. not to assign addresses via DHCP6.

Then setup your internal network with static IPv6 addresses/subnets. Assuming you chose to use ULA (fd00::/8) for your static home network:

you use fd00:1::/64 for the connection Pi<-->R2

Pi will have fd00:1::1/64

R2 has the fd00:1::2/64 and a default IPv6 route via fd00:1::1

Your internal network behind R2 uses fd00::/48

It's assumed below that eth0 is the external interface and eth1 is the internal interface of the Pi

Have the Pi get a dynamic IPv6 address

Assuming you're using Debian:

Set the defaults for forwarding and autoconf by adding these to a file under /etc/sysctl.d:

net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1

net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0

(make sure you reload these with /etc/init.d/procps restart)

Add this to the external interface's iface config:

iface eth0 inet static
  ... ipv4 setup ...
  up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/autoconf || true
  up echo 2 > /proc/sys/net/ipv6/conf/$IFACE/accept_ra || true
  up echo 0 > /sys/devices/virtual/net/$IFACE/bridge/multicast_snooping || true

You most probably want to also setup IPv4 over there. Note that these were set on "inet" and not on "inet6" as inet6 will be auto-configured. Feel free to adapt it.

You also need to setup static routes for the internal network on the pi:

iface eth1 inet static
  ... ipv4 setup ...
  echo 0 > /sys/devices/virtual/net/$IFACE/bridge/multicast_snooping || true

iface eth1 inet6 static
  address fd00:1::1
  netmask 64
  up ip -6 route add fd00::/48 via fd00:1::2 || true
  down ip -6 route del fd00::/48 via fd00:1::2 || true

Have the Pi do SNAT

Since the Pi receives an external address from an /64 IPv6 network and it's the sole user, it's ok to assume that you can use some more IPv6 addresses from that subnet for NAT :-)

You can then do this:

IFEX=eth0
PREFIX=$(ip -6 addr show $IFEX | grep inet6 | grep -v 'inet6 f[de]' | awk '{print $2}' | cut -f 1-4 -d : | tail -1)
PREFIX2="fd00::/48"
NATFROM="::1:1000"
NATTO="::1:2000"

Where PREFIX will hold the /64 prefix of the external (dynamically allocated) network

And then:

from="${PREFIX}${NATFROM}"
to="${PREFIX}${NATTO}"
ip6tables -t nat -A POSTROUTING -o $IFEX -j SNAT --to-source ${from}-${to} --persistent

This will map your fd00::/48 addresses to 4096 IPv6 addresses from the dynamic prefix. You can obviously extend the range considerably, but don't go nuts or you may end up with too many NAT and ND entries.

Have the Pi respond to ND requests

So far the Pi will happily do the NAT and will send the packets to your DSL router, but the DSL router won't be able to send anything back as noone will be responding to ND requests for the NAT range.

To solve the problem we need to use ndppd and a dynamic configuration file.

Grab ndppd from https://github.com/DanielAdolfsson/ndppd, compile it and place the binary somewhere.

First ensure that proxy_ndp is enabled:

echo 1 > /proc/sys/net/ipv6/conf/$IFEX/proxy_ndp

Then create the appropriate ndppd.conf file:

cat << _KOKO > ndppd.conf
proxy $IFEX {
  rule ${PREFIX}::/64 {
    static
  }
}
_KOKO

Then fire up ndppd:

ndppd -d -v -c ndppd.conf

Test it

That's it. If I didn't forget anything then your home network should have IPv6 access to the rest of the world using the fd00::/48 prefix.

Script it

The final step is to script all of this and have it run via cron so that it adapts to IPv6 prefix changes. This is a slimmed down version of what I'm using, adjusted to the fd00::/48 prefix:

[code lang="bash"]
#!/bin/bash

PATH=/bin:/usr/bin:/sbin:/usr/sbin

IFEX=eth0

PREFIX=$(ip -6 addr show $IFEX | grep inet6 | grep -v 'inet6 f[de]' | awk '{print $2}' | cut -f 1-4 -d : | tail -1)
D0="/srv/ipv6" # A directory to work under
PREFIX2="fd00::/48"

NCFG="$D0/ndppd.conf"
NCFGNEW="$D0/ndppd.conf.new"
NCFGOLD="$D0/ndppd.conf.old"

NDPPD="$D0/ndppd/ndppd" # Path to the ndppd executable

NATFROM="::1:1000"
NATTO="::1:2000"
TBL=ip6nat

DEBUG=${DEBUG:-false}
CHANGED=${CHANGED:-false}

debug()
{
$DEBUG && echo "$@"
}

I()
{
debug ip6tables -t nat "$@"
ip6tables -t nat "$@"
}

# Test and set
ITS()
{
if ! I -C "$@" 2> /dev/null ; then
I -A "$@"
fi
}

# Setup NAT
do_nat_start()
{
local from to

if $CHANGED ; then
echo "Reseting NAT rules"
do_nat_stop 2> /dev/null
fi

from="${PREFIX}${NATFROM}"
to="${PREFIX}${NATTO}"

I -N $TBL 2> /dev/null
ITS $TBL -s $PREFIX2 -o $IFEX -j SNAT --to-source ${from}-${to} --persistent
ITS POSTROUTING -j $TBL
}

do_nat_stop()
{
while I -D POSTROUTING -j $TBL > /dev/null ; do : ; done
I -F $TBL
I -X $TBL
}

# Do basic configuration
do_cfg()
{
echo 1 > /proc/sys/net/ipv6/conf/$IFEX/proxy_ndp

cat << _KOKO > $NCFGNEW
proxy $IFEX {
rule ${PREFIX}::/64 {
static
}
}
_KOKO

if ! test -e $NCFG || ! diff -q $NCFG $NCFGNEW > /dev/null ; then
debug "Things changed"
CHANGED=true
test -e "$NCFG" && mv -f $NCFG $NCFGOLD
mv -f $NCFGNEW $NCFG
else
debug "Nothing changed"
fi
}

# Start ndppd
do_ndppd()
{
if $CHANGED ; then
echo "New config. Reloading ndppd. Prefix: $PREFIX"
killall ndppd
sleep 1
fi

if ! pgrep ndppd > /dev/null ; then
if $DEBUG ; then
$NDPPD -vvv -c $NCFG
else
$NDPPD -d -v -c $NCFG
fi
fi
}

doit()
{
if test -z "$PREFIX" ; then
echo "No prefix"
exit 1
else
debug "Prefix: $PREFIX"
fi

do_cfg
do_nat_start
do_ndppd
}

if $DEBUG ; then
doit
else
doit | logger -t 6nat
fi
[/code]

Have fun!

Monday, 11 April 2016

Using TCP-LP with pycurl in Python

Intro

TCP-LP (low priority) is a TCP congestion control algorithm that is meant to be used by TCP connections that don't want to compete with other connections for bandwidth. Its goal is to use the idle bandwidth for file transfers. The details of TCP-LP are here.

With Linux' plugable congestion control algorithms, it is possible to change both the default algorithm for the whole system and the one used per connection. For the latter, one needs to be root.

Note: Changing the CC algorithm will only affect the transmissions. You cannot alter the remote end's behavior. This means that the below only make sense when you are going to upload data.

PyCurl

Changing the CC algorithm is a matter of using setsockopt on a socket. Doing this with pycurl can be a bit tricky. Even though pycurl supports the SOCKOPTFUNCTION, this is only for newer pycurl versions. For older, one can exploit pycurl's OPENSOCKETFUNCTION instead.

The trick is done with this piece of code:

[code language="python"]
import pycurl
import socket

def _getsock(family, socktype, protocol, addr):
    s=socket.socket(family, socktype, protocol)
    s.setsockopt(socket.IPPROTO_TCP, 13, 'lp' )
    return s

c = pycurl.Curl()
c.setopt(c.OPENSOCKETFUNCTION, _getsock)
c.setopt(c.URL, 'http://127.0.0.1/')
c.perform()
c.close()
[/code]

In the above, pycurl will call _getsock and expect it to return a socket. The function creates a new socket, then calls setsockopt with IPPPROTO_TCP and 13 (which is TCP_CONGESTION - see /usr/include/linux/tcp.h, /usr/include/netinet/tcp.h). It then attempts to set the algorithm to "lp" which is the TCP-LP congestion control algorithm.

You most probably want to wrap the setsockopt around a try/except clause as it may fail if "lp" is not available (needs the module tcp_lp loaded) or if the program doesn't run as root.

The _getsock function also depends on the pycurl version, as its arguments have changed over time. Consult the docs for the fine details.

Results

Example of uploading two 500MB files in parallel on an already busy production network. One is with TCP-LP and the other with the default (TCP Cubic):

TCP-Cubic: 9.38 seconds
TCP-LP: 23.08 seconds

Same test, for 100MB files, again in parallel, on the same network:

TCP-Cubic: 3.14 seconds
TCP-LP: 5.38 seconds

Note: The above are random runs, presented to give an idea of the impact. For actual experimental results we would need to have multiple runs and also monitor the background traffic.

Monday, 15 February 2016

Running an NTP server in a VM using KVM

The setup

Having physical server pA, running VMs using KVM. One of theVMs (vA) acts as an NTP server. pA gets the time from vA and vA gets it from the Internet.

It's not a great idea to run an NTP server in a VM, but in this case there was need for it.

The problem

NTP server gets frequently out of sync.

If you use nagios, you may get errors like this:

SERVICE ALERT: pA;ntpd;CRITICAL;SOFT;4;NTP CRITICAL: Offset unknown

Both for the physical server and other servers that fetch the time from vA.

The reason

There's some guessing involved here, but this should be pretty accurate:

VM vA needs to correct its clock every now and then by slowing down or speeding up things per ntpd/adjtimex. As expected, this creates a small discrepancy between vA and pA, as now the physical server gets out of sync and needs to correct its time using vA's reference time.

Once vA attempts to correct its time, again by slowing down or speeding up its clock, this has a direct effect on vA, as vA's clock is now affected by pA's ongoing adjustment. This happens because KVM by default uses kvmclock as its clock source (the source that ticks and not the source that returns the time of the day).

This action sometimes causes pA's ntpd to get even more out of sync and may even make it consider its peers inaccurate and become fully out of sync.

The problem gets even worse if you have two ntp servers (vA and vB) running on two different physical servers (pA and pB), because the amount of desync between the two is mostly random. Assuming that all your servers, including pA and pB, fetch the time from vA and vB, the discrepancy between them will make them mark at least one of them as wrong, as the stratum of vA and vB does not permit such difference between their clocks.

You can see the above by looking at the falsetick result in ntpq's associations:

ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1 33082  961a   yes   yes  none  sys.peer    sys_peer  1
  2 33083  911a   yes   yes  none falsetick    sys_peer  1

Overall, the problem is that the physical servers will try to fix their clocks, thus affecting the clocks of the NTP servers running in VMs under them.

The solution

The problem is with the VMs using the kvmclock source. You can see that using dmesg:

$ dmesg | grep clocksource
Switching to clocksource kvm-clock

The way to disable this is to pass the "no-kvmclock" parameter to the kernel of your VMs. This will not always work though. The reason is that the kernel (at least the CentOS kernels) will panic very early in the boot process as it will still try to initialize the kvmclock even if it's not going to use it, and will fail.

The solution is to pass two parameters to your VM kernels: "no-kvmclock no-kvmclock-vsyscall". The second one is a bit undocumented, but will do the trick.

After that you can verify it through dmesg:

$ dmesg | grep Switching
Switching to clocksource refined-jiffies
Switching to clocksource acpi_pm
Switching to clocksource tsc

Example

Below is the output of a server running in such an environment. In this case the first ntp server (vA) runs with the extra kernel parameters and the other (vB) runs without them. The clock of the physical servers (pA and pB) was slowed down by hand using adjtimex in order to test the effect of the physical server's clock on the VM clocks. As you can see, this server is still in sync with vA and has a very large offset with vB. Note that this server is not a VM under pA or pB.

$ ntpq -nc peers
     remote           refid      st t when poll reach delay   offset  jitter
==============================================================================
*10.93.XXX.XXX   216.218.254.202  2 u   81  256  377 0.433  -87.076  20.341
 10.93.XXX.XXX   216.218.254.202  2 u  290  512  377 0.673  11487.6 9868.84

I.e., what happened is that the first one, using the extra parameters, kept its clock accurate while the second did not.

Sunday, 28 September 2014

Multiple relay configuration based on sender address with sendmail

One of the needs that came up was to be able to use separate relay configurations based on the sender email address, using sendmail. The problem is that sendmail is missing support for most parts of that sentence.

At the end the solution involved a combination of sendmail, smarttable, procmail and msmtp

The idea is the following:

Use smarttable to implement sender based rules

Use the procmail mailer support to use procmail to deliver the emails

Use procmailrc to pipe messages to msmtp

Use msmtp to relay via external hosts

Sender based rules

In order to be able to have sender-based rules I used smarttable.m4 from here.

Download the smarttable.m4 and (assuming sendmail config is under /etc/mail) place it under /etc/mail/m4/. Normally it should be placed along the rest of the sendmail features (/usr/share/sendmail/cf/features) but I don't like polluting system dirs. Then use the following config in sendmail.mc:

dnl Change the _CF_DIR for a bit to load the feature from /etc/mail/m4
dnl then change it back
define(`_CF_DIR_OLD', _CF_DIR_)dnl
define(`_CF_DIR_', `/etc/mail/m4/')dnl
dnl This has to be a hash. I.e. not text.
FEATURE(`smarttable',`hash -o /etc/mail/smarttable')dnl
define(`_CF_DIR_', _CF_DIR_OLD)dnl

Then configure smarttable (/etc/mail/smarttable) like this:

test@test.com    procmail:/etc/mail/persource/test.test.com.procmailrc

You can add as many lines as you like, one for each sender. See smarttable's web page for more information on the supported sender formats. Dont' forget to generate the hashed version (smarttable.db)

Procmail config

Configure sendmail for procmail mailer like this:

define(`PROCMAIL_MAILER_ARGS', `procmail -Y -t -m $h $f $u')dnl
MAILER(`procmail')dnl

You have to override the default procmail parameters in order to add the -t switch. This way delivery errors will be interpreted as softfails, otherwise mails will be rejected on the first failure.

Create /etc/mail/persource and put the procmail configs in there (nice and tidy). In this example create /etc/mail/persources/test.test.com.procmailrc as follows:

:0w
|/usr/bin/msmtp -C /etc/mail/persource/test.test.com.msmtprc -a test@test.com -t

The 'w' flag is essential in order to feed failures back to sendmail.

Msmtp config

Create the msmtp config file (/etc/mail/persource/test.test.com.msmtprc) as follows:

defaults
syslog          on
# logfile /tmp/msmtp-test@test.com.log

account         test@test.com
host            smtp.gmail.com
from            test@test.com
user            test@test.com
password        xxx
auth            on
tls             on
tls_trust_file  /etc/ssl/certs/ca-certificates.crt

Your mileage may vary. They above is good for gmail accounts on a debian system.

Done

And that's it. Sending an email as test@test.com will cause sendmail to use smarttable. This will match the sender and use procmail with our config to deliver the email. Procmail will pipe the email to msmtp which will send the email via google's email servers.

Thursday, 11 September 2014

OpenVPN and remote-cert-tls server

This required a bit of digging into OpenVPN's and OpenSSL's code to figure out.

The problem

This error:

Thu Sep 11 00:12:05 2014 Validating certificate key usage
Thu Sep 11 00:12:05 2014 ++ Certificate has key usage  00f8, expects 00a0
Thu Sep 11 00:12:05 2014 ++ Certificate has key usage  00f8, expects 0088

The condition

Using openvpn with the following option:

remote-cert-tls server

The solution

(for me) to add this to openvpn's config file:

remote-cert-ku f8

The explanation

Background

remote-cert-tls attempts to solve one problem: Lets say you run a CA and you distribute the certificates to 2 people including me and you. Then you setup a VPN server for us to use and you generate another certificate for the VPN server.

As always, the problem that certificates attempt to solve is "how do you know you're connecting to the remote end you assume you are. In normal SSL pages you trust a CA to verify that the CN of the certificate matches the owner of the domain. If you want to achieve the same thing with openvpn then you need to verify the CN of the remote end against either the hostname or a predefined string. If not then you need to use the "remote-cert-tls server" option.

If you don't use any of the above methods then I can fire up an openvpn server using the certificate you provided me with and since both my certificate and the actual VPN server's certificate are signed by the same CA you would be verifying both and be equally willing to connect to both, thus allowing me to spy on you.

To solve this kind of problems X509 has some properties for certificates that designate them for certain purposes. E.g. one of them is to run a VPN endpoint (TLS Web Client Authentication)

To be precise there are 2+1 such designations in X509:

X509v3 Key Usage: 
    Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment, Key Agreement
 
X509v3 Extended Key Usage: 
    TLS Web Server Authentication, TLS Web Client Authentication, IPSec End System, IPSec Tunnel, Time Stamping
 
Netscape Cert Type: 
    SSL Client, SSL Server, S/MIME, Object Signing

"Netscape Cert Type" is kind of old. "Key Usage" is the main one and "Extended Key Usage" is the final addition. Ignoring NS Cert Type, "Key Usage" is a bitmap and thus has limited space for expansion. "Extended Key Usage" on the other hand is a list of object identifiers which allows for unlimited expansion.

The certificate

The certificate I was using for the server-side of the OpenVPN had the above attributes. Ignoring NS Cert Type once more, the other two correspond to the following data:

  494:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Key Usage
  499:d=5  hl=2 l=   4 prim: OCTET STRING      [HEX DUMP]:030203F8
 
  433:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Extended Key Usage
  438:d=5  hl=2 l=  52 prim: OCTET STRING      [HEX DUMP]:303206082B0601050507030106082B0601050507030206082B0601050507030506082B0601050507030606082B06010505070308

Starting with "Key Usage", the actual value is "F8". The meaning of each bit can be found in OpenSSL's code:

#define KU_DIGITAL_SIGNATURE    0x0080
#define KU_NON_REPUDIATION      0x0040
#define KU_KEY_ENCIPHERMENT     0x0020
#define KU_DATA_ENCIPHERMENT    0x0010
#define KU_KEY_AGREEMENT        0x0008
#define KU_KEY_CERT_SIGN        0x0004
#define KU_CRL_SIGN             0x0002
#define KU_ENCIPHER_ONLY        0x0001
#define KU_DECIPHER_ONLY        0x8000

On the other hand, the "Extended Key Usage" part contains the following Object IDs:

06082B 06010505070301 -> serverAuth (TLS Web Server Authentication)
06082B 06010505070302 -> clientAuth (TLS Web Client Authentication)
06082B 06010505070305 -> ipsecEndSystem (IPSec End System)
06082B 06010505070306 -> ipsecTunnel (IPSec Tunnel)
06082B 06010505070308 -> timeStamping (Time Stamping)

The "bug"

The first thing to notice is that the failure is for "Key Usage" and not for "Extended Key Usage" (took me some time to figure out).

After that, a bit of digging into the code confirms that OpenVPN attempts to verify a bitmap with equality. I.e. it gets the certificates' value and compares it against a predefined list of allowed values, which according to OpenVPN's documentation defaults to "a0 88" (which means one of them). However the actual certificates bitmap value is 0xf8 as mentioned above. And thus the comparison fails with the error:

Thu Sep 11 00:12:05 2014 Validating certificate key usage
Thu Sep 11 00:12:05 2014 ++ Certificate has key usage  00f8, expects 00a0
Thu Sep 11 00:12:05 2014 ++ Certificate has key usage  00f8, expects 0088

The reason I'm calling this a bug is because it's not sensible to use equality to compare against a bitmap. Instead one can use AND, in which case there would be:

( <certificate's value> & <desired value> ) == <desired value>
or:
( 0xf8 & 0xa0) == 0xa0  -> True

In order for the validation to succeed with the defaults the certificate should have one of the following designations:

0xa0: Digital Signature, Key Encipherment
0x88: Digital Signature, Key Agreement

The solution

So there, since the comparison is done with equality you can do one of the following:

Use the above Key Usage on the certificate (inconvenient)

Don't use "remote-cert-tls server" (bad)

Use "remote-cert-ku XX" where XX is the value of your certificate which can be seen in OpenVPN's messages (the last octet). In my case it's f8.

Thursday, 31 July 2014

Linux, multicast, bridging and IPv6 troubles (i.e. why my IPv6 connectivity goes missing)

For a long time now I had a very annoying problem with IPv6 under Linux.

My setup is as follows: Linux box <-> Switch <-> Router

The Linux box uses a bridge interface (br0) and usually only has one physical interface attached to it (eth0). That's a very convenient setup.

The problem is that after a couple of minutes the IPv6 connectivity of the host will go away. Now, the host has a static IPv6 assigned to it and it's not that it loses the address or any route. Instead it just stops communicating with everything.

Troubleshooting this showed that the box loses the MAC address of the router and the ND protocol does not work, so it never recovers.

When the problem occurs, the neighbor information becomes stale:

# ip neigh
2a01:XXX:YYY:1::1 dev br0 lladdr 00:11:12:13:14:c4 router STALE
fe80::20c:XXff:feXX:YYYY dev br0 lladdr 00:11:12:13:14:c4 router STALE

I.e the entry remains in a 'STALE' state and never recovers.

My workarounds so far have been:

Enable promiscuous mode on the interface (ifconfig br0 promisc)

Clear neighbors (ip neigh flush)

Everything pointed out to multicast issues (what IPv6 ND uses).

Long-story-short, this was an eye opener: http://troglobit.com/blog/2013/07/09/multicast-howto/

What needs to be done is to disable IGMP snooping on the bridge interface because it causes these issues. This is done with:

# echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping

So do yourself a favor and add this to /etc/network/interfaces, in the relevant interface:

    up    echo 0 > /sys/devices/virtual/net/$IFACE/bridge/multicast_snooping

Thursday, 8 August 2013

Installing package build dependencies from a .dsc file (Debian)

There are cases where one needs to install build-dependencies of a .dsc file in Debian.

Apparently this is not as trivial as:

[code gutter="false"]
# apt-get build-dep package
[/code]

The easiest way I've found so far is to use mk-build-deps (from the devscripts package):

[code gutter="false"]
# mk-build-deps -i vadm_1.0.4ci+r16.dsc -t apt-get --no-install-recommends -y
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_installdocs
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package `vadm-build-deps' in `../vadm-build-deps_1.0.4ci+r16_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
(Reading database ... 41052 files and directories currently installed.)
Preparing to replace vadm-build-deps 1.0.4ci+r16 (using vadm-build-deps_1.0.4ci+r16_all.deb) ...
Unpacking replacement vadm-build-deps ...

Reading package lists...
Building dependency tree...
Reading state information...

0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up vadm-build-deps (1.0.4ci+r16) ...
[/code]

This godly script:

Creates a psudo package that depends on the build-depends of the .dsc file

Does a dpkg -i on the generated deb file (which may fail because of missing build depends)

Does apt-get -f install

Extra points for using -r which will remove the generated package after it's done.

Saturday, 23 March 2013

Raspberry Pi under QEMU

What: Run raspberry pi system under QEMU
Why: I wanted to have builder environment for Raspberry PI. Emulating it it much faster that running something on it
Disclaimer: I am not a qemu/raspberry-pi expert - Some things may be wrong here

Introduction

Running qemu-arm is slightly different than running qemu for x86. There is no BIOS there and it needs to boot with a kernel image directly. The tricky part is that this image is not the standard Raspberry Pi (from now on mentioned as "raspi"). You can either download one from online or create your own.

I ended up creating my own qemu arm image

Disk image

Get the official raspberry pi disk image from here. Place it somewhere and uncompress it.

By default the image does not have much free space. A trick to change that is to extend it with something like the following:

# dd if=/dev/zero of=2013-02-09-wheezy-raspbian.img \
 bs=1024000 conv=notrunc \
 seek=4000 count=1

This will write 1MB after the first 4000 MBytes on the image. This will cause the image to become approximately 4GB but it will leave a hole in the file if the underlying filesystem supports it.

Kernel Image (optional)

If you want to compile your own kernel image for qemu then follow first the instructions here. Follow just the instructions on cloning git and patching the kernel.

Instead of configuring it yourself get this configuration (will speed your life) which has some additional stuff like ~~built-in sound~~, vfat support, IPv6, nice fonts, etc. Put this under the cloned kernel directory with name .config and then run:

# make ARCH=arm oldconfig
# nice make ARCH=arm -j 4   # Adjust for your number of cores
# cp arch/arm/boot/zImage /path/to/raspi/directory

You will need the arm compilation toolchain. If you're running Debian then follow the instructions in http://www.emdebian.org/

The package you are looking for is gcc-4.4-arm-linux-gnueabi

Note: Raspberry Pi supports hardfloat while the above compile is not using hardfloat. There should be no problems with that since the kernel is not using floating point math itself and it won't make a difference. The attached kernel configuration however has hardfloat support and so does qemu.

Booting

I am using vde networking with qemu so your setup may be slightly different. In any case, this is a nice setup to get you started:

# qemu-system-arm \
        -cpu arm1176 \
        -kernel kernel-v13-qemu-130323 \
        -name 'builder-raspi' \
        -hda 2013-02-09-wheezy-raspbian.img \
        -m 256 \
        -M versatilepb \
        -no-reboot \
        -serial stdio \
        -net vde,sock=/var/run/vde2/tap0.ctl \
        -net nic \
        -append "root=/dev/sda2 panic=0 ro

Use the proper path for the disk image and the kernel.

Now, here's the catch: The latest (as of 2013-03-23) raspberry pi image will have some issues booting with the above setup.

Fix latest filesystem image

Change the qemu -append parameter to: "root=/dev/sda2 panic=0 ro single" and boot the image.

This should leave you with a nice prompt. Then:

# mount / -o remount,rw

Next, edit fstab and change mmcblk0p1 and mmcblk0p2 to sda1 and sda2 respectively

Then create the file /etc/udev/rules.d/90-qemu.rules with the following contents:

KERNEL=="sda", SYMLINK+="mmcblk0"
KERNEL=="sda?", SYMLINK+="mmcblk0p%n",

This should ensure that /dev/mmcblk0p1/2 exist and will make other stuff work.

Also edit /etc/ld.so.preload (if it exists) and remove or comment everything in it.

Now shutdown the qemu image, restore the -append parameter and run it again. If raspi-config does not start by itself, run it yourself after logging in as user "pi". You can now select "expand_rootfs" and do another reboot to get more space on the root partition.

Sunday, 30 September 2012

IPsec, Racoon, setkey, Linux, Mikrotik, tunnel, transport and everything

It took me more than 6 months in order to sort all issues, so here are the experiences. Most of the trouble was because I didn't knew or I didn't had things clear in my mind.

I wanted to have IPsec communication between a bunch of servers and a home network. I believe that this includes almost all (if not all) the possible scenarios of IPsec so it's more complicated than it sounds. For obvious reasons I'm presenting a simplified version here omitting all duplicates (i.e. multiple hosts with the same characteristics).

The network

We have the following nodes:

A network behind a DSL line (home network) (normal, home DSL line with non-static IP, with NAT)

A server (srv1) somewhere on the Internet with a static public IP address without NAT.

A server (srv2) in Amazon's EC2 which has an allocated public IP address but uses local IP addresses and thus has NAT. Also Amazon doesn't allow ESP and AH protocol to be carried by IP packets inside their network.

We also have the following systems:

Home network: A bunch of Linux boxes on a private network plus a mikrotik router

srv1 and srv2: Squeeze Debian Linux

The home network uses IP addresses from the network 10.1.0.0/16. A secondary prefix (10.5.0.0/16) is allocated for IPsec addressing only. All home nodes have addresses from the 10.1.0.0/16. Some nodes (including the servers) have addresses from 10.5.0.0/16.

Apart from the above there's a custom CA setup which publishes certificates for all nodes.

The problem

Setup IPsec so that:

srv1 and srv2 can communicate with their public IP addresses with IPsec only

boxes on the home network can communicate both with srv1 and srv2 using IPsec

The setup

Since there are more than one boxes on the home network, the home network needs to be connected with tunneled IPsec to srv1 and srv2. srv1 and srv2 need to be connected with transport mode between them in order to encrypt communication that uses their public IP addresses.

We have setup the DSL router to forward everything to the mikrotik box (routerboard). This is usually referred as DMZ. By doing that it's possible to avoid NAT in IPsec (i.e. UDP encapsulation).

The solution

Mikrotik

In short, Mikrotik's IPsec works quite well and is easy to setup assuming that everything is correct. It is however harder to debug than Racoon. Here's the setup:

Add an IP address from 10.5.0.0/16

Import the box's certificate to the certificate storage, both certificate and public key are needed

Import CA's and other boxes' certificates to the certificate storage. Make sure you use sensible names to be able to look them up later.

Create a new proposal as follows:
- Name: short (or pick something else)
- Lifetime: 00:10:00 - This is essential in older to allow quick recovery when the IP address changes or racoon is restarted.
- Pick your favorite values for everything else

Add two peers, one for each server:
- srv1 (static public IP, no NAT):
  - Address: The public IP of srv1
  - Port: 500
  - Auth method: rsa signature
  - Certificate: Pick the local certificate (mikrotik's)
  - Remote certificate: Pick the certificate of srv1
  - Exchange Mode: main
  - Select: Send Initial Contact
  - Nat Traversal: No
  - My ID User FQDN: Leave empty - isn't needed
  - Proposal check: Claim (remember not to use similar or stricter on remote end)
  - Generate policy: No
  - Lifetime: 08:00:00
  - DPD Interval/Max failures: I use 10/3 but it doesn't make a difference. See notes bellow
- srv2 (static IP, public IP, with NAT): Use the same settings as with srv1
  - I didn't use NAT but it may be worth testing it.

You need to add two policies per peer. One for each local source IP address range (10.1.0.0/16 and 10.5.0.0/16). So you will end up with 4 policies:
- Src Address: 10.1.0.0/16 or 10.5.0.0/16
- Dst Address: srv1's or srv2's public IP address
- Src/Dst Port: Empty
- Protocol: all (255)
- Action: Encrypt
- Level: Unique - very important
- IPsec protocols: ESP
- Tunnel: Yes
- SA Src address: 0.0.0.0
- SA Dst address: srv1's or srv2's IPsec IP address (i.e. allocated addresses from the 10.5.0.0/16)
- Proposal: short (or whatever name you picked for the proposal you created)

Create a script named "ping-servers" (System -> Scripts) as follows:

{
 :local servers
 :local locals

 :set servers {"10.5.1.11";"10.5.1.12"}
 :set locals {"10.1.1.1";"10.5.1.1"}

 foreach loc in=$locals do={
 foreach srv in=$servers do={
 put "ping $srv src-address=$loc count=1"
 ping $srv src-address=$loc count=1
 }
 }
}

servers is the list of server's addresses from the 10.5.0.0/16 network and locals are local addresses to the mikrotik box, one for each of the two networks.

Schedule the script to be executed every minute (System -> Scheduler). This will keep the policies active and also reactivate them if they go down.

srv1 (static public IP, no NAT)

Put the following in /etc/ipsec-tools.d/srv2.conf:

spdadd srv1public srv2public[500] udp -P out none;
spdadd srv2public srv1public[500] udp -P in none;
spdadd srv1public srv2public[4500] udp -P out none;
spdadd srv2public srv1public[4500] udp -P in none;
spdadd srv1public srv2public 50 -P out none;
spdadd srv2public srv1public 50 -P in none;
spdadd srv1public srv2public 51 -P out none;
spdadd srv2public srv1public 51 -P in none;

spdadd srv1public srv2public any -P out ipsec
        esp/transport/srv1public[4500]-srv2public[4500]/require ;

spdadd srv2public srv1public any -P in ipsec
        esp/transport/srv2public[4500]-srv1public[4500]/require ;

Put the following in /etc/ipsec-tools.d/srv2-priv.conf. Somehow it is required in order to establish the IPsec connection when it's triggered by srv2:

spdadd srv1public srv2private[500] udp -P out none;
spdadd srv2private srv1public[500] udp -P in none;
spdadd srv1public srv2private[4500] udp -P out none;
spdadd srv2private srv1public[4500] udp -P in none;
spdadd srv1public srv2private 50 -P out none;
spdadd srv2private srv1public 50 -P in none;
spdadd srv1public srv2private 51 -P out none;
spdadd srv2private srv1public 51 -P in none;

spdadd srv1public srv2private any -P out ipsec
        esp/transport/srv1public[4500]-srv2private[4500]/require ;

spdadd srv2private srv1public any -P in ipsec
        esp/transport/srv2private[4500]-srv1public[4500]/require ;

In the above, srv1public is the public static IP address of srv1, srv2public is the public static IP address of srv2 and srv2private is the private static IP address of srv2.

Setup racoon.conf's section for srv2 and home as follows. Obviously you need to change to match your parameters:

remote "srv2" {
 exchange_mode main,base;
 verify_identifier on;
 peers_identifier asn1dn "Common name of srv2's certificate";
 remote_address srv2public;
 verify_cert on;
 certificate_type x509 "srv1.crt" "srv1.key";
 ca_type x509 "cacert.pem";
 my_identifier asn1dn;
 lifetime time 24 hours;
 nat_traversal on;
 proposal {
  authentication_method rsasig;
  encryption_algorithm 3des;
  hash_algorithm md5;
  dh_group modp1024;
 }
 passive off;
 proposal_check obey;
 generate_policy off;
 dpd_delay 10;
 dpd_retry 10;
 dpd_maxfail 6;
 initial_contact on;
 ike_frag on;
}

Setup racoon.conf's section for the home network as follows:

remote "home" {
 exchange_mode main,base;
 verify_identifier on;
 peers_identifier asn1dn "Common name of mikrotik's certificate ";
 verify_cert on;
 certificate_type x509 "srv1.crt" "srv1.key";
 ca_type x509 "cacert.pem";
 my_identifier asn1dn;
 nat_traversal off;
 proposal {
  authentication_method rsasig;
  encryption_algorithm 3des;
  hash_algorithm md5;
  dh_group modp1024;Â Â Â Â Â  # Group 2
 }
 passive on;
 proposal_check obey;
 generate_policy unique;
 dpd_delay 10;
 dpd_retry 10;
 dpd_maxfail 6;
 initial_contact on;
 ike_frag on;
}

Notice the differences: passive should be onÂ for the home network since it's not possible to trigger that without remote address.

Notice the generate_policy. It must be "unique" and not "on". Otherwise only one policy per remote endpoint will be generated and will also cause problems when an SA becomes bad.

Setup the additional address to a loopback interface and not to a physical interface.

Add static routes for the two networks using the normal gateway and specifying the source IP address. Otherwise you will be using the tunnel with addresses that are not routed via the tunnel and are not protected by IPsec. Obviously this will prevent anything from working on top of IPsec. Surprisingly, this will work occasionally when the traffic is initiated by the remote end just because of the route cache. Your config can be added to the loopback interface as follows:
```
auto lo:1
iface lo:1 inet static
  addressÂ Â  Â  10.5.1.12
  netmaskÂ Â  Â  255.255.255.255
  up          ip route add 10.5.0.0/16 via <gw> src 10.5.1.12 || true
  up          ip route add 10.1.0.0/16 via <gw> src 10.5.1.12 || true
  down        ip route del 10.1.0.0/16 via <gw> src 10.5.1.12 || true
  down        ip route del 10.5.0.0/16 via <gw> src 10.5.1.12 || true
```
where 10.5.1.12 is the address from the 10.5.0.0/16 network for srv1 and gw is the normal gateway of the server.

srv2 (static private IP, static public IP, NAT)

Setup the /etc/ipsec-tools.d/*.conf files in a similar way to the srv1's. You will need an entry for both the private and the public address.

Setup racoon like srv1's except from nat. You will have to set nat_traversal to on for srv1 and the home network.

The Hints / Lessons learned

Either test DPD (Dead Peer Detection) or don't use it at all. It didn't work for me at all.

You need to activate the policies from the home network's side proactively for both the IPsec networks (10.1.0.0/16 and 10.5.0.0/16). Otherwise it will be impossible for the remote ends to connect to local hosts. This is easily done by setting up a ping to run every minute. You need one ping per source IP address using -I.

You need to exclude ISAKMP traffic (UDP ports 500 and 4500) from static IPsec policies or otherwise you will have problems since outgoing traffic will be encrypted and incoming traffic will be dropped if not encrypted, which causes huge issues when one end goes down and requires the IPsec SA to expire from both ends (or flushed) before working again.

If you have firewall rules make sure that you allow ISAKMP traffic and IPsec traffic (protocols 50 (esp) and 51 (ah))

If you get errors that say that a policy is not available then it is not available! I can't stress this enough. While trying to make IPsec to work your brain will enter a bad state and it will start making mistakes. It's extremely easy to confuse static IPsec rules. I've done all sorts of mistakes including (but not limited to): using the wrong direction (in/out), using the address of another server, using tunnel instead of transport (and vice versa), not including the port numbers for esp-udp (UDP encapsulation) mode, not using the .conf extensions for files under /etc/ipsec-tools.d/, etc. Here's an example of that:
```
Sep 27 15:02:04 srvX racoon: ERROR: no policy found: A.B.C.D/32[0] E.F.G.H/32[0] proto=any dir=in
Sep 27 15:02:04 srvX racoon: ERROR: failed to get proposal for responder.
Sep 27 15:02:04 srvX racoon: [I.J.K.L] ERROR: failed to pre-process ph2 packet (side: 1, status: 1).
```

When testing a connection from host A that has both the 10.1.1.1 and 10.5.1.1 addresses to host B with address 10.5.1.2 then you may not be able to ping from B to one of the A's addresses. That's because only one of the IPsec policies is activated. To activate both of them use -I parameter for ping:
```
v13@hostA$ ping -I 10.1.1.1 10.5.1.2
v13@hostA$ ping -I 10.5.1.1 10.5.1.2
```

Pay attention to routing. You need to use the proper source IP addresses.

Tuesday, 24 July 2012

rsync as root with rrsync and sudo

Here's how to rsync something to a remote host as root without allowing root logins and with directory restriction. I did that because I wanted to sync /srv across servers.

In general it will use rsync over ssh, sudo, rrsync and a remote non-root user. I assume that rsync will run from srv1 to srv2.

rrsync

First you will need the rrsync (or rrsync.pl) script ad the server side that's part of the rsync package. In Debian you can find it at /usr/share/doc/rsync/scripts/rrsync.gz. This script acts as the server side and will restrict the destination directory (a'la chroot).

In short the server side will run "rrsync /srv". Then the client side will do something like this:

[code light="true"]
# rsync /srv remote:/
[/code]

and / will be relative to /srv that was defined as a parameter to rrsync.

You can put rrsync under /usr/local/bin.

User on srv2

At the destination server we will need a user that will be used for the ssh session. So go and create a user named 'syncer' on srv2. I'd avoid a username of 'rsync' as it may be used for other reasons at some point.

sudo on srv2

The user on srv2 should be able to run rrsync with sudo and with the -E parameter. -E is required in order to pass the checks of the rrsync script which checks for SSH_ORIGINAL_COMMAND in the environment. Feel free to make this even more strict to allow only this environment variable if you like.

Sample sudoers entry (e.g. to be put in /etc/sudoers.d/syncer):

[code light="true"]
syncerÂ Â Â ALL=SETENV:NOPASSWD:/usr/local/bin/rrsync /srv
[/code]

Obviously we need the user to be able to run this without requiring a password. SETENV will allow for the -E parameter to sudo.

SSH config

Next step is to allow root@srv1 to ssh as syncer@srv2 using public key. If you don't have a key pair generated for root@srv1 then go ahead and create it:

[code light="true"]
# ssh-keygen
[/code]

Then copy the contents of /root/.ssh/id_rsa.pub and paste them in syncer@srv2's authorized_keys file which is most probably at /home/srv2/syncer/.ssh/authorized_keys. Create the directory and the file if they don't exist.

To make rrsync work and make things safer you need to use the command=".." parameter and you should use the from=".." parameter. So your authorized_keys file will look something like this:

[code light="true" wraplines="true"]
from="srv1",command="sudo -E /usr/local/bin/rrsync /srv" ssh-rsa AAAA......siW root@srv1
[/code]

Don't forget to ssh at least once from srv1 to srv2 by hand in order to accept srv2's key and let ssh have it in in known_hosts.

Try it

Finally you are done and you can do the rsync:

[code light="true"]
# rsync --rsh=ssh -a --delete /srv syncer@srv2:/
[/code]

Sunday, 25 March 2012

Linux Containers: Easy LXC

Linux containers (a.k.a. LXC) rock. It's the ultimate way of having multiple Linux boxes with minimal requirements.

Here's how I do it under Debian (and the script I'm using):

Requirements

This guide is for DebianÂ testing as of 25 March 2012. However it should work for other cases as well.

The procedure creates a minimal installation which can then be fully customized by hand or with puppet. The procedure installs Debian under Debian but should be easy to change for other distributions as well (especially Ubuntu).

Packages

You will need to install:

lxc - The linux containers package

bridge-utils - For bridging network interfaces

uml-utilities - For tun/tap interfaces

cdebootstrap - For the bootstrapping of the virtual machines

puppet (optional) - for managing multiple machines

Networking

I prefer networking between lxc installations to be separate from my normal network. It is trivial however to bridge with the outside network as well.

Add the following to /etc/network/interfaces:

[code]
auto virtlxc
iface virtlxc inet manual
tunctl_userÂ Â Â Â root
upÂ Â Â Â Â Â Â Â Â Â Â Â Â ip link set virtlxc up
downÂ Â Â Â Â Â Â Â Â Â Â ip link set virtlxc down

auto brvirt
iface brvirt inet static
bridge_portsÂ Â Â Â Â Â Â Â Â Â Â virtlxc
bridge_maxwaitÂ Â Â Â Â Â Â Â Â 0
bridge_stpÂ Â Â Â Â Â Â Â Â Â Â Â Â off
addressÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 10.3.1.1
netmaskÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 255.255.255.0
dns-searchÂ Â Â Â Â Â Â Â Â Â Â Â Â virt.local
[/code]

Then add the following to /etc/hosts:

[code]
10.3.1.1Â Â Â deb0 deb0.virt deb0.virt.local
10.3.1.11Â Â Â deb1 deb1.virt deb1.virt.local
10.3.1.12Â Â Â deb2 deb2.virt deb2.virt.local
10.3.1.13Â Â Â deb3 deb3.virt deb3.virt.local
10.3.1.14Â Â Â deb4 deb4.virt deb4.virt.local
[/code]

Add as many entries as you like. There should be one entry per virtual machine. It doesn't matter if you have more entries than virtual machines since you may use them in the future. The first (deb) entry is for the local machine.

Bring up the brvirt and virtlxc interfaces and keep reading (ifup virtlxc; ifup brvirt).

You may also want to run something like this to provide network access to the virtual machines (assuming that eth0 is the interface the connects you to the rest of the world):

[code]
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I POSTROUTING -o eth0 -s 10.3.1.0/24 -j MASQUERADE
[/code]

Create the virtual machine

Get the following script and change the desired variables at the beginning as follows (assuming that you followed the network configuration):

SUITE: The Debian suite to use (e.g. squeeze)

MIRROR: A mirror to download debian from. If you use approx like me then you want to use the local machine (i.e. 10.3.1.1)

VIRTUSER: A username you want to have created in the virtual machine. After that you may ssh as that user.

LOCALUSERS: A space separated list of local users to get ssh public keys from and put then in VIRTUSER's authorized_keys file to allow ssh.

PUPPETMASTER: Leave it empty if you don't have a puppet master.

DNSSERVER: The DNS server to use. By default it is the local machine.

Each virtual machine should get a unique MAC address. The MAC addresses are auto-generated from the current y/m/d/H/M, so you should not create more than one virtual machines every minute. You're free to change this of course.

Now run the script at the end of the page and let it create a virtual machine:

[code]
./easylxc deb1
[/code]

The installation will happen under /var/lib/lxc (the default for lxc). You may visit that and fix things by hand if you (i.e.) manage to lock yourself out.

The virtual machine can be started with:

[code]
lxc-start -n deb4
- or -
lxc-start -n deb4 -d
[/code]

However, a bug/feature of rxvt will prevent that for succeeding. In that case you can run:

[code]
sudo lxc-start -n deb4
- or -
sudo lxc-start -n deb4 -d
[/code]

Inside the virtual machine you will be able to su to root by using "su" without password. You will be also able to ssh as root (using the same ssh keys).

Hints'n'tips

I strongly suggest using approx and changing the MIRROR variable as needed. It will speed the creation of many machines by orders of magnitude since there will be no network delays.

The script

[code lang="bash"]
#!/bin/bash

if [ -z "$1" ] ; then
echo "Pass the name of the machine as the first parameter"
exit 1
fi

# The name of the container to create. Also used as the hostname
NAME="$1"

# The name of the parent (local) machine without the domain
PARENTNAME="deb0"

# Distribution
SUITE="squeeze"

# The domain to be used by the virtual machines.
DOMAIN="virt.hell.gr"

# The network prefix (first 3 octets - it is assumed to be a /24 network)
NETPREFIX="10.3.1"

# Since we use approx, this is the approx server. If not, add a mirror.
MIRROR="http://ftp.debian.org/debian/"

# The gateway address for the virtual machine. This is most probably the
# address of the bridge interface.
GW="$NETPREFIX.1"

# The bridge interface to use for networking
BRIDGEIF="brvirt"

# The username of the user to create inside the container
VIRTUSER="v13"

# A list of local users that will have ssh access to the container
# They need to have a public key in the local machine
LOCALUSERS="v13 root"

# The puppet master. This must be the hostname of the master (not an IP addr).
# No puppet if this is empty.
PUPPETMASTER=""

# The DNS server to use.
DNSSERVER="$GW"

IPADDR2=$(getent hosts $NAME.$DOMAIN | awk '{print $1}')

if [ "x$IPADDR2" = "x169.254.1.1" ] ; then
IPADDR2=""
fi

if [ -z "$IPADDR2" ] ; then
echo "Could not resolve $NAME.$DOMAIN"
exit 1
fi

IPADDR="$IPADDR2/24"

MAC=$(date "+4a:%y:%m:%d:%H:%M")

lxc-stop -n $NAME
lxc-destroy -n $NAME

export SUITE
export MIRROR

R0=/var/lib/lxc/$NAME
R=$R0/rootfs

mkdir $R0 $R

# Install base system
echo cdebootstrap -f standard $SUITE $R $MIRROR
cdebootstrap -f standard $SUITE $R $MIRROR

CFG=$R0/config

# Create config file
cat << _KOKO > $CFG
# Auto-generated by: $*
# at $(date)

## Container
lxc.utsname = $NAME
lxc.rootfs = $R
lxc.tty = 6
lxc.pts = 1024

## Network
lxc.network.type = veth
lxc.network.hwaddr = $MAC
lxc.network.link = $BRIDGEIF
lxc.network.veth.pair = veth-$NAME

## Capabilities
lxc.cap.drop = mac_admin
lxc.cap.drop = mac_override
lxc.cap.drop = sys_admin
lxc.cap.drop = sys_module

## Devices
# Allow all device
lxc.cgroup.devices.allow = a
# Deny all device
lxc.cgroup.devices.deny = a
# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m

# /dev/console
lxc.cgroup.devices.allow = c 5:1 rwm
# /dev/fuse
lxc.cgroup.devices.allow = c 10:229 rwm
# /dev/null
lxc.cgroup.devices.allow = c 1:3 rwm
# /dev/ptmx
lxc.cgroup.devices.allow = c 5:2 rwm
# /dev/pts/*
lxc.cgroup.devices.allow = c 136:* rwm
# /dev/random
lxc.cgroup.devices.allow = c 1:8 rwm
# /dev/rtc
lxc.cgroup.devices.allow = c 254:0 rwm
# /dev/tty
lxc.cgroup.devices.allow = c 5:0 rwm
# /dev/urandom
lxc.cgroup.devices.allow = c 1:9 rwm
# /dev/zero
lxc.cgroup.devices.allow = c 1:5 rwm
# /dev/net/tun
lxc.cgroup.devices.allow = c 10:200 rwm

## Limits
#lxc.cgroup.cpu.shares = 1024
#lxc.cgroup.cpuset.cpus = 0
#lxc.cgroup.memory.limit_in_bytes = 256M
#lxc.cgroup.memory.memsw.limit_in_bytes = 1G

## Filesystem
lxc.mount.entry = proc $R/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry = sysfs $R/sys sysfs defaults,ro 0 0

_KOKO

# fix interfaces
T=$R/etc/network/interfaces
mv $T $T.orig
(
cat $T.orig \
| sed "s/^iface eth0.*$//"
echo "
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address $IPADDR2
netmask 255.255.255.0
gateway $GW
dns-nameservers $DNSSERVER
"
) > $T
rm $T.orig

# fix resolv.conf
T=$R/etc/resolv.conf
cat << _KOKO > $T
domain $DOMAIN
search $DOMAIN
nameserver $GW
_KOKO

# add info to hosts
T=$R/etc/hosts
echo "$IPADDR2 $NAME $NAME.$DOMAIN" >> $T
echo "$GW gw gw.$DOMAIN $PARENTNAME.$DOMAIN $PARENTNAME" >> $T

# set debian_chroot (for help)
echo "lxc-$NAME" >> $R/etc/debian_chroot

# create ttys
for i in $(seq 0 6) ; do
mknod $R/dev/tty$i c 4 $i
done

run()
{
echo chroot $R "$@"
LC_ALL=C chroot $R "$@"
}

run2()
{
ssh -o StrictHostKeyChecking=no $IPADDR2 "$@"
}

runmaster()
{
ssh -o StrictHostKeyChecking=no $PUPPETMASTER "$@"
}

# Install locales
run apt-get -y install locales

# disable init scripts
DISABLED="bootlogd bootlogs checkfs.sh checkroot.sh halt hostname.sh \
hwclockfirst.sh hwclock.sh module-init-tools mountall.sh \
mountdevsubfs.sh mountkernfs.sh mountnfs.sh mountoverflowtmp procps \
reboot stop-bootlogd stop-bootlogd-single udev umountfs umountnfs.sh \
umountroot"
for dis in $DISABLED ; do
run update-rc.d $dis disable
done

# disable rsyslog's kernel logging
run sed -i 's/^$.*imklog.*$$/#\1/' /etc/rsyslog.conf

# add user
run adduser --gecos $VIRTUSER --disabled-password $VIRTUSER
run adduser $VIRTUSER root

# fix sources.list
T=$R/etc/apt/sources.list
cat << _KOKO > $T
deb $MIRROR $SUITE main
_KOKO

# Install ssh
run apt-get update
run apt-get -y install openssh-server
run /etc/init.d/ssh stop

# Fix root and su
run passwd -l root
T=$R/etc/pam.d/su
mv $T $T.old
cat $T.old \
| sed 's/^# $.*pam_wheel.so trust$/\1/' \
> $T
rm $T.old

# Add ssh keys
T=$R/home/$VIRTUSER/.ssh/authorized_keys
T2=$R/root/.ssh/authorized_keys
mkdir $R/home/$VIRTUSER/.ssh $R/root/.ssh
for u in $LOCALUSERS ; do
H=$(getent passwd $u | cut -f 6 -d : )
cat $H/.ssh/id_rsa.pub >> $T
cat $H/.ssh/id_rsa.pub >> $T2
done
chown $VIRTUSER.$VIRTUSER $R/home/$VIRTUSER/.ssh $T
chown root.root $R/home/$VIRTUSER/.ssh $T2

# Start it
# Use sudo to bypass file descriptor problems
sudo lxc-start -n $NAME -d
sleep 1

if ! [ -z "$PUPPETMASTER" ] ; then
# Install packages
run2 apt-get -y install puppet

# Clear any existing certificate
runmaster puppet cert clean $NAME.$DOMAIN

# Fix puppet config
T=$R/etc/default/puppet
mv $T $T.old
cat $T.old \
| sed 's/START=no/START=yes/' \
| sed "s/DAEMON_OPTS=\"\"/DAEMON_OPTS=\"--server=$PUPPETMASTER --verbose\"/" \
> $T
rm -rf $T.old

run2 puppet agent --server=$PUPPETMASTER --no-daemonize --onetime

# sign the certificate
runmaster puppet cert --sign $NAME.$DOMAIN

run2 /etc/init.d/puppet start
fi

cat << _KOKO

LXC virtual box is ready!

Config file is at: $R0/config
Root fs is at: $R

Get a console with:
lxc-console -n $NAME

Stop it with:
lxc-stop -n $NAME

Start it with:
lxc-start -n $NAME -d

_KOKO
[/code]
Update: You can use the above code under the GPLv3 license.

#!/bin/bash

if [ -z "$1" ] ; then
echo "Pass the name of the machine as the first parameter"
exit 1
fi

# The name of the container to create. Also used as the hostname
NAME="$1"

# The name of the parent (local) machine without the domain
PARENTNAME="deb0"

# Distribution
SUITE="squeeze"

# The domain to be used by the virtual machines.
DOMAIN="virt.hell.gr"

# The network prefix (first 3 octets - it is assumed to be a /24 network)
NETPREFIX="10.3.1"

# Since we use approx, this is the approx server. If not, add a mirror.
MIRROR="http://ftp.debian.org/debian/"

# The gateway address for the virtual machine. This is most probably the
# address of the bridge interface.
GW="$NETPREFIX.1"

# The bridge interface to use for networking
BRIDGEIF="brvirt"

# The username of the user to create inside the container
VIRTUSER="v13"

# A list of local users that will have ssh access to the container
# They need to have a public key in the local machine
LOCALUSERS="v13 root"

# The puppet master. This must be the hostname of the master (not an IP addr).
# No puppet if this is empty.
PUPPETMASTER=""

IPADDR2=$(getent hosts $NAME.$DOMAIN | awk '{print $1}')

if [ "x$IPADDR2" = "x169.254.1.1" ] ; then
IPADDR2=""
fi

if [ -z "$IPADDR2" ] ; then
echo "Could not resolve $NAME.$DOMAIN"
exit 1
fi

IPADDR="$IPADDR2/24"

MAC=$(date "+4a:%y:%m:%d:%H:%M")

lxc-stop -n $NAME
lxc-destroy -n $NAME

export SUITE
export MIRROR

R0=/var/lib/lxc/$NAME
R=$R0/rootfs

mkdir $R0 $R

# Install base system
echo cdebootstrap -f standard $SUITE $R $MIRROR
cdebootstrap -f standard $SUITE $R $MIRROR

CFG=$R0/config

# Create config file
cat << _KOKO > $CFG
# Auto-generated by: $*
# at $(date)

## Container
lxc.utsnameÂ Â Â Â Â Â = $NAME
lxc.rootfsÂ Â Â Â Â Â = $R
lxc.ttyÂ Â Â Â Â Â Â Â Â = 6
lxc.ptsÂ Â Â Â Â Â Â Â Â = 1024

## Network
lxc.network.typeÂ Â Â = veth
lxc.network.hwaddrÂ Â Â = $MAC
lxc.network.linkÂ Â Â = $BRIDGEIF
lxc.network.veth.pairÂ Â Â = veth-$NAME

## Capabilities
lxc.cap.dropÂ Â Â Â Â Â = mac_admin
lxc.cap.dropÂ Â Â Â Â Â = mac_override
lxc.cap.dropÂ Â Â Â Â Â = sys_admin
lxc.cap.dropÂ Â Â Â Â Â = sys_module

## Devices
# Allow all device
lxc.cgroup.devices.allowÂ Â Â = a
# Deny all device
lxc.cgroup.devices.denyÂ Â Â Â Â Â = a
# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allowÂ Â Â = c *:* m
lxc.cgroup.devices.allowÂ Â Â = b *:* m

# /dev/console
lxc.cgroup.devices.allowÂ Â Â = c 5:1 rwm
# /dev/fuse
lxc.cgroup.devices.allowÂ Â Â = c 10:229 rwm
# /dev/null
lxc.cgroup.devices.allowÂ Â Â = c 1:3 rwm
# /dev/ptmx
lxc.cgroup.devices.allowÂ Â Â = c 5:2 rwm
# /dev/pts/*
lxc.cgroup.devices.allowÂ Â Â = c 136:* rwm
# /dev/random
lxc.cgroup.devices.allowÂ Â Â = c 1:8 rwm
# /dev/rtc
lxc.cgroup.devices.allowÂ Â Â = c 254:0 rwm
# /dev/tty
lxc.cgroup.devices.allowÂ Â Â = c 5:0 rwm
# /dev/urandom
lxc.cgroup.devices.allowÂ Â Â = c 1:9 rwm
# /dev/zero
lxc.cgroup.devices.allowÂ Â Â = c 1:5 rwm
# /dev/net/tun
lxc.cgroup.devices.allowÂ Â Â Â Â Â Â = c 10:200 rwm

## Limits
#lxc.cgroup.cpu.sharesÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â = 1024
#lxc.cgroup.cpuset.cpusÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â = 0
#lxc.cgroup.memory.limit_in_bytesÂ Â Â Â Â Â = 256M
#lxc.cgroup.memory.memsw.limit_in_bytes = 1G

## Filesystem
lxc.mount.entryÂ Â Â Â Â Â = proc $R/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entryÂ Â Â Â Â Â = sysfs $R/sys sysfs defaults,ro 0 0

_KOKO

# fix interfaces
T=$R/etc/network/interfaces
mv $T $T.orig
(
cat $T.orig \
| sed "s/^iface eth0.*$//"
echo "
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
addressÂ Â Â Â Â Â Â Â Â $IPADDR2
netmaskÂ Â Â Â Â Â Â Â Â 255.255.255.0
gatewayÂ Â Â Â Â Â Â Â Â $GW
dns-nameserversÂ Â Â Â Â Â $GW
"
) > $T
rm $T.orig

# fix resolv.conf
T=$R/etc/resolv.conf
cat << _KOKO > $T
domain $DOMAIN
search $DOMAIN
nameserver $GW
_KOKO

# add info to hosts
T=$R/etc/hosts
echo "$IPADDR2 $NAME $NAME.$DOMAIN" >> $T
echo "$GW gw gw.$DOMAIN $PARENTNAME.$DOMAIN $PARENTNAME" >> $T

# set debian_chroot (for help)
echo "lxc-$NAME" >> $R/etc/debian_chroot

# create ttys
for i in $(seq 0 6) ; do
mknod $R/dev/tty$i c 4 $i
done

run()
{
echo chroot $R "$@"
LC_ALL=C chroot $R "$@"
}

run2()
{
ssh -o StrictHostKeyChecking=no $IPADDR2 "$@"
}

runmaster()
{
ssh -o StrictHostKeyChecking=no $PUPPETMASTER "$@"
}

# Install locales
run apt-get -y install locales

# disable init scripts
DISABLED="bootlogd bootlogs checkfs.sh checkroot.sh halt hostname.sh \
hwclockfirst.sh hwclock.sh module-init-tools mountall.sh \
mountdevsubfs.sh mountkernfs.sh mountnfs.sh mountoverflowtmp procps \
reboot stop-bootlogd stop-bootlogd-single udev umountfs umountnfs.sh \
umountroot"
for dis in $DISABLED ; do
run update-rc.d $dis disable
done

# disable rsyslog's kernel logging
run sed -i 's/^$.*imklog.*$$/#\1/' /etc/rsyslog.conf

# add user
run adduser --gecos $VIRTUSER --disabled-password $VIRTUSER
run adduser $VIRTUSER root

# fix sources.list
T=$R/etc/apt/sources.list
cat << _KOKO > $T
deb $MIRROR $SUITE main
_KOKO

# Install ssh
run apt-get update
run apt-get -y install openssh-server
run /etc/init.d/ssh stop

# Fix root and su
run passwd -l root
T=$R/etc/pam.d/su
mv $T $T.old
cat $T.old \
| sed 's/^# $.*pam_wheel.so trust$/\1/' \
> $T
rm $T.old

# Add ssh keys
T=$R/home/$VIRTUSER/.ssh/authorized_keys
T2=$R/root/.ssh/authorized_keys
mkdir $R/home/$VIRTUSER/.ssh $R/root/.ssh
for u in $LOCALUSERS ; do
H=$(getent passwd $u | cut -f 6 -d :)
cat $H/.ssh/id_rsa.pub >> $T
cat $H/.ssh/id_rsa.pub >> $T2
done
chown $VIRTUSER.$VIRTUSER $R/home/$VIRTUSER/.ssh $T
chown root.root $R/home/$VIRTUSER/.ssh $T2

# Start it
# Use sudo to bypass file descriptor problems
sudo lxc-start -n $NAME -d
sleep 1

if ! [ -z "$PUPPETMASTER" ] ; then
# Install packages
run2 apt-get -y install puppet

# Clear any existing certificate
runmaster puppet cert clean $NAME.$DOMAIN

# Fix puppet config
T=$R/etc/default/puppet
mv $T $T.old
cat $T.old \
| sed 's/START=no/START=yes/' \
| sed "s/DAEMON_OPTS=\"\"/DAEMON_OPTS=\"--server=$PUPPETMASTER --verbose\"/" \
> $T
rm -rf $T.old

run2 puppet agent --server=$PUPPETMASTER --no-daemonize --onetime

# sign the certificate
runmaster puppet cert --sign $NAME.$DOMAIN

run2 /etc/init.d/puppet start
fi

cat << _KOKO

LXC virtual box is ready!

Config file is at: $R0/config
Root fs is at: $R

Get a console with:
lxc-console -n $NAME

Stop it with:
lxc-stop -n $NAME

Start it with:
lxc-start -n $NAME -d

_KOKO

Friday, 27 January 2012

Quick fix for X.org screensaver bypass

This vulnerability is quite annoying if you're locking your desktop in work or anywhere else.

In short, one is able to kill xorg's xscreensaver's lock by just pressing alt-ctrl-* or alt-ctrl-/ (both * and / need to be from the keypad).

A workaround that was posted suggests to modify files in the system. If you don't want to (like me - for various reasons) then you can do this on-the-fly.

Put the following script in a file and make it run whenever you log in to your X session (e.g. by putting it in ~/.kde/Autostart/ if you're using KDE):

[code lang="shell"]
#!/bin/bash

xkbcomp :0 - > /tmp/xkbcomp
cat /tmp/xkbcomp \
| sed -n '/key <KPMU> {/,/^ *}/ !p' \
| sed -n '/key <KPDV> {/,/^ *}/ !p' \
> /tmp/xkbcomp.new
xkbcomp /tmp/xkbcomp.new :0
[/code]

On each login, this will get rid of the offending xkb entries.

Friday, 6 January 2012

fix for radeon + opensource driver + kde effects = crash

The problem

Kwin crashes when enabling opengl effects. It doesn't crash immediately but it crashes after specific actions so it is 100% reproducible. For example when exiting from desktop-grid effect.

The situation

I'm using:

Radeon 4870 graphics card (RV770)

Kernel 3.1.5 (but seems irrelevant)

Open source ATI driver with KMS using Gallium

Xorg 1.11.2.902 (but happened with previous versions)

MESA 7.11.2

KDE 4.7.4 from debian

DRM 2.4.29

xserver radeon driver 6.14.3

I'm not using the blur effect

The solution

cd to ~/.kde/env/ (create it if it doesn't exist)

create a file named gl.sh (or any other name) with execute permissions (should not be needed) and with the following contents:

[code]
#!/bin/bash

export LIBGL_ALWAYS_INDIRECT=1
[/code]

The first line should not be needed as this file most probably gets source'd, but it will not hurt.

The drawback

Every GL app you'll be using will inherit the LIBGL_ALWAYS_INDIRECT from environment, which may cause problems. If you want to play (for example) a game then open a terminal and run:

[code]
unset LIBGL_ALWAYS_INDIRECT
nexuiz # or whichever opengl app you want to launch
[/code]
Note: Fireofx is one of the applications that may use GL.

Monday, 2 January 2012

Big nfs_inode_cache

The story

Boxes with various kernel versions have weird free memory problems. After examining the memory usage it seems that processes don't add up to the actual memory that is being used.

Taking a look at /proc/meminfo we see something like this:

[code]
MemTotal:Â Â Â Â Â 8161544 kB
MemFree:Â Â Â Â Â Â Â 115676 kB
Buffers:Â Â Â Â Â Â Â Â Â 3900 kB
Cached:Â Â Â Â Â Â Â Â 200520 kB
SwapCached:Â Â Â Â Â 42336 kB
Active:Â Â Â Â Â Â Â Â 546824 kB
Inactive:Â Â Â Â Â Â 138336 kB
HighTotal:Â Â Â Â Â Â Â Â Â Â 0 kB
HighFree:Â Â Â Â Â Â Â Â Â Â Â 0 kB
LowTotal:Â Â Â Â Â 8161544 kB
LowFree:Â Â Â Â Â Â Â 115676 kB
SwapTotal:Â Â Â Â 2096472 kB
SwapFree:Â Â Â Â Â Â 547480 kB
Dirty:Â Â Â Â Â Â Â Â Â Â Â 1020 kB
Writeback:Â Â Â Â Â Â Â Â Â Â 0 kB
AnonPages:Â Â Â Â Â 453480 kB
Mapped:Â Â Â Â Â Â Â Â Â 66928 kB
Slab:Â Â Â Â Â Â Â Â Â 7250176 kB
PageTables:Â Â Â Â Â 75408 kB
...
[/code]

Notice that Slab is about 7.5GB, almost the whole memory (8GB) (!).

Slab is the kernel memory and we can see where it is allocated by examining /proc/slabinfo. Here's an excerpt:

[code]
# nameÂ Â Â Â Â Â Â Â Â Â Â <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nfs_direct_cacheÂ Â Â Â Â Â 0Â Â Â Â Â 0Â Â Â 136Â Â 28Â Â Â 1 : tunablesÂ 120Â Â 60Â Â Â 8 : slabdataÂ Â Â Â Â 0Â Â Â Â Â 0Â Â Â Â Â 0
nfs_write_dataÂ Â Â Â Â Â Â 62Â Â Â Â 63Â Â Â 832Â Â Â 9Â Â Â 2 : tunablesÂ Â 54Â Â 27Â Â Â 8 : slabdataÂ Â Â Â Â 7Â Â Â Â Â 7Â Â Â Â Â 0
nfs_read_dataÂ Â Â Â Â Â Â 215Â Â Â 297Â Â Â 832Â Â Â 9Â Â Â 2 : tunablesÂ Â 54Â Â 27Â Â Â 8 : slabdataÂ Â Â Â 33Â Â Â Â 33Â Â Â Â 54
nfs_inode_cacheÂ Â 5384386 5399040Â Â 1032Â Â Â 3Â Â Â 1 : tunablesÂ Â 24Â Â 12Â Â Â 8 : slabdata 1799680 1799680Â Â Â Â 40
nfs_pageÂ Â Â Â Â Â Â Â Â Â Â Â 534Â Â Â 750Â Â Â 128Â Â 30Â Â Â 1 : tunablesÂ 120Â Â 60Â Â Â 8 : slabdataÂ Â Â Â 25Â Â Â Â 25Â Â Â 264
rpc_buffersÂ Â Â Â Â Â Â Â Â Â Â 8Â Â Â Â Â 8Â Â 2048Â Â Â 2Â Â Â 1 : tunablesÂ Â 24Â Â 12Â Â Â 8 : slabdataÂ Â Â Â Â 4Â Â Â Â Â 4Â Â Â Â Â 0
...
[/code]

Notice the nfs_inode_cache which is 5.3M objects of 1032 bytes each, adding up to about 5.4GB.

The workaround

Looking a bit about this on the internet we see that this is most probably a bug. Fortunately there are two workaround: A slow and a fast one:

Slow workaround: Login to that box and run "sync". Then leave it alone for a couple of minutes while the nfs_inode_cache memory goes down and down. It make take a couple of minutes before starting going down and there may be pauses in the process. It can take more than an hour to free the memory.

Fast workaround: Login to that box and run:

[code]
# sync
# echo 2 > /proc/sys/vm/drop_caches
[/code]

I'm not sure why the first one works, but it looks like it is triggering a chain reaction that frees the memory.

Sunday, 20 November 2011

Easy one-time git subtree merge

The situation:

Have a git project (A)

Have a second git project (B) that I want to merge to A under a directory

This needs to be done once. After that, project B will not be re-sync to A's subdirectory

Need to preserve history

The actual situation: I have a git project that is named "drlaunch" and another git project that is named "debian". "debian" is the packaging directory for drlaunch. The problem occurred because I used to have two svn trees, one for the project and one for the debian/ directory for making this a package for maemo.

I found a number of related things but all of them were complicated because they were doing more than I wanted. Finally, I came to this simple solution:

Under project drlaunch, there is a subdir drlaunch. I want to include the project debian under a directory named debian in the drlaunch project. The tree looks like this:

drlaunch (project)
\-- drlaunch (dir)

debian (project)

And at the end I want it to look like this:

drlaunch (project)
|-- drlaunch (dir)
\-- debian (dir)

The solution is as simple as this:

Go to the debian's project dir and export all changes with format-patch:[shell]
cd debian/
mkdir ../1
git format-patch --root -o ../1/
rm ../1/0000-*
[/shell]

(Note: The removal of the first file is required. The file should be empty and for me it triggers a git bug causing it to use 100% cpu indefinitely. Feel free to check it yourself)

Go to the drlaunch's project dir and import all changes to a directory:
[shell]
cd drlaunch/
git am --directory=debian/ ../1/*
[/shell]

Ta-da! Ready! Now compare the tree to be sure that nothing bad happend:
[shell]
diff -uR debian ../debian/
[/shell]

Don't forget to commit your changes.

Friday, 6 May 2011

Multiple Monitors with Opensource Radeon Driver and Xorg

Setting up multiple monitors is currently a nice experience. Doing this from krandrtray, which is a very very nice front-end, is easy. But doing it via xorg.conf can be ... well ... interesting. That's mostly because each driver has its own method of properly setting up multiple monitors.

Here's how to setup multiple monitors with xorg.conf when using the opensource radeon driver (tested with 6.14.1). The tricky part (and the one that took me aprox. 1 hour to figure) is to name the Monitors with the exact same name as the card's outputs.

First, you need to launch X at least once with both monitors connected just to find out the output names. Look at /var/log/Xorg.0.log:
[code language="bash"]
$ grep 'Output.*connected' /var/log/Xorg.0.log
(II) RADEON(0): Output HDMI-0 connected
(II) RADEON(0): Output DIN disconnected
(II) RADEON(0): Output VGA-0 disconnected
(II) RADEON(0): Output DVI-0 connected
[/code]
From the above search you can see that the connected outputs are named HDMI-0 and DVI-0. You may be able to determine which monitor is connected to which output either by looking up its resolution:
[code]
(II) RADEON(0): Output HDMI-0 using initial mode 1920x1200
[/code]
or other information in Xorg.0.log

Next you need to create or update /etc/X11/xorg.conf. The required relevant sections are as follows:
[code]
Section "Monitor"
Identifier "HDMI-0"
Option "Primary" "On"
EndSection

Section "Monitor"
Identifier "DVI-0"
Option "LeftOf" "HDMI-0"
EndSection

Section "Device"
Identifier "Card0"
Driver "radeon"
BusID "PCI:1:0:0"
Screen 1
EndSection

Section "Device"
Identifier "Card1"
Driver "radeon"
BusID "PCI:1:0:1"
Screen 1
EndSection

Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "HDMI-0"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1920x1200"
EndSubSection
EndSection

Section "Screen"
Identifier "Screen1"
Device "Card1"
Monitor "DVI-0"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1440x900"
EndSubSection
EndSection

Section "ServerLayout"
Identifier "Layout0"
Screen "Screen0" 1440 0
Screen "Screen1" LeftOf "Screen0"
EndSection
[/code]

Now I'm not sure about the second "LeftOf" because I stopped restarting X and KDM after it worked, but IIRC, it is not required.
As it was mentioned earlier, the tricky part is to named the Monitors as the outputs of the card (HDMI-0 and DVI-0 in my case).
Also, I remember that using the above naming for PciIDs is also required (observer that they are different) and I believe tht the "Monitor" statement inside the "Screen" section doesn't affect anything.

If everything is setup correctly you should see this in Xorg.0.log:
[code]
(II) RADEON(0): Output HDMI-0 using monitor section HDMI-0
(II) RADEON(0): Output DVI-0 using monitor section DVI-0
[/code]
which indicates that monitor sections where properly matched with outputs.

Friday, 2 April 2010

Linux ethernet driver ring buffer

While performing some tests with a congested 10Mbps link, a strange thing happened: The link was congested only on one direction and both endpoint queues were RED queues. Based on the parameters and the queue size, the delay between those two links should be something near 170ms. However, the delay was much larger: >300ms (!).

The "problem" was the ring-buffer of the underlying driver (e1000). This one used a buffer of 128 packets which when added to the average 150 packets in the queue, resulted in >300ms delay.

You can see this buffer by running:


#ethtool -g eth0

And you can modify this buffer by running:


#ethtool -G eth0 tx 80

This is the transmit buffer which (when filled) adds to the delay of the local queue.

Of course, in normal use, this buffer is a good thing as it will allow to get higher transfer rates easier (from the POV of the operating system). But when making experiments, this little thing gets in the way.

Another thing is that there seems to be a minimum value for this number. For example, on this card:

Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller

the minimum value is 80 (using e1000 driver from kernel 2.6.32)

So beware and don't loose a week looking for this thing like I did.

NOTE: This is not related to the transmit queue length that is used on the interface, as shown by:


# ifconfig eth0
# ip link show eth0

Sunday, 21 March 2010

Problems that went away when I switched from fglrx to opensource driver (radeon+kms+2.6.33)

For a long time ago, a computer connected to the Internet had an RV770 ATI card and used to use the proprietary fglrx driver. Yes... That was my pc...

Then the latest fglrx (10.2) wasn't compatible with the latest kernel (2.6.33) and that kernel supported Kernel Mode Setting (KMS) using the radeon driver. Debian also started to have appropriate libdrm and Xorg (+ driver).

So I switched to radeon/KMS driver... At first everything was not working and I blamed the radeon driver. However, as it was proved, even after uninstalling the fglrx driver it kept causing me problems. A couple of files were left behind and there was at least one file left diverted to the fglrx's one. To fix this problem one needs to examine diversions (dpkg-divert --list), use debsusm (e.g. debsums -s libgl1-mesa-glx) and reinstall the packages with checksum problems.

Finally, after switching to radeon/KMS the following things changed:

The used memory after startup reduced from about 2.5-3GB to less than 500MB (!!!!). I'm not talking about card's mapped memory. I'm talking about system's memory.

Everything runs a lot faster. It looks like system latency is greatly reduced. There are two kind of improvements: (a) KDE's desktop effects are smoother and (b) it looks like the latency is reduced. Somehow the effects seem to run faster because there is less delay.

I stopped getting crashes of plasma when logging-in.

KDE's effects stopped being disabled every now and then.

Tearing disappeared (!).

Compositing effects seem to use less CPU.

Some sound-card issues disappeared. Every now and then the sound was muted after system startup, but not any more.

So yeah... I really suggest that you switch to radeon/KMS if you're using the fglrx driver. It sucks.

BTW, I also tested that to a laptop with an older ATI card and it had most of the above improvements as well. A colleague of mine also switched to opensource driver and show the exact same, dramatic reduce of memory usage.

Sunday, 25 October 2009

Why I like Linux (and KDE4)

This video illustrates why Linux is superb:

MP4, 80MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-3.mp4
MP4, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.mp4
AVI, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.avi

In the video you see: KDE 4.3, mplayer, tvtime, konqueror, KVM, nexuiz, wine.
The video was made with kdenlive

Friday, 10 October 2008

Debian i386 to amd64 conversion

After doing this two times, I finally conclude that it *is* possible to convert an i386 debian installation to an amd64 one. At the end of the procedure most of the system will work. Here is how:

NOTE: Don't do anything that is mentioned here unless you fully understand what you are doing. You will most probably need to customize this procedure a bit!

You'll need enough free space for the procedure to work.

While you still have a working system, print this document and download the latest debian netinst image. This can be used as a rescue disk if something goes wrong. It is trivial to restore your system using a rescue CD since we will not delete anything at all.

Step 1: Backup

I cannot stress this enough. Backup all your data, especially databases and anything binary that needs an executable file to be accessed. Most probably files at your home directory don't risk at all but better be safe than sorry.

It is absolutely essential to backup your PostgreSQL database if you have one. The i386 and amd64 disk formats are not compatible (bad postgre... bad bad) and you'll need to restore the database. This most probably comes to:
# # Backup: # su - postgres $ pg_dumpall > mydump # # Restore $ cat mydump | psql template0

I've done this procedure twice without backing up anything :-) except from the PostgreSQL database which is required. The reason I didn't backed-up anything is that I was sure I could rollback to my old system using a rescue CD. If you don't know how just backup.

Step 2: Install an amd64 kernel
You'll need to boot your i386 with an amd64 kernel. Do:
# apt-get install libc6-amd64 linux-image-2.6-amd64 # # Most probably you would like to have X working while you run this procedure so install your favorite closed-source modules too: # apt-get install nvidia-kernel-2.6-amd64
and reboot

Step 3: Install a base amd64 system.
# mkdir /amd64 # dpkg --get-selections '*' > /amd64/sel1 # debootstrap --arch=amd64 lenny /amd64 http://myproxy/
(NOTE: The above code was corrected to include "--arch=amd64" which I forgot to include. Thanks to Slawek Wernikowski)

Step 4: Install an identical amd64 system
Since this is a time consuming procedure and many things can go bad, I suggest you run this under "screen". This way, if the X server crashes you will not loose what you've done.
# chroot /amd64 /bin/bash # vi /etc/apt/sources.list # # Add all sources.list entries that exist in your main system but not the CDs # vi /etc/apt/apt.conf # # Add the line: APT::Default-Release "your-release"; # # if needed # apt-get install locales # mount /proc # mount /sys # mount -t devpts none /dev/pts # export TERM=xterm # dpkg --set-selections < /sel1
Now run the following lines until everything is installed. Most probably parts of the installation may fail and you'll need to rerun those commands. It should be OK. At the end there can be some packages that will not be configured. Just ignore them for now.
# apt-get -o 'Dpkg::Options={"--force-confdef";"--force-confnew"};' dselect-upgrade # dpkg --configure --force-confdef --force-confnew -a
I repeat: You'll have to repeat the above commands until everything is installed. If you have trouble just fix it yourself. Don't forget that you're working in a chrooted amd64 system.

Step 5: Change the system
This is the part that I don't have written so there might be some this missing. Key points are here and you should be able to figure what you need to do if something goes wrong. It is really a very simple procedure:
# init 1 # mkdir /i386 # # Move /usr # mv /usr /i386 # mv /amd64/usr / # # Backup etc # cp -pR /etc /i386 # # Move /lib32 and /lib64 away if they exist # mv /lib32 /i386 # mv /lib64 /i386 # # Create links # ln -s /lib /lib64 # ln -s /emul/ia32-linux/lib /lib32 ## Move other dirs # mv /amd64/emul / # mv /sbin /i386 # mv /amd64/sbin /
Now is the tricky part where you move /lib and /bin:
# # /bin is easy # mv /bin /i386 # /i386/bin/mv /amd64/bin / # # Don't forget that you can only run commands from /i386 from now on until /lib is replaced. # # So, to do ls just write: /i386/bin/ls # # Time for /lib # mv /lib /i386 # # At this point you cannot execute anything at all but don't panic. Just run everything using the runtime linker. This is what happens whenever you run a command. # export LD_LIBRARY_PATH=/i386/lib # /i386/lib/ld-2.7.so /i386/bin/mv /amd64/lib / # # TATA! Welcome to your amd64 system! # ls

Step 6: Fix /var
This step is highly dependent on what you have installed. Most probably you can remove old /var and keep the new one. If you'd like to keep the old one there are some dirs that you need to replace:
# cd /var # mv cache cache.i386 # mv /amd64/var/cache . # # I suggest that you replace the whole /var/lib tree: # mv lib lib.i386 # mv /amd64/var/lib .

Whatever you do you should know that you should replace /var/lib/dpkg and /var/cache/apt with the new one.

Step 7: Make system bootable
Finally you should ensure that your system is bootable. A nice approach is to clean your /boot from whatever kernel is installed (don't forget that with your new system, no package will own existing kernels) and force-reinstall your current kernel:
# cd /boot # mkdir i386 # mv System* initrd* vmlinuz* config* i386 # apt-get install --reinstal linux-image-2.6.26-amd64 # # Replace linux-image-2.6.26-amd64 with your current kernel

This is it. Unless I've forgotten something you should just reboot to your know system. After rebooting you will have to fix the unconfigured packages and possible rerun dselect-upgrade:
# dpkg --configure -a # apt-get dselect-upgrade

Finally, restore your PostgreSQL database and you're done.
MySQL doesn't need dump and restore but wou'll have to copy/move old files from /var/lib.i386/myslq to /var/lib/mysql.