Linux ethernet driver ring buffer

While performing some tests with a congested 10Mbps link, a strange thing happened: The link was congested only on one direction and both endpoint queues were RED queues. Based on the parameters and the queue size, the delay between those two links should be something near 170ms. However, the delay was much larger: >300ms (!).

The "problem" was the ring-buffer of the underlying driver (e1000). This one used a buffer of 128 packets which when added to the average 150 packets in the queue, resulted in >300ms delay.

You can see this buffer by running:

#ethtool -g eth0

And you can modify this buffer by running:

#ethtool -G eth0 tx 80

This is the transmit buffer which (when filled) adds to the delay of the local queue.

Of course, in normal use, this buffer is a good thing as it will allow to get higher transfer rates easier (from the POV of the operating system). But when making experiments, this little thing gets in the way.

Another thing is that there seems to be a minimum value for this number. For example, on this card:

Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller

the minimum value is 80 (using e1000 driver from kernel 2.6.32)

So beware and don't loose a week looking for this thing like I did.

NOTE: This is not related to the transmit queue length that is used on the interface, as shown by:

# ifconfig eth0
# ip link show eth0


Problems that went away when I switched from fglrx to opensource driver (radeon+kms+2.6.33)

For a long time ago, a computer connected to the Internet had an RV770 ATI card and used to use the proprietary fglrx driver. Yes... That was my pc...

Then the latest fglrx (10.2) wasn't compatible with the latest kernel (2.6.33) and that kernel supported Kernel Mode Setting (KMS) using the radeon driver. Debian also started to have appropriate libdrm and Xorg (+ driver).

So I switched to radeon/KMS driver... At first everything was not working and I blamed the radeon driver. However, as it was proved, even after uninstalling the fglrx driver it kept causing me problems. A couple of files were left behind and there was at least one file left diverted to the fglrx's one. To fix this problem one needs to examine diversions (dpkg-divert --list), use debsusm (e.g. debsums -s libgl1-mesa-glx) and reinstall the packages with checksum problems.

Finally, after switching to radeon/KMS the following things changed:

• The used memory after startup reduced from about 2.5-3GB to less than 500MB (!!!!). I'm not talking about card's mapped memory. I'm talking about system's memory.


• Everything runs a lot faster. It looks like system latency is greatly reduced. There are two kind of improvements: (a) KDE's desktop effects are smoother and (b) it looks like the latency is reduced. Somehow the effects seem to run faster because there is less delay.


• I stopped getting crashes of plasma when logging-in.


• KDE's effects stopped being disabled every now and then.


• Tearing disappeared (!).


• Compositing effects seem to use less CPU.


• Some sound-card issues disappeared. Every now and then the sound was muted after system startup, but not any more.

So yeah... I really suggest that you switch to radeon/KMS if you're using the fglrx driver. It sucks.

BTW, I also tested that to a laptop with an older ATI card and it had most of the above improvements as well. A colleague of mine also switched to opensource driver and show the exact same, dramatic reduce of memory usage.

How to disable CDP-4-NATIVE_VLAN_MISMATCH (native vlan mismatch)

Cisco switches support CDP and use it to help us in a number of ways. One of them is to detect native VLAN mismatch between two connected ports. For 99% of the time this is a "good thing to do" (tm) but there are some corner cases where this is not what you want.

For example, if you have a switch that is connected with another switch and their connected ports are configured as access ports (and not trunk ports) then this message doesn't make much sense.

Well... it does...

Cisco switches also support VTP which eases the VLAN management task. For VTP to work, switches that are under the same "local network" are also under the same "VTP domain". A VTP domain logically groups switches.

Now, here is the problem: Two switches connected using access mode that are in the same VTP domain should share the same VLAN configuration, even if they are configured as transparent.

What to do: To bypass this problem you have to change the vtp domain on those switches so that it doesn't match. If you haven't changed that already, they most probably are not in any VTP domain at all or they are in the same VTP domain.

The solution:

1. Configure at least one of the two switches to be in transparent mode. You may not want that, but if you don't know what this means then just do it:
   
Switch(config)# vtp mode transparent



2. Change the VTP domain of that switch:
   
Switch(config)# vtp domain a_unique_name
   
   (you may want to use the hostname)

... and this annoying message:


Oct 27 12:16:29.352 EET: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet2/6 (2), with sw-el0 GigabitEthernet0/8 (1).

will be gone

Why I like Linux (and KDE4)

This video illustrates why Linux is superb:

MP4, 80MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-3.mp4
MP4, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.mp4
AVI, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.avi

In the video you see: KDE 4.3, mplayer, tvtime, konqueror, KVM, nexuiz, wine.
The video was made with kdenlive

The Day The Routers Died

Great video!



And don't forget to visit the IPv4 address report.

Plus a net-geeky video and how the three finger salute started.

hal-acl-tool, pam and high cpu usage

For a long time now i see my CPU usage rising and falling back every some seconds. This is very annoying and it really slows things down. Finally, I found (to some extend) what causes this problem. Using "udevadm monitor" I show that the CPU was rising whenever a uid-add event was created by the kernel:

$ udevadm monitor
KERNEL[1250930556.085871] add      /kernel/uids/111 (uids)
UDEV  [1250930556.087481] add      /kernel/uids/111 (uids)
KERNEL[1250930556.099782] remove   /kernel/uids/111 (uids)
UDEV  [1250930556.099818] remove   /kernel/uids/111 (uids)

I also found that the program that causes the CPU load is "hal-acl-tool" but not directly. In fact, hal-acl-tool wasn't consuming a lot of CPU and top wasn't showing anything. With process accounting I found that there are 1000s of invocations of /usr/lib/policykit/polkit-read-auth-helper:

$ lastcomm |grep polkit-read-aut | wc -l
61161

in a couple of minutes (!). This seems to be a bug that troubles other people too.

Finaly, I concluded that the problem is not related to uid-add events but to session creation. The problem is caused by console kit and its pam hook: pam_ck_connector.so. For newer debian versions (read: testing) you can disable this by running (as root) pam-auth-update and deselecting the "ConsoleKit Session Management".

WARNING: I'm not aware of the drawbacks of disabling "ConsoleKit Session Management", so do this at your own risk.

php foreach by reference

Here's an interesting "feature" (bug?) for php. Recent PHP versions support this syntax for foreach:

foreach ($myarray as &$v)
  $v['koko']='lala';

This allows easy changes to the actual table by using references and not acting on a copy.

- but -

If you do this:

$myarray=array(array('a'=>1), array('a'=>2), array('a'=>3));

foreach ($myarray as &$v)
  $v['b']=1;

foreach ($myarray as $v);

print_r($myarray);

You manage to remove the last element of $myarray (!!!). This is the output:

Array
(
    [0] => Array
        (
            [a] => 1
            [b] => 1
        )

    [1] => Array
        (
            [a] => 2
            [b] => 1
        )

    [2] => Array
        (
            [a] => 2
            [b] => 1
        )

)

Now, if you change the code to:

$myarray=array(array('a'=>1), array('a'=>2), array('a'=>3));

foreach ($myarray as &$v)
  $v['b']=1;

foreach ($myarray as $v2);

print_r($myarray);

The bug is gone. The output is correct:

Array
(
    [0] => Array
        (
            [a] => 1
            [b] => 1
        )

    [1] => Array
        (
            [a] => 2
            [b] => 1
        )

    [2] => Array
        (
            [a] => 3
            [b] => 1
        )
)

To my knowledge, this happens because $v is kept as a reference to the last element when the first foreach is finished. Then, when the second foreach is ran, some assignments are performed to $v, destroying its last element.