ttf-mscorefonts-installer, apt-cacher and .exe files

Since upgrading to 9.04 I was still having problems upgrading ttf-mscorefonts-installer. The 'apt-get dist-upgrade' would die with the following error:

--2009-05-03 12:00:35-- http://heanet.dl.sourceforge.net/sourceforge/corefonts/andale32.exe
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3142... connected.
Proxy request sent, awaiting response... 403 Sorry, not allowed to fetch that type of file: andale32.exe
2009-05-03 12:00:36 ERROR 403: Sorry, not allowed to fetch that type of file: andale32.exe.

Poking around, I found this post which explained that apt-cacher had a list of allowed file types and '.exe' wasn't on it. Unforunately, the apt-cacher script had changed and the patch in the post wasn't relevant any more. Here is my fix:

#cd /usr/share/apt-cacher
#cp apt-cacher-lib.pl apt-cacher-lib.pl.20090503
#vi apt-cacher-lib.pl
# diff -c apt-cacher-lib.pl.20090503 apt-cacher-lib.pl
*** apt-cacher-lib.pl.20090503 2009-05-03 12:06:48.000000000 -0500
--- apt-cacher-lib.pl 2009-05-03 12:07:15.000000000 -0500
***************
*** 57,62 ****
--- 57,63 ----
package_files_regexp => '(?:' . join('|',
('\.deb',
'\.rpm',
+ '\.exe',
'\.dsc',
'\.tar\.gz',
'\.diff\.gz',

The I restarted apt-cacher:

# /etc/init.d/apt-cacher restart

Then I reran 'apt-get dist-upgrade' and it installed fine.

Upgrading Mythbuntu from 8.10 to 9.04

Time to upgrade my Mythbuntu box to 9.04 from 8.10. I followed the Network Upgrade directions.

Not unsurprisingly, update-manage-core was already installed, so on to the upgrade (only showing highlights):

# do-release-upgrade
...
8 packages are going to be removed. 82 new packages are going to be
installed. 731 packages are going to be upgraded.
...
What is the password for the MySQL administrator account 'root':

Hmmm. I just enter here, I got an error about a database connection being refused and the upgrade continued. AFAICT, it worked out OK.

...
Will you be using this webserver exclusively with mythweb? Yes

Configuration file `/etc/services'
==> Modified (by you or by a script) since installation.
Y or I : install the package maintainer's version

I got asked about apt-cacher.conf - I hit "D" to get diffs and I got an error and the install went on. Fortunately I got another chance at this later...

What would you like to do about smb.conf? I kept the local version currently installed
...
Configuration file `/etc/apt-cacher/apt-cacher.conf'
N or O : keep your currently-installed version
...

The ending of the upgrade did not look good:

--2009-05-02 10:23:29-- http://switch.dl.sourceforge.net/sourceforge/corefonts/andale32.exe
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3142... connected.
Proxy request sent, awaiting response... 403 Sorry, not allowed to fetch that type of file: andale32.exe
2009-05-02 10:23:29 ERROR 403: Sorry, not allowed to fetch that type of file: andale32.exe.

andale32.exe: No such file or directory

All done, errors in processing 1 file(s)
dpkg: error processing ttf-mscorefonts-installer (--configure):
subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
ttf-mscorefonts-installer

I'm not sure what ttf-mscorefonts is, but doesn't seem to have any major impact.

Upgrade complete.

The upgrade is completed but there were errors during the upgrade
process.

Ok, not the best message at the end, but at leat things have finished.

First thing I noticed is I started getting emails from cron:

Subject: Cron [ -x /etc/init.d/mythtv-status ] && /etc/init.d/mythtv-status reload > /dev/null
Sorry, failed to fetch http://localhost:6544/xml.

Hmmm. Well, first things first, just on principle:

#reboot

System comes up fine in MythTV. Everything seems to be working well.

Hmmm, another email from cron:

Subject: Cron [ -x /etc/init.d/mythtv-status ] && /etc/init.d/mythtv-status reload > /dev/null
mv: cannot stat `/var/run/motd.new': No such file or directory

Update: This problem was caused by two cron jobs both trying to modify /etc/motd at the same time (every 10 minutes)! The update-motd cron job moves /var/run/motd.new out from underneath the mythtv-status job. You can see the two jobs in /etc/cron.d:

# grep 10 /etc/cron.d/mythtv-status /etc/cron.d/update-motd
/etc/cron.d/mythtv-status:*/10 * * * * root [ -x /etc/init.d/mythtv-status ] && /etc/init.d/mythtv-status reload > /dev/null
/etc/cron.d/update-motd:*/10 * * * * root [ -x /usr/sbin/update-motd ] && /usr/sbin/update-motd 2>/dev/null

I believe what should really be happening is myth-status should be run by update-motd every 10 minutes instead of independently. Here is what I did to make that happen:

# rm /etc/cron.d/mythtv-status
# cd /etc/update-motd.d/
# ln -s /usr/bin/mythtv-status 50-mythtv-status

Test it out...

# /usr/sbin/update-motd
# cat /etc/motd
Linux casey 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686
...typical motd stuff...

MythTV status for localhost
===========================
...typical mythtv-status stuff...

0 packages can be updated.
0 updates are security updates.

End Update.

Checked on vmware and it wasn't running. Usual problem:

# /etc/init.d/vmware start
VMware Server is installed, but it has not been (correctly) configured
for the running kernel. To (re-)configure it, invoke the
following command: /usr/bin/vmware-config.pl.

Usual fix:

# /usr/bin/vmware-config.pl

I took all the defaults. Afterwards, don't forget to fix /etc/vmware/config.

I then checked /usr/share/apt-cacher/apt-cacher-report.pl to see if I needed to patch it and it had been fixed in the distribution - yeah!

Well, I think that's it. Will update this post if I run into any more problems.

Update: This system also acts as my apt-cacher, and when I went to upgrade the next system I got "104 Connection reset by peer" errors. Turns out I had to use the new /etc/apt-cacher/apt-acher.conf file:

# cd /etc/apt-cacher
# mv apt-cacher.conf apt-cacher.conf.8.10
# cp apt-cacher.conf.dpkg-dist apt-cacher.conf
# /etc/init.d/apt-cacher restart
Restarting Apt-Cacher: apt-cacher.

For the record, here's the diff:

# diff apt-cacher.conf.8.10 apt-cacher.conf

26c26
< # optional setting, binds the listening daemon to one specified IP. Use IP --- > # optional setting, binds the listening daemon to specified IP(s). Use IP
40c40
< allowed_hosts="192.168.1.0/24,"> allowed_hosts=*
43c43
< # And similiarly for IPv6 with allowed_hosts_6 and denied_hosts_6. --- > # And similarly for IPv6 with allowed_hosts_6 and denied_hosts_6.
49c49
< # This thing can be done by Apache but is much simplier here - limit access to --- > # This thing can be done by Apache but is much simpler here - limit access to
112a113,117
> # This sets the interface to use for the upstream connection.
> # Specify an interface name, an IP address or a host name.
> # If unset, the default route is used.
> #interface=
>
125,126c130,131
< # To enable data checksumming, install libdbd-sqlite3-perl and set this option < # to 1. Then wait untill the Packages/Sources files have been refreshed once --- > # To enable data checksumming, install libberkeleydb-perl and set this option
> # to 1. Then wait until the Packages/Sources files have been refreshed once
133c138
< # apt-cacher server is beeing relocated (via apt-get's error messages while --- > # apt-cacher server is being relocated (via apt-get's error messages while
140,143c145,147
< # also the only method to use FTP access to the target hosts. The syntax is simple, the part of the beginning to replace, followed by a list of mirror urls, all space separated. Multiple profile are separated by semicolons < # path_map = debian ftp.uni-kl.de/pub/linux/debian ftp2.de.debian.org/debian ; ubuntu archive.ubuntu.com/ubuntu ; security security.debian.org/debian-security ftp2.de.debian.org/debian-security < path_map =" debuntu"> # also the only method to use FTP access to the target hosts. The syntax is
> # simple, the part of the beginning to replace, followed by a list of mirror
> # urls, all space separated. Multiple profile are separated by semicolons
146a151,163
> # path_map = debian ftp.uni-kl.de/pub/linux/debian ftp2.de.debian.org/debian ; ubuntu archive.ubuntu.com/ubuntu ; security security.debian.org/debian-security ftp2.de.debian.org/debian-security
>
> # Permitted package files - this is a perl regular expression which matches all
> # package-type files (files that are uniquely identified by their filename).
> # The default is:
> #package_files_regexp = (?:\.deb|\.rpm|\.dsc|\.tar\.gz|\.diff\.gz|\.udeb|index\.db-.+\.gz|\.jigdo|\.template)$
>
> # Permitted Index files - this is the perl regular expression which matches all
> # index-type files (files that are uniquely identified by their full path and
> # need to be checked for freshness).
> #The default is:
> #index_files_regexp = (?:Index|Packages\.gz|Packages\.bz2|Release|Release\.gpg|Sources\.gz|Sources\.bz2|Contents-.+\.gz|pkglist.*\.bz2|release|release\..*|srclist.*\.bz2|Translation-.+\.bz2)$
>

Had to re-patch apt-cacher-report.pl

Looks like around April 15 a new version of apt-cacher got pushed out, which overwrote my previous fixes. The file apt-cacher-report.pl didn't change though, so I just re-applied the same fixes and saved a copy of the patched script in case it happens again.

BTW, here are the latest numbers. apt-cacher has saved me 10GB in traffic over the past 6 months.

cache efficiency

	Cache hits	Cache misses	Total
Requests	31818 (65.15%)	17015 (34.84%)	48833
Transfers	10.641 GB (72.34%)	4.068 GB (27.65%)	14.71 GB

Fixing apt-cache-report.pl

Update: Looks like this is fixed as of Ubuntu 9.04

I previously reported apt-cache-report was not working. In this post I will set about fixing it.

What I discoved is the apt-cacher log format has changed. Old log:

# gzip -dc /var/log/apt-cacher/access.log.6.gz | head -1
Thu Oct 30 22:01:39 2008|192.168.1.12|MISS|189|security.ubuntu.com_ubuntu_dists_hardy-security_Release.gpg

New log:

# head -1 /var/log/apt-cacher/access.log
Tue Mar 10 01:45:59 2009|19924|127.0.0.1|EXPIRED|189|archive.ubuntu.com_ubuntu_dists_intrepid-updates_Release.gpg

So the new format has an extra column (second column, I have no idea what it is) and rearranged a little. I started in changing /usr/share/apt-cacher/apt-cacher-report.pl to compensate for this. Not that it still needed to be able to parse the old logs, so some logic was involved to distinguish.

# cd /usr/share/apt-cacher/
# cp apt-cacher-report.pl apt-cacher-report.pl.orig
# vi apt-cacher-report.pl
# diff -c apt-cacher-report.pl.orig apt-cacher-report.pl

*** apt-cacher-report.pl.orig   2009-03-13 15:35:14.000000000 -0500
--- apt-cacher-report.pl        2009-03-13 17:04:45.000000000 -0500
***************
*** 109,121 ****
 {
       #$logfile_line =~ s/ /\+/g;
       @line = split /\|/, $logfile_line;
!       $req_date = $line[0];
! #     $req_ip   = $line[1];
!       $req_result = $line[2];
!       $req_bytes  = 0;
!       $req_bytes  = $line[3] if $line[3] =~ /^[0-9]+$/;
! #     $req_object = $line[4];
!
       $lastrecord = $req_date;
       if(!$firstrecord) {
               $firstrecord = $req_date;
--- 109,133 ----
 {
       #$logfile_line =~ s/ /\+/g;
       @line = split /\|/, $logfile_line;
!       if (scalar(@line) == 5)
!       {
!               # 5 columns == old format
!               $req_date = $line[0];
! #             $req_ip   = $line[1];
!               $req_result = $line[2];
!               $req_bytes  = 0;
!               $req_bytes  = $line[3] if $line[3] =~ /^[0-9]+$/;
! #             $req_object = $line[4];
!       } else {
!               # Assume new 6 column format
!               $req_date = $line[0];
!               # I don't know what $line[1] is
! #             $req_ip   = $line[2];
!               $req_result = $line[3];
!               $req_bytes  = 0;
!               $req_bytes  = $line[4] if $line[4] =~ /^[0-9]+$/;
! #             $req_object = $line[5];
!       }
       $lastrecord = $req_date;
       if(!$firstrecord) {
               $firstrecord = $req_date;

Now to test things, by rerunning apt-cacher:

# /etc/cron.daily/apt-cacher
...this takes a while (~5 min)...

And then I browsed to http://localhost:3142/report/ and now I see an updated report:

summary

Item	Value
Report generated	2009-03-13 17:04:50
Administrator	root@localhost
First request	Thu Oct 30 22:01:39 2008
Last request	Fri Mar 13 16:46:42 2009
Total requests	37119
Total traffic	11.487 GB

cache efficiency

	Cache hits	Cache misses	Total
Requests	24007 (64.67%)	13112 (35.32%)	37119
Transfers	8.018 GB (69.8%)	3.468 GB (30.19%)	11.487 GB

If anyone knows where I should submit this patch to, please drop me a comment.

apt-cacher report not updating

Update Mar 13, 2009: I've now fixed apt-cacher-report.pl

I noticed this morning the amounts of hits and data transferred in my apt-cacher report are not updating. See report below, the date has changed but the values in the 'cache efficiency' table have not since my original install. Something is broken some where, either I'm not using the cache or it's not reporting correctly.

Don't have time at this moment to investigate, but will update this post when I do...

Updated: I started by killing apt-cacher to verify the clients were using it...

# /etc/init.d/apt-cacher stop

Stopping Apt-Cacher: apt-cacher.

And, yes, this caused 'apt-get update' on my client to return all sorts of 'Count not connect...' errors, so I know my previous configuration was working:

# cat /etc/apt/apt.conf.d/90local-proxy

Acquire::http::Proxy "http://proxy-host:3142";

So I restarted apt-cacher ("/etc/init.d/apt-cacher start") and looked at the server side harder.

First, I checked to see if there was anything in the cache, and there was lots of stuff including new packages as of today, including wine, which is only on one of my clients and not the server, which would seem to indicate things are working as I should expect:

# grep cache_dir /etc/apt-cacher/apt-cacher.conf

# cache_dir is used to set the location of the local cache. This can
cache_dir=/var/cache/apt-cacher

# du -s /var/cache/apt-cacher

1966256 /var/cache/apt-cacher
# ls -lt /var/cache/apt-cacher/packages | head -3
total 1947272
-rw-r--r-- 2 www-data www-data        0 2009-02-01 09:05 wine.budgetdedicated.com_apt_dists_intrepid_main_binary-i386_Packages.gz
-rw-r--r-- 2 www-data www-data        0 2009-02-01 09:03 wine.budgetdedicated.com_apt_dists_intrepid_Release

Ok, the issue would seem to be reporting. I checked relevant options in apt-cacher.conf and didn't see any issue.

# grep generate_reports /etc/apt-cacher/apt-cacher.conf

generate_reports=1

So I started looking a little harder at the apt-cacher logs and the /usr/share/apt-cacher/apt-cacher-report.pl script that generates the reports, and noticed a problem. Here's what a line in my logfile looks like:

# head -1 /var/log/apt-cacher/access.log
Sun Feb 1 08:22:01 2009|15927|192.168.1.125|MISS||ppa.launchpad.net_network-manager_ubuntu_dists_intrepid_Release.gpg

And here is the section of the apt-cacher/apt-cacher-report.pl script that parses it:

$req_date = $line[0];
# $req_ip = $line[1];
$req_result = $line[2];
$req_bytes = 0;
$req_bytes = $line[3] if $line[3] =~ /^[0-9]+$/;
# $req_object = $line[4];

If you don't understand PERl, basically there is a mismatch. The script is looking for the result (HIT or MISS) in the third column (column 2 starting from 0), but the IP address is there. It expects the number of bytes in column 4, but they are in column 2. So this would seem to be the problem, but why this discrepancy?

Update: See a later post for the fix.

---

Current apt-cacher report output (made narrow to fit into blow):

summary

Item	Value
Report generated	2009-02-01 09:02:18
Administrator	root@localhost
First request	Thu Oct 30 22:01:39 2008
Last request	Sun Feb 1 08:22:05 2009
Total requests	27678
Total traffic	1980.17 MB

cache efficiency

	Cache hits	Cache misses	Total
Requests	2569 (9.28%)	25109 (90.71%)	27678
Transfers	955.718 MB (48.26%)	1024.452 MB (51.73%)	1980.17 MB

Installing apt-cacher

In preparation for upgrading to 8.10 I decided to install a apt-cacher on my home network. Basically I followed the directions at the link in the prior sentence with some tweaks as described below.

First, install:

# apt-get install apt-cacher

Then configure the deamon to always be on by editing /etc/default/apt-cacher and changing AUTOSTART=0 to AUTOSTART=1:

vi /etc/default/apt-cacher

Then I edited /etc/apt-cacher/apt-cacher.conf to restrict access to my local network. I also added the path_map directive, though I'm pretty sure I'm not using it right now.

# cd /etc/apt-cacher
# cp apt-cacher.conf apt-cacher.conf.orig
# diff apt-cacher.conf.orig apt-cacher.conf
40c40
< allowed_hosts=* --- > allowed_hosts=192.168.1.0/24, 127.0.1.1
141a142,143
> path_map = debuntu repository.debuntu.org ; ubuntu archive.ubuntu.com/ubuntu; ubuntu-updates archive.ubuntu.com/ubuntu ; ubuntu-security security.ubuntu.com/ubuntu
>

Ok, first addition to the directions here. Later when the anacron job /etc/cron.daily/apt-cacher ran, I got the following errors:

Use of uninitialized value in concatenation (.) or string at /usr/share/apt-cacher//apt-cacher-lib.pl line 169, <$listpipe> line 244.
Use of uninitialized value in concatenation (.) or string at /usr/share/apt-cacher//apt-cacher-lib.pl line 169, <$listpipe> line 245.

I found this bug report describing the error. Following the report, I edited /usr/share/apt-cacher/apt-cacher-lib.pl as described in the bug report.

Ok, now you are ready to fire up apt-cacher. The missing log messages happen the first time and then went away.

# /etc/init.d/apt-cacher restart
Restarting Apt-Cacher: apt-cacherWarning: /var/log/apt-cacher/access.log missing. Creating.
Warning: /var/log/apt-cacher/error.log missing. Creating.
.

At this point you should be able to point your web broser at http://cache-host:3142/ (replace "cache-host" with the actual IP or hostname of your host) and see the apt-cacher status page. You can also visit http://cache-host:3142/report/ to get a report of how well your cacher is doing, though it won't have any interesting content until it's been running for a day.

Then I imported existing package from /var/cache/apt/archives/ to apt-cacher repository:

# /usr/share/apt-cacher/apt-cacher-import.pl /var/cache/apt/archives

Ok, now to configure the client systems, including the system on which apt-cacher is running, I followed the advice from EWB's comment and created /etc/apt/apt.conf.d/90local-proxy on each system (again, replace cache-host with your actual IP address or hostname):

# vi /etc/apt/apt.conf.d/90local-proxy
# cat /etc/apt/apt.conf.d/90local-proxy
Acquire::http::Proxy "http://cache-host:3142";

Now you can do a apt-get update. Expect this to take a while the first time you run it.

# apt-get update

That's it. Just sit back and enjoy the caching.

Update: After upgrading 5 systems (1 server, 3 VMs and my laptop) to 8.10 using the cacher, here the stats from the report for the day showed it saved me 48% of my apt-get traffic (956MB out of 1980MB).