OpenAFS/OpenAFS with MIT-KRB5

From Gentoo Linux Wiki

[edit] Introduction

This is not a comprehensive guide for setting up OpenAFS with Kerberos 5 authentication. NFSv4 also now uses Kerberos 5 for authentication.

AFS is a distributed network filesystem that also allows for replication. Organizations such as Google and the Internet Archive have been known to use AFS (along with Linux) for its scalability.

From www.openafs.org:

AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for file sharing, providing location independence, scalability, security, and transparent migration capabilities for data.

IBM branched the source of the AFS product, and made a copy of the source available for community development and maintenance. They called the release OpenAFS.

Kerberos offers a network authentication protocol for use in client/server network topologies. AFS originally implemented a version of Kerberos for its authentication purposes and was based on the Kerberos v4 protocol. However, since DES is no longer approved for Federal use, organizations are replacing AFS's authentication server (kaserver) with a Kerberos v5 authentication server.

Most people recommend using the Heimdal Kerberos implementation when using AFS since Heimdal has a library specifically for dealing with AFS (libkafs).

[edit] Conventions

We will use the following names in our exapmles:

EXAMPLE.COM: kerberos realm (please replace with the name of your realm)

example.com: afs cell

afsadmin: afs (and kerberos) administrator

[edit] Install MIT Kerberos

Just like anything else the first step is to emerge! Before emerging be sure to check the USE flags using emerge -p to see if there are any relative USE flags for your environment.

emerge app-crypt/mit-krb5

[edit] Setup a Kerberos Realm

Before you begin, now would be the time to decide on a Kerberos realm name if you haven't done so already. The usual convention is to name your realm after your domain name, using all UPPER CASE letters, however it can be any ASCII string.

e.g. EXAMPLE.COM

If you need to create multiple Kerberos realms then its best to use a descriptive string or name before your Kerberos realm, much like a domain name.

e.g. SPRINGFIELD.EXAMPLE.COM and ATLANTA.EXAMPLE.COM

Next you'll need to create (or edit) a basic krb5.conf file which is usually located in the /etc directory. This file (as well as other Kerberos config files) uses an INI style config.

In the example below admin_server, default_domain, and kdc should all be DNS names pertaining to the relevant Kerberos servers. If this is your first Kerberos server then kdc and admin_server will most likely be _this_ server.

Under the section [default_realm] you will need to add DNS to Realm name mappings.

[libdefaults]
        default_realm = EXAMPLE.COM

[realms]
        EXAMPLE.COM = {
                admin_server = server.example.com
                default_domain = example.com
                kdc = server.example.com
        }

[domain_realm]
        .example.com  = EXAMPLE.COM
        example.com = EXAMPLE.COM
 

The same /etc/krb5.conf file can then be copied to any clients that will be a part of or need access to your Kerberos realm.

[edit] Creating a KDC

Next, you'll need to create the Key Distribution Center (KDC) which is responsible for housing Kerberos principals, passwords and access control lists.

Now create a directory for storing the KDC files and databases:

hostname ~# mkdir /var/lib/krb5kdc

This location is really arbitrary as it will be specified in /etc/kdc.conf. If you choose to create the KDC database and files elsewhere just be sure to change /etc/kdc.conf accordingly.

Below is an example of a kdc.conf file. Be sure to create (or edit) this file in the /etc directory. For more information regarding this file be sure to read the man page.

hostname ~# man kdc.conf

[kdcdefaults]
        kdc_ports = 750,88

[realms]
        EXAMPLE.COM = {
                database_name = /var/lib/krb5kdc/principal
                admin_keytab = FILE:/etc/kadm5.keytab
                acl_file = /var/lib/krb5kdc/kadm5.acl
                key_stash_file = /var/lib/krb5kdc/.k5.EXAMPLE.COM
                kdc_ports = 750,88
                max_life = 10h 0m 0s
                max_renewable_life = 7d 0h 0m 0s
        }
 

Granting access to the Kerberos database is controlled by the kadm5.acl file which is specified by acl_file in the example above. Below is a quick and dirty example that allows ANY principal that has an instance name of admin access to everything. For more detailed information about the ACL file syntax, be sure to read the kadmind man page.

hostname ~# man kadmind

*/admin@EXAMPLE.COM *

Now its time to actually create the database using the kdb5_util command.

hostname ~# cd /var/lib/krb5kdc

hostname ~# kdb5_util create -r EXAMPLE.COM -s

You will be prompted for the Master Password. This will be stored as the principle K/M@EXAMPLE.COM.

[edit] Test KDC Installation and add an admin principal

At this point you should test your Kerberos installation by using the kadmin.local command. At the kadmin.local: prompt, you should then create an admin principal for kerberos and afs administration.

kadmin.local: ank afsadmin@EXAMPLE.COM
kadmin.local: ank afsadmin/admin@EXAMPLE.COM

[edit] Add Kerberos Principals

hostname ~# kadmin.local -q "addprinc -randkey afs/example.com"

[edit] Create the AFS Keyfile

Each afs server needs a kerberos keyfile, so we create one here:

kadmin.local:  ktadd -e des-cbc-crc:normal -k /etc/krb5.keytab.afs afs/example.com

It is important to remember (or write down) the key version number (kvno) that this command gives in its output, you will have to use it later.

[edit] Install OpenAFS

USE="kerberos" emerge net-fs/openafs

[edit] Server Setup

The next step then is to tell afs some details about your cell, both for the server and the client:

echo "example.com" > /etc/openafs/server/ThisCell
echo ">example.com #Example Cell" > /etc/openafs/server/CellServDB
echo "192.168.1.1 #afs.example.com" >> /etc/openafs/server/CellServDB

cat /etc/openafs/server/CellServDB>>/etc/openafs/CellServDB
cat /etc/openafs/server/ThisCell>/etc/openafs/ThisCell

Now that openafs is installed, the asetkey command should be available, so that you can set the correct key version number in the keytab:

asetkey add 3 /etc/krb5.keytab.afs afs/example.com

Replace the 3 with the kvno you got from the ktadd command above.

You can now start the basic overseer server to configure and start all other afs services:

bosserver -noauth

Once BOS server is running you can tell it some database services, like protection server and volume location server:

bos create afs.example.com ptserver simple /usr/libexec/openafs/ptserver -cell example.com -noauth
bos create afs.example.com vlserver simple /usr/libexec/openafs/vlserver -cell example.com -noauth

Now it's time to create the admin user:

pts createuser -name afsadmin -cell example.com -noauth
pts adduser afsadmin system:administrators -cell example.com -noauth
bos adduser afs.example.com afsadmin -noauth

After that you can setup the remaining services (file server, volume server and salvager) and stop the bos server:

bos create afs.example.com fs fs /usr/libexec/openafs/fileserver \
     /usr/libexec/openafs/volserver /usr/libexec/openafs/salvager \
     -cell example.com -noauth
bos shutdown afs.example.com -noauth
pkill -9 bosserver

Before you can start your new afs server via Gentoo's init-script, you should create at least one vice partition (or better: logical volume, to make growing it later a little bit easier) and mount it as /vicepa. You can choose any filesystem type available on Linux (I use xfs). You can also have multiple vice partitions or volumes if you want. They should be mounted as /vicepb, /vicepc, etc. Don't forget to add entries for your vice partitions to /etc/fstab.

You can now start all openafs services:

/etc/init.d/openafs-server start
rc-update add openafs-server default

[edit] Client Setup

Since you emerged openafs at the beginning of this section, you should also have openafs-kernel installed, which is needed on the client. This package provides the kernel module needed to access AFS file space.

To get a working client, you will now have to setup a new partition or logical volume as a cache for the cache manager. This partition or LV must not be formatted with a filesystem other than ext2! In gentoo, the default is to mount this filesystem at /var/cache/openafs.

After mounting this filesystem, we have to tell the cache manager where/ to find it and what should be the threshold value for deleting cache entries before the filesystem gets filled up. This value should usually be around 80 to 85% percent of the filesystem size. Setup properly /etc/openafs/cacheinfo.

 # Copyright 1999-2005 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2

 # ENABLE_AFSDB and ENABLE_DYNROOT determine whether AFSDB support and
 # Dynroot support (dynamically generated /afs), respectively, should be
 # enabled in the AFS client.
 ENABLE_AFSDB="yes"
 ENABLE_DYNROOT="no"
 

To start openafs-client, you need to:

1. initialize the cache manager with afsd;
2. authenticate as administrator and get a token;
3. create the principal volume to be mount on /afs root point;
4. restart afsd and kernel module (recommended)

emerge openafs-kernel
modprobe -v libafs
afsd -nomount

/etc/init.d/mit-krb5kdc start
rc-update add mit-krb5kdc default
kinit -V afsadmin
aklog -d

vos create afs.example.com vicepa root.afs -cell example.com

afsd -shutdown
modprobe -rv libafs
/etc/init.d/openafs-client start
rc-update add openafs-client default

[edit] Create Basic AFS Directory Structure

Create basic volumes and set access permissions:

vos create afs.example.com a root.cell -cell example.com
fs sa /afs system:anyuser rl
fs mkm /afs/example.com root.cell
fs sa /afs/example.com system:anyuser rl
fs mkm /afs/.example.com root.cell -rw

[edit] Obtain Tokens at Login

[edit] Install PAM_Krb5

emerge pam_krb5

[edit] Install PAM-AFS-Session

pam-afs-session is a PAM module intended for use with a Kerberos v5 PAM module to obtain an AFS PAG and AFS tokens on login. It puts every new session in a PAG regardless of whether it was authenticated with Kerberos and runs a configurable external program to obtain tokens. It supports using Heimdal's libkafs for the AFS interface and falls back to an internal Linux-only implementation if libkafs isn't available.

emerge pam-afs-session

[edit] Modify PAM Service(s)

#%PAM-1.0

auth required pam_env.so
auth sufficient pam_krb5.so
auth optional pam_afs_session.so program=/usr/bin/aklog
auth sufficient pam_unix.so try_first_pass likeauth nullok
auth required pam_deny.so

account sufficient pam_krb5.so
account required pam_unix.so

password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password sufficient pam_krb5.so use_authtok ignore_root
password sufficient pam_unix.so nullok md5 shadow use_authtok
password required pam_deny.so

session required pam_limits.so
session optional pam_krb5.so ignore_root debug
session required pam_afs_session.so program=/usr/bin/aklog
session required pam_unix.so
 

[edit] Running Services That Require Access to an AFS cell

[edit] k5start

k4start (for Kerberos v4) and k5start (for Kerberos v5) are modified versions of kinit that can use srvtabs or keytabs to authenticate, can run as daemons and wake up periodically to refresh a ticket, and can run single commands with their own authentication credentials and refresh those credentials until the command exits. Make sure that you have "afs" in your USE flags.

emerge app-crypt/kstart

[edit] Working Example

start() {
        checkconfig || return 1
        ebegin "Starting apache2"
        [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
        /usr/bin/pagsh -c "/usr/bin/k5start -b -K 10 -l 10h -p /var/run/apache2.k5start.pid \
        -f /etc/krb5.keytab -t apache/server.example.com; ${APACHE2} ${APACHE2_OPTS} -k start"
        eend $?
}

stop() {
        checkconfig || return 1
        ebegin "Stopping apache2"
        ${APACHE2} ${APACHE2_OPTS} -k stop
        [ -f /var/run/apache2.k5start.pid ] && kill `cat /var/run/apache2.k5start.pid`
        eend $?
}
 

[edit] Resources

Kerberos V5 Installation Guide
OpenAFS-mini-guide for newbies (mostly for Debian)
Gentoo Linux OpenAFS Guide
kstart -- Great for running services that require access to an AFS file system!