Talk:HOWTO Using a shared portage via NFS

[edit] Readwrite?

Why is the remote filesystem mounted read-write? What do clients have to write?

grtz

Simple, if you emerge Xorg on your desktop, you need the tgz/tbz2 's. You could fetch them on your server, but that's time consuming. (Or you install the same packages on your server, and that becomes really silly). So you need rw on distfiles dir. That's almost what I did actually. I think it's more secure.

If distfiles is RW, corrupt tgz's aren't really a problem. md5's are calculated/checked on a sync and stored in the RO part of /usr/portage/*. (Ok corruption is possible if you'd replace the files after download/before md5 calc, but that is hardly possible).

However, instead of mounting it RO (you need to chmod distfiles/ -R and distfiles/.* -R +rwx) I mount it without the no_root_sqash. Now only a lot less users are allowed to write.

Only problem I found so far was with lockfiles and the like (that would be a problem especially with RO nfs share). It appears to me that emerge (on the desktop /nfsclient) wants to store lockfiles for the emerge. I belive it does this in /usr/portage/metadata (/cache maybe?) So i just chmod a+rwx -R on /usr/portage/metadata for now, but it appears a sync resets this? I'm still (very slowly) trying to figure out that part.

But I think, yes, having RW on NFS is needed. No, no_root_squatch however shouldnt' be on. I don't think it's needed. maybe with a little tweaking of emerge anyway.

oliver

I use this technique to manage a large number of systems. There is no need to export read-write. You will notice on the clients however that when emerging certain things (portage package comes to mind) it will try and 'fix' permissions in the portage tree, which will of course throw some errors. These errors can be safely ignored and everything will work fine.

Dan-

I export read-write because I want my client systems to have a place to put their downloaded distfiles. My fileserver is just that, and doesn't have most of the software that the client systems do.

-dcb

--alternately, you could set up a read-write distfiles share outside of the portage share, and keep the portage share itself read-only.

[edit] Security issues with wireless?

What if someone changes his ip on the network to access your portage? Could anyone alter your ebuilds and exploit this to let you install a backdoor?

If you export read-write, then yes this wouldn't be very difficult at all. The easy solution would be to export read-only. Securing your wireless network as much as possible (eg. use VPNs for point to point communication instead of relying on WEP, etc.) wouldn't hurt either.

Dan-

[edit] Might use a portage binhost instead

My NFS server is occasionaly corrupting file transfers. I've done everything exactly the same, except I have used rsize=8128,wsize=8128 (which improve performance and are reconmended in the man pages. why are they not in your howto? ).

Anyway bzip and gzip keep sensing errors after file transfers. This doesn' thappen after using ftp for the same file. (intermittant fault).

>>> extracting cups-1.1.23 tar: \uffff\uffff\uffff\uffff \uffff\uffff\uffff\uffff\uffff\uffff \uffff\uffff\uffff\uffff \uffff\uffff tar: Archive contains obsolescent base-64 headers

bzip2: Data integrity error when decompressing. tar: Read 520 bytes from - tar: Error exit delayed from previous errors !!! Error extracting /usr/portage/packages/All/cups-1.1.23.tbz2

Cheers,

Luke

You might try using 8192 instead of 8128. NFS may prefer multiples of 1K.

BTW, here are some interesting limits from Linux NFS faq

Maximum read/write sizes
NFSv2         8192
NFSv3 (udp)  57344 (theoretical)
NFSv3 (tcp)  32768

As of 2.4.20 the max NFSv3 block size is 32768 (udp or tcp).

And note: it seems to be a good idea to set the [r|w]size at least as large as the OS page size (usually 8K on x86 Linux systems)

-dcb

One further suggestion: Look at the date of the nfs(5) man page: 20 November 1993.

JeR --80.127.65.12 13:10, 16 August 2005 (GMT)

[edit] UDP and/or TCP

Q. After following this HOWTO I spent 3 or four hours trying to get the mounting to work. Turns out tcp works and udp doesn't I'm assuming that this is because I have NFS over TCP compiled into both kernels. If this is the case, shouldn't it be mentioned. Otherwise congrats on a great HOWTO.

A. From my research for the past months, TCP/IP is the only functional option for NFSV4. From what the devs had blurbed, UDP is too troublesome compared to TCP/IP these days, or they just haven't had time to work on UDP. You can still trying by specifying "proto=tcp" or "proto=udb" as a mount/fstab option. My experience today with vanilla-2.6.22.1 showed a long >2 minute hang before any packets would transfer. TCP was, by far, extremely more robust! And, since it looks like you're working with a customized NFS as well, have a look below at "15) Slow Emerge -ugDNv world on Clients?".

[edit] Does "emerge sync" only download files?

When you access your portage dir via NFS, you won't do an emerge sync anymore. But isn't "emerge sync" doing more stuff than just downloading files?

Performing global updates or updating the portage cache for example. Or are those tasks unnecessary?

emerge --metadata on the client systems will update the portage cache (this is mentioned at the very end of Step 4).

-dcb

[edit] Placement of '--nospinner' and command ordering

In the crontab line, '--nospinner' is an argument to 'true' and not to 'emerge'
"emerge sync > /dev/null 2>&1 || true --nospinner" && emerge world -vup

I ask for my own curiosity why it was done that way since 'true' would silently discard the option. I would have expected to see it written as:
"emerge sync > /dev/null 2>&1 || true && emerge --nospinner world -vup"

Not meant to nitpick, just curious if I somehow have misunderstood something with fundamental argument parsing.

It would also appear that emerge --nospinner world -vup is going to run regardless if the "emerge sync" succeeds or fails. For example:

$ true || echo 1 && echo 2
2

$ false || echo 1 && echo 2
1
2

In both cases, the number "2" was printed.

It would appear the short circuit operators need to be reversed:

$ true && echo 1 || echo 2
1

$ false && echo 1 || echo 2
2

That should provide for a true '?:' conditional expression, leading us to this:
"emerge sync > /dev/null 2>&1 && emerge --nospinner world -vup || true"

Unfortunately, if "emerge --nospinner world -vup" fails, it will still return as true (e.g.: true && false || echo 2) and may not be what you want.

Thank you for the informative article.

[edit] Time problems

I would like to note that it is imperative to sync up your servers to one time source.

If your servers attempt to emerge at the same time you will get access violation errors and many problems.

My advice is to setup NTP on one of the servers or use time.nist.gov.

I don't get this. What servers are you refering to?

If you share the Portage tree through a mounted filesystem, you only need one (NFS) server, and multiple clients can then do emerge --metadata very safely.

If you share a local Portage tree through rsync, multiple clients can safely do emerge --sync at the same time.

The only case I can think of where access violations would occur would be when you run emerge --sync on multiple clients while they share a single Portage tree. Also, you may expect minor problems if FEATURES=distlocks and you happen to use an older/other NFS implementation (such as on ppc-macos).

However, setting up a single reliable local timesource and synchronising all local systems to that timesource regularly is definitely a good idea. :)

--80.127.65.12 12:59, 16 August 2005 (GMT)

[edit] Concurrency issue with the /usr/portage/distfiles directory

'''''''''''''''''''Isn't there a potential concurrency issue with the distfiles directory that needs to be addressed here? What if 2 of your servers are emerging the same package at approximately the same time? One begins the source tarball download but while that's happening, the other thinks it needs to download the source package and begins stepping on the download currently in progress. Or, as also might happen, one begins trying to emerge with a partially downloaded source tarball. That's a distinct possibility if 2 or more of the emerge clients are kicked off at the same time. One way around this is to set up a distinct DISTFILES directory for each server using portage on your LAN, but that destroys the advantage of only having one copy of a downloaded source tarball on your system. I don't see any way to avoid this without a change to portage. Any ideas here?'''''''''''''''''''

Yeah, that will cause a problem as the tarballs get stamped on. Some kind of lock file might be the way to go. At the moment, I am careful to emerge -avf (fetchonly) on one box and then let the emerge -av on multiple boxes once I am confident I've got all the source code down.

-- I'm pretty sure that portage implements a fairly simple locking system using the .locks directory, and I've seen my concurrent fetch attempts yield to the first one to get there, waiting for the lock before using the file. So I don't think it's that much of an issue these days.

[edit] Need to recreate /etc/make.profile symlink

I found in order to emerge anything, I had to recreate the /etc/make.profile symlink. the command needed is:

1. ln -s /mnt/nfs_portage/profiles/defualt-linux/<arch>/<gentoo-version> /etc/make.profile

otherwise all my boxes complain it's missing.

probably should declare it in your .zshrc (or .bashrc if that's what you use)

NQS

Not needed if you mount directly in /usr/portage -- Kalin

[edit] Metadata caching?

I am kind of concerned with the very slow `emerge --metadata` on every client... Isn't there any other way to use a centrally updated metadata? -- Kalin

Also wondering why (or if) it would be bad to just export /var/cache/edb/dep/usr/portage, too. Wouldn't it be possible to skip the whole emerge --metadata thing and just mount the cache dir directly? -matt

I've tried it and it works great! -Stefan

No. It doesn't work great now with NFS4. Reason being, exported folders are mounted into your /export folder using "bind" type filesystem. Invoking eix-sync (emerge --sync) removes the parent /var/cache/edb/dep folder, causing the binded filesystem mounted /export/var/cache/edb/dep/usr/portage to not update with the local filesystem. All depending networks will read the exported folder as empty. To partially resolve this while using NFS4, you can export a higher folder, which is also more appropriate if you also have a /usr/local/portage tree. Export /var/cache/edb/dep. But now, my remote boxes can't change permissions on this higher folder as it likes, as such, emerge borks with a permissions chown error. When I "ls -alh" the NFS4 folder(s) on the client side, I see numbers instead of user:group. This might be a NFS4 related issue. (Oh & I am using the no_root_squash arg too.) One thing I've been able to do is, get the server to resolve uid:guid to /etc/idmapd.conf for NFS4 instead of getting 4294967294:4294967294 for uid:guid. To resolve this, I unmasked the latest nfs-utils, libnfsidmap & portmap. But this still leaves with with a permission problem for chown on /usr/portage! Would be great if we could designate /usr/portage within /etc/idmapd.conf to resolve specifically to root:portage. --Roger 20:15, 27 July 2007 (UTC)

One thing I've noticed, make sure you have /etc/make.conf properly configured for clients. If the client can't find an already built package (within PORTAGE_BINHOST / PKGDIR), emerge will then try to download to /usr/portage/distfiles, of which might be exported from the server. Emerge trying to chown/chmod while trying to download source will fail on the exported folder. Make sure you have defined DISTDIR (see below) to download sources to an alternate location instead of ontop of of the export. This should hack around the issue for now, but might still be present.

/etc/make.conf
PORTAGE_BINHOST="http://localhost2/gentoo/grp/pentium3/"
PKGDIR="/mnt/portage/packages/"
DISTDIR="/usr/local/portage/distfiles"

Seems separating the local DISTDIR from the NFS share /usr/portage resolved this issue. (When the client needs to rebuild something, it will have assigned & write access to /usr/local/portage/distfiles. Other then this, the mounted /usr/portage share from the server will provide read access to it's already downloaded files within the server's /usr/portage/distfiles folder.)

The only other problem I've run into, is stuff utilizing /usr/portage/distfiles/svn-src such as mythtv ebuilds running svn update. Even though emerge is using /usr/local/portage/distfiles/svn-src, permissions for the /usr/local/portage/distfiles/svn-src/mythtv/mythtv/.svn/tmp/ were set to readonly causing SVN to hang during emerging mythtv. "chmod a+rwX" the folder temporarily resolves the problem. Users will see error messages pertaining to permission access errors. All in all, this NFS4 section under Metadata Caching has resolved into something else and (I) should really sub-topic is "NFS4 Permissions Errors". All in all, things seem to be working much better now. ;-) --Roger 18:46, 4 January 2008 (UTC)

I've just resolved the issue with uid:gid exporting NFS4 shares to nobody:nobody for uid:gid. I found /etc/init.d/rpc.idmapd not starting along with /etc/init.d/nfs. Once rpc.idmapd was started (/etc/init.d/rpc.idmapd start), all exports/shares reported proper uid:gid instead of nobody:nobody. (I've filed a bug concerning why rpc.idmapd isn't starting when nfs rc starts.) This seems to have resolved all issues with Portage working on NFS4 shares that I'm aware of, especially chown/chmod usage by Portage! I'm also just mounting NFS4 shares over /usr/portage and /etc/portage and keeping all package configuration central, on the server except for /etc/make.conf. Only lacking is a method of assigning package names within /etc/portage/package.* by domain name, so I can avoid installing certain versions of nvidia-drivers on two clients. ---Roger 01:59, 8 August 2008 (UTC)

Ok. Finally getting NFS4 mounted metadata working here.

As to the original topic concerning sharing of the metadata folder, I have not yet tried. I have just been running eix-sync -mvv or update-eix. with the mounted/shared /usr/portage and /etc/portage. --Roger 02:06, 8 August 2008 (UTC)

Wow. Just did some research, and found some information on the following url: http://forums.gentoo.org/viewtopic-t-622949.html The posts explain how to write /etc/portage/modules with "portdbapi.auxdbmodule = cache.metadata_overlay.database" onto the clients and wiping the /var/cache/edb/dep/usr/portage folders on the clients. Hence, no need to sync metadata. However, I also share /etc/portage via NFS! (This keeps package keywords/masks synced.) Hence... I can't use this method either because I cannot assign specific settings to only specific hosts! What a mess. The only thing I can probably do, is share metadata via NFS as well as /usr/portage & /etc/portage. At least finally, the posts at the above url dated 2004 finally makes this wiki. :-/ ---Roger 01:50, 11 August 2008 (UTC)

Ok. Finally getting successful results with NFS4 mounted metadata working here. The key was as I said before, making sure "/etc/init.d/rpc.idmapd" is started to make sure NFS shares report the proper uid:gid instead of nobody:nobody. Here's my config:

file:/etc/fstab

/usr/portage       /export/usr/portage              none    bind    0 0
/etc/portage       /export/etc/portage              none    bind    0 0
/var/cache/edb     /dep/export/var/cache/edb/dep    none    bind    0 0

file:/etc/exports

/export/etc/portage             192.168.1.0/255.255.255.0(rw,insecure,no_subtree_check,async,no_root_squash)
/export/usr/portage             192.168.1.0/255.255.255.0(rw,insecure,no_subtree_check,async,no_root_squash)
/export/var/cache/edb/dep       192.168.1.0/255.255.255.0(rw,insecure,no_subtree_check,async,no_root_squash)

Stuff for sharing the metadata folder over (NFS4) should be pushed to the wiki page. Cheers. ---Roger 05:37, 11 August 2008 (UTC)

[edit] RPC: Program not registered error.

Q: I followed every step in the "setup a home server" howto, and after getting the programs installed, and setting my exports, when I call mount -t nfs <server>:<dir> <dest-dir> command on the client, I get the following. mount: RPC: Program not registered.

Can anyone assist me with solving this?

A: This is most likely a problem with the server's tcpwrapper not allowing the client to access portmap. To check if you have the necessary access from the client run:

rpcinfo -p <server ip address>

You should see a list of services. If you do not, on the server, try add this to hosts.allow:

portmap : ALL : <client ip address>

You also may need to add mountd to the server's hosts.allow

[edit] PKGDIR separation

attention note that when there are different CHOST or CFLAGS for only specified target (like i686-pc-linux-gnu,ppc-*,sh-* OR -march=target) on around portage shared machines, package backup like quickpkg is invited breakage system. also USE-flags difference income problem (lib dependency missing).

i suggest change PKGDIR to /var/lib/portage/pkg/ or someplace when make.conf isn't same on all machines.

[edit] Faster, mostly transparent method

Modify make.conf so that PKGDIR=/local_pkg_dir or, if the make.conf's parameters is common to many machines, PKGDIR=/mnt/NFS_pkg_dir_for_ARCH.

Leave the rest as-is and simply mount the portage tree via NFS _over_ /usr/portage. This will give you the avantage of very little modifications to make.conf and the flexibility of keeping a "fallback" portage tree (useful for laptops)... Just don't forget that they aren't in sync.

All other modifications to (eix, sync, metadata) still do apply though).

So...who votes for this...any counter-indications?

[edit] Slow Emerge -ugDNv world on Clients?

Q: For the past year or so since I've created a host/client shared portage, my laptop (client) does not start emerging packages until an hour or so after a emerge -ugDNv is executed. I've exported /etc/portage, /usr/portage and /var/cache/edb/dep/usr/portage. I also took time in the past months and sync /etc/make.conf files (USE flags) as all my boxens are Pentium3 based. Today, I just exported /var/cache/edb/dep/usr/portage so finally decided to post something on this. Also, CPU and Network bandwidth all seem to be <= 25% or not even being used. Disk bandwidth on the laptop is not noticeable, nor does it seem noticeable on the host. --Roger 08:58, 14 July 2007 (UTC)

A: Perform the following.

1) Remove or comment-out any static NFS port settings within lilo.conf & grub.conf. (ie. lockd.nlm_udpport=32768 lockd.nlm_tcpport=32769)

2)Remove or comment-out any settings within /etc/conf.d/nfs & /etc/conf.d/portmap.

3) Mounting Portage related NFS folders with the async option produces a small increase in performance. (This increase in performance with using async option was noticeable prior to performing the above. Besides, since we're mainly just reading /usr/portage and the issue with corruption with the async option is probably negligible. So using async with the shared portage folders is more then likely desirable!)

4) And, just for good trouble shooting method since I am seeing port related issues, I shutdown the firewall. (ie. Stop the shorewall rc service and issue "# shorewall clear".)

In conclusion, it seems assigning static port assignments within the lilo & grub config files when I was using nfs3 to permit usage with a firewall caused a severe issue with NFS performance.

Restarting the firewall, Shorewall, with dynamic ports now shows me the firewall was more then likely not the cause.

--Roger 01:00, 22 July 2007 (UTC)

I'm completely stumped on this issue. I've filed a bug in case somebody else has performance issues with using NFS with a fire or such. The bug has been marked, I think, as either "in active" or "needs info" for historical reasons. I do not believe I've seen this bug since, but does have a tendency of reoccurring once a year or so. --Roger 02:12, 8 August 2008 (UTC)

[edit] Useful Script to share only distfiles

I wrote a script that using NFS share only distfiles, It also merges the local distfiles (if any) before mounting remote distfiles , you can find it here http://forums.gentoo.org/viewtopic-t-579466.html I Hope it is useful to somebody. --Magowiz 16:47, 1 September 2007 (UTC)

[edit] Genkernel

"I WOULD HIGHLY RECOMMEND NOT USING GENKERNEL." from the article makes this assertion but does not back it up anywhere. The link to the article on genkernel explains what it is, but not the merits or demerits of using it.

Many gentoo users, and especially new users, do use genkernel. Unless there is a specific demerit to this (which should be linked to or noted elsewhere), it is confusing and against the 'gentoo way'.

I'm removing it for this reason. Please feel free to tell me I'm wrong here and add additional information about why genkernel is a bad idea in regards to nfs mounting portage.

You're probably correct. The only demerit I can think of, is how it is configured or patched. If there's a patch in genkernel inhibiting nfs from properly working or there's a config setting doing something similar, but I cannot see this happening on such a popular kernel feature. --Roger 18:12, 9 July 2008 (UTC)

[edit] Hacked Configuration instead of using a Common Configuration Hierarchy

I don't like the following configuration listed at the end of "Portage Setup":

bash_client # rm -rf /etc/make.profile && ln -sf /mnt/nfs_portage/profiles/default-linux/x86/2006.1/server /etc/make.profile

Some scenarios, no "fall back" as mentioned by somebody simply exporting over /usr/portage. And, if they are using /mnt/nfs_portage, and one is on a laptop and wish to sync without using NFS but the default method, not only do they have to edit /etc/make.conf folder locations to default locations, but they also now have to remember to point /etc/make.profile to the default location. Currently, I just leave my /etc/make.profile link pointing to /usr/portage/profiles/... as links are followed through with NFS at my end here. --Roger 18:29, 9 July 2008 (UTC)

[edit] Configuration files do not allow for independent Server/Client Package Masking

Portage should have a server/client variable defined for packages listed within /etc/portage/package.* configuration files allowing for independent server/client package masking. With this feature, package masking can be centralized on the server and exported to other clients via NFS.

For Example:

Server has a recent Nvidia card using the latest driver version. Client has an old Nvidia card using the older driver versions. In order to keep all binary dependencies in sync (ie. using latest Firefox), /etc/portage/package.* files need to be synced. However, doing so inhibits either the client to use the latest nvidia-driver which does not work on the client, or the server needs to use the older driver version.

As such, probably best to have a configuration variable defined, such as the following syntax and example:

Syntax:
HOSTNAME="PACKAGE NAME with optional VERSION"

Usage:

localhost2="x11-drivers/nvidia-drivers"

localhost3=">=x11-drivers/nvidia-drivers-97.0.0"

Doing so, would allow us better control over which computer gets a package upgrade, or we can mask packages for a specific computer which do not work on that computers hardware or particular setup. (albeit, the nvidia-drivers is probably a bad example, as it should have been split already into nvidia-legacy-drivers (older) and nvidia-drivers (newer). --Roger 18:55, 9 July 2008 (UTC)