Subject: chflags - immutable|appendonly files come to 2BSD (#197 - Part 1 of 14)
Index:	bin,etc,lib,sys/many_files 2.11BSD

Description:
	Files/directories can not be declared append-only, immutable, archived 
	or not to be dumped under 2.11BSD.

	dump(8) lacked the ability to bypass files marked for 'nodump'.

	Upper case only and Hazeltine terminal support made the tty driver 
	(and several utilities - 'vi' and 'getty' to name two) needlessly
	complicated and larger (especially since those devices are obsolete).

	open(2) can not provide for atomically obtaining a flock(2)'d
	file descriptor.

	open(2)ing with O_NDELAY (now called O_NONBLOCK) would not wait
	for carrier on the first i/o operation.

	Obsolete stub system calls 'setdopt' and 'getdopt' were still
	present in the kernel even though they have never done anything
	except take up space.

Repeat-By:
	This section does not apply this time since this is an 'update'
	which adds new functionality rather than fixing an existing bug.

Fix:
	Please read this entire file before proceeding further.  It is
	lengthy but the changes made to the system are widespread.  In
	particular the kernel is heavily modified because several internal
	calling conventions had to be changed (this is explained in some
	detail further on).

	This preamble will not be repeated in each of the following
	13 parts of this update.  Each of the remaining parts (patches #198
	thru #210 inclusive) will contain: 

		1) A reference back to #197 (this file).

		2) A list of files modified by that patch file.

		3) The actual patch file, shell script or shar file.

	This is part 1 of 14 (#197).  Contained in this file is is a 
	brief description of the new functionality being added to the 
	2.11BSD system. 

	The contents of this file are:

		1) The introduction (this section).

		2) General notes on what was done (my chance to improve
		   my typing skills ;-)).

		3) A manifest of the update kit contents - what is in each
		   of the 14 parts.

		4) Directions how to apply the update kit.

	The 4.4BSD 'chflags' capability has been ported (using the 4.4-Lite
	CD as a source of inspiration and guide) to 2.11BSD.  Perhaps an 
	excerpt from the manual page (as modified from 4.4 for 2.11) for 
	chflags(2) would be useful at this point:
	
	---------
	       "int chflags(path, flags)
	       		char *path;
	       		u_short flags;

	       int fchflags(fd, flags)
	       		int fd;
	       		u_short flags;

	       The flags specified are formed  by  or'ing  the  following
	       values

	       UF_NODUMP      Do not dump the file.
	       UF_IMMUTABLE   The file may not be changed.
	       UF_APPEND      The file may only be appended to.
	       SF_ARCHIVED    File is archived.
	       SF_IMMUTABLE   The file may not be changed.
	       SF_APPEND      The file may only be appended to.

	       The  UF_IMMUTABLE  and UF APPEND flags may be set or unset
	       by either the owner of a file or the super-user.

	       The SF_IMMUTABLE and SF_APPEND flags may only  be  set  or
	       unset by the super-user.  They may be set at any time, but
	       normally may only be unset when the system is  in  single-
	       user mode.  (See init(8) for details.)"
	----------

	NOTE: the mention of 'single user mode' is premature, the security
	level features of 4.4BSD have not been implemented yet in 2.11BSD.
	This is on the TODO list and scheduled for some time in 1995.  At
	present the 2.11BSD kernel runs with "securelevel" (to use 4.4's
	terminology) set to -1.

	What might this feature be used for?  Logfiles are a good example.
	You may wish to may a file publically writeable so that any program
	can log activity to it - but you do not want programs to be able
	to write anywhere but at the end.  By setting one of the APPEND
	bits all programs must specify O_APPEND when opening the file or the
	open(2) will fail.  Files marked as 'append only' may not be deleted
	until the append flag is turned off.  Directories marked as APPEND
	may only grow - no files may be removed until the APPEND flag is
	turned off.

	Immutable files may not be rename(2)'d, unlink(2)'d, or open(2)'d
	for writing.  Period.  Any access except opening for reading will
	be denied.  It is almost eerie to see a file with 666 permissions
	that you can not remove, overwrite, rename, etc.

	In the process of modifying the tty driver to support the new
	calling convention for read/write/ioctl entry points the support
	for 'LCASE' and 'TILDE' (uppercase only and Hazeltine terminal
	tilde character handling) were removed for three reasons:  1) the 
	space was needed for more useful features, 2) the logic was simplified
	(and made a bit faster), 3) terminals which do not know that '~' is
	an ASCII character or which do not generate lowercase characters
	are obsolete.

	Since the kernel's handling of open() was being modified anyways
	(to support APPEND and IMMUTABLE) it was almost trivial to add
	O_SHLOCK and O_EXLOCK support to the open(2) syscall.

General Notes:
	The changes to the kernel ended up being _far_ more extensive than
	originally planned.  Originally (back in May/June 1994) when the
	thought of porting 'immutable/append-only' files struck it appeared
	to be a simple matter of putting a 'flags' word in the inode and
	testing a couple bits in several places in the kernel.

	Alas things rapidly became more complex.  The 'append' and 
	'nonblocking' flags of the file descriptor needed to be passed
	down from the top level (user program does a read/write) to the
	lower levels of the kernel.  In the case of 'character' devices
	the 'ioflags' needed to be passed all the way down to the device
	driver.

	In addition to 'append' and 'nonblocking' there is the concept of
	'synchronous' I/O.  Directory I/O is always synchronous - when the
	kernel writes a directory it must specify the IO_SYNC flag to
	'rdwri()'.

	Thus many of the device drivers had their read, write and ioctl
	entry points augmented with another argument.  The kernel's
	internal read/write routines - rdwri() and rwip() - were modified
	to accept the additonal parameter.

	As if that was not enough the 'uio' structure changed too!  The
	type of request (UIO_READ or UIO_WRITE) is now encapsulated in
	the 'uio' structure rather than being passed along as a separate
	calling parameter.  This did have a pleasant side effect of reducing
	the code size in several places.

	The 'tty' driver was a bit of a mess.  The 'uio' handling worked, but
	had quite a bit of needless code present.  Writing to a tty port that
	was opened in nonblocking mode will now wait for carrier correctly
	(it did not before).

	All in all the changes ended up being ~300kb of diffs and new programs.

	Adding 'chflags' and 'fchflags' to the kernel was the starting goal,
	but along the way quite a bit of cleanup was done as well.

	To reduce the size of several modules (so that changes to existing
	overlay structures would not be needed or would be minimal) much of 
	the 'inlining' (use of ILOCK() instead of ilock() for example) was 
	removed.  Some of the macros generated quite a bit of code and the 
	benefit was quite small (performance wise).  The function call 
	overhead was likely over estimated or was much higher on the Vax.  
	Even with almost all of the inlining removed from the kernel 2.11BSD 
	has not slowed much at all - kernel recompile times are within seconds 
	of what they always have been.

	The incore inode structure grows by two bytes (i_flags is a unsigned
	short).  To make room for this (new features should at least attempt
	to pay their own way!) the following steps were taken:

		1) a couple panic strings were shortened or ifdef'd out 
		   (since they'd never occured during the last 7 years)

		2) repeated error message strings were assigned to a static
		   'char []' variable (tmscp.c and ts.c are good examples of
		   this).  The savings were enough that the GENERIC kernel
		   is actually smaller now than it was before.

		3) The 'sccs' string was removed - this saved almost 100 bytes
		   alone!

		4) The floating exception table was changed from 'int' to
		   'u_char' since the codes are all small in size (hey - it's
		   16 bytes reclaimed!).

Manifest:

	Part  Patch    Contents
	----  -----    ----------
	1     #197     introduction   description of the update kit.

	2     #198     remove	      shell script to remove old files and
				      prepare for part 2.

	3     #199     new.shar       shar file of _new_ sources (chflags.c,
				      new modules for 'ls', and so on)

	4     #200     sys1.patch     first half of /sys/sys patch file

	5     #201     sys2.patch     second half of /sys/sys/patch file

	6     #202     include.patch  patch file for /usr/include.  NOTE:
				      this patches /usr/include, _not_
				      /usr/src/include!

	7     #203     pdp.patch      patch file for /sys/pdp

	8     #204     pdpuba.patch   patch file for /sys/pdpuba

	9     #205     lib.patch      patch file for /usr/src/lib

	10    #206     bin.patch      patch file for /usr/src/bin

	11    #207     ex.patch       patch file for /usr/src/ucb/ex

	12    #208     etc.patch      patch file for /usr/src/etc

	13    #209     man.patch      patch file for /usr/src/man

	14    #210     misc.patch     small patch file for remaining (3)
				      programs (tabs, usw.)

Directions:

	And now the moment you have all been waiting for! ;-)

	To install the update kit make sure you have all the parts.  There are
	14 files as shown in the table above.

	Optional (but recommended) step:  purge the system of files created
	by previous patches.  This patch kit is large, you may need the space.

	find / \( -name '*~' -o -name '*.rej' -o -name '*#' \) -print > /tmp/XXX

	then examine /tmp/XXX to see if there are any files you really wish
	to preserve (if there are files you want to save, edit /tmp/XXX and
	remove their names).  Finally if all the files in /tmp/XXX are to
	be deleted do:

		rm `cat /tmp/XXX`

	Parts 2 and 3 _must_ be done first and in that order.  This is because
	part 2 creates directories and renames files needed by part 3.

	A convention I have found useful is to save the parts of the kit into
	/tmp files and name those files after the patch number - thus you should
	have the files: /tmp/198, /tmp/199, /tmp/200, ... /tmp/210 if you have
	the entire kit present.

	Part 2 is a shell script - you apply it by one of two methods (first
	removing any mail/news headers):

		chmod +x /tmp/198
		/tmp/198

	or
		sh /tmp/198

	Part 3 is a shar file, after removing the headers you install it
	with:

		chmod +x /tmp/199
		/tmp/199

	or

		sh /tmp/199

	ALL of the remaining parts (4 thru 14, files 200 thru 210) are
	'patch' files made with absolute pathnames.  While not strictly
	necessary it is recommended that the patches be applied in order
	like this:

		patch -p0 < /tmp/200
		patch -p0 < /tmp/201
		...
		patch -p0 < /tmp/210

	Now look for any rejected patches.  There should be none if the
	system was currently at patch level 196 before starting the update.

		find / \( -name '*.rej' -o -name '*#' \) -print

	will give a list of patches which did not apply correctly.  You will
	need to manually examine the files to find out what was rejected and
	why.  If you get stuck send me a mail item with the details and I will
	try to help out.

	IMPORTANT:  If any kernel patches fail to apply properly do not
	attempt to build and use a new kernel!  You may proceed with the
	building of the libraries and utilities.

	Next you will need to recompile the kernel.  NOTE: many of the kernel
	modules changed size - it is highly probable that the overlay structure
	will need minor adjustments.  Moving 'mem.o' to an overlay (or to a
	different overlay) is all that was necessary for the GENERIC kernel,
	your changes may be similarily trivial.

	You do not need to reconfigure the kernel - a simple recompilation is
	enough.  Then install the kernel and reboot.  The 'pstat' program
	will return garbage until it is recompiled, but the new kernel can
	be used to perform any future recompilations:

		cd /sys/YOUR_KERNEL
		make tags		# time to recreate the tags file
		make
		mv /unix /ounix		# save the old kernel
		mv /netnix /onetnix	# only if making a networking system
		mv unix /
		chmod 744 /unix
		mv netnix /
		chmod 744 /netnix	# only for networking systems 

	After the system successfully reboots you should 'rm /ounix /onetnix'.

	Optional: remake the GENERIC kernel and install as /genunix.  This
	can be deferred until later but it is something all sites should do.
	Having a copy of the GENERIC kernel present is inexpensive insurance
	against losing /unix or /netnix.

		cd /sys/GENERIC
		make
		install -m 744 unix /genunix

	The next step is to recompile and install the libraries.  If you
	are familiar with the proceedure it is possible to recompile the
	new/modified files and update the libraries manually.  It is HIGHLY
	recommended that you simply recompile the libraries from source to
	avoid leaving something out:

		cd /usr/src/usr.lib/libcurses
		make
		make install

		cd /usr/src/lib/libc
		make
		make install

		cd /usr/src/usr.bin/lint
		./libs

	NOW it is time to recompile and install the new and modified programs:

		cd /usr/src/bin/sh
		make 
		make install

		cd /usr/src/bin/csh
		make 
		make install

		cd /usr/src/bin/ls
		make
		make install

		cd /usr/src/bin/chflags
		make all
		make install

		cd /usr/src/bin
		make stty
		install -s -m 755 stty /bin/stty

		cd /usr/src/etc
		make pstat
		install -s -m 2755 -g kmem pstat /etc/pstat

		cd /usr/src/etc/getty
		make
		make install

		cd /usr/src/ucb/ex
		make
		make install

		cd /usr/src/ucb/tset
		make all
		make install

		cd /usr/src/usr.bin
		make tabs
		install -m 755 tabs /usr/bin/tabs

	then clean up:

		cd /usr/src
		make clean

	and lastly the man pages:

		cd /usr/src/man/man2
		make chflags.0 fcntl.0 open.0
		install -m 444 chflags.0 fcntl.0 open.0 /usr/man/cat2
		ln /usr/man/cat2/chflags.0 /usr/man/cat2/fchflags.0

		cd /usr/src/man/man1
		make ls.0
		install -m 444 ls.0 /usr/man/cat1/ls.0

		cd /usr/src/man/man4
		make tty.0
		install -m 444 tty.0 /usr/man/cat4/tty.0

		cd /usr/src/man/man8
		make dump.0
		install -m 444 dump.0 /usr/man/cat8/dump.0

	Congratulations - you are now done.  Have fun with 'chflags'!
