Also, suppose you need the memory only for startup and shutdown (things like logfiles, network connections, command line parsing, etc).
Things like network connections, logfiles are used all the time, so they won't be swapped out (actually file handles are kernel side so never swapped anyway). You can free the command line parse after setting the options.
And clean shutdown is overrated: long running programs can just terminate fairly gracelessly if necessary, the OS cleans everything up.
If you increase the size of the memory available to you (sbrk) you can only decrease it if no memory is allocated between the new area and the end of it.
In practice the memory is never returned, and applications rely to swap to deal with that.
It's not the logfile (and network) handle that is swapped out - it's the code for deciding where it is, and opening it. Also initialization code.
Some programs can abort, but others will require a (slow) consistency check of their data if that happens to them.
And finally theory is all well and good, but in actual practice about 3/4 of the memory used by running programs can be swapped out.
