The network port is faster than the CDVD drive or any of those accessories with the exception of the HDD. The ethernet PHY links at 100Mbit, but the processors inside the PS2 are not really capable of pushing that speed, the best I ever saw when installing games over the network with a hyper-optimized IP stack (on the IOP, IIRC) was something like 5MiB/s.
The HDD is the fastest form of I/O one can use on the PS2. It might not even need to be modified - depending on how well it's coded, it may be possible to run this software via Open PS2 Loader, which will replace CDVDMAN with a custom version that will access USB/ETH/HDD (and as mentioned in sibling comments, USB on the PS2 is version 1.1 and is much slower than even the CDVD drive).
Both network and HDD will also greatly minimize the cost of seeking the CDVD, which may be an issue depending on how the CD is laid out. CD access is up to 24x, DVD-ROM access is at 4x. DVD is thus slightly faster, and can be further increased by pushing the used data to the edge of the disc via a dummy file (traditionally, developers and game modders used Sony's own CD/DVD Generator software to determine the order of files being added to the disc, thus allowing the boot files to come first, followed by the dummy file, then any data files that need the extra speed).
