QUIC File Transfer Service, a CLI and srv for transferring large files (opens in new tab)

(github.com)

40 pointsHydrocarb0n1y ago17 comments

17 comments

Hey, not sure how this came up but I’m the original author of the repository…thanks for taking a look at my code and thinking it was cool enough to post on here. If you have any questions let me know, or create an issue on the repository and I’ll try to take a look at it.

bradknowles1y ago

Looks like the last time the code itself was touched was about seven months ago. And there are no open issues. I did spot the following in the README.md file: ``` An optional MD5 checksum can be calculated as well for the transferred file to verify that the content is the same as the source file. The server provides its own MD5 for comparison once the file is written. However, this would only be an additional level of redundancy as quic has a reliability guarantee already built into the protocol. ``` Could we make this SHA-256 or something else a little more modern and robust than MD5?

sirgallo1y ago

Hey man, had no reason to work on the code further, was a one man experimentation with no incentive to work on it further. The use of MD5 was not meant for security guarantees but instead as a way to ensure the integrity of content. There would be no reason to use SHA-256 in this case, not worried about security since that would be handled through use of SSL/TLS between server and client, which is already available.

tarasglek1y ago

Weird to be comparing vs rsync. Rsync is notorious for maximizing latency as it has no pipelining.

Better baseline would be transferring data via ssh + tar. Or serving a directory over http and concurrently downloading a file with axel

sirgallo1y ago

Sure it may be weird, but my company utilized rsync for almost everything, and we were transferring files that were 100+Gb on the regular so this was actually a great tool to compare against since we were already using it for large file transfers.

yjftsjthsd-h1y ago

I would be more interested in seeing throughput vs rsync or tar+ssh (or scp, really).

nunobrito1y ago

Well.. I'm not a go programmer. Just wanted the usual download/install and syntax to use the tool.

Nowhere to be found. I guess I'll just open a ticket there.

sirgallo1y ago

Hey, you can already do this. It is designed to be portable and usage is described in the cmd folder in separate markdown file. Can be used in docker as well, with server and client usage described.

vips7L1y ago

How does QUIC compare to the proprietary Aspera protocol?

sirgallo1y ago

Haven’t looked into it, there are upsides and downsides to everything, there is never a one size fits all solution

Hydrocarb0nOP1y ago

The File Transfer Service utilizes the quic-go implementation of the quic Protocol, built on top of UDP. Since quic allows for multiplexing of streams on a single connection, the service takes advantage of this to attempt to speed up file transfers by processing and writing the file from the remote host (the server) to the destination (the client) concurrently

01HNNWZ0MV43FF1y ago

That's clever. I'd wondered how you can transfer files with "any order, as long as the entire file completes as fast as possible" semantics, that is, without head-of-line blocking, without incurring a lot of overhead. I guess the answer is to use lots of big chunks and lots of streams, instead of lots of tiny packet-sized chunks in 1 stream.

adgjlsfhk11y ago

it's exactly the way torrents have worked for decades, so it's pretty well tested.

GauntletWizard1y ago

Is it time to move QUIC past UDP yet? Does QUIC deserve it's own Protocol number? There's only one byte in IP for protocol number, so it makes sense to limit the number of applications, but I believe QUIC has reached that level.

https://www.iana.org/assignments/protocol-numbers/protocol-n...

foresto1y ago

I'm under the impression that UDP is what makes QUIC valuable, by keeping it compatible with the world's existing network hardware and firewall configurations.

If that issue didn't exist, SCTP would surely see a lot more use.

01HNNWZ0MV43FF1y ago

Hm. Wikipedia tells me that a UDP headers has 4 parts: Source port, dest port, length, and checksum.

I suppose with a new protocol you could ditch the length and checksum, which (I don't know much about networking) might be redundant with the IP header and with TLS respectively. That would save you 4 bytes per packet. Nothing huge.

The reason it uses UDP is that the Internet is full of middleboxes built 10 years ago that won't get any firmware updates for another 10 years, and those boxes love TCP and we're lucky if they even pass UDP and ICMP packets. Anything else might just get dropped, and then you have to fall back. Is UDP underneath QUIC really hurting much?

AndrewDucker1y ago

The problem is that most routers don't support anything other than UDP and TCP.

j / k navigate · click thread line to collapse

17 comments

sirgallo1y ago

bradknowles1y ago

sirgallo1y ago

tarasglek1y ago

Weird to be comparing vs rsync. Rsync is notorious for maximizing latency as it has no pipelining.

Better baseline would be transferring data via ssh + tar. Or serving a directory over http and concurrently downloading a file with axel

sirgallo1y ago

yjftsjthsd-h1y ago

I would be more interested in seeing throughput vs rsync or tar+ssh (or scp, really).

nunobrito1y ago

Well.. I'm not a go programmer. Just wanted the usual download/install and syntax to use the tool.

Nowhere to be found. I guess I'll just open a ticket there.

sirgallo1y ago

Hey, you can already do this. It is designed to be portable and usage is described in the cmd folder in separate markdown file. Can be used in docker as well, with server and client usage described.

vips7L1y ago

How does QUIC compare to the proprietary Aspera protocol?

sirgallo1y ago

Haven’t looked into it, there are upsides and downsides to everything, there is never a one size fits all solution

Hydrocarb0nOP1y ago

01HNNWZ0MV43FF1y ago

adgjlsfhk11y ago

it's exactly the way torrents have worked for decades, so it's pretty well tested.

GauntletWizard1y ago

https://www.iana.org/assignments/protocol-numbers/protocol-n...

foresto1y ago

I'm under the impression that UDP is what makes QUIC valuable, by keeping it compatible with the world's existing network hardware and firewall configurations.

If that issue didn't exist, SCTP would surely see a lot more use.

01HNNWZ0MV43FF1y ago

Hm. Wikipedia tells me that a UDP headers has 4 parts: Source port, dest port, length, and checksum.

AndrewDucker1y ago

The problem is that most routers don't support anything other than UDP and TCP.

j / k navigate · click thread line to collapse