TensorRT-LLM runtime now open-source (opens in new tab)

(github.com)

4 pointsmmoskal1y ago1 comments

1 comments

Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).

j / k navigate · click thread line to collapse

TensorRT-LLM runtime now open-source (opens in new tab)

(github.com)

4 pointsmmoskal1y ago1 comments

1 comments

mmoskalOP1y ago

Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).

j / k navigate · click thread line to collapse