TinyChat: Large Language Model on the Edge (opens in new tab)

(hanlab.mit.edu)

2 pointsenduku2y ago1 comments

1 comments

TinyChat is an efficient, lightweight, Python-native serving framework for 4-bit LLMs by AWQ. It delivers 2.3x generation speed up on RTX4090.

j / k navigate · click thread line to collapse

(hanlab.mit.edu)

2 pointsenduku2y ago1 comments

endukuOP2y ago

TinyChat is an efficient, lightweight, Python-native serving framework for 4-bit LLMs by AWQ. It delivers 2.3x generation speed up on RTX4090.

j / k navigate · click thread line to collapse