You can accelerate so slow down. The flyby of a planet creates an accelleration that, if in the correct direction, results in a slower orbital velocity in the sun's reference frame. In really simple terms, if you approach the planet from the right (in a counterclockwise solar system) you speed up, if you aim to the left you slow down. The velocity is transferred to/from the planet.

https://youtu.be/hKjcbmlAaYY

Notice how the probes orbit gets more eccentric but still smaller each time it approaches a planet on the left. The gravity assist is slowing it down, again in the solar reference frame.