undefined | Better HN

0 pointsAjtevhihBudhirv4y ago0 comments

There're still faster approaches than naïve and probably common "copy valid byte sequences one by one into a resizable result buffer". For instance, scan through the input bytes all at once, keeping track of position and length of valid sequences, then memcopy each valid sequence into a preallocated buffer.

Edit: Although, it looks like Rust's std already does this, except for preallocating an exactly correct size result buffer: https://doc.rust-lang.org/src/alloc/string.rs.html#538

0 comments

duckerude4y ago

> preallocating an exactly correct size result buffer

Looks like it just uses the size of the original slice. If the average broken chunk is less than three bytes (maybe quite common?) then it'll have to grow the buffer, at least doubling it.

  >> let bytestring = b"foobar\xcc";
  >> bytestring.len()
  7
  >> let cleaned = String::from_utf8_lossy(bytestring).into_owned();
  >> cleaned.len()
  9
  >> cleaned.capacity()
  14

j / k navigate · click thread line to collapse

0 pointsAjtevhihBudhirv4y ago0 comments

Edit: Although, it looks like Rust's std already does this, except for preallocating an exactly correct size result buffer: https://doc.rust-lang.org/src/alloc/string.rs.html#538

0 comments

duckerude4y ago

> preallocating an exactly correct size result buffer

Looks like it just uses the size of the original slice. If the average broken chunk is less than three bytes (maybe quite common?) then it'll have to grow the buffer, at least doubling it.

  >> let bytestring = b"foobar\xcc";
  >> bytestring.len()
  7
  >> let cleaned = String::from_utf8_lossy(bytestring).into_owned();
  >> cleaned.len()
  9
  >> cleaned.capacity()
  14

j / k navigate · click thread line to collapse