Replibyte – Seed your database with real data (opens in new tab)

The answer is always “it depends,” but I think if a date time is a UTC timestamp, such as a record of when an event happened, then with random sampling, it shouldn’t matter? It’s just a timestamp. The amount of information it contains might include location, might include timing to other events, could be correlated, but… on its own? It doesn’t need anonymization. Likewise the sequence of events, should be safe to use.

I get that you can look up or de-anonymize an event by its timestamp and the same is true of ID numbers. But it’s worse for ID numbers because these are often permanent and re-used for multiple events.

But yeah, the risk in anonymized data is that it’s never truly both anonymous and useful. Truly anonymous data might be considered junk or random data.

Anonymized data has some utility purpose to fulfil. Perhaps “realistic” analytics is required, or you want to troubleshoot a production issue without revealing who did what to engineers. So you anonymize the fields they shouldn’t see, and create a subset of data that reproduces the issue…?

Anonymized data is almost always a bad approach compared to generating data from algorithmic or random sources, but sometimes we need anonymized or restricted data to start that process.

The user tells it what fields need replacing with the yaml config.

Hi, author of Replibyte here :)

Yes, transformers is the way to go. I plan to add a way to detect schema changes and at list not trying to create a dump in case of change. I don't think it can be done in a safe way without human admin check.

(Thank you for your PR)

You may want to check out Snaplet at https://docs.snaplet.dev. I'm the co-founder, but we're not open-source (yet.) Our goal is to give developers a database, and data, that they can code against.

We identify PII by introspecting your database, suggest fields to transform, and provide a JavaScript runtime for writing transformations.

Besides transforming data, you can reduce, and generate data. We are most excited about data-generation!

The configuration lives in your repository, and you can capture the snapshots in GitHub Actions. So you get "gitops workflow" for data.

A typical git-ops workflow:

  1. Add a schema migration for a new column. 
  2. Add a JS function to generate new data for that column.
  3. Add core to use the new column.
  4. Later, once you have data, use the same function to transform the original value. (Or just keep generating it.)

We do this via Copycat (https://github.com/snaplet/copycat). We generate static "fake values" by hashing your original value to a number, and map that to a fake-value.

This will not work, at least not if we’re talking PII as it is defined by a Somewhat Sane (TM) privacy legislation.

Sure, passwords and credit card info is obscured with your methodology, but names, dates of birth, sexual orientation, telephone numbers, email and ip will remain unique. This uniqueness is what allows you to potentially identify a person given enough data.

I hate to be so self promoting (I swear I'm just trying to be helpful), but Gretel has that as a transformer you can use[0]. You can test out a lot of our stuff without payment info through our console[1] if you just want to mess around and see if tools like it ( and Replibyte of course :) ) would fit your use case. That being said, you can run into issues using direct transforms like this, depending on the correlated data, because of various known deanonymization attacks. There are some pretty gnarly examples out there if you Google around.

[0]https://docs.gretel.ai/gretel.ai/transforms/transforms-model...

[1]https://console.gretel.cloud/login

What you're asking for is similar to what goes by the term "tokenization"[1], a technique often used by payment processors to avoid leaking credit card numbers and similar sensitive data. Using the proper transformer might provide the behavior you need.

1 https://www.tokenex.com/resource-center/what-is-tokenization

Hi, author of Replibyte here. Feel free to open an issue and explain what is your use case. I will be happy to consider a solution with the community.

I will take a look for Replibyte. Thanks for sharing

Also if it's possible to run everything without uploading it to S3. For a smaller time dev with projects in production I would find this really interesting for debugging the production database data, but in development. Uploading it and having it in S3 would needlessly complicate it for me (even though I can understand enterprise customers might prefer it that way)

You have a local storage option https://www.replibyte.com/docs/datastores#local-disk