We’re Everest, Andrei and Sabera, the founders behind Fuzzbuzz (https://fuzzbuzz.io) - a fuzzing as a service platform that makes fuzzing your code as easy as writing a unit test, and pushing to GitHub.
Fuzzing is a type of software testing that generates & runs millions of tests per day on your code, and is great at finding edge cases & vulnerabilities that developers miss. It’s been used to find tens of thousands of critical bugs in open-source software (https://bugs.chromium.org/p/oss-fuzz/issues/list), and is a great way to generate tests that cover a lot of code, without requiring your developers to think of every possibility. It achieves such great results by applying genetic algorithms to generate new tests from some initial examples, and using code coverage to track and report interesting test cases. Combining these two techniques with a bit of randomness, and running tests thousands of times every second has proven to be an incredibly effective automated bug finding technique.
I was first introduced to fuzzing a couple years ago while working on the Clusterfuzz team at Google, where I built Clusterfuzz Tools v1 (https://github.com/google/clusterfuzz-tools). I later built Maxfuzz (https://github.com/coinbase/maxfuzz), a set of tools that makes it easier to fuzz code in Docker containers, while on the Coinbase security team.
As we learned more about fuzzing, we found ourselves wondering why very few teams outside of massive companies like Microsoft and Google were actively fuzzing their code - especially given the results (teams at Google that use fuzzing report that it finds 80% of their bugs, with the other 20% uncovered by normal tests, or in production).
It turns out that many teams don’t want to invest the time and money needed to set up automated fuzzing infrastructure, and using fuzzing tools in an ad-hoc way on your own computer isn’t nearly as effective as continuously fuzzing your code on multiple dedicated CPUs.
That’s where Fuzzbuzz comes in! We’ve built a platform that integrates with your existing GitHub workflow, and provide an open API for integrations with CI tools like Jenkins and TravisCI, so the latest version of your code is always being fuzzed. We manage the infrastructure, so you can fuzz your code on any number of CPUs with a single click. When bugs are found, we’ll notify you through Slack and create Jira tickets or GitHub Issues for you. We also solve many of the issues that crop up when fuzzing, such as bug deduplication, and elimination of false positives.
Fuzzbuzz currently supports C, C++, Go and Python, with more languages like Java and Javascript on the way. Anyone can sign up for Fuzzbuzz and fuzz their code on 1 dedicated CPU, for free.
We’ve noticed that the HN community has been increasingly interested in fuzzing, and we’re really looking forward to hearing your feedback! The entire purpose of Fuzzbuzz is to make fuzzing as easy as possible, so all criticism is welcome.
Other use cases include using fuzzing to compare implementations of libs that require the same functionality, detecting invariant violations, testing implementations that are meant to work together (i.e. serialize(deserialize(x)) == x).
In general fuzzing C/C++ libraries for memory bugs is the most commonly described use-case, but I think there are tons of fuzzing use cases that haven't been thoroughly explored yet.
We support a couple of languages that OSS-Fuzz doesn't (Go & Python as of now), which is why I thought this was worth mentioning :)
https://github.com/oilshell/oil/blob/master/bin/osh_parse.py
https://github.com/oilshell/oil/issues/171
I'm already testing it by running it on more than a million lines of shell [1], which I imagine should provide a very good starting point for the AFL algorithm. I've only fuzzed one thing before but that's my understanding.
If anyone is itching to try out Python fuzzing, this might be a nice and realistic intro.
I made note of Fuzzbuzz on the bug.
They opened an office in Seattle to fuzz for Microsoft. As soon as they proved that they could succeed, Microsoft hired away all the people, leaving the company with a lease for an empty office.
Generally, companies don't trust outsiders and/or don't see a need. You're up against internal politics too. People within the company don't want to compete with you and don't want to be embarrassed by you.
Also, 20MM/yr is not a revenue number to sneeze at. Enterprise security is a huge and mature product space and most aspirants in it do not hit that number. These companies aren't Uber, where every dollar coming in is going back out the door with an extra couple dimes to boot.
There's a substantial counterargument to this I need to type up. I understand where you're coming from as someone who once ran a consultancy, Tom, but from the perspective of someone who hires security firms and consumes their services—and this is essentially a TL;DR for the opinion I need to flesh out here—we don't do it because we trust. We do it out of necessity.
TODO: Bryant to flesh this out in between laundry rounds tonight.
So, this company definitely has a chance. Further corroboration is the uptake Hypothesis was getting in Python shops. There's definitely a demand. I just don't know the numbers for this sector.
I think the key difference though, is that we don't do any consulting/training/manual pentesting. We're more of a dev tool company than a security company in that we don't aim to replace security engineers but to make their lives easier.
The company was created starting from early fuzzing research at Florida Institute of Technology. The whole point of the company was to fuzz things for software companies. That mostly didn't work out.
That all might not be your fate, but consider it a warning. You could do a better job of making things accessible, or you could offer a more acceptable price point, or you could advertise better, or maybe 2019 is different enough from 2002 that such a business is more viable.
I don't know much about fuzzing but I'm inspired to give your tool a try if/when I get a chance.
If you're interested in learning more, our docs [0] explain everything from the ground up.
Wasn't our idea though - we stole it from Labelbox [0] :)
func BrokenMethod(Data string) bool {
return len(Data) >= 3 &&
Data[0] == 'F' &&
Data[1] == 'U' &&
Data[2] == 'Z' &&
Data[3] == 'Z'
}
def BrokenMethod(strInput):
if len(strInput) >= 2:
return strInput[0] == 'F' and strInput[1] == 'U'
Next, it's not clear to me how you indicate success/failure of a test. Is success just any program than exits 0 and failure any program that exits non-zero? That would be my guess but the docs don't say.
Typo: https://github.com/fuzzbuzz/docs/pull/3
This page is missing a link to the find-your-first-bug-in-Python example:
https://github.com/fuzzbuzz/docs/blob/master/getting-started...
The docs site loads slowly for me on an older iPad, and there's even a slight delay on a recent Macbook. Looks like it's maybe a font loading issue? (Oh, it's gitbook. How awful. I guess there's nothing you can do about that other than use a different doc provider.)
1) The BrokenMethods are simple examples of programs that crash on buffer overflows/index out of range errors. If you were to pass "FUZ" into the Go method, it would check Data[3], thus causing a panic since there are only 3 elements in the string.
ninja edit: that python method in your comment IS a valid method with no error - a bit of a brain fart on my end when writing out the docs. It's been changed :)
2) In general a failure is any non-zero exit. We do this to be flexible in the way you report bugs. For C/C++ and Python this is usually with assertions, and in Go you can achieve something similar with:
if !x {
panic("Error")
}
3) Thanks for the fixes - much appreciated. And yeah, we know GitBook is pretty slow, and we're in the process of moving to another docs provider.
If you've got any more questions please let me know!
Shoot me an email at everest@fuzzbuzz.io and I'll let you know when we launch ruby fuzzing.
I am curious about how you use AFL under the hood - how do you scale? Do you use a shared kernel and run a worker process for each physical core, or do you do some virtualization or perhaps run with a kernel patch such as https://github.com/sslab-gatech/perf-fuzz/ ? My experience is that you will hit a wall pretty quickly unless you start multiple kernels by using virtualization, or simply having a very slow binary so you don't get a high number of execs/s to start with.
All of this was done to try and keep the scaling as linear as possible, so that when you double your CPU count you're doubling your execs/second as well.
This sounds like a good solution. I was trying to solve this problem on my own as well and ended up making a minimal kernel+afl image that I then boot multiple times using deduplication features in order to save RAM (I don't have 256 gig ram like you do in a proper server). Each instance ends up eating quite a lot of RAM even with a limited root filesystem so that's why I wanted to keep it low. I'm on a 2990wx rig which was kind of a disappointment from a fuzzing perspective because of the limited memory bandwidth, but that's for another discussion.
Do think it would be worth trying out that snapshot approach that I linked in the parent comment or it might not be worth it? I was thinking of rebasing their patch onto the latest master and getting it to work again - sadly the authors seem to have abandoned the project, at least there hasn't been any public changes since a year back.
So let's say I upload some FOSS project and end-up finding some crashes|potential vulnerabilities. Have you considered some sort of tie-in|integration to bug bounty programs so that I could get a small pay-out without having to go through the trouble of figuring out how exploitable a given crash might be, and more importantly to actually have to deal with trying to get the attention of the project?
[0] https://www.microsoft.com/security/blog/2013/06/13/exploitab... [1] https://github.com/jfoote/exploitable
We've been thinking about the best way to use Fuzzbuzz to benefit the OSS/bug hunting community, and the integration idea is a great one. We're also providing free plans with extra CPU power for security researchers & bounty hunters.
[0] https://gitlab.com/akihe/radamsa
In regard to CPUs - my laptop reports 4 CPUs, my workstation 16 - where the value for someone involved in fuzzizng would come in my mind would be if you could take away the hassle of scaling fuzzing 'transparently' to 100 or 1000 CPUs. What I am suggesting here is that on your pricing page you might be off by factor of 100 in regard to what number of CPUs actually make offering compelling to someone who would consider outsourcing their fuzzing infrastructure.
This is even better.
https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
Congrats Everest (and team) for making it easier to make software more secure!
If you send me an email (andrei@fuzzbuzz.io), I can let you know when we launch Java support
Thanks!
We are in the process of building a fuzzer for generic web-apps, so watch this space :)
I recall there is|was a company based out Santa Cruz, CA called Fuzz Stati0n with a pretty similar concept. Might be good idea to ‘compare’ experiences.
Best wishes and happy to talk.
Do you guys support fuzzing by protocols? Syscalls, REST, or SQL? It might be faster to extend protocol fuzzing than fuzzing by language (I'm not sure though). It'd be cool to have a fuzzer for Apache Calcite; it's a library to slap on a SQL interface to your database.
Any plans to extend fuzzing to property-based testing?
Do you guys fuzz your fuzzer (dogfood)? Probably useful, but also funny :)
We're also thinking about extending to property-based testing. There are some really awesome hybrid testing tools out there, such as DeepState (https://github.com/trailofbits/deepstate) which combines Symbolic Execution and Fuzzing behind one clean interface, and we'd really like to push the boundaries of that type of testing.
And yep, we do! Everything is written in Go, which is part of the reason it's one of the first languages we support.
On scaled display it shows up fairly small and so it's hard to see what's going on without going to full screen. As a quick fix could you enable the full screen button. And as a longer term fix consider recording another version where the windows are smaller or there's some sort of magnification?
Also, yeah, we're not video experts, so we had a feeling the video might not be the correct format. We just wanted to put something up that showcased the platform without forcing you to sign up, but we'll definitely make sure our next video is in a better format.
One of the things that became a source of frustration is writing the specs that define the shape of the inputs.
Does FuzzBuzz make that any easier?
We do have plans to build some tools to make this easier. I'd like to see a scenario where defining inputs is as simple as specifying the data types that your code requires. (Or perhaps even automatic detection, for less complex cases)
So you can do: auto s = datasource.Get<std::string>();
For each type I need I override Get<T> for that type. If the data source class is out of data, I throw a specific exception, that I catch at the global level. This works like a breeze and the advantage over protobufs is that it's faster (no Protobuf parsing overhead, errors) and you never consume more than you need.
#OnWisconsin