Please do ping me if you have any questions about the API or want to learn more! sean.herron@fda.hhs.gov
Also, here's a direct link to the API documentation: https://open.fda.gov/drug/event
What's your take on government and open source projects like this one?
Beyond improving our own site, it would be absolutely fantastic if someone took openFDA and spun up their own copy. That could be another government agency using it to serve up different data, an external group mirroring openFDA in case of government shutdown or other issue, or a company that uses our code to build something innovative.
I know that sentiment is shared among a lot of agencies right now. In particular, 18F (https://18f.gsa.gov) is a new digital services delivery unit that is looking to do this at a huge scale across the federal government.
seanherron, do you work for a contractor or are you an in-house developer for the FDA or another federal agency?
I've been doing luigi pipeline work recently, I might see if I can get yours running and get some pull requests in :)
As noted in the documentation, if you need more than 60,000 per day, give us a ring at open@fda.hhs.gov.
Huge shout out to api.data.gov as well - all of our key authentication and analytics are powered by their open source API Umbrella platform.
Looks like they're including some data we didn't previous have access to or know about. Anyways, make no mistake this is huge and will be incredibly useful for doctors and patients everywhere, this is some great data. This is the type of data that should be investigated before you take any medication, prescription or not. I can see some of this data becoming common label information shortly.
The openFDA website is built on Jekyll (https://github.com/FDA/open.fda.gov) and its API is powered with Python and Node.js (https://github.com/FDA/openfda)...It's not just the framework/current-tooling that is nice, but that such systems use open, readable formats (such as Markdown for the web pages).
The current administration has always paid lip-service toward open-source...they won't satisfy people who think "open source" and "government" means hand over just about everything...but they're doing a good job making inroads on the parts of the U.S. data interfaces that were well-intended, but so obfuscated by poor design that it was a job in itself to parse/scrape their sites.
(FDA has always had really exhaustive dumps of their data...strewn about their legacy site...the API isn't as interesting to me as the documentation for the API and the pipeline of data)
* I don't want to just slag on Drupal...but Drupal was what Obama's head tech officer wanted in place, and to their credit, they did open-source parts of their custom Drupal modules...which were not particularly useful, because of the particulars of Drupal's module system and its quickly changing API...nevermind being only useful for other Drupal installations. But a lot of credit has to go to the U.S. gov't for pivoting off of Drupal to a mix of WordPress, Jekyll, and even node.js sites with less coupled components. It's been only about two or so years since Data.gov open-sourced its Drupal components before promptly switching to WordPress and CKAN modules...considering how a non-significant number of the fed sites are built on 12+ year-old code...the turnaround in the U.S. gov's stack is pretty amazing...(when it's not attempted on a service-critical site, such as healthcare.gov)
https://www.nlm.nih.gov/research/umls/
It's a very big semantic database of health terminology. Among other things, it has a subset called RxNorm that contains all currently prescribe-able prescription drugs.
I've been very impressed with it and I feel like not enough people have heard of it.
http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInforma...
They don't respond: https://twitter.com/statwonk/status/413355130461761536
Feel free to tweet at us at @SocialHealthIs, @Geek_Nurse, and/or @Skram
We support gzip on the api json response for clients that support it. Given that, I'd expect the size improvements would be minimal for whitespace stripping but let us know if you have evidence to the contrary!
http://i.imgur.com/U1O4Xg5.png
The raw json contains the whitespaces, while they were removed in the minified json. So there is a 47% improvement for the uncompressed version, and a 21% improvement for the compressed version.
What would be interesting to see is how the second (compressed) number scales with the filesize (I don't know enough about compression algorithms to guess that).
EDIT: I really don't know how to format a table in plaintext...
