Joan Didion, "On Keeping a Notebook"
Apologies for the long quote but this is pretty much how I feel about all attempts at "personal analytics" so far. As other posters have mentioned, analytics are great for businesses who have their FOMs and TPMs and KPPs and whatever to maximize already, but on a personal level the best use of diaries and journals is to force yourself to realize what your goals are, not to help achieve them.
Personal analytics software is something I would never use. It must be nice to have a life worth quantifying.
Its exactly focused on insight and de-cluttering the mind and finding focus. Being happier and more creative.
It really works and lead me to significant periods of peace and contentment and working on meaningful things.
In the last year I did a lot of work to turn it into a proper business, but got caught up in another startup which took up all my time.
I am planning on making it into a product with great design.
That's really the primary work now: crafting a design that is communicates the data in a clear way.
Some messy bells and whistles thing with all the data and graphs and features.... nope. Not interested. I want clarity.
I use DidSum to track running, blogging, eating healthy, sex, coffee consumption, and more.
DidSum allows you to define the actions you want to improve at. You can track publicly with friends and family and encourage/compare/compete with each other. You can also track privately (some actions I only track with my wife). It has basic analytics, which will get more powerful with time.
We're in a closed beta right now, if you're interested in helping test I'd love anyone interested in this thread's feedback: http://didsum.com/auth/register.html
You need to separate the signal from the noise, and ramping up the noise isn't the solution. Quality takes precedence over quantity. Benjamin Franklin identified thirteen 'virtues', and those signals seemed to work for him.
> I propose a simple data entry framework: click a button, and that event gets tracked.
The problem with data entry is that it requires the motivation and initiative of the user, which is precisely what the user lacks. That's the reason people get personal trainers and life coaches: you need someone else to keep you accountable and motivated.
> Machine Learning Matters
Here's where I think almost all of these personal-analytic type applications go wrong: it's trying to solve a very subjective human problem with a very objective machine program.
Here's an alternative startup idea: a web application for life coaches and their clients, focused on providing them the tools they need to communicate better with one another. I'm betting that life coaches use email, text messages, phone calls, a calendar, and maybe a spreadsheet. They probably could use some help integrating all those tools and all the data they are collecting.
And as far a motivation, the mobile component is essential because it will literally buzz in your pocket to remind you to do things.
Everyone wants to improve something (or everything!) about themselves. Most people have a sense of what that is, and what they should be doing. Stop smoking, go to the gym, get more sleep, eat better, spend less money, make more money, etc. A personal analytics system may help someone optimize their life from seeing the hidden details, but I claim that the steps to self-improvement for most people are fairly obvious. The problem in self-improvement is not lack of data, it's how to execute on the obvious known fixes (how to stop smoking, etc).
This is doubly true in a social setting where your friends/spouse/mom/coach can _see_ when you are or are not taking actions toward what you want to improve at.
friends <- [user.alice, user.bob, user.carol] # array of friends friends_movies <- apply(friends, movies)
When a friend updates its movies everything is recalculated like in a spreadsheet. The core idea behind this is to end with emergent behaviour from the small scripts of millions of people.
To simplify the programming language everything can be written as an s-expression. I've published the parser (using o-meta) with an article here: http://blog.databigbang.com/parsing-s-expressions-in-c-using... more food for thought in:
i) Egont, A Web Orchestration Language: http://blog.databigbang.com/ideas-egont-a-web-orchestration-...
ii) Egont Part II: http://blog.databigbang.com/egont-part-ii/
For the moment we're starting small and just trying to pull in activity tracker data (Fitbit, Jawbone, Withings etc) and some secondary services that provide useful quantifiable data - things like Foursquare checkins, Gmail, task management tools, productivity trackers, etc. It's hard to interface with every service when you're writing custom API clients for each of them. And on the other hand, if you try to dictate a common standard without any clout to back it up, you just end up with http://xkcd.com/927/. We're working on an API to export all your data too, so people can build visualisations and apps on the aggregates of their data.
Tracking things has to be as frictionless as possible, so I'm wary of manual tracking unless you're already quite committed, or it takes mere seconds - the interface in #2 looks like a good way to approach this. But we're avoiding this for now.
The other thing we strongly agree on is #4 - you need insights, not just data! Numbers alone are of limited value, and a lot of people already into Quantified Self end up with "number fatigue" where they're no longer motivated to put in the effort to track things, let alone work towards a goal, just so they can see the number go up. People need to qualify their data and make it part of a narrative about their life. They also need actionable feedback, which is where we're trying to focus on providing value - if you're trying to be more productive, on which days is that working, and what are the factors behind it? We'll try and tell you so you can focus on that. Same process for becoming more active or anything else.
It's a big problem to solve, but so many possibilities too.
Other than that, I totally agree. Someone please make this app-- then let me do the machine learning for you. :)
I know some people are trying food tracking via taking a photo. Taking a photo per meal is a lot of overhead! That said, 4 Hour Body recommends it to stick to a nutrition goal.
Also, I disagree with your premise. I know what I shouldn't eat, and the main questions I want answered are what correlates to my making bad decisions. How does sleep relate? Alcohol? Being around friends? Tracking specifically what I eat seems irrelevant.
I am seeking to apply the "easy data entry" + "subjective good/ok/bad" measures to help folks discover patterns of bad spending decisions. While most every budgeting/spending app will show you "how much you have left" or "safe to spend" the real measure of improvement is whether you made good decisions or bad decisions given your life circumstances. This also has the effect of softening the penalty for missing data. If you miss some data, you just pick up where you left off. You're not trying to balance your budget. You're tracking decisions. Although I'm not (yet) biting off the notion of correlating it with other personal analytics, that's an interesting extension of the idea.
I'm really convinced that a behavioral approach to spending/budgeting would lead to longer lasting positive changes than fretting about exceeding your grocery budget by $19 due to unexpected company joining you for lunch.
The general gist is described here: http://www.spendlight.com/land-3.html
Sites like MyFitnessPal and MyNetDiary would likely disagree. They have hundreds of thousands of users. However, I agree that if you're in generally good health then it probably is too high of a burden.
> I know what I shouldn't eat
I'd argue you probably do not know what you should or should not eat. Sure, in general you know you should avoid eating candy for every meal, but I suspect you don't really know which particular ingredients are resulting in a change of your state.
For instance, many people are gluten intolerant for years before realizing it. If you were able to track foods to the approximate level of knowing the level of gluten in your body at a given time, you could identify gluten as a cause of some symptoms. I would also make similar arguments about the long-term effects of certain diets (semi-vegetarian, low-carb, etc.).
I guess there is a divide between the kind of myopic analytic insight you're looking for and the more long-term ones I'm interested in. Both are valuable, but they require different data sources.
Yes Fitbit is a great pedometer that gets people to walk more but last time I checked they weren't recording that much data and linking to connected scales is useful but its barely scratching the surface.
One issue with Personal Analytics is we need more data and it needs to be EFFORTLESS to record. The ROI is only high enough when the effort is very low. This is a challenging problem that requires "true innovation". I don't think a tracking app on my mobile phone is going to be the answer.
Another issue is the ability to improve some facet of my life with this data(weight, sleep, etc). Even if I can correlate drinking coffee in the evening is causing me sleep problems unless I can change my habits its not useful information. We need to be able to change our habits and processes based on this data. There are scientific strategies to get people to change habits and simply telling them they need to walk more or eat less do not pass scientific muster. Trigger -> Habit -> Reward. If we need to change Habits we must replace them something. If I need to stop drinking coffee then I need to replace it with something.
Basis although is lagging in the software department. Their mobile apps are not very useful and their API is not open so people can't grab their data without doing unofficial hacks that accesses the web apps json data directly.
I think although people are going to be using something like the iPhone 5S's M7 motion tracker more than something like a fitbit, because they carry their phones everywhere and don't have to buy or manage a separate device.
This would be the best current combo if basis opened it's data silo:
1. A basis watch for sleep and general activity.
2. A wifi weight scale for weight & easy to measure BF%
3. Runkeeper, etc to track your cardio activities. Basis & fitbit doesn't track actual fitness activities (like biking or hiking) too well compared to runkeeper.
4. Also lets you manually input personal measurements such as your navel, hips, waist, etc from the mobile app. Fitbit does not do this.
All of these items in one cloud-managed website & mobile app will be pretty powerful. The closest you have for this is fitbit right now.
In the future, a credit card / payment system that automatically tracks nutrition purchasing info will be very powerful. This needs to be combined with more nutrition estimates in restaurants although.
Passive 24/7 location tracking can also help co-relate events. It can track the effects of vacations, ski trips, trips to the pool, more grocery shopping less restaurants, etc to put more meaning into your graphs. It can co-relate you going to the gym frequently or not going to burger king anymore with you losing weight.
I think just awareness that your slowly gaining weight can help snip weight gain trends in the bud, along with warnings your eating a lot of high calorie crap that you might not be aware just how high calorie it actually is.
But, with the new wave of smart watches, glasses and other smart technology there is an opportunity for that data to be easily imported into a personal analytics application.
If someone can create a system that nearly automates the tracking of what you eat you'll make gazillions and help a lot of people along the way. Maybe a bluetooth food scale in conjunction with a tablet that can do "facial recognition" on a plate of food to estimate what I'm eating and the carb/protein/fat/calorie breakdown. Then based on my physical activity for the day and calorie consumption it could recommend I skip desert or have a carrot instead.
I would focus things around a goal, and make clear that the service will help you stick to it over time. The benefit is over time and not immediate, so I think you're point is really valid.
It also depends on the quality of the insight. Maybe you can tell from my email and social media patterns when I'm sleeping, and make recommendations a few moments after you auth.
If we could better understand what we actually enjoy then we could design a lifestyle that maximizes periods of real well-being and enjoyment. Happiness is a loaded word but wouldn't you contend that the modernity, and all the alienation contrived therein with life reduced to private experiences, could be improved for all by an optimization of private experience?
If a web platform is developed to provide storage and an API, the only way I would trust it is if it were completely open source and users were encouraged to run their own nodes (as well as supporting easy data migration between nodes).
But frankly, requiring users to run their own nodes is a terrible idea and would doom a startup that followed this advice to failure. It would please some early adopters, but make growth essentially impossible.
Basically, set up a scenario where the startup is service-oriented and must stay honest because it competes with its own community for users on the official node. Most users will prefer to trust the startup with their data for ease of use, and there will be an enormous amount of credibility gained by having a mechanism to keep the startup's greed in check.
I've written a response, but I suppose I am hellbanned for new posts. No clue why unfortunately.
The basic idea: prompting is of great importance in this field. You can't get GREAT data through passive measures, but you can get much more consistent data if you consistently prompt. (This is my personal method, at least.)
[edit] It's not so much of a response as it is related - a way to capture data about self. Not so much related to the startup viability discussed in OP's article.
And therein lies a potential headache. What if the API for those 100 services break? What if just 10 of them change their API every month? Are you going to update them? And oh btw, you're gonna keep doing this before you actually have paying customers.
I want to track how much spend, consume electricity or gas, what are my goals.
The problem I found that I forget to enter the data into the app so it is fairly sparse. I can only get meaningful data on a monthly base.
I imagined it would be capable to compare gas (heating) consumption with the weather on the same day but it is not really working (without some automatic data input, that I don't have).
Among others I though about adding mood tracking, attaching 1-3 picture/day so when you want to watch back on your vacation you are not overwhelmed with 5000 photos, tracking locations. I would like to think about it like a social network where I am alone or something like this. (Codename: Track My Sheet :-) )
Showing trends, adding watchers like "you now need 1 year 3 months on your Y Account to reach Z amount at your current rate +Q USD / day, but these kind of things are faster with an excel sheet. Here is a dashboard mockup: http://snag.gy/OozTi.jpg
It is similar to the reportr.io app, actually I started mine on the same week before I first saw it on HN. :-)
Honestly I think this is too much data to make use of. I'm just one person. How much meaning can you get out of what I do? If it is a lot, do I really want that platform to exist on the web?
If the data entry is quick and easy, then I'm not gathering empirical usable data. If I try to gather good data the user will become annoyed.
I think data gathering apps work for a small scope when the user is engaged enough to provide the data. Fitness or body building for example. Or sports car races etc.
This was making me think that maybe a generic statistics app with user generated values/measures/timers might be interesting. But once again, non technical folks probably aren't interested in statics too much.
Maybe gamification and a great UI is the answer here or maybe one should start with a specific problem (workout, drinking, sex or whatever) and broaden the scope as he analyzes user interaction.
I would give such an app a shot, but I'm not sure if I wouldn't forget about it too quickly. Just as I forget about using Lift (the app) and other interesting stuff.
Completely agree that "graphs without the ability to manipulate data are worse than nothing"--it's hard to gain much insight without being able to drill down and filter the data. Simple questions such as "how strong is the correlation between room temperature and sleep" end up having to be refined e.g. with "ignore weekends" and "correct for different amounts of physical activity".
Being able to "get as much data as possible" sounds good, but garbage in, garbage out applies...
Finally, I don't think it's possible to create a single mobile app that suits everyone, which is why Zenobase has a flexible, generic data model and an API.
http://blog.stephenwolfram.com/2012/03/the-personal-analytic...
The biggest challenge imo is privacy and security: How do you safely collect and process everything (from emails to texts)? While I could implement something for my personal use, would anyone give access to all their communications to a third-party service? I wouldn't.
Maybe there is some better way to collect this information (just metadata? do some analytics and store results instead of raw messages?).
http://activitystrea.ms/ is a good first step in opening some of this stuff up for consumption.
The way it is now, each front-end service has to do a ton of work for each data source they want to integrate with. There should be services that translate data into a shared format. And then front-end services can focus on their idea instead of having to parse all the data.
Where do you get your revenue for personal analytics like this? Therein comes super targeted ads with pinpoint accuracy, using predictive analytics at a personal level. If you buy a brand of milk only on Saturdays, rivals have a higher incentive to strut their wares on next Saturday.
In addition to the Quantified Self crowd, we are also (for now, primarily) focusing on the educational space. We are currently hiring developers. Most of the code is Python, but we need front end help also. Contact me at personalanalyticsdc@drbacon.33mail.com if you are interested in learning more.
Maybe it's because the consumer startup crowd and the data science crowd don't overlap enough. If http://prediction.io supported time series data you might see a few of these products pop up overnight.
How does it seem easy? If you just want to track one or two data points and put them on a graph sure, it's easy. If you want to track 100 sources of data, each in a different format with different types of data it becomes quite hard. Then try taking that data and analyzing it. Try looking for correlation between your manually recorded caffeine consumption and your sleep patterns logged with FitBit / BodyMedia / another service.
It's a hard problem with a limited market and limited options for monitization.
I disagree; I think there is enormous potential in this market; having a set of concise and insightful information that shows you are making progress with a particular goal is something a large number of people would be interested in.
Machine learning, simple design, scaling 100 API integrations, and mobile apps are all really, really hard.
Mint is surprisingly bad at automatic categorization (failing on things like large transfers that trump all my other data), so I think there is an opportunity here.
The Secret Digital Ocean: The Data We Aren't Addressing
Ask yourself questions Receive email everyday with the set of questions Reply to email with answers
Login to your dashboard for an overhead view (charts, lists, etc).
An API and Mobile support are in the works.
Send me a note for an invite.
Dispersed services, central location, central analytics.
