1st rule of Big Data Club
Decide what knowledge you want to get — Then explore how you can generate information that can support this give you this knowledge, and finally you can begin to exploit the data you have at your disposal to generate this information.
The trick is not to generate lots of data, but to know how to extract information from it. Big Data™ Tools can assist in exploiting the data you have, to generate new more meaningful data, but that doesn't make it information, it's just more data.
Note that I distinguish between knowledge, information, and data.
One of the biggest challenges we had was introducing all kinds of new technology but minimal impact on bottom-line. In fact, it probably caused a great deal of productivity and cultural issues that just slows everything down.
Regarding your point however, I think you might have missed my point by a little bit, so I'll try to clarify.
I'm not advocating for existing tech. I love new technologies, and they're often a better fit than what came before. What I'm advocating for is that instead of spending time and money on investigating new technologies and trying to find problems you could solve with them, you instead spend time on finding and identifying problems that your organization is running into, or developing a vision on where you want your company to be in 5+ years.
I believe that the best and most suitable technology is always already out there, as long as you know what you need to build.
The thing is of course that looking at new technologies is easy, and identifying worthwhile problems to solve is very, very hard. But that shouldn't stop us from considering that way of thinking.
(1) blockchain: most arguments about blockchain always being replaceable with a traditional single node central database conveniently leaves out the constraint for decentralized control. Yes, if you change the ideal goals for the solution, of course you can propose the traditional solution. (E.g. a government central database of property records does not solve the same problem as a decentralized blockchain of property records.)
The problems that blockchains are attempting to solve is real. The more interesting discussion is whether the costs of implementation (whether Proof-of-Work or Proof-of-Stake) will ever deliver on that promise. Two possibilities: (1) subsequent failed attempts of blockchains eventually lead computer scientists to a theorem that states any decentralized scheme always costs more than the economic value returned. (The theorem would be similar to the ironclad conclusions of CAP theorem or Godel's Incompleteness Theorem and also similar to discovering you _can_ synthesize gold but the cost (of a nuclear reactor) to do it costs more than the gold itself.) Or (2) more computer science thinking finally makes blockchains feasible for real world applications. Feasibility includes cpu costs, transactions-per-second, 51% attacks, rogue forks, etc.
(2) Hadoop map reduce: he writes...
>"The entire business plan of a ton of companies [...] set up these huge data clusters based on Hadoop and write complex MapReduce queries for basic operations, and wait for the spice to flow. But there’s no end game. When asked, there’s no vision on what to do with the data that will actually make money"
Author should spell out exactly which data companies are spending millions gathering hundreds of terabytes/petabytes with zero clue what to do with it.
Yes, there are all sorts of examples out there of "solutions looking for a problem" but this particular essay is empty of insightful data points.
He specifically says that virtual currencies themselves are a valid reason to use the Blockchain.
For his blockchain example, I couldn't find his Dutch handicap parking story via Google for more details so I don't know if a traditional central database would solve the same exact problem.
I'm saying that virtually all dismissals of a blockchain in favor of central databases almost always removes the benefits of decentralization. The ironic part is that the skeptics don't realize that the central db "solution" is incomplete when compared to all the decentralized goals (e.g. multiple witnesses guarding against tampering, potentially lower cost ownership verification, etc). Yes, if one changes the rules or moves the goal posts around, one can replace a decentralized blockchain with MySQL. If somebody is the skeptic, does he even notice that he performed that sleight-of-hand in his argument?!?
So, a theorem which says decentralised scheme "costs" more than economic value is not possible to have.
For example, say gold is very expensive to synthesise. But if it is absolutely necessary for some job the cost becomes worth i9t and the price would reflect that.
The point is that many of the proposed uses don't actually benefit from decentralization in any meaningful way, so you might as well save a pile of headaches and use a standard database.
Mining is centralized but bitcoin consensus is decentralized.
blockchain != Bitcoin
blockchain != Ethereum
Maybe on a Google scale of data, that doesn't work as easily. Maybe when you have a billion dollar infrastructure bill, big data works better. But it leaves out 99% of companies who's data isn't that big.
But seriously, I just came off a project last year that used Hadoop "because" and for no other discernible reason. I personally did a ton of studying on Data Science and then...crickets. Couldn't find anyone who really wanted that kind of work done. Maybe in time.
From what I've seen so far it seems like the super big tech companies pay _all the money_ for ML/DS people; outside of that the pickings seem to get slim quick.
Hadoop provided a data-centric approach to parallel computing. Using Cascading, a high level pipe/filter library for Hadoop, we could make complex, locally testable models. Using eventing we could plug those models into a self-managing workflow. Adding a new model meant starting a JVM for that model that hooked into the event system and ran Hadoop jobs as needed. If any one model failed, we could rerun it without affecting the rest of the system.
This scaled to 60 some odd fraud models that looked over up to 5 years worth of data (5 TB). Some were quick since they only looked at a day's worth of data. Some took several hours. In the end, Hadoop made the entire process easier to handle mentally, testable, and scalable.
The Amish decide 200 years ago 'lets stick with current technology'. It works, they have houses, food, friends, a living.
Why go further? This topic is almost existential :)
https://en.wikipedia.org/wiki/Amish#Use_of_technology_by_dif...
The valley sickness in a nutshell.
I argue for creating actual value for users, on the long term.
finance -> spreadsheets
"helping kids with the homework" -> word processing
"the little lady can store her recipes" -> small single-user databases
Sounds to me like they all panned out except home automation.
Using a computer to store recipes was wildly impractical. Using it to do basic CRM or point-of-sale tasks with a small relational or navigational database was much more appropriate.
That's a great perspective, and I'm really glad you mentioned this in particular. It's one of those things that was so often repeated, and I always found it to be really bizarre. Recipes? Really?
I'd add music synth to your list of real world uses, in some sense that was a killer app for microprocessors.
https://en.wikipedia.org/wiki/Honeywell_316
Seriously.
You're right. The Altair 8080 famously didn't do anything useful at all upon its release. It wasn't until someone in the Homebrew Computer Club discovered that the radio interference its CPU generated could play tones on an AM radio by buzzing the CPU in loops that a use was found for it.
Now, you could argue that it makes more sense to try to make the iPhone instead of a me-too system like android originally was. But most companies aren't apple of 2005, and they know it. They know they aren't really leading in anything - they just don't want to be left in the dust. So they ask their tech teams to investigate stuff that might be transformational, like the blockchain, because if it did create a new green field, they want to have access to the new market and at least near-first mover advantage.
I think new technologies are awesome. But I think you shouldn't look at a technology and ask "which problems should I solve with this?", but rather at the problems you have and then consider "which technologies should I use to solve this problem?".
The last one does mean you have to become good at identifying problems in your company or product offering, which is much harder than finding technologies. But I think that is a very worthwhile goal.
http://webcache.googleusercontent.com/search?q=cache:bZt3Y3m...
Which is not that bad in Google translate.
Essentially:
Blockchain when requesting wheelchairs Blockchain is a new technology whose impact is compared to the arrival of the Internet. The Quality Institute of Dutch Municipalities (KING) organizes pilots to investigate what Blockchain technology can mean for municipalities. Stichtse fought with the research question whether Blockchain is useful when requesting a wheelchair. The pilot shows that Blockchain can offer many benefits. For example, the process of requesting issue with Blockchain is faster and more transparent. Residents with Blockchain have more views on the status of their application, as with Track & Trace in a mail package. In addition, the administration around the application becomes a lot easier for all parties.
On the original page there is a link to a dedicated site:
http://hostedby.frogjump.nl/blockchain-magazine#!/stichtse-v...
where there is a link to an actual .pdf (largely in English):
http://cdn.instantmagazine.com/upload/6826/blockchain-sticht...
with a detailed process analysis from which it is clear how the "blockchain" has actually no meaning/usefulness in the whole stuff the "problem" (or "non problem") can seemingly be solved by much more traditional technical approaches (which is the article Author's thesis).
Taking a step back and looking at it more generally I see this as problem solving vs problem finding (in Alan Kay's words), where a perfect example would be the hype with self driving cars and high speed car tunnels. This is problem solving (traffic jams, car accidents, "I need to get there faster"). Whereas problem finding will ask the bigger questions that will lead to better results. In the question of transportation I think you should set up yourself to the problem "How do we build car free cities?"
The actual underlying issue however is that finding and identifying problems to be solved is _hard_. It's much easier to focus on something concrete like a piece of technology.
An example is, well, your example. The problem "How do we build car free cities?" is already much better than "How do we avoid traffic jams". But it's simple to solve that problem, for example by banning all cars; that doesn't solve the actual problem though (and is a bit silly).
The actual problem (I think) is not that there are cars, the problem is that there are too many traffic movements necessary over a physically too limited space to get everyone's needs satisfied.
Rephrasing the problem like that allows you to consider the various aspects of the problem; how can we reduce traffic movements (public transport, carpooling)? How can we more optimally use our limited space (smaller cars, stimulate bike usage and bike lanes)? And what are the needs that are being solved by traffic movements, and could we satisfy those without them (working remotely, grocery deliveries)? And what would it be worth to us to reach these goals, compared to the problem we are solving?
Perhaps I'll write an article on this specific mode of thinking. If I can actually find a way to get it on paper of course.
For example, I have 12 5/4 cedar boards in my wife's car for storm window frames. I had to drive her Vibe to Jacksonville, which is an hour drive one way. No one around here sells cedar, let alone in 5/4 width. I can't get that delivered. Even if I wanted to, I need to pick the boards since I have aesthetic requirements.
How do we solve the small town problem?
The solution is public transport; it seems you already have a train that goes to Jacksonville, though it's rather slow; the Brussels-Bruges line is about the same distance and takes only 1h, even with intermediate stops.
As for not having delivery, that's solved by the question itself: if people didn't have cars, they would definitively deliver!
As for choosing the boards, that's hardly a problem, you go to the shop, choose them and they'd package those specific boards and deliver them. That's fairly common for large items, at least here in Europe, where people (even small town dwellers) tend to have small cars.
Real consensus about a specific technology stack takes at least a couple of years to build up. If a tech product becomes popular very quickly, it is purely because of its superficial qualities - Often at the expense of finer details.
- microservices architecture
- using document data stores despite your data being relational
