I guess its not surprising given how Jim Simons of Renaissance Technologies fame indicated that of his first 10 employee's almost half were cleaning data.
It's alot more difficult than people realize and you know right away that someone has no real idea of the scale and difficulty of the problem when they suggest that a shell script can solve most of the data issues.
I think Renaissance Technologies actually illustrates just how much a good data cleaning and back testing platform is a real competitive advantage.
A couple of former RenTech people left for Millennium partners and for a couple of years.
Even though these employees were good enough to work at RenTech and had insights into the strategies employed there, they weren't able to be successful on their own without the huge backtesting and data cleaning framework at RenTech.
If your data is from disparate sources then you may need to normalize timestamps across records from different sources, you may be dealing with different languages, identical tokens that mean different things depending on the source, different formatting of numeric fields, etc.
This is an incomplete list, the GP probably has a more exhaustive list of problem types...
https://news.ycombinator.com/item?id=16057016
We get a little bit of luck with people who started hacking back when it was normal to care about assembly language. For example, somebody on our team used to write cartridge-based games for the Atari 800XL computer, which was an 8-bit home computer system. There are not a lot of people like that though, and many of them don't want to move because they have settled down with houses and family.
We get a little luck with people fresh out of high-end engineering schools like Carnegie Mellon and Georgia Tech. Those schools still prepare students for dealing with low-level hacking.
I think much of the trouble is that many people entering college have a focus on games, web sites, and phone apps. Writing low-level code (hypervisor, emulator, exploits, boot loader, OS kernel, compiler back end...) isn't something that seriously enters the mind of the typical student. People pass (or avoid?) their "Computer Architecture CS351" course with MIPS code, and their "Operating Systems CS302" course with Minix, and then they forget that stuff as fast as they can.
One thing that jumps out is that I know many of the tools you list, but I'm basically AVR + Intel these days, so seeing all the architectures listed is a bit overwhelming.
When I see posts like that though I always imagine it comes down to location and salary. For the right price many people would move, if they're not local.
I've seen too many adverts where people want the kind of skills you'd learn over 20+ years of industry employment, with a salary a teenager could live off, and its not too hard to understand why the same jobs get posted year-in, year-out. (Not that I'm accusing you of that, but it's a slippery slope, and filling "impossible" jobs gets easier every time you increment the salary.)
Hmmm. Maybe I'm scaring people off.
For example, we'd hire somebody who is fully focused on x86 if they are good at that. The sort of level we'd be looking for is a person who can recognize the common string.h functions in bare x86 assembly code.
The Arduino usually uses a ATmega8 CPU. I just encountered that CPU, not in an Arduino, and might soon be dealing with it. Adaptability is really desirable; this is a CPU that I've never dealt with before and I'm not about to wimp out.
The ESP8266 has a core based on the Tensilica Xtensa, which I've dealt with.
I think I heard somebody around the office dealing with a Z80, but I'm not sure if I remember that right. Chances are, we've done Z80 work.
It's kind of fun to encounter a new CPU. It's especially neat to encounter one for which step 1 is to write a disassembler.
I certainly do post the same jobs year-in, year-out. That doesn't mean we got nobody. We need more than one person.
Lots of people really won't move. Maybe for a $million they would... but only "maybe"! It is particularly hard to convince people on the west coast that there is civilization elsewhere. There is a fear of being stuck if the job doesn't work out, yet here I am in a city with at least half a dozen large defense contractors and a whole bunch of cyberwar-related startup companies and even some space program work.
