I'll try to address the comments on here, so ask any questions you have!
My approach is a tad different though. My approach works by using the microphone. And I have the code to extract notes from a microphone via a FFT. Allowing for typing with say a guitar.
Extracting single notes works quite well. I haven't gotten around to test chords just yet, but I think I have the fundamentals there.
I'm a bit short on musical knowledge and instruments so so far I've only tested it with my voice (ugh), synthesized sounds and partially filled water bottles.
Also I have a suggestion for a graphical interface that would make the learning curve easier and making it language/alphabet independent.
Please reply to this comment if you are interested in integrating this.
RE: integration with what I did - I'm hoping to release the source code one day, then you'd most likely be free to use that code however you want (I'll probably MIT-license it). But as for my involvement, I'd like to be done with this project.
Slight tangent--I've often wondered if the principles behind audio keyloggers, where you record typing sounds and try to reconstruct what was typed, could be used to make a wireless computer keyboard that never needs batteries. Seems like you're attacking an inverted version of that problem.
Interesting stuff, thanks!
Kudos!
Which can be seen in action here: https://www.youtube.com/watch?v=HrUNrb3ihl0
And no, no April Fools here ;-)
Q: Can you play a real song so that we can see what it types out?
A: This isn't as satisfying as you may think since the entire typing range is only about two octaves, and most songs exceed that range, so nothing actually gets typed when you play those notes. Anyway, here's Wonderwall: Iuiuigaezobzgbxgebiuiuigaezobzgbegz
(that's actually the right hand of Für Elise)
Video of the stenotype machine at the Italian Senate (audio in Italian) https://youtu.be/e-z1ZAsQjio?t=18
Wikipedia page about the writing method (unfortunately only available in Italian, you may try Google Translate) https://it.wikipedia.org/wiki/Metodo_Michela
How to build a Michela Stenotype machine by modifying a MIDI keyboard: https://www.youtube.com/watch?v=q9Y9jtOB7G0
OpenSteno SW that supports various Stenotype machines including the Michela machine built using a MIDI keyboard as seen above http://www.openstenoproject.org/
https://en.wikipedia.org/wiki/File:Telegraph_Keyboard.jpg
https://en.wikipedia.org/wiki/File:Printing_Telegraph.jpg
See the article for some words on the topic: https://en.wikipedia.org/wiki/Printing_telegraph
> Hughes telegraph devices, which also had piano style keyboards, were very popular in France, where there were likely many more piano and harpsichord players than telegraphers.
But I'm not too sure how useful it is for efficiency even after practicing so much with it; there are a lot of false positives and negatives since keypress velocity doesn't matter for regular typing. With enough data, you could figure out a specific velocity threshold for each key so that you make fewer mistakes, and then maybe it'd be more helpful.
It can be made in a mechanical way, like a tristate button -- off, slighly-on, on.
I also suspect that the keyboard layout is more standardized between keyboards than the force threshold would be.
(And, it occurs to me that we're essentially chording with the shift key.)
Someone needs to make a velocity-sensitive mechanical qwerty keyboard :)
Edit: apparently "bigram" is the word I was looking for instead of "pair of letters".
To do that same process for musicality would be challenging since your average English sentence contains so many different letters. I would probably do something like assign each letter a set of sounds, then pick one of those sounds based on the rest of the sentence.
Here is the twitch channel where he broadcasts daily the game development: https://www.twitch.tv/adam13531
Here is the game he is developing ("An online strategy game with a focus on automation"): https://play.bot.land
3 pedals and 2 feet gives you 6 I think.
I briefly tried to find a youtube video of someone using it, but no joy-- I guess it's too boring to watch someone type. :)
https://www.youtube.com/watch?v=sjtZxjEH6cM
(if you've not seen that video, it's a little mind-bending)
(it's not really worth watching if you're satisfied with the YouTube video )