Typing with a Piano [video] (opens in new tab)

Do not be afraid to release bad source code. It's fine.

You beat me to it! I've been working on similar thing, so I'm a bit jealous that you published this before me! Then again it makes me happy that I'm not the only one with far out ideas like this!

My approach is a tad different though. My approach works by using the microphone. And I have the code to extract notes from a microphone via a FFT. Allowing for typing with say a guitar.

Extracting single notes works quite well. I haven't gotten around to test chords just yet, but I think I have the fundamentals there.

I'm a bit short on musical knowledge and instruments so so far I've only tested it with my voice (ugh), synthesized sounds and partially filled water bottles.

Also I have a suggestion for a graphical interface that would make the learning curve easier and making it language/alphabet independent.

Please reply to this comment if you are interested in integrating this.

Love the details for completeness, like modes, and your discussion of adjustments you found necessary along the way. Great job on the project and great explainer video too.

Slight tangent--I've often wondered if the principles behind audio keyloggers, where you record typing sounds and try to reconstruct what was typed, could be used to make a wireless computer keyboard that never needs batteries. Seems like you're attacking an inverted version of that problem.

Interesting stuff, thanks!

This was really cool to watch! The youtube video offered a nice explanation and I actually enjoyed the audio/visual-feedback from watching you type on it.

Kudos!

If you love these kind of things you will be blown away by this: https://codeklavier.space/

Which can be seen in action here: https://www.youtube.com/watch?v=HrUNrb3ihl0

And no, no April Fools here ;-)

It might be fun/interesting to pipe pieces transcribed to MIDI into your system and see what falls out. I'm sure it'll be nonsense, but perhaps fun nonetheless. (e.g. http://www.piano-midi.de/chopin.htm)

What were the key or note chording patterns you used to make this?

My friend is a professional pianist and wants to see how high he can get his WPM on typing - is the source available?

I just wanted to add that, in order to use the Michela Method with Plover (the software from OpenSteno), you will also need the Plover-MIDI plugin: https://pypi.org/project/plover-midi/ (both "plover" and "plover-midi" are on Pip)

Wow the stenotype machine at the Italian Senate is actually a piano keyboard!

What's fun is that it's the only time when typing in all caps actually makes you feel like you're shouting!

But I'm not too sure how useful it is for efficiency even after practicing so much with it; there are a lot of false positives and negatives since keypress velocity doesn't matter for regular typing. With enough data, you could figure out a specific velocity threshold for each key so that you make fewer mistakes, and then maybe it'd be more helpful.

I don't think I would. I can coordinate moving two fingers (shift and the character) easier than I can control the force that I hit a key with... at least I think I can. Some of that may be training, though.

I also suspect that the keyboard layout is more standardized between keyboards than the force threshold would be.

(And, it occurs to me that we're essentially chording with the shift key.)

Yea that's a good one. Same with numbers and special characters.

Someone needs to make a velocity-sensitive mechanical qwerty keyboard :)

I find that space being always the same note does bring some more musicality actually! It brings structure to the sound instead of sounding like a mess of random notes

Obviously space should be remapped to a hit on a snare or bass drum.

Something I didn't mention in the video is that I had to tweak the positioning of certain letters due to common digrams in English. For example, I originally had "v" as a right-hand thumb key due to its frequency as a standalone letter, but some common English words have the digram "ev" ("ever", "every", "never"), and "e" was also a thumb key, so I remapped the "v" to my pinky to make those words much more smooth.

To do that same process for musicality would be challenging since your average English sentence contains so many different letters. I would probably do something like assign each letter a set of sounds, then pick one of those sounds based on the rest of the sentence.

Here's the chapter on chord keybords http://billbuxton.com/input06.ChordKeyboards.pdf from Bill Buxton's work-in-progress book Human Input to Computer Systems: Theories, Techniques and Technology http://billbuxton.com/inputManuscript.html . Steno keyboards https://en.wikipedia.org/wiki/Stenotype are chording keyboards. The Velotype http://www.velotype.com/en/ and Veyboard http://www.veyboard.nl/en_main.html use a similar system that would be a lot more suitable for general-purpose text input. Then there's the important niche of one-handed chording keyboards, like the one Doug Engelbart designed for the NLS system, and the mobile, wireless Twiddler3 https://twiddler.tekgear.com/ .

404?

The April Fool's joke was that I'd even bother spending my time trying to learn this, but I assure you, it's all real! I linked to this elsewhere in the thread, but here's a Twitch VOD of practically nonstop typing tests for a few hours: https://www.twitch.tv/videos/404585365

(it's not really worth watching if you're satisfied with the YouTube video )

This is a thing, of course. It can be used for playing midi on your laptop, and such.