Implementing an Augmented Reality Piano

Question

I´m working on an AR piano and I got stucked on trying to solve this problem. The project consists on a paper piano in which theuser will be able to play the paper piano and via AR the app will recognize what keys are being played.

My image target is pretty similar to this one:

I´m using Vuforia along with Unity 3D for the project, and what I´ve done until now is to add a virtual button to each key. This works pretty good with the camera right above the image target, but I´m getting the problem that when the user plays chords there are other keys that are hidden, making the app play notes it should not play. The same happens when the user plays a black key.

I saw other similar projects on youtube and found that this one does a really good job recognizing black notes and chords:

https://www.youtube.com/watch?v=FHJ_I6DmFNw

The big difference between the video and my project is that they don´t put the phone right above the image target. Putting the phone in front of the piano as in the video helps because the camera now sees which specific keys are being pressed without the rest of the hand hiding other keys. Another thing I noticed is that the piano does not have the letters on the keys, and in my project if I remove the names of the notes I can´t get the virtual buttons to work because the virtual buttons have to be placed on rich images that differ from the rest of the image target.

The problem is that when I put my image target and camera in this position, I can´t get the app to recognize the image target nor the buttons.

Do I have to change my image target to put it in perspective? Or I´m I using the wrong AR SDK for this purpose? Also how can I be able to remove the notes names and recognize the position of the user´s fingers?

I have tried lots of things now and any help regarding how to solve this and achieve something similar to what´s one the video above will be really useful for me.

Thank you

Answer 1

I just saw the video. Looks like a hoax. But since you're out to make it real, one thing I noticed is that they don't play the black keys.

So if you're trying to get to play the black keys you could problem have a block() function that detects when a black key, as well as the two white keys below/next to it are covered. And hence play only the tone of the black key if that should happen.

For eg: If F# is played, then

if(pressed(Fsharp) && pressed(F) && pressed(G)){
playsound(Fsharp);
}

And just to make it works efficiently, try making sure you check for these conditions first before checking if a single key is clicked. That way, you could start at CheckNumberofKeysDown(); and only then start a thread that checks for if the keys are together by using the keys within an enum or array keys[] and use conditions like

//Second touched key compared with the first key's neighbor
if(key[touched[1]]!=key[touched+1]){ 
playsound(touched[0]);
playsound(touched[1]);
}

So these are just a few ideas that crossed my mind while trying to figure it out. The angle they've used it almost unreal. But having the camera face you from the other side would be ideal since you would get a better view of all the keys.

Yes, clearly you'll need a lot of R&D to work perfectly.Hope this helps you somehow. And good luck!