I would like to access the array with something like array[5000][440] meaning 5000ms from the start and 440hz and it would give me a value of the frequency's amplitude at this very position.
I could not find something like that here, if there is, please point me to it.
You basically want a spectrogram. To get you started, go through your sound file in small chunks, where each chunk is, say, 1/10th of a second, and FFT each of these chunks. (Then, of course, to look up 5000ms and 440Hz, go to the FFT of the appropriate chunk.)