Pulseaudio/alsa : slow playback device wake-up

Question

I have a Debian machine (3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux) on which I have some audio devices connected. The devices that are stereo works well, but I have a problem with a mono headset. When I type the command

aplay -v -D plughw:2,0 ~/piano2.wav

the device waits up to 3-4 seconds before starting to output sound. If I retype the command in the following 5 seconds, the sound is played directly, but if I wait a bit more, I have to wait 3-4 seconds again before hearing anything.

Here is the output when I run the command above :

Playing WAVE '/home/console/piano2.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
Plug PCM: Route conversion PCM (sformat=S16_LE)
  Transformation table:
    0 <- 0*0.5 + 1*0.5
Its setup is:
  stream       : PLAYBACK
  access       : RW_INTERLEAVED
  format       : S16_LE
  subformat    : STD
  channels     : 2
  rate         : 48000
  exact rate   : 48000 (48000/1)
  msbits       : 16
  buffer_size  : 24000
  period_size  : 6000
  period_time  : 125000
  tstamp_mode  : NONE
  period_step  : 1
  avail_min    : 6000
  period_event : 0
  start_threshold  : 24000
  stop_threshold   : 24000
  silence_threshold: 0
  silence_size : 0
  boundary     : 6755399441055744000
Slave: Rate conversion PCM (16000, sformat=S16_LE)
Converter: libspeex (builtin)
Protocol version: 10002
Its setup is:
  stream       : PLAYBACK
  access       : MMAP_INTERLEAVED
  format       : S16_LE
  subformat    : STD
  channels     : 1
  rate         : 48000
  exact rate   : 48000 (48000/1)
  msbits       : 16
  buffer_size  : 24000
  period_size  : 6000
  period_time  : 125000
  tstamp_mode  : NONE
  period_step  : 1
  avail_min    : 6000
  period_event : 0
  start_threshold  : 24000
  stop_threshold   : 24000
  silence_threshold: 0
  silence_size : 0
  boundary     : 6755399441055744000
Slave: Hardware PCM card 2 'Jabra PRO 9460' device 0 subdevice 0
Its setup is:
  stream       : PLAYBACK
  access       : MMAP_INTERLEAVED
  format       : S16_LE
  subformat    : STD
  channels     : 1
  rate         : 16000
  exact rate   : 16000 (16000/1)
  msbits       : 16
  buffer_size  : 8000
  period_size  : 2000
  period_time  : 125000
  tstamp_mode  : NONE
  period_step  : 1
  avail_min    : 2000
  period_event : 0
  start_threshold  : 8000
  stop_threshold   : 8000
  silence_threshold: 0
  silence_size : 0
  boundary     : 9007199254740992000
  appl_ptr     : 0
  hw_ptr       : 0

And this is my .asoundrc file :

    pcm.!default {
        type plug
        slave {
                pcm "hw:0,0"
        }
}

pcm.device1 {
        type plug
        slave {
                pcm "hw:1,0"
        }
}

pcm.device2 {
        type plug
        slave {
                pcm "hw:2,0"
        }
}

pcm.device3 {
        type plug
        slave {
                pcm "hw:3,0"
        }
}

ctl.!default {
        type hw
        card 0
}

ctl.device1 {
        type hw
        card 1
}

ctl.device2 {
        type hw
        card 2
}

ctl.device3 {
        type hw
        card 3
}

Does anybody have and idea why I get such a delay when my device wakes-up ?

Thanks