I don't know what the problem is because this works perfectly for SIN and COS. For TAN, it returns 0.0000 for 50.0 radian. But if I enabled the commented line, it works as expected. That's weird because TAN is supposed to return a double in XMM0, not RAX.
;./fasm testing.asm
;gcc -s testing.o -o testing -lm
format elf64
extrn printf
extrn tan
section '.data' writeable align 16
rad dq 50.0
fmt db "%.5lf",0ah,0
section '.text' executable align 16
public main
main:
    push rbp
    mov rbp,rsp
    pxor xmm0,xmm0
    movq xmm0,[rad]
    call tan
    ;movq rax,xmm0  ;works if I enable this. but SIN and COS don't need this
    mov rdi,fmt
    call printf
    mov rsp,rbp
    pop rbp
    ret
What could be the problem here?
 
                        
When calling any function in x86-64 assembly,
ALmust contain the number of registers used. It's the convention, you can't avoid it.You can consult the System V Application Binary Interface, chapter 3.2.3 Parameter Passing.
Therefore, you need to specify the number of parameters you use in
rax.should suffice.