How do I print the half-precision / bfloat16 values from in a (binary) file?

Question

This is a variant of:

How to print float value from binary file in shell?

in that question, we wanted to print IEEE 754 single-precision (i.e. 32-bit) floating-point values from a binary file.

Now suppose that I want to print half-precision (i.e. 16-bit) floats. od doesn't seem to like doing this:

$ od -t f2 c.bin
od: invalid type string ‘f2’;
this system doesn't provide a 2-byte floating point type

and neither does perl's pack...

Bonus points if your answer also covers binary files with bfloat16 (also 16-bit) values.

Answer 1

The following is code generated by SO's "Ask with AI" for IEEE 754 half precision (I asked 2 different questions, merged the codes, and added a missing import):

#!/usr/bin/env python3

import struct
import sys

def half_to_double(half):
    # Pack the half-precision number into bytes
    packed_half = struct.pack('H', half)

    # Unpack the bytes into the corresponding format
    unpacked_half = struct.unpack('e', packed_half)[0]

    # Convert the unpacked value to double-precision
    double = float(unpacked_half)

    return double

# Read 2 bytes from stdin
data = sys.stdin.buffer.read(2)

# Unpack the bytes into a 2-byte integer
half_number = struct.unpack('<h', data)[0]

double_number = half_to_double(half_number)
print(double_number)

Tests under bash or zsh (for the printf \x support), with the 2 bytes ordered for a little-endian machine:

vlefevre@cventin:~$ printf "\x00\x00" | ./tst.py
0.0
vlefevre@cventin:~$ printf "\x00\x3c" | ./tst.py
1.0
vlefevre@cventin:~$ printf "\x01\x3c" | ./tst.py
1.0009765625
vlefevre@cventin:~$ printf "\xff\x7b" | ./tst.py
65504.0

This can be checked on the examples given at Half-precision floating-point format on Wikipedia.

Here's also a version for Perl, but only for normal numbers. The code was also generated by SO's "Ask with AI", but it was wrong (it mixed up 32-bit and 64-bit numbers), so I had to fix it (and also adapt the beginning for the question).

#!/usr/bin/env perl

use strict;

# Code from SO's "Ask with AI" with various fixes.
# For normal numbers only!

# Read the binary16 number from stdin
my $binary16;
read(STDIN, $binary16, 2) == 2 or die;

# Convert the binary16 number to an unsigned short integer
my $unsigned_short = unpack 'S', $binary16;
printf "%04X\n", $unsigned_short;

# Extract the sign bit, exponent, and significand from the unsigned short
my $sign = ($unsigned_short & 0x8000) >> 15;
my $exponent = ($unsigned_short & 0x7C00) >> 10;
my $significand = $unsigned_short & 0x03FF;

# Convert the binary16 components to binary32 components
my $exponent_bias = 15;  # Bias for the binary16 exponent
my $exponent_offset = 127;  # Offset for the binary32 exponent

my $binary32_sign = $sign;
my $binary32_exponent = ($exponent - $exponent_bias) + $exponent_offset;
my $binary32_significand = $significand << 13;

print "$sign $exponent $significand\n";

# Combine the binary32 components into a binary string
my $binary32_string = pack 'N',
  $binary32_sign << 31 | $binary32_exponent << 23 | $binary32_significand;

# Unpack the binary string as a float
my $converted_number = unpack 'f>', $binary32_string;

print "$converted_number\n";

For bfloat16, this is even simpler as it can be viewed as the truncation of a binary32 binary string. So one just needs to shift the 2-byte integer by 16 bits to the left (i.e. insert 16 zeros on the right). Here's the code:

#!/usr/bin/env perl

use strict;

# Read the bfloat16 number from stdin
my $bfloat16;
read(STDIN, $bfloat16, 2) == 2 or die;

# Convert the bfloat16 number to an unsigned short integer
my $unsigned_short = unpack 'S', $bfloat16;
printf "%04X\n", $unsigned_short;

my $binary32_string = pack 'N', $unsigned_short << 16;

# Unpack the binary32 string as a float
my $converted_number = unpack 'f>', $binary32_string;

print "$converted_number\n";

This can be checked on the examples at bfloat16 floating-point format on Wikipedia.

Answer 2

GNU coreutils will add support for this in version 9.5, with the -tfH and -tfB types respectively:

$ printf '\x3F\x80\x00\x00' | od -An --endian=big -tfH -tf2 -tfB -tfF
           1.875               0
           1.875               0
               1               0
                               1