Why do a bitwise-and of a character with 0xff?

Question

I am reading some code that implements a simple parser. A function named scan breaks up a line into tokens. scan has a static variable bp that is assigned the line to be tokenized. Following the assignment, the whitespace is skipped over. See below. What I don't understand is why the code does a bitwise-and of the character that bp points to with 0xff, i.e., what is the purpose of * bp & 0xff? How is this:

while (isspace(* bp & 0xff))
    ++ bp;

different from this:

while (isspace(* bp))
    ++ bp;

Here is the scan function:

static enum tokens scan (const char * buf)
                    /* return token = next input symbol */
{   static const char * bp;

    while (isspace(* bp & 0xff))
        ++ bp;

        ..
}

Answer 1

From the C Standard (7.4 Character handling <ctype.h>)

1 The header <ctype.h> declares several functions useful for classifying and mapping characters.198) In all cases the argument is an int, the value of which shall be representable as an unsigned char or shall equal the value of the macro EOF. If the argument has any other value, the behavior is undefined.

In this call

isspace(* bp)

the argument expression *bp having the type char is converted to the type int due to the integer promotions.

If the type char behaves as the type signed char and the value of the expression *bp is negative then the value of the promoted expression of the type int is also will be negative and can not be representable as a value of the type unsigned char.

This results in undefined behavior.

In this call

isspace(* bp & 0xff)

due to the bitwise operator & the result value of the expression * bp & 0xff of the type int can be represented as a value of the type unsigned char.

So it is a trick used instead of writing a more clear code like

isspace( ( unsigned char )*bp )

The function isspace is usually implemented such a way that it uses its argument of the type int as an index in a table with 256 values (from 0 to 255). If the argument of the type int has a value that is greater than the maximum value 255 or a negative value (and is not equal to the value of the macro EOF) then the behavior of the function is undefined.

Answer 2

Your question:

How is this:

while (isspace(* bp & 0xff))
    ++ bp;

different from this:

while (isspace(* bp))
    ++ bp;

The difference is, in the first example you are always passing the lowermost byte at bp to isspace, due to the result of a bitwise AND with a full bitmask (0b11111111 or 0xff). It's possible that the argument to isspace contains a type that is larger than 1 byte. For example, isspace is defined as isspace(int c), so as you can see the argument here is an int, which may be multiple bytes depending on your system.

In short, it's a sanity check to ensure that isspace is only comparing a single byte from your bp variable.

Answer 3

while (isspace(* bp & 0xff))
    ++ bp;

&&

while (isspace(* bp))
    ++ bp;

Strictly speaking, both are incorrect if bp does not reference unsigned char.

In this case it should be:

while (isspace((unsigned char)(*bp & 0xff)))
    ++ bp;

or better

while (isspace(*bp == EOF ? EOF : (unsigned char)(*bp & 0xff)))
    ++ bp; 

isspace is undefined if parameter is not EOF or it does not have the value of unsigned char

if *bp references char it has to be:

while (isspace((unsigned char)(*bp)))
    ++bp;

Answer 4

From cppreference isspace(): The behavior is undefined if the value of ch is not representable as unsigned char and is not equal to EOF.

When *bp is negative, for example it's -42, then it is not representable as unsigned char, because it's negative and unsigned char, well, must be positive or zero.

On twos-complement systems values are sign extended to bigger "width", so then they will get left-most bits set. Then when you take 0xff of the wider type, the left-most bits are cleared, and you end up with a positive value, lower or equal to 0xff, I mean representable as unsigned char.

Note that arguments to & undergo implicit promotions, so the result of *bp is converted to int before even calling isspace. Let's assume that *bp = -42 for example and assume a sane platform with 8-bit char that is signed and that int has 32-bits, then:

*bp & 0xff               # expand *bp = -42
(char)-42 & 0xff         # apply promotion
(int)-42 & 0xff          # lets convert to hex assuming twos-complement
(int)0xffffffd6 & 0xff   # do & operation
(int)0xd6                # lets convert to decimal
214                      # representable as unsigned char, all fine

Without the & 0xff the negative value would result in undefined behavior.

I would recommend to prefer isspace((unsigned char)*bp).

Basically the simplest isspace implementation looks like just:

static const char bigarray[257] = { 0,0,0,0,0,...1,0,1,0,... };
// note: EOF is -1
#define isspace(x)  (bigarray[(x) + 1])

and in such case you can't pass for example -42, cause bigarray[-41] is just invalid.

Answer 5

In c char can be signed or unsigned https://en.wikipedia.org/wiki/C_data_types

When passed to isspace, bp will be promoted to an integer. If it is signed and the high bit is set then it will be sign extended to become a negative integer. This may mean it is not an unsigned char or EOF as required by the isspace function https://linux.die.net/man/3/isspaceNo

See http://cpp.sh/9mp2i for how it changes the bitwise and changes value of that isspace sees

Answer 6

If we assume bits of char type are always 8,
then the code bitwise-and operator with 0xff here will confuse us.

But what about that if char type is not always 8-bits?
Then 0xff may have another meaning, right?

Actually, the char type is not always 8-bits and we can see the detail in C99 standard. The char type in standard is not defined as 8 bits.

The following is how C99 standard describe the size of char type.

6.5.3.4 The sizeof operator When applied to an operand that has type char, unsigned char, or signed char, (or a qualified version thereof) the result is 1. When applied to an operand that has array type, the result is the total number of bytes in the array.) When applied to an operand that has structure or union type, the result is the total number of bytes in such an object, including internal and trailing padding.

6.2.5 Types An object declared as type char is large enough to store any member of the basic execution character set. If a member of the basic execution character set is stored in a char object, its value is guaranteed to be positive. If any other character is stored in a char object, the resulting value is implementation-defined but shall be within the range of values that can be represented in that type.

For example, TMS320C28x DSP from Texas Instruments has a char with 16 bits.
For the compiler specifies here, CHAR_BIT as 16 on page 99.

This appears to be a modern processor (currently being sold), compilers supporting C99 and C++03.