Wrong result in a determinant calculation using int data type in C

141 views Asked by At

i want to know

why is not possible to calculate all this using int data type?

So, im trying to calculate triangle area using determinants in C, im using turbo C 2.0 (i know is old), im getting a calculation inaccuracy when i use int data type, dont really know why, heres the code im using to calculate determinants

#include <stdio.h>
#include <conio.h>

int Area(int ax, int ay, int bx, int by, int cx, int cy)
{
    return ax * by - ay * bx + bx * cy - by * cx + cx * ay - cy * ax;
}

int main()
{
    int ax, ay, bx, by, cx, cy;

    ax = 548;
    ay = 125;
    bx = 544;
    by = 344;
    cx = 68;
    cy = 209;

    clrscr();
    printf("%d", Area(ax, ay, bx, by, cx, cy));

    getch();
    return 0;
}

using int data type it returns -26288 when it should be 104787.

i know this returns the area * 2 and it can be negative, but thats not the problem here, since im using it for a convex hull algorithm.

int data type in my system is 16 bits so i know i can store 2^16 numbers, i thought it could be a max int error but i tried using long int as well which is 32 bits long, it didnt work, it only works when i use floating point data types such as double or float like so:

float Area(float ax, float ay, float bx, float by, float cx, float cy)
{
    return ax * by - ay * bx + bx * cy - by * cx + cx * ay - cy * ax;
}

with this i get the right answer, but the question i have is, again,

why is not possible to calculate all this using int data type?

am i using int in a wrong way?

Thank you!

1

There are 1 answers

0
Antonio Escorcia On

the answer was simple, it is because I was still using int as arguments, since int here was just 16 bits, the maximum positive number It can represent is given by 2^16 / 2 - 1 = 65,536 / 2 - 1 we divide by 2 cause it also represents negative values and - 1 cause it also represents the number 0 so we get 65,536 / 2 - 1= 32,767 the result 104784 clearly overflows the maximum positive number It can represent so it truncates to -26288 to fit int data type.

I was kind of tired to notice I was still using int variables as arguments, I already knew this but thanks a lot to @MikeCAT for making me double check so I could realize.

I thought I was using long int as arguments for some reason when I wasn't.

long int Area(int ax, int ay, int bx, int by, int cx, int cy)
{
    return ax * by - ay * bx + bx * cy - by * cx + cx * ay - cy * ax;
}

so ax * by - ay * bx + bx * cy - by * cx + cx * ay - cy * ax expression was still being computed as int, I just needed to change arguments data type as well. right way:

long int Area(long int ax, long int ay, long int bx, long int by, long int cx, long int cy)
{
    return ax * by - ay * bx + bx * cy - by * cx + cx * ay - cy * ax;
}