Fisheye effect in GameMaker Ray Casting

Question

I've recently made an easy ray caster in Game maker just with a 2d game where you walk and collide with walls and then i added a ray caster to it:

function cast_ray(sx, sy, sdir) {
    var rayx=sx
    var rayy=sy
    
    while true {
        if collision_point(rayx, rayy, obj_wall, false, true) or rayx<0 or rayx>room_width or rayy<0 or rayy>room_height {
            break
        }
        
        rayx+=lengthdir_x(1, sdir)
        rayy+=lengthdir_y(1, sdir)
    }
    
    return [point_distance(sx, sy, rayx, rayy), [rayx, rayy]]
}

This fires a ray and continues it until it touches a wall or exits the game room. When that happens, the function returns the length of the ray and it's collision point. I then added a draw_rays() function that fires the number of rays that corresponds to the variable resolution, gets their length and finally calculates the height they should have on screen (rotation corresponds to player rotation):

function draw_rays() {
    var rot_start=rotation-resolution/2
    var ray_width=room_width/resolution
    
    for (var r=0;r<resolution;r++) {
        var ray=cast_ray(x, y, rot_start)
        var rayd=ray[0]
        //show_debug_message(rot_start)
        var rayh=clamp(50000/rayd, 0, room_height)
        var col=make_color_rgb(clamp(rayh, 0, 255), clamp(rayh, 0, 255), clamp(rayh, 0, 255))
        
        draw_set_color(col)
        draw_rectangle(r*ray_width, height-rayh/2, r*ray_width+ray_width, height+rayh/2, false)
        rot_start+=1
    }
}

Everything works fine untill you approach a wall:

I tried changing one part of my code as everyone was doing to counter this effect (6th line of second code): var rayd=ray[0] to var rayd=ray[0]*cos(degtorad(rotation-rot_start)). It worked only for the walls you are facing but the walls on your side got worse

Answer 1

This is a classic problem with ray casting, as described here.

Should be solvable by doing something like

rayd *= dcos(rotation - rot_start);

before the rayh calculation.