Getting "Return from incompatible pointer type" warning, but i don't see why

Question

I'm trying to make a maze generator in C, but i get a warning for some reason. If the answer has something to do with memory allocation, please explain pretty in depth because i have no idea on how that works.

#include <stdlib.h>
#include <stdio.h>

unsigned char ** gen(int w, int h){
    unsigned char maze[w+2][h+2];

    for(int i=0; i<w; i++){
        if(i==0 || i==w-1){
            for(int j=0; j<h; j++){
                maze[i][j]=1;
            }
        }else{
            for(int j=0; j<h; j++){
                if(j==0 || j==h-1){
                    maze[i][j]=1;
                }else{
                    maze[i][j]=0;
                }
            }
        }
    }
    return maze;
}

I have looked around on the internet but i have not found anything that works.

edit: fixed typo in code

Answer 1

You have two main problems here.

First you're trying to return an array of type unsigned char [w+2][h+2] from a function that expects the return type to be unsigned char **. These types are incompatible.

While an array does decay to a pointer to its first element, this only applied to the outermost dimension, not all of them. So an array of type unsigned char [w+2][h+2] actually decays to a pointer of type unsigned char (*)[h+2], i.e. a pointer to an array of usigned char of size h+2, and this type is not compatible with unsigned char **.

Second, you're attempting to return a pointer to a local variable, in this case the 2D array. That array's lifetime will end when the function returns, so attempting to use the returned pointer will trigger undefined behavior.

You can fix the local return issue by dynamically allocating space for the 2D array:

unsigned char (*maze)[h+2] = malloc((w+2) * sizeof *maze);

For the return type mismatch, you can change the function to return a pointer to an array:

unsigned char (*gen(int w, int h))[] {

And assign the return value to a pointer of the proper type:

unsigned char (*m)[h+2] = gen(w,h);

Note here that the function returns a pointer to an array of unspecified size. This is necessary because the size of the pointed-to array is not known at compile time. However it is valid code because it is allowed to convert between a pointer to an array of known size and an array of unspecified size.

On a side note, you have a bug at the start of the function:

for(int i=0; 1<w; i++){

This should be:

for(int i=0; i<w; i++){

And this does nothing:

maze[i][j]==0;

It should instead be:

maze[i][j]=0;

Answer 2

You declared a local variable length two-dimensional array with automatic storage duration:

unsigned char maze[w+2][h+2];

that will not be alive after exiting the function.

So returning it from the function does not make sense. Dereferencing the returned pointer from the function will invoke undefined behavior.

You should allocate the array dynamically.

As for the compiler message then the array used in the return statement:

return maze;

is implicitly converted to the type unsigned char ( * )[h+2].

but the return type of the function is unsigned char **:

nsigned char ** gen(int w, int h){

These pointer types are not compatible.

Note that your code has other problems; for example, a typo in this statement:

for(int i=0; 1<w; i++){
             ^^^

And it is unclear why the array is defined using sizes w + 2 and h + 2:

unsigned char maze[w+2][h+2];

but in for loops there are used w and h.

Also maybe you should initialize elements of the array with characters '0' and '1' instead of integers 0 and 1.

You could initialize your array without using if-else statements.

For example if you have an array declared like:

unsigned char maze[w][h];

then you can write:

    for (int i = 0; i < w; i++)
    {
        for (int j = 0; j < h; j++)
        {
            maze[i][j] = !( i && j ) || !( w - 1  - i && h - 1 - j ) ;
        }
    }

Or of you want to initialize the array with characters '0' and '1' then the assignment statement can look like:

maze[i][j] = ( !( i && j ) || !( w - 1  - i && h - 1 - j ) ) + '0';