Flutter ScrollNotification Only working for one scroll Direction

Question

I have a floating action button I am trying to get to disappear on down scroll but re-appear on scrolling up.

When I test, it disappears perfectly when I scroll down. BUt when I change scroll direction it does not come back:

Expanded(
            flex: 9,
            child: NotificationListener<ScrollNotification>(
              onNotification: (notification) {
                if (notification.metrics.axisDirection == AxisDirection.down) {
                  print("end of scroll");
                  setState(() {
                    fabVisibility = false;
                  });
                } else if (notification.metrics.axisDirection ==
                    AxisDirection.up) {
                  print("scrolling up");
                  setState(() {
                    fabVisibility = true;
                  });
                }

                return true;
              },
              child: ListItems(
                info1: one,
                info2: two,
                info3: three,
                info4: four,
                info5: five,
              ),
            ),
          ),