SwiftUI: Pager view with vertically draggable pages

Question

I am trying to create a pager view where each page can be dragged vertically. For that I am using TabView where each page is VerticalDraggableView. The later has a viewBuilder whose resulting view we shall call Label.

I almost have it working. The pages can be dragged vertically when I tap directly on the Label, but then they cannot be dragged horizontally, i.e. TabView does not recognize the touch.

If I tap slightly under the included Label I can change pages, but I would like it to work when I tap on the included view as well.

Here's my code, followed by a short video

TabView {
    ForEach(users) { user in
        VerticalDraggableView {
            UserCardView(user: user)
        } onSwipe: {
            self.viewModel.didSwipeUser(user, swipeDirection: $0)
        }
    }
}
.tabViewStyle(.page(indexDisplayMode: .never))

And for VerticalDraggableView

enum SwipeDirection {
    case up
    case down
}

struct VerticalDraggableView<Content: View>: View {
    @State private var dragOffset = CGSize.zero
    @State private var isDragging = false
    
    private let content: () -> Content
    private let onSwipe: (SwipeDirection) -> Void
    
    init(@ViewBuilder content: @escaping () -> Content,
         onSwipe: @escaping (SwipeDirection) -> Void) {
        self.content = content
        self.onSwipe = onSwipe
    }
    
    var body: some View {
        content()
            .offset(x: 0, y: dragOffset.height)
            .scaleEffect(isDragging ? 1.1 : 1.0)
            .simultaneousGesture(
                DragGesture()
                    .onChanged { value in
                        self.dragOffset = value.translation
                        self.isDragging = true
                    }
                    .onEnded { value in
                        if value.translation.height > .dragThreshold {
                            onSwipe(.down)
                        } else if value.translation.height < -.dragThreshold {
                            onSwipe(.up)
                        } else {
                            self.dragOffset = .zero
                            self.isDragging = false
                        }
                    }
            )
            .animation(.easeInOut, value: dragOffset)
    }
}