UIScrollView doesn't scroll upwards sometimes

Question

I am using 2 scroll Views When the inner Scroll View is scrolled upwards, outer scrollView also moves upwards(positive contentoffset.y) and when inner SV's contentoffset.y goes less than y, both scroll Views come back to their original positions.

Here is the code: //secondView is a UIView inside outerScrollView and above inner SV(scrollView).

var scollView: UIScrollView!
var yOffset: CGFloat!
override func viewDidLoad() {
    super.viewDidLoad()

    // Do any additional setup after loading the view.

    self.automaticallyAdjustsScrollViewInsets = true
    self.outerScrollView.delegate = self

    self.scrollView = UIScrollView(frame: CGRect(x: 0, y: self.secondView.frame.origin.y + self.secondView.frame.height, width: self.view.frame.width, height: self.view.frame.height - (264)))
    self.scrollView.delegate = self
    self.outerView.addSubview(self.scrollView)
    self.scrollView.setContentOffset(CGPoint.zero, animated: true)
}

func scrollViewWillBeginDecelerating(_ scrollView: UIScrollView) {
    if (scrollView == self.scrollView) {
        if (scrollView.contentOffset.y > 0) {
            if (self.outerScrollView.contentOffset.y <= 0 ) {
                let contentnOffset:CGPoint = CGPoint(x: 0.0, y: self.yOffset)
                DispatchQueue.main.async {
                    self.outerScrollView.setContentOffset(contentnOffset, animated: true)
                    self.scrollView.frame.size = CGSize(width: self.view.frame.width, height: self.view.frame.height - self.qLabel.frame.height - 9)

                    if (scrollView.contentSize.height < self.view.frame.height - self.qLabel.frame.height - 9) {
                        let downPanGesture: UISwipeGestureRecognizer = UISwipeGestureRecognizer(target: self, action: #selector(self.handlePan))
                        downPanGesture.direction = .down
                        self.scrollView.addGestureRecognizer(downPanGesture)
                    }
                }
            }
        } else {
            DispatchQueue.main.async {
                self.outerScrollView.setContentOffset(CGPoint.zero, animated: true)
                self.scrollView.frame.size = CGSize(width: self.view.frame.width, height: self.view.frame.height - (self.scrollView.frame.origin.y))
            }
        }
    }
}

Although everything works fine, there is just 1 small problem. After the inner scrollView is scrolled upwards and contentoffset has changed and I want to keep on scrolling this view, it sometimes just doesn't scroll(upwards). So, I have to first scrooll it downwards and then it will again start scrolling fine. It only happens rarely but it's quite annoying.

Can anybody tell me what could be the possible problem? Thanks!

P.S. I have tried all other scrollView's delegate methods like didenddecelerating() or didenddragging() etc.