Chromedp working in non-headless but doing nothing in headless

Question

I am using Chromedp to post some stuff occasionally on Social Media. When I run the script in non-headless mode it is working perfect.

When I try to run it in headless mode, nothing happens. Obviously something is going wrong, but since I cannot "see" what is going wrong I am clueless.

Can anyone tell me how one should debug something when not able to visually see what Chromedp is doing?

And maybe explain to me why non-headless is working but headless not, because I do not understand how that can be.

I tried it with this code (minimized it), and in non-headless it works perfect. As soon as I go headless is doesn't do anything.

opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.Flag("headless", false),
    )
    actx, acancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer acancel()

    ctx, cancel := chromedp.NewContext(
        actx,
        chromedp.WithLogf(log.Printf),
    )
    defer cancel()

    ctx, cancel = context.WithTimeout(ctx, 120*time.Second)
    defer cancel()

    err = chromedp.Run(ctx,
        emulation.SetUserAgentOverride("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"),
        //chromedp.ResetViewport(),
        chromedp.Navigate("https://example.com/"),

        chromedp.Sleep(time.Second*5),
        chromedp.MouseClickXY(1075, 33),
        chromedp.Sleep(time.Second*5),
                // A LOT MORE CLICKS AND KEY PRESSES GOING ON
        chromedp.MouseClickXY(850, 61),
        chromedp.Sleep(time.Second*10),

        // end
        chromedp.Stop(),
    )
    if err != nil {
        fmt.Println(err)
    }

I added a "debug" line and now I see tons and tons of code (HTML I guess) output in my console but I still cannot figure out where it goes wrong in all honesty.

Answer 1

Can anyone tell me how one should debug something when not able to visually see what Chromedp is doing?

Enable the debug log

ctx, cancel := chromedp.NewContext(
    actx,
    chromedp.WithDebugf(log.Printf),
)

The output will be quite a lot. Redirect the output to a file so that you can examine it carefully and share it to others if in need.

And read the Chrome DevTools Protocol to understand the log.

Take a screenshot

You can take a screenshot with one of the following funcs:

• chromedp.CaptureScreenshot: take a screenshot of the current browser viewport.
• chromedp.FullScreenshot: take a full screenshot of the entire page.

You can even start a screencast to save a series of images. See an example here.

---

And maybe explain to me why non-headless is working but headless not

The viewport has a different size in headless mode

If the page is responsive and adapt to different window sizes, then chromedp.MouseClickXY with a hard-coded position could fail. Take a screenshot to confirm that. Avoid using chromedp.MouseClickXY directly. Use chromedp.Click to click a specific element.

If in need, you can use chromedp.WindowSize to set the initial size of the window. Or use chromedp.EmulateViewport to change the size of the viewport later.

The browser is detected as a bot

Another common reason is that the browser in headless mode is detected as a bot and the website refuses to serve the browser. See Bot detection engines.

To understand what it is, see some of the reported issues below:

• https://github.com/chromedp/chromedp/issues/1163
• https://github.com/chromedp/chromedp/issues/877
• https://github.com/chromedp/chromedp/issues/1069
• https://github.com/chromedp/chromedp/issues/1102

---

The issue is most likely caused by the usages of chromedp.MouseClickXY with hard-coded position. Since the question does not contain enough information, I can not do much about it. If you're stuck but can not share your use case publicly, you can send me an email with concrete information so that I can look into it.