Pandoc conversion to pdf not working on heroku

Question

I have a ruby on rails app that uses pandoc-ruby to convert markdown files into pdf. The pandoc-ruby requires pandoc installation. To successfully convert to pdf, pdflatex needs to be present as well. Locally (tested on Mac and Ubuntu 18.04) everything is working if pandoc, texlive-latex-recommended and texlive-fonts-recommended packages are installed. Things get a little bit tricky when deploying to heroku. To install all the packages on heroku I've used the Aptfile approach and I have not been able to solve this.

Approach 1: Aptfile

I've specified this Aptfile:

# Aptfile
texlive-latex-recommended
texlive-fonts-recommended
https://github.com/jgm/pandoc/releases/download/2.11.3.2/pandoc-2.11.3.2-1-amd64.deb

The error that I'm getting is:

2021-01-22T21:46:03.478937+00:00 app[web.1]: [6ac45926-62f3-48bf-98b5-8f7c1022a02f] RuntimeError (warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
2021-01-22T21:46:03.478938+00:00 app[web.1]:
2021-01-22T21:46:03.478938+00:00 app[web.1]: kpathsea: Running mktexfmt pdflatex.fmt
2021-01-22T21:46:03.478940+00:00 app[web.1]: warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
2021-01-22T21:46:03.478940+00:00 app[web.1]: /app/.apt/usr/bin/mktexfmt: kpsewhich -var-value=TEXMFROOT failed, aborting early.
2021-01-22T21:46:03.478942+00:00 app[web.1]: BEGIN failed--compilation aborted at /app/.apt/usr/bin/mktexfmt line 28.
2021-01-22T21:46:03.478942+00:00 app[web.1]: warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
2021-01-22T21:46:03.478942+00:00 app[web.1]:
2021-01-22T21:46:03.478943+00:00 app[web.1]: kpathsea: Running mktexfmt pdflatex.fmt
2021-01-22T21:46:03.478943+00:00 app[web.1]: warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
2021-01-22T21:46:03.478944+00:00 app[web.1]: /app/.apt/usr/bin/mktexfmt: kpsewhich -var-value=TEXMFROOT failed, aborting early.
2021-01-22T21:46:03.478944+00:00 app[web.1]: BEGIN failed--compilation aborted at /app/.apt/usr/bin/mktexfmt line 28.
2021-01-22T21:46:03.478945+00:00 app[web.1]: Error producing PDF.
2021-01-22T21:46:03.478947+00:00 app[web.1]: This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
2021-01-22T21:46:03.478947+00:00 app[web.1]: I can't find the format file `pdflatex.fmt'!
2021-01-22T21:46:03.478947+00:00 app[web.1]:
2021-01-22T21:46:03.478948+00:00 app[web.1]: ):

I've tried to use heroku run bash to try and figure out what's going on. All the packages seem to be installed, but a simple pandoc command fails, see the output below.

~ $ pandoc -v
pandoc 2.11.3.2
Compiled with pandoc-types 1.22, texmath 0.12.1, skylighting 0.10.2,
citeproc 0.3.0.3, ipynb 0.1.0.1
User data directory: /app/.local/share/pandoc or /app/.pandoc
Copyright (C) 2006-2020 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
~ $ pdflatex -v
pdfTeX 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian)
kpathsea version 6.3.1
Copyright 2019 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.01
~ $ pandoc test.md -o test.pdf
warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.

kpathsea: Running mktexfmt pdflatex.fmt
warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
/app/.apt/usr/bin/mktexfmt: kpsewhich -var-value=TEXMFROOT failed, aborting early.
BEGIN failed--compilation aborted at /app/.apt/usr/bin/mktexfmt line 28.
warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.

kpathsea: Running mktexfmt pdflatex.fmt
warning: kpathsea: configuration file texmf.cnf not found in these directories: /etc/texmf/web2c:/usr/local/share/texmf/web2c:/usr/share/texmf/web2c:/usr/share/texlive/texmf-dist/web2c:/app/.apt/share/texmf/web2c.
/app/.apt/usr/bin/mktexfmt: kpsewhich -var-value=TEXMFROOT failed, aborting early.
BEGIN failed--compilation aborted at /app/.apt/usr/bin/mktexfmt line 28.
Error producing PDF.
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/Debian) (preloaded format=pdflatex)
I can't find the format file `pdflatex.fmt'!

Approach 2: custom buildpacks

I've also tried with 2 custom buildpacks for TeX Live found here and here. It was looking more promising but it still produces an error when converting to pdf. See the output from heroku run bash below.

~ $ pandoc -v
pandoc 2.11.3.2
Compiled with pandoc-types 1.22, texmath 0.12.1, skylighting 0.10.2,
citeproc 0.3.0.3, ipynb 0.1.0.1
User data directory: /app/.local/share/pandoc or /app/.pandoc
Copyright (C) 2006-2020 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
~ $ pdflatex -v
pdfTeX 3.14159265-2.6-1.40.21 (TeX Live 2020)
kpathsea version 6.3.2
Copyright 2020 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.02
~ $ pandoc test.md -o test.pdf
Error producing PDF.
! LaTeX Error: File `xcolor.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

Enter file name:
! Emergency stop.
<read *>

l.36 \IfFileExists

Answer 1

After quite a bit of trial and error, I have found a solution that works.

As @mb21 mentioned, Docker image would probably be the best option long term. Docker images are supported on Heroku. However, I wanted to avoid dockerizing the whole application to solve this issue.

After finding a TeX Live buildpack for Heroku that supports adding custom TeX Live packages (one example of such buildpack), the error on conversion was ! LaTeX Error: File 'xcolor.sty' not found.

I used tlmgr to get some info on the missing file. Running tlmgr search --global --file xcolor.sty does the trick and reveals that there is a package called xcolor. After installing that we come to the next error, and the next, and the next. In the end I ended up installing 2 collections that are small enough for Heroku (mind the 500MB slug size limit) and contain everything pandoc needs for a successful conversion. Those 2 are collection-fontsrecommended and collection-latexrecommended.

Adding a texlive.packages file to the root of the application does the trick. It is recognized by the buildpack and it installs all the specified packages for you using tlmgr.

# texlive.packages

collection-fontsrecommended
collection-latexrecommended