Creating a Distributable Tarball from a Node.js Project

33 views Asked by At

I have a complex monorepo containing a yarn workspace with multiple libraries and backend services with interdependencies but thankfully nothing circular. I wish that I could actually do the correct thing and decouple/separate all of these into their own Git repositories, but I cannot do that at this point.

I'm using the latest yarn release, and the libraries/services are written in TypeScript and transpiled, and everything 'works' right now. Transpiled files are copied into a local dist directory for each service.

The build process takes an extremely long time and requires a massive filesystem cache, which again, sadly, I cannot do anything about at the moment. Additionally, due to the situation, creating a Dockerfile which does not simply do COPY . . (with appropriate .dockerignores) would result in an extremely brittle setup and an extremely long build time, so in order to reduce complexity, I've decided that in the short term, the most time-effective solution for now is to:

  1. run the build step exactly once
  2. produce a tarball containing the transpiled JS files, config files, and all dependencies for each service in an included node_modules folder.
  3. save the built tarball as an artifact in the GitHub Actions CI job which performs yarn install and yarn run build for the service.
  4. in my Docker build step, grab the tarball from the build step, COPY it into the Docker build and extract it.

I know very well that most, if not all, of these things are not the way to generally do it, but sadly I cannot do anything about it at the moment.

I have been having trouble learning how to create such a tarball, as I have multiple node_modules folders and workspace dependencies, so I have service/${service_name}/node_modules, which is for some reason only 30 MiB, and node_modules at the root of the repository which is 2 GiB.

I've been searching high and low for a way to create a simple tarball using only the dependencies needed by the application, and I'm a bit confused as to why this hasn't been done much; perhaps it's because my use case is highly contrived.

Is there a yarn command or perhaps some other tooling in the Node.js space which can help me create a tarball with everything (except for Node.js itself of course) that is needed to run my application? Alternately I can use a Makefile, but I'm having trouble weeding out unnecessary dependencies and not creating 2-3GiB tarballs. I know that there will be compiled native libraries that must match the architecture, operating system, and distribution, so I'm aware of that hurdle.

0

There are 0 answers