Is it a bad idea to use a web api instead of a tcp socket for master/slave communication?

Question

TL:DR; Are there any drawbacks / pitfalls to use a RESTful API instead of a TCP/Socket connection for a master-slave pattern?

---

I'm writing a web application that fetches data from an 3rd party API, calculates statistics (and more), and presents it. The fetched data is stored in a database.

I'm writing this web application with NodeJS and AngularJS.

So there are 2 basic opertions the app will do:

1. Answer HTTP requests from the front end
2. Update the database. This includes: Fetching data, saving data to database, and caching results of some heavy queries every 5 minutes in a in-memory database like redis.

I want to seperate these operations in two applications. The HTTP server shall be the master. The second application is a just a slave, of which as many instances can be spawned.

The master will implement some kind of task-processor which just distributes tasks to idle slaves. The Slave is very dumb. It can report about its status (idle/busy and some details like current load etc). You can start tasks on a slave. The master will take care of queueing tasks and so on.

I guess this is common server/agent pattern.

So I've started implenting a TCP Server and Client, but it just feels like so much overhead to do this. A bit like reinventing the wheel. So I thought I just could use a HTTP server on my client which just has two endpoints like

[GET] /status 
[POST] /execute/:task

Am I on the right track here?

Answer 1

TL;DR; There are drawbacks to rolling your own REST API for master-slave architecture.

Your server/agent pattern is commonly referred to as microservices.

Rolling your own REST API might work, but is probably not optimal. Dealing with delivery semantics (for example, at most once vs at least once), transient failures, polling, etc will likely cause a lot of pain before you get it right.

There are libraries/services to provide varying levels of convenience:

1. Seneca - http://senecajs.org
2. Pigato - http://prdn.github.io/pigato/
3. Kong by Mashape - http://getkong.org
4. Webtask by Auth0 (paid) - https://webtask.io