Add background job processor.

[papodaca]

If the intent of this application is to be scaleable, it isn't ideal to process all the uploaded files in the main web server process. After a few (100-1000 not sure) simultaneous uploads performance may suffer handling other requests.

I've found a few options for achieving this easily:

• kue
• node-resque
• bull

To achieve this the code in this async block would be moved to a background job.

[mayeaux]

The current approach I have to this, and let me know your take, is that the app is kind of partitioned into different sections. Check the top of .env.settings.sample, you can see :

LIVESTREAM_APP=false
FILE_HOST=false
FRONTEND=true

Basically you can run the app just as a file host which receives uploads, and the frontend is decoupled and can run as a standalone app.

Is that what you were getting at?

[dmdboi]

This is something we should look at if we plan on scaling nodetube.live

[papodaca]

@mayeaux Not sure I follow 100% of what you are saying. Does this FILE_HOST/FRONTEND flag have to do with that http path certain processes are responding to? If that is the case at larger scales this may be insufficient.
This isn't a feature I require or anything just an architecture suggestion that I thought might be helpful.

[mayeaux]

@papodaca Yup, you can see it here: https://github.com/mayeaux/nodetube/blob/master/config/routes.js

So you can decouple the frontend from other functionality, I agree eventually a job queue will be needed for scale but it's at least something for now

[Yemeni]

we are going to need this too for #341 to resume quality convert