• [5/5/] How big things get done: This was a very, VERY, good read. This is how a book, filled with real experience and backed by data looks like. One of the very few management books that I have read that it didn't feel like "vague stories" but as real, hard earned insights from actual projects. I strongly recommend reading it. If I had to keep one thing from the whole book is this: Think slow, act fast.
• [5/5/] The Tyranny of Merit: What's Become of the Common Good?: This book was a great eye-opener regarding the dangers of meritocracy. The core idea is that meritocracy breeds hubris among winners and humiliation among losers. I agree with the author’s argument that meritocracy ignores a crucial factor: luck. It convinces people that their success is entirely of their own making, while blinding them to the advantages that allowed them to succeed in the first place. The result is not just inequality, but a moral divide that erodes solidarity. A strong and unsettling read.
• [5/5/] The Pragmatic Programmer: From Journeyman to Mastery, 20th Anniversary Edition: It has been some time since I read the original version of the book and the anniversary edition was sitting there in my bookshelf so I felt like re-reading it. I am glad I did. This is a book that every software engineer should read. Years of distilled experience are written in these pages and the important part is not the code examples, it's the philosophy of the book with which I agree 100%. YOU are responsible for the quality of your work. Not your tools, not your manager, not the process. YOU. Be adaptable and never stop improving yourself by treating software development as a continuous learnign and decision-making discipline.
• [4/5/] The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully: This is probably one of the few books I have read about consulting and genuinely liked. Its core idea is that consulting is fundamentally a human and political activity, not a technical one. Success depends less on having the "right" answers and more on managing expectations, relationships, and responsibility. I completely agree with this perspective. The book does a great job of describing common pitfalls consultants fall into and stays practical throughout. I highly recommend it, especially if you’re considering venturing into the consulting world.
• [4/5/] Rework: I really like the way these guys (Jason Fried and David Heinemeier Hansson) work. They get things done and consistently cut through the BS to focus on what really matters: delivering value. The book challenges many assumptions people take for granted about work and growth. Its core message is that simplicity, focus, and restraint often beat scale, speed, and ambition. Some ideas felt quite opinionated and context-dependent, but overall it offers a refreshing perspective on running a business without the hustle and overcomplication that most organizations fall into these days. Definitely an interesting read.

• [5/5/] Peopleware: Productive Projects and Teams: If you are a manager in a software company then you need to read this NOW. The important lesson here is that people are the core of any development process, something that a lot of managers forget most of the time. The book was written in 1987 and still applies today.
• [5/5/] The Phoenix Project: Loved it. I started reading it a couple of years ago and left it in the middle so decided to start from the beginning again. I am glad I did that because it was definitely worth my full attention. I am pretty sure that anyone long enough in IT will appreciate it and resonate with Bill's (the hero of the book) daily "adventures" in the crazy world of technology mixed with business. I also recommend reading The Goal, by Eliyahu M. Goldratt which I believe it's what basically inspired the writing of this book as well.
• [4/5] How to Solve It: A New Aspect of Mathematical Method: A very insighful book which applies even today, even if it's written in 1945 and it's around mathematics and geometry. Although it gets a bit repetetive, it's definitely worth it since it's filled with examples in order to help the reader understand its underlying framework for problem solving. Polya manages to translate his approach from the mathematics field to other domains that apply to our daily lifes.
• [4/5/] Sapiens: A Brief History of Humankind: A VERY complex and extensive topic that I think the author manages to summarize it pretty well in 500 pages and keeps the reader engaged. Lot's of interesting information and ideas, and even though I am sceptical about some, the fact that it does give an interesting perspective to human history is definitely intriguing and worth the read. The central idea around the book is that the humans did a huge mistake by turning from from being hunter-gatherers to agriculture which made them settle down, lose important knowledge and skills that all anchestors posesed and made us more miserable and harsh. Really enjoyed it, definitely recommend.
• [3/5] The Art of Shralpinism: Lessons from the Mountains: When a master speaks, you listen. Jeremy Jones is definitely a master in snowboarding and mounteneering and I was anxious to read his book. I really, really wanted to love it but I am afraid, it failed short to my expectations. First of all, there are a lot of story sections which due to the chosen font and background, are either very hard to read or compltely unreadable. Secondly, I was expecting to gain some deep knowledge which someone can only gain by doing it, and even though Jeremy Jones has been out there for a long time and have gained years and years of experience, he has failed to transfer at least some of it. Although there are tips and technical notes in the book here and there, in general it felt a little bit vague. Sorry Jeremy, I still respect you deeply for doing what you are doing, but for the next book, get a writing consultant!
• [1/5] Why Are We Yelling? The Art of Productive Disagreement: A very interesting question which the author has failed to answer. At first, the book looks promising but as you keep going the author starts rambling about gun violence, politics and other completely unrelated things with the topic. Do not waste your time.

• [5/5/] Code: This book was in my bookshelf for some time now and I finally got around it. Although most of the concepts were known to me this is the first time that I felt everything coming together and got plenty of "Aha!" moments. I must admit, that a couple of chapters got too deep technically and I've only read them without trying to fully understand them, but they were definitely worth going through them. This book is a MUST for anyone into computer science
• [4/5/] Good Strategy Bad Strategy: I really enjoyed this book and how it defines strategy and makes a clear distinction between a good and a bad one. Lots of good real world examples and actionable ideas. I felt that a few sections could have been more compact but it definitely wasn't tiresome. I strongly recommend reading it
• [4/5] Refactoring UI: This book is not only for designers, but for developer as well. In fact, I would probably say that a developer needs to read this more than a designer. It definitely gives an interesting pespective on design and how to keep things clean and simple, to get things going instead of losing yourself into too much details. I strongly recommend reading it
• [4/5] The Goal: A very well written process improvement novel. The story was very realistic with practical lessons that can be applied to the real world. For me, an important lesson here was that open-minded teams which want to solve problems and improve, will find the way by communicating well and by being willing to go through an iterative process. The story will probably be more interesting to people that are in the manufacturing industry but in general I find it useful for anyone that deals with process flows. It was a breez reading it and if you liked it and you are in IT, I also strongly recommend reading The Phoenix Project by Gene Kim
• [4/5] Seven and a Half Lessons About the Brain: A short and very interesting book on human brain, a subject that I find fascinating. Two things really hit me when I was reading this book. The first one is the way the book phrases the fact that our brain can create reality and that we live in a made up world. The second one, which it's the first time I read about it although it seems it is known in this circle for some time now, is that we do not have a triune brain in which we can blame our primitive behavior when e.g get angry, but we simple have a differently evolved brain from other species and our irational thoughts can simple be attributed to body budgeting reasons. A very interesting idea that if accepted world wide, it will affect society at it's core, especially in law cases. If you are interested about the human brain, then definitely read this
• [4/5] Phoenix in Action: A nice and easy read about the Phoenix framework which gives you a good overview with detailed examples of what is possible with it. I feel that the Elixir sections could have been more compact in order to make you read an Elixir-specific book that covers the subject better and maybe write more about of Phoenix, but I definitely didn't mind. This book is a good entry point for learning about the framework but things are moving within the Phoenix community and some code/techniques might not apply at the time you read it, so I strongly recommend that you also have a look at the very well written, official Phoenix documentation
• [4/5] Programming Ecto: A well written and helpful resource about Ecto. Altough Phoenix in Action had a lot of information about it, I felt that I had some gaps which this book definitely filled. Lot's of practical examples. What I also liked about the book is that it specifically covered real scenarios which in my opinion are pretty common. I strongly recommend reading it after Phoenix in Action
• [4/5] RxJS in Action: Reactive programming is a very interesting and powerful programming technique, especially in UI where a lot of things are happening asynchronously at the same time, and this book will definitely help you get into it by using RxJS. Lot's of real world examples and practical ideas. Some sections were a little bit more dense and more tiring than others but definitely worth the trouble reading them. I feel that some examples which showed a bad way to achieve something were missing their counterpart, the good way to actually do it and not just explaining it in theory. I definitely feel though that I now know much more about RxJS than before reading it
• [3/5] The Art of Readable Code: I have mixed feelings about this book since I am not sure to whom is addressed. Lot's of legit advice and ideas, but an experienced developer should already know all of this. A junior developer however, will not, but some of the advice in here might not be understood until the developer actually experiences it in a real code base. I definitely recommend that any junior developer should read this, and perhaps a more senior should go through it and refresh what they already know. I mostly agree with everything described in here but I would really love to if the author showed some functional programming as well
• [3/5] CTO Excellence in 100 Days: I am not even sure if the book is worth the 3/5 I gave it. It's probably somewhere between 2 and 3. It was very basic and it was mostly about people having lack of confidence into becoming CTOs than actually being CTOs. I believe there are better books out there that can enhance your understanding of a technical role better, especially the role of a CTO. For example, I found The Startup CTO's handbook by Zach Goldberg much much better. I wouldn't recommend this one
• [2/5] Nobody Wants to Read Your Shit: A very mediocre book with just a catchy title. I didn't even manage to finish it. Stopped at around 2/3 of the book. Even if there was some solid advice here and there, it was VERY basic and kinda boring. Don't waste your time
• [1/5] Rich Dad, Poor Dad: This book is just, sad. Among other things, the author basically says to pray on the weak in order to create wealth. Even if I completely put my feelings aside and judge the book more objectively, it only contains repetetive information without any actionable advice. Just a generic "learn what is an asset and what is a liability" mantra that is thrown around in most of the pages. I really do not understand why this book is a best seller. Definitely learn about investing and finance, just not from this. He did manage to do something right though; to properly market the book and take our money

Authorization? How hard can it be? I am pretty sure that others have already solved it. We are not the first ones doing microservices. It should be easy to integrate what's already out there.

- Everybody when they started designing their microservices, before they cried

Fine-grained authorization in microservices is hard. Definitely not impossible, but hard. You would expect that a more standardized, all-around, full-proof solution is out there, but I am afraid there isn't. It's a complex matter and depending on what you are building, implementation varies.

You will probably start with a boolean admin flag in your User model and then you will replace it with a role field, as we all did. However, as things progress and the business model becomes more and more complex, so do the solutions that we need to implement in order to deal with that complexity.

But how do you actually go from a simple flag to Role Based Access Control (RBAC) and then to Attribute Based Access Control (ABAC), especially in a microservices environment? In the following post I hope to help you get there.

(UPDATE: 02/04/2022): This article made it to the HackerNews frontpage and some interesting comments can be found here. Feel free to participate!

The monolith​

The first time I had to deal with a more complex authorization system was in a monolithic CMS application written in PHP 5.x, on top of Zend Framework (now known as Laminas Project), with a MySQL backend, a billion years ago. The app was following the hot and trendy Model-View-Controller (MVC) pattern and the requirement in the specific project was, except hierarchical role-based access control (H-RBAC), to have more fine-grained permissions as well as to be able to answer two specific questions:

1. Can the user do X on a Y resource if their role allows it or if the resource is owned by them?
2. Can the user A, with role X, access the data of user B, that has the same or a higher level role?

(For the sake of simplicity and because it does not really offer any extra value, I will skip the implementation details of question 2 and I will just use question 1 for my example below).

After thinking about the situation, and the fact that we had a VERY specific problem to solve, that I KNEW it wouldn't change in the future, and ALL the data models were associated with a single user, this is what a very simplified version looked like (forgive my PHP, it's a bit rusty):

class ItemsController extends Zend_Controller_Action {
  // ...omitted code
  
  public function listAction() {
    $page = $this->_getParam('page', 1);
    $orderBy = $this->_getParam('order_by', 'created_at');
    $order = $this->_getParam('order', 'desc');
    
    $filters = array('drafts' => false);
    $user = Session::getUser();
    
    if ($user->isAuthorized('can_list_drafts', 'items')) {
      $filters['drafts'] = $this->_getParam('drafts', false);
    
    $result = ItemModel::find($filters, $page, $orderBy, $order);
  
    return $result;
  }
  
  public function destroyAction() 
  {
    $itemId = $this->_request->getParam('itemId');
    $item = ItemModel::findById($itemId);
  
    if (!$item) {
      throw new ItemNotFoundException();
    }
    
    $user = Session::getUser();
    
    if (!$user->hasRoleOrSelfPermissionOn('can_delete_items', 'can_delete_own_items', 'items', $item)
      throw new AuthorizationException();
    
    $result = $item->destroy();
    
    return $result;
  }
  
  // ... rest of code
}

class User {    
  // ...omitted code
  
  public isAuthorized($permission, $resource) 
  {
    // The ACL singleton class is a wrapper around Zend ACL which is a module that provides you
    // with hierarchical, role based, access control lists which you can query and check
    // if a role has the requested permission on a resource.
    if (ACL::getInstance()->isAuthorized($this->role, $permission, $resource)
      return true;      

    return false;
  }
  
  public hasRoleOrSelfPermissionOn($rolePermission, $selfRolePermission, $resource, $model)
  {
    if ($this->isAuthorized($rolePermission, $resource) {
      return true;
    }
    elseif ($model.userId == $this.id) {
      if ($this->isAuthorized($selfRolePermission, $resource))
        return true;
    }
    
    return false;
  }
  
  // ...rest of code
}

A few things are going on above:

1. The decision point and enforcement of authorization is happening in the controller, getting mixed with application code that does not care about all these things, plus, it makes it harder to test. We do not want that. We want our controllers to be lean (in terms of responsibilities) and just return appropriate responses which are related to what was requested in the first place.
2. In listAction we need to check individual filters in order to decide if the user is authorized to use them or not. If multiple filters exist that need to be checked, the controller will become too messy, too fast.
3. The application data that the authorization requires (in this case $item) are locally available and easily loaded with our ItemModel class. As you will understand later, loading necessary application data is the biggest pain point.
4. The User class is suddenly coupled with authorization. It shouldn't. It should only care about the user entity itself, like the class name implies.
5. hasRoleOrSelfPermissionOn makes the assumption that all models have a userId field which means that we should know beforehand that this will be true (forever) or else the function will not work as expected.

The authorization code is tedious, poluting the controllers, and error prone. However, we were also lucky because this was the ugliest it could get (based on project requirements) and therefore it was acceptable (or at least we persuaded ourselves that it was), instead of pushing to a more complex solution. If, for example, there was the requirement of checking multiple model attributes (or attributes from different models), depending on the controller action, then hasRoleOrSelfPermissionOn wouldn't be able to cover everything and the controllers would immediately become much messier. It did serve us well back then but I no longer like it because avoiding tangling code at some point became a priority.

What I am describing above is not inherent to monoliths. The same implementation could have been done in a microservice (and I have seen it in multiple occasions, with everything done in the controllers) if the application data that the authorization mechanism required, lived under the same roof. But more on that later.

So, what could we have done to improve the above situation?​

Push the authorization code higher up in the stack. That would mean that you could push it to somewhere like a middleware, as most web frameworks call it, which is basically code that is executed before your controllers. And this, is where the fun begins and a whole new set of implementation problems rises!

But before we dive into that, we need to think about our authorization flow in a more abstract level.

Authorization flow overview​

So, what would this authorization mechanism look like in order to not force a single architecture but be more flexible instead? The following diagram will give you an idea:

The flow goes like this:

1. User requests to view record A.
2. The request is intercepted by the Policy Enforcement Point (PEP). This is usually a middleware, or generally speaking, a layer in your stack, as high as possible, in order for the request to stop right there in case it's not authorized, instead of allowing it to travel deeper in the system.
3. PEP makes a request to the Policy Decision Point (PDP) in order to figure out whether or not the request is authorized to move forward. PDP will probably be a library that keeps track of roles, permissions and resources and expose an interface in order to query it and get a boolean answer whether or not someone can do something on a resource.
4. PDP might need extra information in order to decide if the request should be allowed or denied so it needs to ask the Policy Information Point (PIP) for that extra information. That extra information can be retrieved from a database, from a flat file, from an external service or from any other source you need. PIP could be just another library which PDP can use internally if it needs to.
5. PIP loads the extra information and returns it to the PDP.
6. PDP, in combination with the extra information it got from PIP, evaluates the defined policy (which can be stored anywhere you like, e.g. database or flat file), and decides whether or not the request has access to the underlying resource based on what the policy says.
7. The answer is returned to PEP, which either allows or denies the request to move forward.

On the diagram above there is another piece, the Policy Administration Point (PAP). This is basically an optional interface that helps you manage the policies. It can be anything, a web interface, a command line tool, a desktop GUI or you can just skip it entirely if you do not want to provide it and do everything manually instead.

Architectures​

Now that we have defined our authorization flow, let's go through a few real world architectures together. All the following architectures include a diagram and within this diagram I make clear what plays which role based on what I showed you earlier.

With an authorization service​

So, you are there thinking, I am doing microservices, right? So the logical thing to do is to implement an authorization service and everybody would be able to use that and keep your precious service boundaries, right? WRONG. Your hell, has just begun!

How will PIP fetch the application data that PDP needs?

1. Direct DB reads: I am pretty sure someone will suggest this, and then will say, "I know we shouldn't be doing this in microservices but let's just make this exception", but that's how all big problems are created. With a small exception here, with another one there, and then things start to blow up, or even worst, become unmaintenable. Direct DB reads, outside the domain of each service, is a bad idea and it will be obvious to you why on the first schema change. Please don't be lazy and just say no to this.
2. Attach all the extra data to the authorization service call:. By doing this you will probably increase the network traffic and you might cause a bottleneck on the service if every request has a lot of data attached. However, these - manageable - disadvantages are not even my main concern with this approach. My main concern is the fact that the caller now needs to know the authorization logic that will be used internally by the authorization service in order to send the appropriate data. This will probably lead to callers sending more data than they should, or not removing data from the call whenever a policy changes, "just to be sure", and as a result the code maintenance will become harder. Even in a perfect world, were the data are ALWAYS exactly what is needed, the extra work of choosing the data is a burden that the caller should not have.
3. Keep a copy of the data the authorization service needs: This basically means syncing the data from all the other services, to the authorization service database, with a generic model that will be able to fit (in terms of a schema) everything. I can tell you right now that, this is difficult to achieve, especially if the syncing is experiencing network delays and/or conflicts. There are others who are doing this (check Google Zanzibar), but for most mortal companies I think dealing with the complexities of syncing on your first implementation, is an overkill. Besides, even Zanzibar is not a silver bullet and it's higly opinionated with its models, so you might need to build extra parts that communicate with it in order to achieve what you want.

With an authorization service and a data service​

This is probably the most complex architecture but it definitely clears things up a bit. If the data state and the mutations are all handled by the same layer (a data service), and all other services use that layer, it's like they have direct access to the database and the PIP can now fetch whatever it needs.

Personally, I like this architecture because now there is a very clear seperation of who is doing what, even if it means that more code will be required in order to set it up. Building a data service is beyond the scope of this post so I will just keep it simple and say that you shouldn't start with this, unless you already have a data service implemented or if you have other needs for it that go beyond authorization.

Variations​

There are a couple of variations with this architecture that I would like to mention since someone might find them interesting.

Instead of doing the authorization within each internal service, you could do the authorization at the API Gateway. You can use an authorization middleware within the API Gateway in order to communicate with the authorization service, or, you can even get rid of it completely and communicate with the data service directly from the middleware.

This has the advantage of stopping the request as early as possible before it has the chance to reach the internal services. In general, I like that idea.

However, it also has a couple of disadvantages:

1. When services make internal calls, they will be bypassing the authorization checks. It might be acceptable in your case but personally I do not like it. I want each service to allow only what is expected and nothing more. You could enforce in your implementation to make all the calls go through the API Gateway but then you increase the network latency and you lose all the advantages of actually having an internal network.
2. API Gateway will need to have all the application data loaders (for PIP) implemented there and things might get messy if you are implementing too much. Especially if you have a lot of services you might be changing API Gateway too often for reasons unrelated with the service itself.

With an authorization middleware and library per service​

In my opinion this is probably a good balance between complexity and a scalable solution. You basically keep the authorization logic in each service and each service is responsible for the authorization rules of its own domain.

One thing you will probably need is to have the common code implemented as a library (or libraries) since it will be shared among all services but it shouldn't be a problem. Just make sure you cover it with enough tests!

The real question with this approach is; how homogeneous is your system? If it's 100% then you are in luck. You will only need to implement (or find) everything you need once. if it's not, you will need to do it for every different language you are using in your system. Even though there are plenty of libraries out there, depending on your system details, they might not integrate very well or they might be missing a couple of features that you might need. Before you are too eager to implement your own, make sure if an existing solution covers your needs. If it does not, maybe you can get away with it by doing a few changes on your side, or by adding those missing features to the library you found. Whatever you do, take the time to decide carefully.

Implementation​

RBAC​

If you are doing RBAC, then implementing this is straight forward and clean. You just need a middleware (PEP) in which you must check if the user role (usually attached to a JWT that comes with the request) has access to the specified route path. The policies can easily be stored in a database or a configuration file, load them upon service initialization and let the RBAC library (PDP) that you will use do its work from within your middleware. In this case, no PIP is required.

There are a lot of good solutions out there for RBAC, for any modern web framework from any language, so I will not go into details about this, but I just wanted to briefly mention it. The only tip I do wanna share though, in case you are implementing something custom, is that the resource in your policies will probably be the regex route path and the permission will be the HTTP verb (GET, DELETE, etc). By doing that, you can easily get the route path from within the middleware and use a regex route matcher library to check if the resource is a match.

RBAC/ABAC​

So let's assume that you go with option 3, with an authorization middleware and library per service. How would such implementation look like?

The following implementation approach is neither the best, nor the only one. It is however applicable to the real world and that's what is important for me. What we are basically trying to answer with our implementation is:

1. How will my authorization mechanism load the necessary application data it needs?
2. How will I express my policies, in a non confusing way, and be able to access dynamically loaded data from within those policies?

By answering the above questions we will also be able to provide a better answer back to our friend the monolith and its question, how will we allow a user to do X on Y resource if their roles allows it, or if it's owned by them?

The example is written in Node.js, but I am confident you can do something similar in any web framework of your choice. Also, please do keep in mind that I have omitted a lot of code and kept a flat structure in order to try to reduce any confusion with unecessary implementation details. I understand that it can be frustrating but I will try to explain in detail what everything is supposed to do.

import express from 'express'; // express is a web framework for Node.js 
import logger from '/your/logger';
import {
  AccessControl, 
  ROLE, 
  PREDICATE
} from '/your/access/control/lib';

// `express.js` router handler for listing items and assigned to route `GET /items`. 
// The assignment is done in our `routesObject` further below.
async listItemsHandler(res, req, next) {
  const items = await ItemModel.find(req.query);
  
  // ... rest of handler; no authorization checks required 
}

// `express.js` router handler for getting a specific item and assigned to route `GET /items/:id`. 
// The assignment is done in our `routesObject` further below.
async getItemHandler(res, req, next) {
  const {id} = req.params;
  
  const item = await ItemModel.findById(id);
  
  if (!item) {
    throw new ItemNotFoundError();
  }
  
  // ... rest of handler; no authorization checks required
}

/*
  This is a function that represents an `express.js` middleware. It's basically code that will run BEFORE 
  the `listItemsHandler` and the `getItemHandler` implemented above. The middleware acts as the PEP 
  and  internally uses our access control library in order to decide if it should allow a request or not.
  
  Check here for extra info on middlewares: https://expressjs.com/en/guide/using-middleware.html
*/
function authorizationExpressMiddleware(request, response, next()) {
  // Get the user object from the JWT token that is attached to the request.
  // Usually this happens within a middleware higher up in the middleware chain.
  const user = request.locals?.user? || null;
  const userRole = request.locals?.user?.role? || ROLE.GUEST;

  // This is just an object with data that we want to pass to our access control library that is related
  // with this request. Everything in this object will be accessible by specifying the appropriate 
  // object path in the `conditions` key of our policy definitions as we will see later.
  const context = {
    user,
    request
  };

  const action = request.method; // GET, DELETE, PUT, etc
  const resource = path.join(request.baseUrl, request.route.path); // e.g. /items/:id

  // Check if the current request is authorized, by also including our `context` object. 
  const isAuthorized = await accessControl
    .withContext(context)
    .can(userRole, action, resource);
    
  // If the request is authorized, then pass the control to the next middleware in the chain.
  if (isAuthorized) {
    return next(); 
  }
  
  // Request is UNAUTHORIZED. Set the HTTP response status and stop the rest of the middleware 
  // chain from executing by returning an error. The request will stop here.
  response.status(401); // Unauthorized
  next(new AuthorizationError());
}

/*
  Initialization function for our access control library. This is an imaginary custom PDP library
  that internally uses an imaginary custom PIP library. Using the PIP part of the access control 
  library should be optional.
*/
function initAccessControl(endpointsObject) {
  // PIP loader function in order to load a specific item and provide to
  // the access control library the extra information it needs.
  const pipItemLoader = async (itemId) => {
    const item = await ItemModel.findById(itemId);
    
    if (item) {
      return item; 
    }
    
    return null;
  }

  // Initialize the access control library with two roles. The library should be able 
  // to support hierarchical roles in order to inherit permissions from a parent role 
  // instead of repeating them, just like below.
  const roles = [{
    name: ROLE.USER
  }, {
    name: ROLE.ADMIN
    inherits: [ROLE.USER]
  }];
  
  const accessControl = new AccessControl(roles);
  
  // Add our loader to the access control library in order to be able to use it internally when necessary.
  accessControl.addLoader(PIP_LOADER_TYPE.ITEM, pipItemLoader)
  
  // `buildPermissions` is a custom utility function that parses the `accessControl` key object value from 
  // our endpoints object and creates a list of permission objects, formatted the way our access
  // control library expects them.
  const permissions = buildPermissions(endpointsObject);

  accessControl.addPermissions(permissions);

  return accessControl;
}

// The following object contains everything we need to describe all the exposed routes. Within the object,
// we have also embedded our authorization policies via the `accessControl` key. Within this key, we specify
// which role can do what and under which conditions.
const routesObject = {
    middlewares: [authorizationExpressMiddleware],
    endpoints: {
     '/items': {
      '/': {
        GET: {
          handler: listItemsHandler,
          accessControl: {
            permissions: [{
              role: ROLE.USER,
              // The 'user' role can only list items which the `user_id` filter matches the user who request them.
              condition: {predicate: PREDICATE.EQUAL, args: ['$.request.query.user_id', '$.user.id']}
            }, {
              role: ROLE.ADMIN,
              // The admin role can list items with any filters they like.
              condition: * // Any filter is allowed
            }]
          }
        },
        '/:id': {
          GET: {
            handler: getItemHandler,
            accessControl: {
              loader: {fn: PIP_LOADER_TYPE.ITEM, args: ['$.request.params.id']},
              permissions: [{
                role: ROLE.USER,
                // The 'user' role can only get an item that is owned by the user who requested it.
                condition: {predicate: PREDICATE.EQUAL, args: ['$.resource.userId', '$.user.id']}
              }]
            }
          }
        }
      },
     }
    }
  }
};

// Initialize access control and our app.
const accessControl = initAccessControl(routesObject.endpoints);
const app = express();

// `buildRoutes` is a custom utility function that parses the routes object and adds all the specified routes 
// to our express.js router by also associated them with the respective handler. The function will also apply
// the `middlewares` array from the object to all routes which means that each request will need to go
// through our authorization middleware.
//
// Check the express.js router here: https://expressjs.com/en/guide/routing.html
buildRoutes(app, routesObject);

app.listen(3000, () => {
  logger.info(`kitten app listening at http://localhost:3000`)
})

I tried to keep the example code as succinct as possible in order for everyone to understand, even if they are not familiar with Node.js. The important take from the code above is that you could use an object to describe your policies, along with your routes, and pass values to those policies by expressing paths from dynamically loaded data that can be used during the evaluation step within our access control library.

You have probably noticed that in the condition key some weird stuff are happening in the args that are prefixed with $. What is happening there is that I am using JSON path syntax in order to specify the location of the data (within the context object) that will be passed as arguments to the predicate; In this case, PREDICATE.EQUAL, which as the name implies, is a predicate for equality check.

If you re-read the code, you will see that the authorizationMiddleware passes a context object to the access control library that includes the request and user objects. That's how they are available for access with the JSON path syntax. As for the $.resource in the args, it's just a key name I choose to attach the data returned from the pipItemLoader and automatically attached to the context (this happens inside our imaginary custom PIP library). That means, that the resource represents an item object with a field named userId which we can now access with JSON path syntax.

Like I have already mentioned, the above is one way to do it, and it works well. You could go with something that is more general-purpose and advanced like a policy engine (e.g OPA) but I assure you that they are definitely not a free meal either, nor without limitations. Once again, it comes down to the details of your own system and what you are trying to achieve.

(UPDATE 23/05/2022): Cerbos recently came to my attention which follows this "contextual" approach I am describing above. Maybe you can check it out as well!

Potential problems​

I would like to briefly mention a few situations that you might come up against as a heads up and what you can try to do in order to solve them.

Bulk enforcement​

This becomes apparent as soon as you have some kind of listing endpoint. How will you check if the user is authorized to access all the items in the result? Checking them one by one is probably the first thing that comes to mind, but that means you will have to load them first, plus, it might be slow if you need to keep loading 1000 items on every request even for users that are not allowed to access that data.

In our implementation above this is solved with our routes map and the condition key. When you are listing items, you are probably passing some kind of filters to the endpoint via query parameters, which means you can easily use those filters and decide what to do before you even load the data. For roles that are allowed to pass whatever filters they want (e.g. an admin role), you can just omit the condition key.

The important thing here is that the authorization mechanism is not getting coupled with internal details of how you actually implement your data filtering. It just stays on a very shallow layer of your service which is publicly available to use.

Not Found requests in PIP loaders​

This is something that I am still working on myself. I have tried various approaches but I am not 100% satisfied with any of them.

Let's asume that you have your pipItemLoader in which you load an item with id X. What should happen if the item does not exist? Should we return an authorization error? Should we return an ItemNotFound error? If we do the first one, isn't it misleading for the API caller? Especially if the user actually has access to load items. If we do the second one, before we check if the user is authorized for this action, wouldn't it be bad from a security perspective to provide such info? Maybe we should return a special value from the loader and if it's detected to let the request go through, and HOPEFULLY, our controller will be doing checks if the item exists or not.

You can try any of the above and something might fit your needs, or you can be creative and let me know if you find something interesting as well!

Duplicate database round trips​

As you might have noticed on our implementation above for the GET endpoint, is that we are loading the same item twice; once in the pipItemLoader and once in getItemHandler. Why hit the database twice when you can just avoid it?

One solution is to cache the data in order for the second call to return the cached data. But why hit the cache when you can avoid that as well? How? Just use a data loader and persist the data in RAM (and cache them as a side effect) for as long as the request lives and clear them upon finish with a middleware. For example, for Node.js you could use the dataloader package. I am pretty sure there are similar solutions in whatever you might be using.

Of course, none of this is necessary and might be too much. Once again, it depends on how everything is implemented on your side and more importantly, your incoming traffic load.

Restricting resource fields​

Depending on user access level, return resources with specific fields removed. Some authorization libraries that I have seen provide a way to do this by exposing a function/method, but in my opinion this does not really belong there. I feel like this is a job for an authorization middleware, or at least something other than the library itself which should be kept as tight as possible.

If you are using the routes map approach above, then you can easily extend it to support this by adding an extra key that describes the fields that are allowed and let the middleware remove the rest.

Conclusion​

As you saw, implementing authorization is hard, but it's also a very interesting challenge.

Although there is a popular belief that in a monolith is easier to implement authorization, in my opinion both the monolith and a microservices architecture have a lot in common when it comes to the authorization mechanism when that same mechanism has access to the application data it needs. The main difference is, as long as that mechanism no longer has a straight forward way to get that data, things start to get more complicated. Due to the independent nature of microservices, this can be easily observed, but the same complexity can be also observed within the same organization if multiple monoliths need to have a common authorization system.

I am definitely not done with looking for better solutions than the ones I have already suggested and I am looking forward to hearing what others are doing.

The good thing is, that more and more people are starting to take this more seriously and spend more time on it. Hopefully, at some point we will solve it more elegantly and have a more unified approach. Until then, good luck!

PS: I know you are going to ask me, so I am telling you right now that I made all the diagrams with draw.io.

Let's assume that you have a service in which you:

1. Accept a request
2. Fork a process with child_process.fork
3. Execute some code within that process
4. Exit from the child process
5. Complete the request

Probably the first thing you tried was to receive the request, spin up a process, do whatever you need in the processor, and exit. You timed the whole thing and your jaw dropped that it took a million years for the request to complete, even if you are just doing a console.log('I love kittens') inside your processor.

Don't bother. I will tell you right now that the bottleneck is the forking.

"OUTRAGEOUS! I should have used [INSERT_OTHER_TECH_HERE] which is super awesomely fast and all the cool kids are using it! Some random dude on medium.com says it is true!

-- your loud voice

Well, instead of re-writting the whole thing in a different language, I have an alternative for you; use a pool of forked child processes.

Approach​

The idea is straightforward; Upon service initialization, fork a bunch of processes, and whenever a request comes in, get a resource (a child process) from the pool and use IPC communication to send commands to it. After the processor completes, return it to the pool.

The first thing for all this of course, is a pool. The good news is that there is no need to implement your own, you can just use this lovely npm package, generic-pool, or the native cluster module depending on your use case.

For this post, I will use generic-pool so lets start with the processors pool which will look something like this:

import {fork} from 'child_process';
import GenericPool from 'generic-pool';
import logger from '/your/logger';

const commandProcessorsPool = GenericPool.createPool({
  create: () => {
    const modulePath = path.join(__dirname, 'processor.js');
    const commandProcessor = fork(modulePath);

    logger.debug(`Forked command processor with pid ${commandProcessor.pid}`);

    return commandProcessor;
  },
  destroy: (commandProcessor) => {
    logger.debug(`Destroying command processor with pid ${commandProcessor.pid}`);

    commandProcessor.removeAllListeners();
    commandProcessor.kill('SIGKILL');
  },
  validate: commandProcessor => commandProcessor.connected && !commandProcessor.killed
}, {
  testOnBorrow: true,
  min: 2, // Depending on your load, set a MINIMUM number of processes that should always be available in the pool
  max: 5 // Depending on your load, set a MAXIMUM number of processes that should always be available in the pool
});

commandProcessorsPool.on('factoryCreateError', logger.debug);
commandProcessorsPool.on('factoryDestroyError', logger.debug);

And of course, you will also need to implement the actual processor which looks like this:

import {serializeError} from 'serialize-error';
import logger from '/your/logger';
import {MESSAGE_STATUS} from './pool.js';

process.on('message', async (message) => {
  try {
    // Read and validate input data from `message` and do whatever you need to do...
    const {name} = message;
    const result = await petKitten(name);
    
    // All went well, send the result of your function to the parent process...
    process.send({status: MESSAGE_STATUS.OK, data: result});
  } catch (e) {
    logger.debug(e);

    /* 
      In the real world, your processor will probably need to handle errors as well and pass those errors to
      the parent process. Unfortunately, sending `Error` instances via `IPC` is not possible, BUT, we can just
      serialize them and then deserialize them on the parent process!
    */
    process.send({status: MESSAGE_STATUS.ERROR, data: serializeError(e)});
  }
};

Going back to our pool implementation, the only thing left now is the function that will be executing our commands:

// ... omitted previous code for succinctness

import {deserializeError} from 'serialize-error';

const MESSAGE_STATUS = {
  OK: 'ok',
  ERROR: 'error',
};
  
async function executeCommand(params) {
  const commandProcessor = await commandProcessorsPool.acquire();
  
  try {
    const commandProcessorTask = () => {
      return new Promise((resolve, reject) => {
        // https://nodejs.org/api/child_process.html#child_process_event_error
        commandProcessor.on('error', reject);

        commandProcessor.on('message', (message) => {
          const {status, data} = message;

          const handlersMap = {
            [MESSAGE_STATUS.OK]: () => resolve(data),
            
            // Don't forget to deserialize the error first!
            [MESSAGE_STATUS.ERROR]: () => reject(deserializeProcessorError(data)           
          };

          const handler = handlersMap[status];

          if (!handler) {
            return reject(new Error(`Unknown command processor message status '${status}'`));
          }

          handler();
        });

        commandProcessor.send(params);
      });
    };
    
    const result = await commandProcessorTask();
    
    return result;
  } finally {
    // Make sure that the command processor is returned to the pool no matter what happened
    await commandProcessorsPool.release(commandProcessor);
  }
}

export {
  MESSAGE_STATUS,
  executeCommand
}

For the shake of this example, I am going to assume that you are using express.js. The following is a very simple and short snippet on how to bring everything together, and execute a command when a request is received:

import express from 'express';
import {executeCommand} from './pool.js'; // By importing the file, our pool will be initialized
import logger '/your/logger';

const app = express();

app.post('/pet_kitten', (req, res) => {
  const {name} = req.body;
  
  // If this is a long-running process, then we shouldn't really block the request by using await.
  // Depending on your case here, you might want to handle the actual command execution differently.
  executeCommand({name}).catch(logger.error);

  res.status(202); // HTTP Status Accepted
  res.json({message: `Petting ${name} is underway...`});

app.listen(3000, () => {
  logger.info(`kitten app listening at http://localhost:${port}`)
})

Conclusion​

Although the above was a simple real world example on how to use a pool of processes, it can be the basis for more advanced usage. For example each processor can be modified to accept various commands instead of being limited to just one function. You could also add an onMessage event handler to support commands that send updates while they are being executed. I will leave those for another blog post!

Depending on your use case though, you can do a lot of different things by expanding the approach I have just described.

The important thing here is to remember that most of the time, the problem is not the tool, it's your attitude towards the tool. Be creative and solve shit.

The reason I am writing this is to address some cases which the library does not cover out of the box and share our experience on how we solved them in case others have the same needs.

I assume that you are already familiar with bull and how to use it so I will not waste time providing instructions. The official documentation can be found here, so I strongly recommend looking at it first before you read any further.

Background​

In a system we were working on, we had the requirement for a jobs service. That jobs service had to be written in Node.js for reasons that are irrelevant to this article. The important part is that the users of the system would be able to submit jobs of different types (copies, moves, compressing/extracting files, etc) to the service via a UI and then track their progress and manage them.

Even though bull looked promising, with a nice set of features out of the box, we still had to solve a few problems on our own. After using this library for some time now with success, and seeing that others have similar requirements, I thought I should share our experiences in case someone finds them useful.

We were trying to avoid using an external database and keeping in sync the jobs with bull since we didn't need any advanced querying, plus, it is always a pain point to keep two different sources in sync. One source of truth is always easier...unless the code complexity increases a lot (without even providing a competitive advantage) and suddenly the problems that come with syncing do not sound that bad...

Although we did have a few interesting iterations and used them in production for some time, we ended up going with a database after all.

Fetching jobs by user id​

Approach #1 - Using the bull API​

The first pain point in our quest for a database-less solution, was, that the bull API does not expose a method that you can fetch all jobs by filtering the job data (in which the userId is kept). Even if it did, it would probably be expensive to fetch the data by going through all the Redis HSET structures that the library uses.

Your only API-based option here is to fetch ALL the jobs with Queue.getJobs and do the filtering on the application level. This, of course, is not optimal at all, unless you know that your dataset will always be small. Not our case unfortunately but it might be OK for yours.

Approach #2 - Using a Redis secondary index​

Another approach, since you will already have Redis deployed, is to use a Redis secondary index. The general idea is to store a Redis LIST with all the job ids associated with a user id as the key. Then, you would load that list, loop it, and fetch each job with Queue.getJob.

Once again, nothing is for free. You would need to add logic to maintain those lists, cleanup scripts to ensure that nothing in those lists is dead, etc. Depending on your load, this might have a performance penalty since fetching multiple jobs with one call is not available in the bull API, but, if you are not overdoing it I think you will be fine.

Approach #3 - One queue per user​

Another, more complex approach, if you do not want to use a database, is to use one queue per user in combination with connections re-use. Depending on your use case and your user base, this might be just fine for you. You would still need to have all the queue objects loaded in memory at the same time, route each user request to the appropriate queue (e.g. by maintaining a Map with userId->Queue association), load all existing queues upon service initialization (and since there is no Queue.list method you need to scan Redis manually to get the queue names, and then instantiate the queues), cleanup, etc. If your load is relatively small, with a couple of thousand users, you will probably be ok with this.

If you need to have multiple queues per user, then I am telling you already that this will be far more complex than you think and you should probably avoid it. We have already done that, it worked, but, we weren't very happy with it, it does not scale well, and it will not work in a distributed environment.

If you really want to give it a go, however, the following example code should give you the general idea.

The UserQueue class​

import Queue from 'bull';
import _ from 'lodash;

// Keep track of all our user queues
const userQueues = new Map();

// The `UserQueue` class can serve as a layer between `bull` and your application if you need
// multiple queues per user and implement any method that you need here in order to manage
// the underlying queues.
class UserQueue {
  constructor(userId) {
    this.queues = new Map();
  }
  
  initQueues() {
    // Initialize all your queues and for each queue add an entry to the this.queues
    // map with the queue type as a key and the Queue object as value. Each queue needs
    // to include the userId in it's name along with the queue type in order to be easily
    // identifiable e.g. somePrefix:45:copies
    
    // Make sure that you re-use Redis connections here or else you will end up with A LOT
    // of open connections since each queue requires at least 3.
  }
  
  async getJobs() {
    // You basically need to loop all queues and get the jobs from each queue. No need from filtering
    // on the application level since all the jobs belong to this user. This is a very simple example
    // however, check Queue.getJobs documentation since you might need to pass extra parameters.
    const promises = _.map(this.queues, queue => queue.getJobs());
    const jobs = await Promise.all(promises);
    
    return jobs;
  }
  
  // Add any other methods you might need to interact with the underlying queues
}

// Check if the queue is already in our Map or else instantiate it and add it before returning it
function loadUserQueue(userId) {
  if (userQueues.has(userId) {
    return userQueues.get(userId);
  } else {
    const userQueue = new UserQueue(userId);
    userQueues.set(userId, userQueue);
    
    return userQueue;
  }
}

/* 
   Upon service initialization
*/

// Load all queue key names and extract the user id from the key matches
const userIds = await getUserIdsFromQueueKeys();

// Instantiate all queues in order to start processing jobs
_.forEach(userIds, loadUserQueue);

/* 
   During normal operation
*/

// A request comes in to fetch the jobs for user 45
const userQueue = loadUserQueue(45);

const jobs = await userQueue.getJobs();

Approach #4 - Using a database​

If you want to use a database, you need to built a layer in front of bull to bridge them together and control the flow. That layer, can be a service class, or separate functions, or whatever fits your codebase best.

Of course, there is the issue of data synchronization. If, for example, you have some kind of cleanup script that runs Queue.clean every now and then, you need to remove the jobs from your database as well. That's where the queue event handlers come in handy!

The JobService class​

So, an approach with a service class would look like this:

import Queue from 'bull';

class JobService {
  constuctor() {
    this.queues = new Map();
    
    this.initQueues();
  }
  
  /**
    Initialize all service queues.
    
    @private
  */
  initQueues() {
    // Initialize all your queues and update `this.queues`
    
    // For each queue run `this.attachQueueHandlers(queue)`
  }
  
  /**
    Attach event handlers to each queue.
    
    @private
  */
  attachQueueHandlers(queue) {
     queue.on('removed', (job) => {
       // Remove from your database
    });
  }
  
  async add(jobData) {
    // Add the job to your database
    
    // Add the job to the appropriate queue but remember to use the id from database by passing
    // it to the `Queue.add` method!
    
    return job;
  }
  
  async find(filters = {}) {
    // Use your database to find the jobs you want
  }
  
  // Add any other methods you might need to manage your jobs
}

// Global service object
const jobService = new JobService();
const job = await jobService.add(....);

From what you have probably already figured out, if you want more than just fetching the jobs of a specific user, then the place to add more code is here!

Removing an active job in sandboxed environments​

Overview​

This was another pain point when we started using bull; the inability to remove - basically abort - a running job in a sandboxed environment. The first approach we tried, although it did work for us, it was a bad solution since our fork was straying way far out from the core boundaries of the library. That means, that after a while, maintenance starts becoming a nightmare.

And of course, the nightmare became true after bluebird (which supports promise cancellation) was completely removed from the library. That would mean that our fork would not be able to be in sync with the official repo from now on. To be fair here, I think that move was 100% correct from the bull side and it helped us push in a far better direction. Besides, I do love a challenge!

Approach​

So, the idea is simple:

1. Send a kill signal to the sandboxed job processor
2. Handle the signal from within the processor
3. Discard the job (in order for bull to not retry it)
4. process.exit from within the processor and let bull do the rest

As with all recipes, these are the ingredients to make it work:

• Job.update to be available in the sandboxed environment (added in v3.23.0)
• Job.discard to be available in the sandboxed environment (added in v3.24.0)
• A high order function to wrap all our processors with the common functionality

The wrapProcessor function​

The following code snippet shows a simplified version of a higher order function that can be used to wrap all your processors in order to enhance them with your functionality and keeping all that code in one place. In our own wrapper version, we do other things as well, like, processor validation, setting the state of the jobs in our database, wrapping any errors thrown, etc. Any extra functionality that you need to run before and/or after executing the processor, can be done here.

import logger from '/your/logger';

/**
 * Wraps a processor and adds common functionality to avoid code duplication.
 * Make sure that ALL processors are exported by being wrapped first.
 *
 * @param {Function} processor
 * @throws {Error}
 * @returns {Function}
 */
function wrapProcessor(processor) {
  async function wrappedProcessor(job) {
     const exitHandler = (exitCode) => {
      logger.debug(`Received SIGTERM for job id '${job.id}' with exit code '${exitCode}' and PID '${process.pid}'`);

      // Discard the job first to ensure that it will not be retried after a process kill.
      job.discard();
     
      process.exit(exitCode);
    };

    process.on('SIGTERM', exitHandler);

    try {
      // Store the process pid in order to be able to abort the process at any time by simply killing it.
      await job.update({...job.data, pid: process.pid});

      const result = await processor(job);

      return result;
    } finally {
      // Bull internally uses a child pool of forked processors that are being re-used so we need to make sure
      // that we remove the listener before the processor returns to the pool or else we will cause memory leakage.
      process.removeListener('SIGTERM', exitHandler);
    }
  }

  return wrappedProcessor;
}

Usage​

// myProcessor.js
import wrapProcessor from '/some/path/wrap_processor.js'

async function myProcessor(job) {
   // Your processor code here
   
   return result;
}

// Wrap your processor!
export default wrapProcessor(myProcessor);

// main.js
import Queue from 'bull';
import kill from 'tree-kill';

function killJob(queue, jobId) {
  return new Promise((resolve, reject) => {
    try {
      const job = await queue.getJob(jobId);
      
      if (!job) {
        return resolve(false);
      }
      
      // You can also add an extra check here if the PID does not exist
      // and return resolve(false) if it's true.
      
      kill(job.data.pid, 'SIGTERM', (err) => {
        if (err) {
          reject(err);
        } else {
          resolve(true);
        }
      });
 	} catch (e) {
 	  reject(e);
 	}
 });
}

const queue = new Queue('downloader');
queue.process('/some/path/processors/my_processor.js'')
const job = await queue.addJob({url: 'https://example.com/file1.mov'});

// The following will send a SIGTERM to the underlying job process, and if it's still active, the
// job processor will receive it, exit itself, and the job will be moved to the failed queue by bull.
await killJob(queue, job.id); 

And of course, you can always update the JobService class and implement an abort method like so:

class JobService {
  // ... omitted previous code for succinctness
  
  async load(id) {
    // Load from database
    
    // If id does not exist, throw error
    
    return job;
  }
  
  async abort(id) {
    const job = await this.load(id);
    
    await this.kill(job.data.pid);
    
    return job;
  }
  
  kill(pid) {
    // Kill code here
  }
}

Attaching current state to jobs​

Overview​

The only way to get the job state with the bull API at the moment is Job.getState. That means, that first, you need to load the job(s) and then call the method. Not very convenient I am afraid, but again, for performance reasons from the bull side, this is how it is.

Approach #1 - Dynamically attach a state field to jobs​

Although this is not very robust since it can break after a bull upgrade, it worked fine for us at the beginning (before we ended up using a database) and it might do the trick for you as well. The following function can be used to attach a state field on each Job object, which is way faster than calling .getState for every each one of them:

import _ from 'lodash';

/**
 * Enum for possible job states.
 *
 * @readonly
 * @enum {String}
 */
 const JOB_STATE = {
   ACTIVE: 'active',
   WAITING: 'waiting',
   DELAYED: 'delayed',
   COMPLETED: 'completed',
   FAILED: 'failed',
   PAUSED: 'paused',
   STUCK: 'stuck',
   UNKNOWN: 'unknown'
 };

/**
 * The function will use some of the job fields in order to calculate the current state.
 *
 * @param {Job} job - The job object.
 * @see {@link https://github.com/OptimalBits/bull/issues/1076}
 */
 attachStateField(job) {
   const {
     processedOn = null,
     delay = 0,
     finishedOn = null,
     failedReason = null
   } = job;

   const statesPredicateMapping = {
     [JOB_STATE.COMPLETED]: () => _.isInteger(finishedOn) && _.isNull(failedReason),
     [JOB_STATE.ACTIVE]: () => _.isInteger(processedOn) && _.isNull(finishedOn) && _.isNull(failedReason),
     [JOB_STATE.DELAYED]: () => _.isNull(processedOn) && delay > 0,
     [JOB_STATE.WAITING]: () => _.isNull(processedOn) && delay === 0,
     [JOB_STATE.FAILED]: () => _.isString(failedReason)
    };

   job.state = _.findKey(statesPredicateMapping, predicate => predicate()) || JOB_STATE.UNKNOWN;

   return job;
}

// Usage
const queue = new Queue('downloader');
const jobs = await queue.getJobs();
_.forEach(job, attachStateField);

You can also check this discussion thread which includes the above plus a few more ideas in case you want to persue this any further.

Approach #2 - Using a database​

If you are using a database, then this is pretty straightforward. You add the job in the database with a default value, like JOB_STATE.PENDING, and then with the use of the queue events, we can update the job state.

The following is using the JobService from our previous section to show you an example of how to do it:

class JobService {
  // ... omitted previous code for succinctness
  
  /**
    Attach event handlers to each queue.
    
    @private
  */
  attachQueueHandlers(queue) {
    // Attach on queue all events that can change state like so:
     queue.on('active', (job) => {
       // Update `state` field in database
    });
    
    // Add all the necessary queue event handlers that change state
  }
}

Using a custom Error object​

Another problem that we had to solve was the need to expose custom error objects from within the processors in order to be able to pass them higher in the call chain and handle them accordingly.

Unfortunately, bull does not have some kind of mechanism at the moment to do that and the only thing you have access to when an error occurs, is the job.failedReason which is a string. So, why not use that to our benefit?

Approach​

You basically need to create a small mechanism to JSON.stringify the error object you want to store and JSON.parse it before consumption.

One way to do this is by taking advantage of our previously mentioned higher order function, wrapProcessor in combination with a custom Error object that will use the message property of the Error to store a JSON stringified value. The reason we will use the message property is because that's what bull uses to automatically store the failed reason.

After you wrap the error thrown by the processor, you can just re-throw it and parse it in your failed queue event handler or when you load a job with the Queue class. In the following code I will use our JobService class as an example for parsing and handling the error:

// The custom error object
import ExtendableError from 'es6-error';

class JobError extends ExterndableError {
  constructor(code, message) {
    this.errorObject = {
      code,
      message
    };
    
    super(JSON.stringify(this.errorObject));
  }
  
  getErrorObject() {
    return this.errorObject;
  }
}

// The updated wrapProcessor function
import logger from '/your/logger';

function wrapProcessor(processor) {
  async function wrappedProcessor(job) {
    // ... omitted previous code for succinctness

    try {
     // ... omitted previous code for succinctness
     
      const result = await processor(job);

      return result;
    } catch (e) {
      logger.error(e);
      
      if (e instanceof JobError) {
        throw e;
      } else {
        /* 
          Wrap the error as `JobError` and throw it. You can also create a mapping of "known" errors and assign them
          a specific code and error message. For example if you are doing validation within your processor, you could
          handle the validation error here and then wrap it as a `JobError`.
        */
      }
    } finally {
      // ... omitted previous code for succinctness
    }
  }

  return wrappedProcessor;
}

// myProcessor.js
async function myProcessor(job) {
   // Your processor code here
   
   // Something bad happened, throw error!
   throw new JobError('myErrorCode', 'Something very bad happened!');
   
   return result;
}

// Wrap your processor!
export default wrapProcessor(myProcessor);

// The updated `JobService` 
import _ from 'lodash';

class JobService {
  // ... omitted previous code for succinctness
  
  /**
    Attach event handlers to each queue.
    
    @private
  */
  attachQueueHandlers(queue) {
    // Attach on queue all events that can change state like so:
     queue.on('failed', (job) => {
       const errorObject = this.parseFailedReason(job.failedReason);
       
       // Do whatever you want with `errorObject`
    });
    
    // Add all the necessary queue event handlers that change state
  }
  
  /**
   * Parses failed reason which occurs when the job process gets killed or an error is thrown from within
   * the processor. Bull uses the `message` property from any errors inherited from `Error`, so, if that's the case 
   * we should be able to get a JSON object since we stringified our error object, or else we will just handle 
   * the `failedReason` parameter as a plain string.
   *
   * @private
   * @see {@link https://github.com/OptimalBits/bull/blob/v3.28.1/lib/job.js#L276}
   * @param {String} failedReason
   * @returns {Object}
  */
  function parseFailedReason(failedReason) {
    // https://github.com/OptimalBits/bull/blob/develop/lib/process/sandbox.js#L42
    const unexpectedExitString = 'Unexpected exit code';

    try {
      return JSON.parse(failedReason);
    } catch (e) {
      // If we are here, then it means that something out of our control happened
      // and the processor never returned a json result string as we expected.
      return {
        code: _.startsWith(failedReason, unexpectedExitString) ?
          'UNEXPECTED_EXIT_ERROR':
          'INTERNAL_ERROR',
         message: failedReason
      };
    }
  }
}

Per-user, per queue concurrency granularity​

Overview​

The final and toughest pain point was that the concurrency of the named processors was not working as we thought it would.

For our case, we needed to have a concurrency limit per-user, per queue, and at first, the named processors in combination with the concurrency option seemed that it might do the trick. Unfortunately, that was not the case. The plan was to use one named processor per user to apply those limits. Not sure if that would have worked well anyway, but it wasn't a viable option so we didn't pursue it any further.

The following from the bull documentation explains the named processors' concurrency limitation:

/***
 * For each named processor, concurrency stacks up, so any of these three process functions
 * can run with a concurrency of 125. To avoid this behaviour you need to create an own queue
 * for each process function.
 */
const loadBalancerQueue = new Queue('loadbalancer');
loadBalancerQueue.process('requestProfile', 100, requestProfile);
loadBalancerQueue.process('sendEmail', 25, sendEmail);
loadBalancerQueue.process('sendInvitation', 0, sendInvite);

Approach #1 - One queue per queue type​

If you need a per queue concurrency, then the simplest way to do it is to use separate queues. if your environment has some kind of limitation then you can also re-use your connections as I have already mentioned.

In combination with the UserQueue class, you could also achieve a database-less solution for a per-user, per queue concurrency granularity.

Approach #2 - Using a database​

Let me start by saying that for our own needs, we have a more complex implementation so I will try to simplify this as much as I can to give you a general idea of what you can try on your own.

The idea is this:

1. Keep track of all jobs in a database with a state and a userId field on each entry
2. Every time a job is created, completed or failed, a JobService method (let's call it addNextPendingJobs) will take a count of currently active jobs of the user and if they have empty slots, add some more to bull

Sounds straightforward, right? Wrong. If your service accepts parallel requests, you will need to take into account any possible race conditions each time you count the active jobs of a user and deciding whether or not you should add it to the bull queue.

Example scenario:

1. Request A comes in, you execute a database count
2. Request B comes in, you execute a database count
3. Count B returns, slots are available, code starts adding jobs to bull
4. Before the jobs from Count B have been added to bull and the state has been updated, *Count A *returns. Slots are available (even though they are not) and code starts adding jobs to bull
5. You now have more active jobs than your limit allows

So, depending on the database you are using, the atomicity level it offers, whether or not it supports transactions and exclusive locking, you will need to implement your addNextPendingJobs accordingly. In other words, you need to ensure that the entries that are being read during the counting are locked like they are being updated and not allow other operations from modifying them until the transaction ends.

For example, in the SQL world, this can be done with SELECT...FOR UPDATE but this is way out of the scope of this article. If you have NO CLUE what I am talking about then I recommend starting reading material on locking and the documentation of your database before you try anything further.

If your database does not support any of the above, or if you want to use Redis to implement the whole thing, you could also use something like redlock instead.

And of course, you can always process everything in sequence and avoid any possible race conditions, but it will create a bottleneck if you have a big load. As always, depending on your case, it's up to you.

Finally, you will need to use the queue events to handle the completed and failed events and add more jobs whenever necessary.

The updated code in our JobService will look like this:

import logger from '/your/logger';

class JobService {
  // ... omitted previous code for succinctness
  
  /**
    Attach event handlers to each queue.
    
    @private
  */
  attachQueueHandlers(queue) {
    // Attach on queue all events that can add next pending jobs like so:
     queue.on('completed', (job) => {
       // this.addNextPendingJobs
    });
    
    queue.on('failed', (job) => {
      // For the failed scenario, we need to check whether or not the failed job will automatically
      // re-run by `bull` before we try to add more jobs
      if (job.isDiscarded() || job.attemptsMade >= job.opts.attempts) {
        // this.addNextPendingJobs
      }
    });
  }
  
  async create(jobData) {
    // ... code
    
    // Add next pending jobs whenever possible without blocking
    this.addNextPendingJobs(jobData.userId, jobData.type).catch(logger.error);
  }
  
  async addNextPendingJobs(userId, jobType) {
     // Start a transaction/take lock
     
     // Count all ACTIVE entries of the specified `userId` and `jobType`
     
     // If the active are less than MAX ALLOWED then update those entries to WAITING
     // and start adding them to the appropriate `bull` queue
     
     // Commit transaction/release lock
  }
}

The above is simplified but it should give you some idea on how to proceed. As a final remark, the whole approach is not bulletproof. You need to ensure that queue stagnation will never happen in case something goes wrong in addNextPendingJobs and more jobs are not added to the queue. A retry strategy can be used or maybe the code that adds more jobs can live in a setInterval if you are using a single service.

Like I said plenty of times before, depends on your use case and there is always room for improvement.

Conclusion​

This has been a long post but I wanted to show you that with a little creativity a lot can be accomplished. No magic tool, or a magic library exists out there that will cover all your requirements out of the box, and being eager to jump into a custom solution from the first sign of limitation is a bad tactic. Of course, you will need to spend some time figuring out how an existing solution works, but if its core fits your needs and you can build on top of it, then you will just need to solve a specific set of problems instead of the whole spectrum of what the existing solution is solving.

If you have any questions/feedback for any of the above, then do not hesitate to email me!

I am astounded by the number of candidates that come for an interview and have no clue what the company does exactly. Except of what the – clueless recruiter – might have told them, they haven’t even bothered to check the company site in detail. There is occasionally someone that has read the first paragraph of the home page, but that’s usually it. If you don’t give a damn or don’t even have the curiosity to check out the place you will be spending a large part of your day, then I do not want to work with you.

If on the other hand you do give a damn, then keep reading.

Don’t be late. I hate when people are being late. I do not care about any excuses, and unless there was a real emergency, you should be on time. It’s a sign of respect, professionalism and that you understand that your time is not more important than the other person’s time.

I will be polite upon your arrival. I will offer you something to drink, probably give you a few options but please choose the water. It’s not a cafeteria and I am not your barista. I am your interviewer so do not abuse my hospitality. That means you should not ask me for a latte like someone did once. Yes, that has really happened. He wanted to look "cool" as they advised him in the computer science school he went to during his "how to prepare for an interview" course. Nope, I did not find it cool.

We will be moving to the meeting room where the interview will take place. I am an easy person to talk to so try to relax. We will start with some casual conversation about your past experience, things that interest you, what happened to your last job, what you expect from your new job, how you try to improve yourself every day, what was the last book you read, what hobbies you have and basically anything else that might give me the slightest clue about the person you are.

I want to be absolutely clear about this. What type of person you are and how well you will fit in the team is the most important thing for me. Of course, your technical skills are important as well, but let me put it this way; if it comes down to two candidates, and the first one has an open personality and qualities that I am looking for, but the second one does not, although their technical skills are better, I will ALWAYS choose the first one. You can teach a skill but you cannot teach someone how to be a human being.

What qualities am I looking for? Honesty, patience, someone who is observant, reliable, and respectful of those around them. I want to see a passionate, trustworthy person. A person that is not always looking for excuses or whines about everything but rises up to the occasion. And of course, to have some sense of humor, because, what is life without humor?

Try to be as sincere as possible about yourself. Even if you pretend to be someone you are not during the interview, or if you manage to give the answers I want to hear, you won’t be able to hide your real self after a couple of months of working closely together. Under pressure, all characters are truly revealed. And you will be pressed. So please, if you plan to bamboozle me, do both of us a favor and walk away because I will be reluctant to work with you.

At some point, after I feel I have heard enough about you – or too little – we will move to the technical part of the interview. Please do not suck. No seriously, please do not suck. By sucking I mean that when I ask you to write a for loop (for a simple “FizzBuzz” test) on the whiteboard just to check that you are actually able to write ANY code at all before we proceed, please do write it without hesitation. Unfortunately, it has happened more than once people tell me “I haven’t written code for the last couple of months and I do not remember how to write a loop at the moment” or “I did not expect to write any code. Nobody asked me in the other interviews”. SERIOUSLY? This is an interview for a software development position. I definitely want to see SOME code.

Yes, I know that phone screens exist, yes I know that you can give to the recruiter a cut-off test, yes I know all about the awesome ways to avoid wasting your time, but sometimes, these people just slip through and my answer to these people is, find a different work field. I know that someone told you that computer science “is the future” and that all the cool kids are doing it, but obviously, you do not like it. If you did, you would know how to write a simple for loop. Find something you like instead, be great at it, stop wasting your time, and even worst, others’ time!

Depending on the level of the position you are applying the questions will vary, but there will be something in common. Through the questions, I want to understand what kind of knowledge you were able to acquire from your experiences. I need details. It is not the amount of knowledge you have acquired that interests me, it is the ability of actively trying to recognize and acquire that knowledge through the challenges/opportunities that were presented to you.

I want to hear about your personal preferences and opinions about software. Do you have a piece of code that you like to show me and take me through it? Why did you do it with X way instead of Y way? Do you have a favorite editor? Which tools do you use? Do you have a favorite programming language? Why do you like it? Do you have a least-favored programming language? Why do you hate it? Have you done a project that you are proud of? How was the code organized? What challenges did you face? What do you think you could have done better? Are you familiar with functional programming? If yes, do you prefer it over OOP and why?

If your answer to most of the questions above – or similar ones that I will ask – is “I haven’t really thought of it” or some other passive answer, then I do not want to work with you.

In most interviews, there are theoretical questions like, what is OOP? What is a class? What is polymorphism? What is a linked list? What is the difference between stack and heap? What is a binary tree? What is inheritance? What is encapsulation? What is a singleton? What is the difference between merge sort and quick sort? I will not linger a lot on this kind of questions but I will definitely ask a few. To be honest, I do not expect an exact definition from Wikipedia, but I do expect you to be able to give some kind of answer in your own words or through an example. These questions should be easier for a person that just got out of University and has all this theory fresher, but seniors, please, do brush it up a bit as well.

The one thing that I will not be very kind with bad answers is recursion, especially from senior developers. In fact, I will ask you to implement recursion. The most common exercises with recursion are to find the factorial of a positive number n or calculating a Fibonacci sequence or walking a directory tree. Do not worry about the math in the first two, I will give exact instructions. In fact, you will not need any math knowledge at all. As long as you understand recursion, you will have no problem implementing any of the exercises. If you are a junior candidate and you are not very comfortable with it, please do try without whining that “I am not a mathematician” like someone told me once. There will be no math. I will give you all the definitions and clarifications you need. It’s just another programming problem you need to solve and you will have all the time in the world. JUST TRY.

Although theoretical questions are some kind of indicator, at the same time it does not really show a person’s real experience. You could have just memorized all the theory, read all the “How to pass a coding interview” books and manage to succeed with a little bit of luck and by manipulating the interviewer on a weak day. I really don’t understand why a person would do that but it can happen. In fact, some people have the nerve to see how far they can go by trying to fool everybody even after they get hired. That can mostly work in companies with a lot of employees. In small teams the value of their work has a more direct effect and it’s harder to hide.

Therefore, the following part of the interview will be the most important one. Through real-world use cases, to show me your experience. To show me that you have tried to solve actual problems and you have bled while doing it. These can be from "What fields would a database table need for a user authentication system?", “What is the difference between authentication and authorization?“, "Why would you use a finally clause in a try-catch-finally statement?", “What are some common logger levels?” to very specific ones like “How do you enter and exit the Node.js REPL?”, “How do you run the debugger with a keyboard shortcut in your favorite IDE?”.

Do you have some specific examples on your own? Even better, please share them with me! I always enjoy empathizing with the pain and joys that come with software development. I understand that you are worried that you might not have experience with any of the questions I might ask. I assure you that if you have actually worked on ANY real problem with ANY programming language, framework or technology, we will find some common ground.

I wish you good luck!

PS: I have humor.

How do you add caching to a class that is used very often within the rest of the codebase in the least intrusive way? You wrap the prototype of the class of course! And how do you wrap the prototype of the class? With another function that does the wrapping of course!

Our main business model (which is just a fancy name for a class that has business logic in it) requires some initialization every time it is instantiated. Therefore, we have a function that initializes and configures an instance of the BusinessModel which is used throughout the service. Let’s say that it looks like this:

// Our dummy 'BusinessModel' ES6 class
class BusinessModel {
  findItem() {
    // code
  }

  findItems() {
    // code
  }

  getBasicInfo() {
    // code
  }
}

// The utility function that instantiates and initializes a 'BusinessModel'
function getBusinessModel() {
  const businessModel = new BusinessModel();
  
  // Do some initialization here and other magic things...

  return businessModel;
}

So, the above function is basically being called any time we need an instance of the BusinessModel instead of directly constructing it. If for some reason we do not want to wrap the prototype and go with another approach instead, this function is the perfect candidate to add caching capabilities to the freshly instantiated businessModel. But first things first and more about this later.

We will need the code that actually adds caching capabilities to our class so I will just dump it here and try to explain a few things afterward:

'use strict';

import _ from 'lodash';
import objectHash from 'object-hash';

const debug = require('debug')('cachify');

const KEY_PREFIX_DELIMITER = ':';

/**
 * @callback preCache
 * @param {CacheManager} cacheManager - The cache manager instance.
 * @param {String} cacheKey - The cache key used, including the prefix.
 * @param {Sting} keyPrefix - The cache key prefix.
 * @param {String} fnName - The name of the function that was wrapped.
 * @param {Array} [fnArgs=[]] - The arguments of the function that was wrapped.
 */

/**
 * @callback postTransformer
 * @param {*} data - The cached data.
 */

/**
 * Cachify configuration object.
 *
 * @typedef {Object} CachifyOptions
 * @property {preCache} [preCache=null] - The pre cache hook to call after the original function has been called and
 *   before the data have been actually cached.
 * @property {Function} [postTransformer=null] - The post data load transformer hook. Called after the data have
 *   been loaded from cache.
 * @property {String|Array|Function} [keyPrefix=cachify] - The default prefix for cache keys. If it is a function
 *   the first argument of the function will be the context object in which the function was called.
 */

/**
 * Returns a cache key prefix string.
 *
 * @param {String|Array<String>} keyPrefix - The key prefix to use for the cache key.
 * @returns {String}
 */
function getCacheKeyPrefixString(keyPrefix) {
  const keyPrefixString = Array.isArray(keyPrefix) ? keyPrefix.join(KEY_PREFIX_DELIMITER) : keyPrefix;

  return keyPrefixString;
}

/**
 * Returns a cache key which is computed by hashing the target function name and
 * the target arguments list.
 *
 * @param {String|Array<String>} keyPrefix - The key prefix to use for the cache key.
 * @param {String} fnName - The function name.
 * @param {Array} fnArgs - The function arguments list.
 * @returns {String}
 */
function getCachifyCacheKey(keyPrefix, name, args) {
  const keyPrefixString = getCacheKeyPrefixString(keyPrefix);
  const hash = objectHash({name, args}, {replacer});

  return ${keyPrefixString}${KEY_PREFIX_DELIMITER}${hash};
}

/**
 * Wraps target function with caching capabilities. The wrapped function will always return a promise
 * since it needs to check the cache first asynchronously and decide whether or not it will call
 * the original function.
 *
 * @param {CacheManager} cacheManager - The cache manager instance.
 * @param {Function} targetFn - The function we will be wrapping with a caching layer.
 * @param {CachifyOptions} options - The cachify configuration object.
 * @returns {Function} - The asynchronous wrapped function.
 */
function cachify(cacheManager, targetFn, options = {}) {
  const {
    keyPrefix = 'cachify',
    postTransformer = null,
    preCache = null,
    ttl = null
  } = options;

  const wrappedFn = async function(...args) {
    const keyPrefixString = _.isFunction(keyPrefix) ?
      getCacheKeyPrefixString(keyPrefix(this)) :
      getCacheKeyPrefixString(keyPrefix);

    const targetFnName = targetFn.name;
    const cacheKey = getCachifyCacheKey(keyPrefixString, targetFnName, args);

    const result = await cacheManager.wrap(cacheKey, async () => {
      debug({cachify} Calling original function '${targetFnName}' with cache key '${cacheKey}'...);

      const originalResult = await targetFn.apply(this, args)

      if (preCache) {
        preCache(originalResult, {
          cacheManager,
          cacheKey,
          args,
          keyPrefix: keyPrefixString,
          fnName: targetFnName
        });
      }

      return originalResult;
    }, {ttl});

    return postTransformer ? postTransformer(result) : result;
  };

  return wrappedFn;
}

/**
 * Wraps specified methods of the targetObject with caching capabilities.
 *
 * **NOTE:** Function mutates `targetObject`.
 *
 * @param {CacheManager} cacheManager - The cache manager instance.
 * @param {Object} targetObject - The object we will be wrapping.
 * @param {Object} cacheableMap - The object describing which methods of `targetObject` should be wrapped.
 * @param {CachifyOptions} [options={}] - The cachify options object.
 * @returns {Object} - The cachified object.
 */
function cachifyObject(cacheManager, targetObject, cacheableMap, options = {}) {
  _.forEach(cacheableMap, (functionConfig, functionName) => {
    if (!functionConfig) {
      debug(`{cachifyObject} Cachification for function '${functionName}' is disabled...skipping...`);

      return;
    }

    const originalFn = targetObject[functionName];

    if (!originalFn) {
      debug(`{cachifyObject} Couldn't find function '${functionName}' on target object...skipping...`);

      return;
    }

    const functionOptions = {...options, ...(_.isPlainObject(functionConfig) ? functionConfig : {})};
    const wrappedFn = cachify(cacheManager, originalFn, functionOptions);

    targetObject[functionName] = wrappedFn;
  });

  return targetObject;
}