Node.JS "Require" plays hide-and-seek

on node.js, require, require.cache, const, decache, cache, clone, javascript, js, json

Some times ago (I don’t really remember when), working on a Node.Js project, I realised something about require …​

The situation …​

Imagine we have some datas in a Json file we want to access in different modules.

datas.json
{
  "version":"0.1",
  "name":"MyApplication"
}

This datas.json is quite short, but for explaining the situation it is more than enough !

Now the code :

index.js
var module1 = require('./module1.js'),
    module2 = require('./module2.js');

module1.log();
module2.log();
module1.js
var myDatas = require('./datas.json');

module.exports = {
  log:function(){
    console.log(myDatas);
  }
}
module2.js
var myDatas = require('./datas.json');

module.exports = {
  log:function(){
    console.log(myDatas);
  }
}

As you can see, module1 and module2 are exactly same.

So if we execute the code, we will have the following output :

output 1
⇒  node index.js
{ version: 'O.1', name: 'MyApplication' }
{ version: 'O.1', name: 'MyApplication' }

Ok, now imagine module1 will modify some value before logging :

module1.js (version 2)
var myDatas = require('./datas.json');

module.exports = {
  log:function(){
    lowerCaseName() (1)
    console.log(myDatas);
  }
}

function lowerCaseName(){
  myDatas.name = myDatas.name.toLowerCase();
}
1 We just add this simple function that will lowerCase the name.

In my idea, this should not affect the module2 as they should be independant, so when I execute the code, here is the output :

output 2
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'myapplication' }(1)
1 Damned !

Source code

Why

Actually the reason is quite simple, when you require the file datas.json, the require function will create the object and store it in the require.cache object, then next time you use require('./datas.json') to retreive the datas, require will return you the value in that require.cache object.

So it mean that it gives you the reference to the value, so if you change one property of that value, all other modules are affected as you are actually doing the change on the reference in the cache.

This is quite well documented in the Node.js documentation :

Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.

— Node.Js Documentation
https://nodejs.org/api/modules.html#modules_caching 
Remember that Node.Js is single thread, it mean for example that the require cache is shared accross all requests

Reflections on solutions to have the desired behaviour

Now I will try to make the code work as desired.

Some of the solutions are not solving the problem, but I wanted to try to see how it works …​

The Const way …​ FAIL

First I wanted to use the new ES6 const instead of var in the line

// replace
var myDatas = require('./datas.json');
// with
const myDatas = require('./datas.json');
output 3
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'myapplication' }(1)
1 Damned !

But actually, const do not prevent the variable to be modified, but prevent reafectation.

FAIL

Source Code, branch const_way

The delete way …​ SUCCESS

With this solution, you will delete the reference in the require.cache object before require the datas.json file like this :

module2.js
delete require.cache[require.resolve('./datas.json')];
var myDatas = require('./datas.json');
// Code continue ...

As you may not know how your modules are inserted, you need to add the line on every files that need to use the datas.json file.

In my opinion, this is not very convenient, but it works :

output 4
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'MyApplication' } (1)
1 Great !

I Think that this solution is the "Quick And Dirty" solution.

SUCCESS

Source Code, branch delete_way

The Proxy Way …​ SUCCESS

With this solution I imagine to create a Proxy around the json datas, and override the set method, in order to forbid the manipulation of the value.

This approach can be usefull to throw excpetion if someone try to set the property value.

Here is the code :

datasProxy.js
var datas = require('./datas.json'); (1)

module.exports = new Proxy(datas, {
  set:function(){
    return; (2)
  }
});
1 This should be the only location in the app where you require the json file.
2 Here, we just do nothing, but we could throw exception here to reject any modification of any properties.

We also need to change the reference in module1 and module2 :

// Replace
var myDatas = require('./datas.json');
// with
var myDatas = require('./datasProxy');

Can you see a big problem ?

YES, the code in module1 should be updated because now we can not set the property (even locally)

Let’s first execute the code without any modification :

output 5
⇒  node index.js
{ version: 'O.1', name: 'MyApplication' }
{ version: 'O.1', name: 'MyApplication' }

So there is a problem, the first line should display the text MyApplciation in lowercase. So let’s edit the code in module1 to have the desired behaviour.

Here is a working code :

module1.js
var myDatas = Object.assign({}, require('./datasProxy')); (1)

module.exports = {
  log:function(){
    lowerCaseName()
    console.log(myDatas);
  }
}

function lowerCaseName(){
  myDatas.name = myDatas.name.toLowerCase();
}
1 We use Object.assign() to create a "copy" of the object in the module1

If we look at the output :

output 6
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'MyApplication' }

So it works !

But, as the MDN web site says:

"The Object.assign() method only copies enumerable and own properties from a source object to a target object."

— Mozilla Developper Network
http://devdocs.io/javascript/global_objects/object/assign

Maybe you should use a "clone" function that allow to clone objects deeply (check this with lodash _.clone() for example)

I think this solution is not too bad, but the thing is that you delegate to the module the need to create a copy, maybe this suits your needs, or maybe we can do it in the proxy itself.

I think both solution can be justified, you just need to make a choice.

SUCCESS

Source Code, branch proxy_way

The Clone Way …​ SUCCESS

This solution is the solution I considered previously.

So with this solution, the "Proxy" (or you can call it, the "wrapper") will create the copy and returns it to the modules :

So edit the Proxy code, and the modules :

datasProxy.js
var datas = require('./datas.json');

module.exports = function(){ (1)
  return clone(datas);
}

function clone(datas){
  return JSON.parse(JSON.stringify(datas)); (2)
}
1 We export a function that need to be called in module to return a copy of the datas
2 Here we use a hack to clone "deeply" an object.

Then edit the modules to call the exported function

module1.js
var myDatas = require('./datasProxy')(); (1)

module.exports = {
  log:function(){
    lowerCaseName()
    console.log(myDatas);
  }
}

function lowerCaseName(){
  myDatas.name = myDatas.name.toLowerCase();
}
1 Call the function to get the copy.
module2.js
var myDatas = require('./datasProxy')(); (1)

module.exports = {
  log:function(){
    console.log(myDatas);
  }
}
1 Call the function to get the copy.

Then execute the code :

output 7
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'MyApplication' }

Great it works !

This is my favorite solution.

SUCCESS

Source Code, branch clone_way

The decache way …​ SUCCESS

This idea was given to me by Jérémy Morin.

Decache is a Node Module, that remove module from the require cache.

This solution has exactly the same effect than the delete solution I present before. But it make it easier to do.

First you need to install the decache node module :

npm init (1)
npm install decache --save
1 This is not necessary if you already initialise a node project.

Then edit the code :

index.js
var module1 = require('./module1.js'),
    module2 = require('./module2.js');

module1.log();
module2.log();
module1.js
var myDatas = require('./datas.json');

module.exports = {
  log:function(){
    lowerCaseName()
    console.log(myDatas);
  }
}

function lowerCaseName(){
  myDatas.name = myDatas.name.toLowerCase();
}
module2.js
var decache = require('decache'); (1)
decache('./datas.json'); (2)
var myDatas = require('./datas.json');

module.exports = {
    log: function() {
        console.log(myDatas);
    }
}
1 First we need to require the decache module
2 Then we use decache to remove the datas.json file from the cache
Maybe we should use decache from both module1.js and module2.js because if we invert the the two require lines in the index.js file the cache is not removed as we expect.
output 7
⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'MyApplication' }

It works !

SUCCESS

Source code, branch decache_way

The import Way (aka. the ES6 way) …​ FAIL

This idea was also given to me by Jérémy Morin.

As Node.JS (version 7.4.0 on my laptop) is not compatible with the ES6 (import) syntax, we will need to "babelize" (transpile with babel) the code, and for that we need to refactor our code, and initilaze a npm project.

  1. First create a src doirectory and copy all files inside that directory

  2. In a terminal window run the npm init command and answer all question

    npm init

  3. Then install the babel-cli dependency

    npm install babel-cli --save-dev

  4. Create a "build" script

    package.json
    {
  "name": "require_strange",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "babel src -d lib" (1)
  },
  "author": "",
  "license": "ISC",
  "dependencies": {},
  "devDependencies": {
    "babel-cli": "^6.24.1"
  }
}
    1 Now you can use npm run build to build from ES6 code

  5. Configure babel :

    1. Install the presets

      npm install babel-preset-env --save-dev

    2. Create the .babelrc

      /.babelrc
      {
  "presets":["env"]
}

  6. Edit the Code to convert it to ES6

    /src/datas.js
    export default {
  "version": "0.1",
  "name": "MyApplication"
};
    /src/index.js
    import module1 from './module1';
import module2 from './module2';

module1.log();
module2.log();
    /src/module1.js
    import myDatas from './datas';

export default {
    log: function() {
        lowerCaseName();
        console.log(myDatas);
    }
};

function lowerCaseName() {
    myDatas.name = myDatas.name.toLowerCase();
}
    /src/module2.js
    import myDatas from './datas';


export default {
    log: function() {
        console.log(myDatas);
    }
}

  7. Build

    npm run build

  8. Run

    output 9
    ⇒  node index.js
{ version: 'O.1', name: 'myapplication' }
{ version: 'O.1', name: 'myapplication' }

As you can see the execution output exactly the same, so this is not a good solution.

FAIL

Source code, branch import_way

Conclusion

As a conclusion, I would say that it is important to understand how the require cache works, that’s why I wrote this article.

In order to solve the problem presented in the introduction, I would use the "clone" solution, by creating a "wrapper" object, maybe using the Proxy from ES6.

Please feel free to make any comments …​

comments powered by Disqus