Node Redis Publisher Consuming Too Much Memory

I wrote a small redis publisher in node by using node_redis library. After the program is done publishing 1M messages, it continues to hold up around 350 MB of memory. Can anyone p

Solution 1:

There are two questions here.

Why does the program require so much memory?

I think it is due to the lack of back pressure.

Your script just sends 1M publish commands to Redis, but it does not process any reply to these commands (which are therefore just discarded by node_redis). Because it never waits for any reply, the script will accumulate a lot of context in memory for all these commands. node_redis needs to keep a context to keep track of the commands, and associate Redis commands and replies. Node.js is faster to enqueue commands, than the system is to convey those commands to Redis, process them, build replies, and convey replies back to node.js. The context is therefore growing, and it represents a lot of memory.

If you want to keep the memory consumption to an acceptable level, you need to throttle down your code to give a chance to node.js to process Redis replies. For instance, the following script also processes 1M items, but it publishes them as batches of 1000 items, and waits for the replies every 1000 items. It therefore consumes very little memory (the context contains at most 1000 pending commands).

var redis = require("redis"),
    publisher = redis.createClient();

functionloop( callback ) {
   var count = 0;
   for ( i=0 ; i < 1000; ++i ) {
        publisher.publish("rChat", i, function(err,rep) {
        if ( ++count == 1000 )
            callback();
        });
   }
}

functionloop_rec( n, callback ) {
    if ( n == 0 ) {
        callback();
        return;
    }
    loop( function() {
        loop_rec( n-1, callback );
    });
}

functionmain() {
    console.log("Hello");
    loop_rec(1000, function() {
        console.log("stopped sending messages");
        setTimeout(function(){publisher.end();},1000);
        return;
    });
}

publisher.ping(main)

setTimeout(function() {
    console.log("Keeping console alive");
}, 1000000);

Can the memory be released?

Usually, it cannot. As all C/C++ programs, node.js uses a memory allocator. When memory is freed, it is not released to the system, but to the memory allocator. Generally, the memory allocator is not able to give back the unused memory to the system. Please note it is not a leak, because if the program performs a new allocation, the memory will be reused.

Writing a C/C++ program which can actually release memory to the system generally involves designing a custom memory allocator. Few C/C++ programs do it. On top of that, node.js includes a garbage collector with v8, so it should put additional constraints on the memory releasing policy.