Nodejs – Streams, Buffers and pipes

What exactly are the Streams?

Stream is a collection of data – like an arrays, most common difference is that, it might not available all at once. Imagine when you are watching a video online, sometime middle of the play time, spinner start the show that still loading the content, that mean player is still trying to get next video segment( Buffer) from server. Normally video or another large files do not fit to memory. so what nodejs does, backend will read file as small chunk for data and send it to bowser or another place. That small chunk call buffer.

Stream is very powerful technique to work with larger file specially YouTube, Netflix or any other content provide use this technique to deliver larger file to destination even in low bandwidth.

Stream at a glance

Basically there are two types of stream in Nodejs, Readable and Writeable. You can find these stream object in following places.

How to import file Stream module in Nodejs?

When you work with files, one of Stream comes from fs module ( File System) in nodejs. It will give many ways of reading and writing data as Stream. Since Nodejs is Single Threaded, asynchronies programming language. fs module gives synchronise and asynchronies way to deal with large data.

This is how fs module ( File System) import to your module.

var fs = require("fs")

How to read a file using Stream?

This sample code shows you how to read file using Stream object and data will be received as chunk on data event.

// import file system module
var fs = require("fs")

// create readable stream object and send encoding format
var readableStream = fs.createReadStream(__dirname + "/<<file name>>", {encoding:'utf8'});

// Base on file size, data will be received through below callback
readableStream.on("data", function(datachunk){
    console.log(datachunk);
});

// This event will be called when all data has been consumed
readableStream.on("end", () => {
    console.log("Stream ended");
});

Stream paused mode

Stream object provides very useful methods to pause and resume Stream.

readableStream.pause();

readableStream.isPaused();

readableStream.resume();

Stream vs Buffer

This is the overview of Stream and Buffer. Buffer is small portion of data in stream.

Pipes

The pipe method is really useful to reduce the code changes and easily link readable stream with writable stream together. What you have to remember is,

readableStream.pipe(writableStream);

Complete code as below. This code can be used to duplicate the files.

// import file system module
var fs = require("fs")

fs.createReadStream(__dirname + "/<<file name>>", {encoding:'utf8'})
.pipe(fs.createWriteStream(__dirname + "/<<new file name>>"));

Leave a Reply