Nodejs – Streams, Buffers and pipes

What exactly are the Streams?

The stream is a collection of data – like an array, the most common difference is that it might not available all at once. Imagine when you are watching a video online, sometime middle of the playtime, the spinner starts the show that still loading the content, which means player is still trying to get next video segment( Buffer) from the server. Normally video or other large files do not fit memory. so what nodejs does, the backend will read a file as a small chunk for data and send it to bowser or another place. That small chunk call buffer.

The stream is a very powerful technique to work with larger files especially YouTube, Netflix, or any other content provider uses this technique to deliver a larger file to destinations even at low bandwidth.

Stream at a glance

Basically there are two types of stream in Nodejs, Readable and Writeable. You can find these stream object in following places.

How to import file Stream module in Nodejs?

When you work with files, one of Stream comes from fs module ( File System) in nodejs. It will give many ways of reading and writing data as Stream. Since Nodejs is Single Threaded, asynchronies programming language. fs module gives synchronize and asynchronies way to deal with large data.

This is how fs module ( File System) import to your module.

var fs = require("fs")

How to read a file using Stream?

This sample code shows you how to read files using Stream object and data will be received as chunks on data event.

// import file system module
var fs = require("fs")

// create readable stream object and send encoding format
var readableStream = fs.createReadStream(__dirname + "/<<file name>>", {encoding:'utf8'});

// Base on file size, data will be received through below callback
readableStream.on("data", function(datachunk){
    console.log(datachunk);
});

// This event will be called when all data has been consumed
readableStream.on("end", () => {
    console.log("Stream ended");
});

Stream paused mode

Stream object provides very useful methods to pause and resume Stream.

readableStream.pause();

readableStream.isPaused();

readableStream.resume();

Stream vs Buffer

This is the overview of Stream and Buffer. Buffer is a small portion of data in the stream.

Pipes

The pipe method is really useful to reduce the code changes and easily link readable streams with writable streams together. What you have to remember is,

readableStream.pipe(writableStream);

Complete code as below. This code can be used to duplicate the files.

// import file system module
var fs = require("fs")

fs.createReadStream(__dirname + "/<<file name>>", {encoding:'utf8'})
.pipe(fs.createWriteStream(__dirname + "/<<new file name>>"));