How to Efficiently Store and Retrieve Large Files in MongoDB Using GridFS.

JavaScriptWeb DevelopmentDatabase

3 Minute Read

Storing huge files in a database could get unthinkable and chaotic. While traditional databases struggle in storing binary files, MongoDB has a secret sauce: GridFS, a powerful mechanism for storing and retrieving large files efficiently.

In this article, we will peek in to how GridFS slices and dices large files and stores, retrieves them efficiently. We will also see how to optimize its performance.

Why Use GridFS?

MongoDB’s 16 MB document size limit is well known, which is definitely not ideal for storing large files. Additionally, when those files are stored in BSON format, querying and retrieving them can make your database dizzy.

GridFS comes handy when you need to store such large files in MongoDB. GridFS is great not just for files larger than 16 MB, but also for any file where you want to access it without loading the whole thing into memory at once. GridFS splits large files into smaller chunks (default 255KB) and stores them across multiple collections:

  • fs.files – Stores metadata about the file (filename, length, upload date, etc.)
  • fs.chunks – Stores binary file data in 255KB chunks.

When retrieving a file, GridFS assembles chunks into a single file and streams it efficiently.

Setting Up GridFS in Node.js with MongoDB & Storing Files

Step 1: Install Required Packages

npm install mongoose mongodb multer-gridfs-storage gridfs-stream multer

Step 2: Connect to MongoDB and Initialize GridFS

Connect to MongoDB using Mongoose and initialize GridFS to handle file uploads by creating a GridFSBucket instance.

const express = require("express"); const mongoose = require("mongoose"); const { GridFsStorage } = require("multer-gridfs-storage"); const multer = require("multer"); const Grid = require("gridfs-stream"); const app = express(); const mongoURI = "mongodb://localhost:27017/mydatabase"; // Connect to MongoDB const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true, }); // Initialize GridFS let gfs; conn.once("open", () => { gfs = new mongoose.mongo.GridFSBucket(conn.db, { bucketName: "uploads", }); });

Step 3: Configure Multer for GridFS Storage

Use multer-gridfs-storage to configure Multer for storing uploaded files in MongoDB via GridFS, specifying the bucket name and file metadata.

const storage = new GridFsStorage({ url: mongoURI, file: (req, file) => { return { filename: file.originalname, bucketName: "uploads", }; }, }); const upload = multer({ storage });

Step 4: Create an Upload API Endpoint

Set up a POST API endpoint (/upload) to handle file uploads, using Multer middleware to save files directly to GridFS.

app.post("/upload", upload.single("file"), (req, res) => { res.json({ file: req.file }); });

Retrieving and Downloading Files from GridFS

Step 5: Fetch File Metadata

Create a GET API (/files) to retrieve metadata of uploaded files, querying the fs.files collection for details like filenames and file size.

app.get("/files", async (req, res) => { const files = await gfs.find().toArray(); res.json(files); });

Step 6: Download a File

Create a GET API to stream a file by its filename.

app.get("/file/:filename", async (req, res) => { const file = await gfs.find({ filename: req.params.filename }).toArray(); if (!file || file.length === 0) { return res.status(404).json({ error: "File not found" }); } const readStream = gfs.openDownloadStreamByName(req.params.filename); readStream.pipe(res); });

Deleting Files from GridFS

Step 7: Delete a File

Create a DELETE API to delete a file from GridFS by its file ID, ensuring the file is removed from both fs.files and fs.chunks collections.

app.delete("/file/:id", async (req, res) => { await gfs.delete(new mongoose.Types.ObjectId(req.params.id)); res.json({ message: "File deleted" }); });

Performance Optimisation

LL Here are some ways to turbocharge your GridFS performance:

  • Use Indexes for Faster Lookups.
  • Stream Data Instead of Loading Entire Files.
  • Adjust the chunk size for better performance with large files.
  • For files larger than say 100MB, use alternative storage such as AWS S3.

To wrap it up, MongoDB GridFS is your weapon for handling large files like a pro.

Find the whole code snippet here for your reference.

As I’m still on my journey of learning and leveling up, I’d love to hear your suggestions or tips! Got a trick I missed? Share it with me.

About Author

Kaveri T S

And I was running far away Would I run off the world someday? Nobody knows I was dancing in the rain I felt alive, But now take me home..

Latest Posts