Storing huge files in a database could get unthinkable and chaotic. While traditional databases struggle in storing binary files, MongoDB has a secret sauce: GridFS, a powerful mechanism for storing and retrieving large files efficiently.
In this article, we will peek in to how GridFS slices and dices large files and stores, retrieves them efficiently. We will also see how to optimize its performance.
Why Use GridFS?
MongoDB’s 16 MB document size limit is well known, which is definitely not ideal for storing large files. Additionally, when those files are stored in BSON format, querying and retrieving them can make your database dizzy.
GridFS comes handy when you need to store such large files in MongoDB. GridFS is great not just for files larger than 16 MB, but also for any file where you want to access it without loading the whole thing into memory at once. GridFS splits large files into smaller chunks (default 255KB) and stores them across multiple collections:
- fs.files – Stores metadata about the file (filename, length, upload date, etc.)
- fs.chunks – Stores binary file data in 255KB chunks.
When retrieving a file, GridFS assembles chunks into a single file and streams it efficiently.
Setting Up GridFS in Node.js with MongoDB & Storing Files
Step 1: Install Required Packages
npm install mongoose mongodb multer-gridfs-storage gridfs-stream multer
Step 2: Connect to MongoDB and Initialize GridFS
Connect to MongoDB using Mongoose and initialize GridFS to handle file uploads by creating a GridFSBucket instance.
const express = require("express");
const mongoose = require("mongoose");
const { GridFsStorage } = require("multer-gridfs-storage");
const multer = require("multer");
const Grid = require("gridfs-stream");
const app = express();
const mongoURI = "mongodb://localhost:27017/mydatabase";
// Connect to MongoDB
const conn = mongoose.createConnection(mongoURI, {
useNewUrlParser: true,
useUnifiedTopology: true,
});
// Initialize GridFS
let gfs;
conn.once("open", () => {
gfs = new mongoose.mongo.GridFSBucket(conn.db, {
bucketName: "uploads",
});
});
Step 3: Configure Multer for GridFS Storage
Use multer-gridfs-storage to configure Multer for storing uploaded files in MongoDB via GridFS, specifying the bucket name and file metadata.
const storage = new GridFsStorage({
url: mongoURI,
file: (req, file) => {
return {
filename: file.originalname,
bucketName: "uploads",
};
},
});
const upload = multer({ storage });
Step 4: Create an Upload API Endpoint
Set up a POST API endpoint (/upload) to handle file uploads, using Multer middleware to save files directly to GridFS.
app.post("/upload", upload.single("file"), (req, res) => {
res.json({ file: req.file });
});
Retrieving and Downloading Files from GridFS
Step 5: Fetch File Metadata
Create a GET API (/files) to retrieve metadata of uploaded files, querying the fs.files collection for details like filenames and file size.
app.get("/files", async (req, res) => {
const files = await gfs.find().toArray();
res.json(files);
});
Step 6: Download a File
Create a GET API to stream a file by its filename.
app.get("/file/:filename", async (req, res) => {
const file = await gfs.find({ filename: req.params.filename }).toArray();
if (!file || file.length === 0) {
return res.status(404).json({ error: "File not found" });
}
const readStream = gfs.openDownloadStreamByName(req.params.filename);
readStream.pipe(res);
});
Deleting Files from GridFS
Step 7: Delete a File
Create a DELETE API to delete a file from GridFS by its file ID, ensuring the file is removed from both fs.files and fs.chunks collections.
app.delete("/file/:id", async (req, res) => {
await gfs.delete(new mongoose.Types.ObjectId(req.params.id));
res.json({ message: "File deleted" });
});
Performance Optimisation
LL Here are some ways to turbocharge your GridFS performance:
- Use Indexes for Faster Lookups.
- Stream Data Instead of Loading Entire Files.
- Adjust the chunk size for better performance with large files.
- For files larger than say 100MB, use alternative storage such as AWS S3.
To wrap it up, MongoDB GridFS is your weapon for handling large files like a pro.
Find the whole code snippet here for your reference.
As I’m still on my journey of learning and leveling up, I’d love to hear your suggestions or tips! Got a trick I missed? Share it with me.