JSON Streaming with NDJSON: Newline-Delimited JSON for Big Data

NDJSON (Newline Delimited JSON) and streaming JSON are essential techniques for handling large datasets that would otherwise consume too much memory. Instead of loading an entire JSON document into memory, streaming processes records one at a time. This guide covers NDJSON format, JSON streaming parsers, real-world use cases, and implementation in multiple languages. Use our JSON Formatter to inspect NDJSON samples and JSON Validator to validate individual records.

What is NDJSON?

NDJSON (also called JSON Lines or newline-delimited JSON) is a variant where each line is a separate, complete JSON object. Multiple JSON objects are delimited by newline characters. Unlike a JSON array ([{...}, {...}]), NDJSON can be processed line-by-line without loading the entire dataset.

// NDJSON format: one JSON object per line
{"id": 1, "name": "Alice", "timestamp": "2025-01-15T10:00:00Z"}
{"id": 2, "name": "Bob", "timestamp": "2025-01-15T10:01:00Z"}
{"id": 3, "name": "Charlie", "timestamp": "2025-01-15T10:02:00Z"}

// Equivalent JSON array (requires loading all objects at once)
[
  {"id": 1, "name": "Alice", "timestamp": "2025-01-15T10:00:00Z"},
  {"id": 2, "name": "Bob", "timestamp": "2025-01-15T10:01:00Z"},
  {"id": 3, "name": "Charlie", "timestamp": "2025-01-15T10:02:00Z"}
]

JSON Array vs NDJSON Comparison

Feature	JSON Array	NDJSON
Memory usage	Entire file loaded at once	One record at a time (constant)
Parsing	Single JSON.parse() call	Line-by-line parsing
Append data	Must re-write entire array	Simple append: add new line
Partial reading	First byte to last byte needed	Read from any offset
File size overhead	Square brackets + commas	One newline per record
Error resilience	One bad byte corrupts entire array	Bad lines can be skipped
Streaming support	Complex (must track array state)	Natural (line-based)
Compression ratio	Slightly better (shared structure)	Slightly worse (per-line overhead)

Streaming NDJSON in Node.js

const fs = require('fs');
const readline = require('readline');
const { Transform } = require('stream');

// Stream NDJSON file line by line
async function processNDJSON(filePath) {
  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });

  let lineCount = 0;
  let errorCount = 0;

  for await (const line of rl) {
    if (line.trim() === '') continue; // Skip empty lines

    try {
      const record = JSON.parse(line);
      // Process one record at a time
      await processRecord(record);
      lineCount++;
    } catch (error) {
      errorCount++;
      console.error('Error parsing line ' + (lineCount + 1) + ':', error.message);
      lineCount++;
    }
  }

  console.log('Processed ' + lineCount + ' records with ' + errorCount + ' errors');
}

// Transform stream for NDJSON
class NDJSONTransform extends Transform {
  constructor() {
    super({ readableObjectMode: true });
    this.buffer = '';
  }

  _transform(chunk, encoding, callback) {
    this.buffer += chunk.toString();
    const lines = this.buffer.split('
');
    this.buffer = lines.pop() || ''; // Keep incomplete line

    for (const line of lines) {
      if (line.trim()) {
        try {
          this.push(JSON.parse(line));
        } catch (e) {
          this.emit('parse-error', { line, error: e });
        }
      }
    }
    callback();
  }

  _flush(callback) {
    if (this.buffer.trim()) {
      try {
        this.push(JSON.parse(this.buffer));
      } catch (e) {
        this.emit('parse-error', { line: this.buffer, error: e });
      }
    }
    callback();
  }
}

NDJSON in Python

import json

def read_ndjson(file_path):
    with open(file_path, 'r', encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                yield json.loads(line)
            except json.JSONDecodeError as e:
                print(f"Error parsing line: {e}")
                continue

def write_ndjson(records, file_path):
    with open(file_path, 'w', encoding='utf-8') as f:
        for record in records:
            f.write(json.dumps(record, ensure_ascii=False) + '
')

# Usage
for record in read_ndjson('large_dataset.ndjson'):
    process(record)

NDJSON Production Use Cases

Use Case	Example	Why NDJSON
Log aggregation	Splunk, ELK stack, Datadog	Logs arrive one at a time, must be appendable
Data export	Database dumps, SaaS exports	Multi-GB datasets need streaming
Real-time analytics	Clickstream data, sensor readings	Records arrive continuously, no batching
ETL pipelines	Data warehouse ingestion	Each record can be transformed independently
Machine learning	Training data feeds	Process shards in parallel, skip bad records
API streaming	Twitter API, GitHub events	Events arrive as stream of JSON objects

NDJSON vs JSONL vs JSON Lines

The terms are often used interchangeably, but there are subtle differences:

• NDJSON — Newline Delimited JSON, the formal specification
• JSON Lines — Popularized by the jsonlines.org specification
• JSONL — Common file extension (.jsonl) for NDJSON files

All three follow the same principle: one JSON value per line, with a newline character as delimiter.

Best Practices for NDJSON

• Always handle parse errors gracefully — skip bad lines, log them, continue
• Use CR-LF compatible readers for cross-platform compatibility
• Validate NDJSON with JSON Validator during development
• For compression, gzip NDJSON files — gzip compresses repeated structures well
• Include a trailing newline on the last line for compatibility
• Sort or partition NDJSON files for efficient parallel processing
• Use NDJSON instead of JSON arrays for files larger than 100MB

Next Steps

Format individual NDJSON records with JSON Formatter. Validate records with JSON Validator. Compress NDJSON files with JSON Compress.