NDJSON for Streaming and Big Data: A Practical Guide

NDJSON (Newline Delimited JSON) is the preferred format for streaming and processing big data. Unlike standard JSON arrays that must be fully loaded into memory, NDJSON processes records one line at a time, enabling parallel processing, backpressure handling, and memory-efficient operations on datasets of any size. This guide covers NDJSON for big data architectures, streaming pipelines, and integrations with Spark, Kafka, and cloud storage. Use our JSON Formatter to inspect individual records and JSON Validator for quality control.

NDJSON vs JSON for Big Data

Aspect	Standard JSON	NDJSON
Memory usage	O(n) — entire dataset in memory	O(1) — one record at a time
Streaming support	Complex (must track array boundaries)	Natural (line-delimited)
Parallel processing	Must split file first	Split by byte offset (lines)
Append performance	Slow (rewrite entire array)	Fast (single append write)
Error resilience	One bad byte = corrupt file	Bad line can be skipped
File size (1M records)	~200 MB + overhead	~180 MB (no array overhead)
Compression (gzip)	~20 MB	~22 MB
Schema evolution	Requires migration	Per-record schema versioning

NDJSON Streaming Architecture

// Producer: Write records as NDJSON stream
const fs = require('fs');
const stream = fs.createWriteStream('events.ndjson');

function writeEvent(event) {
  stream.write(JSON.stringify(event) + '
');
}

// Consumer: Read and process with backpressure
const readline = require('readline');
const { Transform, pipeline } = require('stream');

const rl = readline.createInterface({
  input: fs.createReadStream('events.ndjson')
});

const processor = new Transform({
  readableObjectMode: true,
  writableObjectMode: true,
  transform(line, encoding, callback) {
    try {
      const event = JSON.parse(line);
      const processed = transformEvent(event);
      callback(null, JSON.stringify(processed) + '
');
    } catch (err) {
      // Skip malformed lines, log error
      console.error('Skipping bad line:', err.message);
      callback();
    }
  }
});

// Pipeline with backpressure
pipeline(
  fs.createReadStream('events.ndjson'),
  rl,
  processor,
  fs.createWriteStream('processed.ndjson'),
  (err) => console.log('Done:', err)
);

NDJSON with Apache Spark

// Scala/Spark: Read NDJSON
val df = spark.read
  .option("multiLine", "false")  // NDJSON = one record per line
  .json("s3://data-bucket/events/*.ndjson")

// Process with Spark SQL
df.createOrReplaceTempView("events")
val result = spark.sql("""
  SELECT
    event_type,
    COUNT(*) as count,
    AVG(duration) as avg_duration
  FROM events
  WHERE date = '2025-01-15'
  GROUP BY event_type
  ORDER BY count DESC
""")

// Write as NDJSON
result.write
  .option("lineSep", "
")
  .json("s3://data-bucket/results/")

// Python/PySpark
// df = spark.read.json("events.ndjson", multiLine=False)
// df.write.json("output/", lineSep="
")

NDJSON with Apache Kafka

// Kafka messages as NDJSON
// Producer (Node.js)
const { Kafka } = require('kafkajs');
const kafka = new Kafka({ clientId: 'producer', brokers: ['localhost:9092'] });
const producer = kafka.producer();

async function sendEvent(event) {
  await producer.send({
    topic: 'user-events',
    messages: [{
      key: event.userId,
      value: JSON.stringify(event)  // Each message is one JSON object
    }]
  });
}

// Kafka Connect: Sink connector writes NDJSON to S3
// Connector config:
// {
//   "name": "s3-sink",
//   "config": {
//     "connector.class": "io.confluent.connect.s3.S3SinkConnector",
//     "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
//     "storage.class": "io.confluent.connect.s3.storage.S3Storage"
//   }
// }

NDJSON in Cloud Storage

// AWS S3: Partition NDJSON by date
// s3://bucket/year=2025/month=01/day=15/events-0001.ndjson
// s3://bucket/year=2025/month=01/day=15/events-0002.ndjson

// Google Cloud Storage: NDJSON for BigQuery
// BigQuery loads NDJSON directly
// bq load --source_format=NEWLINE_DELIMITED_JSON dataset.table gs://bucket/data.ndjson schema.json

// Azure Data Lake: NDJSON as input to U-SQL
// DECLARE @input string = "wasb://container/data/*.ndjson";
// @data = EXTRACT ... FROM @input USING Extractors.Json();

NDJSON File Format Best Practices

Practice	Why	Implementation
Include trailing newline	Some parsers require it for last record	Add after last line
Sort or partition by key	Enables efficient range scans	Sort by timestamp before writing
Compress with gzip	50-80% size reduction	gzip events.ndjson
Use .jsonl extension	Clear file type identification	data.jsonl
One record per line	No multi-line JSON objects	Minify each record before writing
Handle bad lines	Graceful error recovery	Skip + log, don't stop processing

Performance: NDJSON Processing

Dataset	JSON (full parse)	NDJSON (stream)	Improvement
1 GB (5M records)	45s, 2.5 GB RAM	38s, 50 MB RAM	50x less memory
10 GB (50M records)	OOM (out of memory)	6m, 50 MB RAM	Processable vs impossible
100 GB (500M records)	Impossible	~60m distributed	Feasible with Spark

Next Steps

Format individual NDJSON records with JSON Formatter. Validate records with JSON Validator. Compress NDJSON files with JSON Compress.