Tables, joins, normalisation and ACID may feel manageable until an interview or PSU paper asks what happens after one database becomes many machines. The single-server model breaks when data or traffic no longer fits comfortably on one server. The core design choices are where to place each record, how to survive node failures, what consistency to trade during a partition, and how to process data beside the storage nodes.
Big data systems: what the 5 Vs really mean
Big data combines scale, speed and quality problems that one conventional database may not handle economically or reliably.
V | Meaning | Concrete example |
|---|---|---|
Volume | The stored data is too large for one practical disk or server | A multi-terabyte log archive growing towards petabytes |
Velocity | New data arrives continuously and must be handled quickly | Millions of click events arriving each hour |
Variety | One system receives several data shapes | Relational rows, JSON documents, images and server logs |
Veracity | The input can be incomplete, duplicated or wrong | Two customer records for one person, with a missing address |
Value | Processing must produce a useful outcome | Turning event logs into a fraud alert or recommendation |
Vertical scaling adds RAM, CPU and storage to one server, but retains one failure point and a hardware ceiling. Horizontal scaling adds machines and divides data and work. Traditional relational systems grew mainly through vertical scale; big data systems make many horizontal nodes behave like one database.
Partitioning and sharding across database nodes
Sharding splits a table across nodes. Range partitioning might place keys 1 to 1,000,000 on A and the next range on B. Hash partitioning maps a hashed key to a node, often using modulo.
Suppose four nodes, N0 to N3, use node = user_id mod 4. For five users:
101 mod 4 = 1, because101 = 4 × 25 + 1, so user 101 goes to N1.102 mod 4 = 2, because102 = 4 × 25 + 2, so user 102 goes to N2.103 mod 4 = 3, because103 = 4 × 25 + 3, so user 103 goes to N3.104 mod 4 = 0, because104 = 4 × 26, so user 104 goes to N0.105 mod 4 = 1, because105 = 4 × 26 + 1, so user 105 goes to N1.
The placement is N0 = {104}, N1 = {101, 105}, N2 = {102}, N3 = {103}. Because five rows cannot divide evenly across four nodes, one node must receive the extra row; this tiny remainder is not evidence of skew. A shard is genuinely skewed or hot when an imbalanced key distribution or workload persists at scale. Consistent hashing helps a real system add a node while moving only part of the key space.
A shard-key query can reach one node, but a non-key filter or cross-shard join may fan out to all nodes. Denormalisation is therefore common, the opposite direction from DBMS Normalization MCQs.
Replication and the CAP theorem
Partitioning spreads different data for scale. Replication copies a shard across nodes for survival and read capacity. Networked replicas create the CAP problem.
CAP means Consistency, where reads observe the latest completed write; Availability, where each request gets a non-error response; and Partition tolerance, where operation continues despite dropped messages. During a network partition, the system cannot guarantee both C and A, so it must choose.
Take replicas A, B and C storing x = 5. A partition isolates C, and a write x = 9 reaches only the isolated replica C. A CP design rejects that write at C because C cannot coordinate with a quorum, so a request to the minority side loses availability. An AP design accepts x = 9 while A and B return x = 5, sacrificing consistency until reconciliation.
A bank balance leans towards CP because conflict is dangerous. A social feed can lean towards AP because temporary disagreement may be acceptable.
Quorum consistency with N, W and R
Quorums tune this trade-off. N is the replica count, W the write acknowledgements, and R the read responses. With comparable replica versions, W + R > N forces every R-replica read set to overlap the W replicas that acknowledged a completed write. The reader must still compare versions correctly, and the inequality alone does not resolve concurrent-write conflicts.
For N = 3, W = 2 and R = 2, W + R = 2 + 2 = 4 > 3. Minimum overlap is W + R - N = 4 - 3 = 1, so every two-replica read touches an acknowledging replica.
With W = 1 and R = 1, W + R = 1 + 1 = 2 ≤ 3. A read can hit a replica that missed the write. Higher W and R improve consistency but cost latency and failure tolerance.
MapReduce: move computation to the data
MapReduce sends computation to nodes holding the data. Map transforms records into key-value pairs, shuffle groups equal keys, and reduce aggregates each group.
Use two input lines: data is big and big data data.
Map line 1 to
(data,1), (is,1), (big,1).Map line 2 to
(big,1), (data,1), (data,1).The map phase emitted
3 + 3 = 6pairs.Shuffle produces
data → [1,1,1],big → [1,1],is → [1].Reduce produces
data → 3,big → 2,is → 1.
The output total, 3 + 2 + 1 = 6, matches the six pairs. Each map runs beside its shard, so only intermediate pairs move. Spark and stream processors make this pattern faster or continuous.
NoSQL data models versus relational tables
NoSQL is a group of models, not one product or one promise.
Family | Data shape | Suitable use |
|---|---|---|
Key-value | Distributed map from a key to a blob | Sessions and caches |
Document | JSON-like records with nested fields | Product catalogues |
Column-family | Sparse, wide rows grouped by column family | Event logs and time-series data |
Graph | Nodes and edges as first-class data | Social networks and recommendations |
These models can scale horizontally more naturally than rigid tables. Costs include weaker multi-row transactions, fewer rich joins, more application logic or eventual consistency. NoSQL does not beat SQL; it chooses different trade-offs. DBMS Transactions MCQs revises the ACID foundation being relaxed.
Big data systems: four common traps
Treating CAP as “pick two forever”. Calling a distributed system CA ignores unavoidable partitions. During a partition, describe it as CP or AP and name the sacrificed guarantee.
Assuming replicas guarantee consistency. Raising N while keeping
W = R = 1permits stale reads. CheckW + R > Nand version selection.Assuming sharding speeds every query. Cross-shard joins and non-key filters fan out. Choose the shard key from the dominant query pattern.
Confusing partitioning with replication. Partitioning stores different subsets for scale; replication copies a subset for survival. Most distributed databases use both.
Big data systems for GATE CS and interviews: the short version
The official GATE 2026 Computer Science and Information Technology syllabus lists the relational model, SQL, integrity constraints, normal forms, file organisation, indexing, transactions and concurrency control. It does not list CAP, sharding, quorums, MapReduce or NoSQL. Treat those distributed-system topics mainly as interview and university extensions: trace a shard key, choose CP or AP during a partition, test a quorum inequality, or compute a MapReduce output. Check the official syllabus again for the cycle you will attempt.
The spine is simple: horizontal scaling adds machines; partitioning spreads data; replication survives failures; CAP forces a consistency-versus-availability choice during partitions; W + R > N creates quorum overlap; MapReduce moves compute towards data; and NoSQL changes the model to accept another trade-off.
Build the relational base through Relational Algebra for GATE, then continue through the Database Management System learning path. For a structured preparation sequence, use GATE Guidance by Sanchit Sir.
