parquet-java: A Java Library for Parquet File Handling

Parquet-Java: An Efficient Columnar Storage Solution for Big Data Scenarios

If you've worked with large-scale data in the Java ecosystem, you may have encountered issues like high storage costs, excessive IO consumption during queries, and cumbersome handling of complex nested data. These pain points are particularly pronounced in big data analytics scenarios—traditional row-based formats (e.g., CSV, JSON) require loading entire rows even when querying a few columns, leading to massive unnecessary IO. Meanwhile, general-purpose compression algorithms offer limited efficiency for structured data. Apache Parquet, a representative columnar storage format, was built to solve these problems, and parquet-java, its Java implementation, is the core tool for handling Parquet files in the big data ecosystem.

What is Parquet-Java?

Simply put, parquet-java is the Java implementation library of Apache Parquet, providing APIs for reading and writing Parquet files, along with integration capabilities with the Java big data ecosystem (e.g., Spark, Flink, Hadoop). Parquet itself is a column-oriented binary file format designed for efficient storage and fast retrieval of large-scale structured/semi-structured data. Unlike row storage, Parquet stores data from the same column contiguously, which means:

• Only required columns are read during queries, reducing IO by over 80% (especially ideal for analytical scenarios with SELECT on a few columns);
• Columns with consistent data types enable better compression (e.g., Snappy, Gzip), typically achieving 3-5 times higher compression ratio than CSV;
• Supports complex nested structures (e.g., object arrays in JSON) without requiring data flattening.

Core Capabilities: Beyond "Columnar Storage"

The value of Parquet-Java extends far beyond "storing by columns"—its core advantages lie in deep performance optimizations and ecosystem compatibility:

1. Type-Aware Encoding and Compression

Parquet-Java uses specialized encoding for different data types instead of generic compression. For example:

• Numeric data uses a combination of "Run-Length Encoding (RLE)" and "Bit Packing"—duplicate values are stored once, and small integers are bit-compressed to save space;
• String/enumerated data uses adaptive dictionary encoding—high-frequency values are mapped to a dictionary, while low-frequency values are stored directly, balancing compression ratio and decoding speed;
• Time-series data uses Delta encoding, storing differences between adjacent values, making it ideal for logs, sensor data, etc.

In practical tests, storing 100 million user behavior logs (including timestamps, user IDs, action types, etc.) in Parquet reduced file size by 70% compared to CSV, and Spark SQL query time decreased by 60%.

2. Native Support for Multiple Data Formats

Parquet-Java directly reads/writes data structures defined in Avro, Thrift, and Protobuf without additional conversion tools. For example, nested data defined with an Avro Schema:

// Avro Schema example: User data with nested structure
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "orders", "type": {"type": "array", "items": {
      "type": "record", "name": "Order",
      "fields": [{"name": "orderId", "type": "long"}, {"name": "amount", "type": "double"}]
    }}}
  ]
}

Via the parquet-avro module, Avro objects can be directly written to Parquet files or read from Parquet files as Avro objects, eliminating data format conversion overhead.

3. Predicate Pushdown and Column Statistics

Parquet files include "column statistics" (e.g., min/max values, null counts per column) and "index pages," allowing query engines to filter invalid files/blocks before data reading. For example, when executing WHERE create_time > '2024-01-01', Parquet-Java first checks the maximum value in the time column's metadata, skipping all non-matching files and reducing IO operations by 90%.

4. Seamless Integration with Apache Arrow

Arrow is an in-memory columnar format, and Parquet-Java can load Parquet files directly into Arrow in-memory format via the parquet-arrow module, avoiding frequent conversions between Java objects and binary data. This is critical for in-memory computing engines like Spark and Flink—"zero-copy" data transfer from disk to memory significantly accelerates computation.

Technical Highlight: Efficient Handling of Nested Structures

Processing nested data (e.g., arrays, objects in JSON) is Parquet's "unique skill," enabled by the "Record Shredding and Assembly Algorithm" from Google's Dremel paper. In short, Parquet-Java "shreds" nested structures into flat columns for storage while preserving metadata for reassembly. For example, the orders array in the User data above is shredded into orders.orderId and orders.amount columns, stored contiguously by column but retaining reassembly metadata.

This design combines columnar IO efficiency with elimination of manual data flattening (avoiding data bloat). In contrast, traditional row storage either stores nested data as JSON strings (requiring full parsing during queries) or splits it into multiple tables (high JOIN costs).

Comparison with Alternatives: Parquet vs ORC

When discussing columnar formats, Hive's ORC format often comes to mind. While their定位 overlaps, Parquet-Java has distinct advantages:

• Cross-ecosystem compatibility: Parquet is supported by nearly all big data tools (Spark, Flink, Hive, Impala), whereas ORC is more Hive-centric;
• Nested structure support: Parquet handles nested data more maturely—early ORC versions lacked complex nested support;
• Memory efficiency: Through Arrow integration, Parquet performs better in in-memory computing scenarios, while ORC focuses more on disk storage optimization.

However, ORC is preferable in specific cases: e.g., Hive native tables or when ACID transaction support is needed, as ORC offers more robust metadata management.

Practical Advantages and Limitations

Ideal Scenarios:

• Analytical queries: Data warehouses, BI reports—queries typically involve few columns;
• Large-scale data archiving: Historical logs, user behavior data—long-term storage with occasional queries;
• Stream processing intermediate results: State storage for Flink/Spark Streaming—columnar storage + compression saves space.

Less Suitable Scenarios:

• High-frequency small data writes: Parquet files require "block" writes (usually 64MB-1GB), so frequent small-batch writes cause numerous small files, degrading performance;
• Full table scans: If queries need all columns, columnar storage offers little advantage and may underperform row storage (columnar storage requires merging multiple columns).

Ease of Adoption:

Parquet-Java’s API is relatively intuitive, especially when integrated with Avro/Protobuf—basic read/write operations require just a few lines of code:

// Example code for writing Parquet with Avro
ParquetWriter<User> writer = AvroParquetWriter.<User>builder(path)
    .withSchema(userSchema)
    .withCompressionCodec(CompressionCodecName.SNAPPY)
    .build();
writer.write(user);
writer.close();

Conclusion: "Infrastructure" for Big Data Storage

Parquet-Java isn’t a silver bullet, but it’s nearly a "standard tool" in big data scenarios. Its value lies in resolving the core conflict between storage cost and query performance through columnar storage, intelligent encoding, and cross-ecosystem support. If you work with TB-scale data in the Java ecosystem—whether building data lakes, developing ETL pipelines, or optimizing analytical queries—parquet-java is worth mastering. It’s not just a library but an excellent case study in big data storage optimization.

Final tip: In practice, start by analyzing data characteristics (e.g., column cardinality, value distribution) with parquet-tools (Parquet’s command-line tool) before tuning encoding and compression parameters—you’ll often see unexpected performance gains.