Introduction to File Formats (CSV and JSON)

CSV (Comma-Separated Values)

CSV is a simple, widely-used format for storing tabular data in plain text. Each line in a CSV file represents a row in the table, and columns are separated by commas (or other delimiters). CSV files are commonly used for data exchange between systems and applications.

Example CSV File:

name,age,city
Alice,30,New York
Bob,25,Los Angeles
Charlie,35,Chicago

JSON (JavaScript Object Notation)

JSON is a lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate. JSON files are structured as key-value pairs and are often used for configuration files and data exchange in web applications.

Example JSON File:

[
    {
        "name": "Alice",
        "age": 30,
        "city": "New York"
    },
    {
        "name": "Bob",
        "age": 25,
        "city": "Los Angeles"
    },
    {
        "name": "Charlie",
        "age": 35,
        "city": "Chicago"
    }
]

Using Perl Modules for Data Parsing

Perl has robust modules for handling both CSV and JSON file formats. This section covers how to use Text::CSV for CSV files and JSON for JSON files.

Parsing CSV Files with Text::CSV

The Text::CSV module is a powerful tool for reading and writing CSV files in Perl. It handles various CSV parsing nuances, such as quoting and delimiters.

Installation:

cpan install Text::CSV

Reading a CSV File:

#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;

# Create a new Text::CSV object
my $csv = Text::CSV->new({ sep_char => ',' });

# Open the CSV file
open my $fh, '<', 'data.csv' or die "Cannot open file: $!";

# Read the header
my $header = $csv->getline($fh);

# Process each row
while (my $row = $csv->getline($fh)) {
    my ($name, $age, $city) = @$row;
    print "Name: $name, Age: $age, City: $city\n";
}

close $fh;

Writing a CSV File:

#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;

# Create a new Text::CSV object
my $csv = Text::CSV->new({ sep_char => ',' });

# Open the CSV file for writing
open my $fh, '>', 'output.csv' or die "Cannot open file: $!";

# Write header
$csv->print($fh, ['name', 'age', 'city']);

# Write data rows
$csv->print($fh, ['Alice', 30, 'New York']);
$csv->print($fh, ['Bob', 25, 'Los Angeles']);
$csv->print($fh, ['Charlie', 35, 'Chicago']);

close $fh;

Parsing JSON Files with JSON

The JSON module provides a simple way to decode and encode JSON data in Perl.

Installation:

cpan install JSON

Reading a JSON File:

#!/usr/bin/perl
use strict;
use warnings;
use JSON;
use File::Slurp;

# Read JSON file
my $json_text = read_file('data.json');

# Decode JSON data
my $data = decode_json($json_text);

# Process each item in the JSON array
foreach my $item (@$data) {
    my $name = $item->{name};
    my $age = $item->{age};
    my $city = $item->{city};
    print "Name: $name, Age: $age, City: $city\n";
}

Writing a JSON File:

#!/usr/bin/perl
use strict;
use warnings;
use JSON;

# Create data structure
my $data = [
    { name => 'Alice', age => 30, city => 'New York' },
    { name => 'Bob', age => 25, city => 'Los Angeles' },
    { name => 'Charlie', age => 35, city => 'Chicago' }
];

# Encode data as JSON
my $json_text = encode_json($data);

# Write JSON data to file
open my $fh, '>', 'output.json' or die "Cannot open file: $!";
print $fh $json_text;
close $fh;

Reading, Processing, and Extracting Data from Files

Reading CSV Data and Converting to JSON:

#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
use JSON;

my $csv = Text::CSV->new({ sep_char => ',' });
open my $csv_fh, '<', 'data.csv' or die "Cannot open CSV file: $!";
my $header = $csv->getline($csv_fh);

my @data;
while (my $row = $csv->getline($csv_fh)) {
    push @data, {
        name => $row->[0],
        age  => $row->[1],
        city => $row->[2]
    };
}
close $csv_fh;

my $json_text = encode_json(\@data);
open my $json_fh, '>', 'data.json' or die "Cannot open JSON file: $!";
print $json_fh $json_text;
close $json_fh;

Reading JSON Data and Converting to CSV:

#!/usr/bin/perl
use strict;
use warnings;
use JSON;
use Text::CSV;

my $json_text = do {
    local $/; # Slurp mode
    open my $json_fh, '<', 'data.json' or die "Cannot open JSON file: $!";
    <$json_fh>;
};

my $data = decode_json($json_text);

my $csv = Text::CSV->new({ sep_char => ',' });
open my $csv_fh, '>', 'data.csv' or die "Cannot open CSV file: $!";
$csv->print($csv_fh, ['name', 'age', 'city']);

foreach my $item (@$data) {
    $csv->print($csv_fh, [$item->{name}, $item->{age}, $item->{city}]);
}
close $csv_fh;

Converting Between File Formats

CSV to JSON

The previous example shows how to convert CSV data to JSON format. The process involves reading the CSV file, transforming it into a data structure, and then encoding that structure as JSON.

JSON to CSV

Similarly, converting JSON data to CSV involves reading the JSON file, decoding it into a data structure, and then writing that structure to a CSV file.

Practical Examples for Handling Large Datasets

Efficient CSV Handling

When working with large CSV files, consider using streaming techniques to process data in chunks.

#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;

my $csv = Text::CSV->new({ sep_char => ',' });
open my $csv_fh, '<', 'large_data.csv' or die "Cannot open CSV file: $!";

while (my $row = $csv->getline($csv_fh)) {
    # Process each row
    print "Processing row: @$row\n";
}
close $csv_fh;

Explanation:

• Uses Text::CSV to handle large files efficiently by processing one row at a time.

Efficient JSON Handling

For large JSON files, use streaming techniques like JSON::XS's decode_json with filehandles or JSON::Stream for more advanced cases.

#!/usr/bin/perl
use strict;
use warnings;
use JSON::Stream;

open my $json_fh, '<', 'large_data.json' or die "Cannot open JSON file: $!";

my $json_stream = JSON::Stream->new;
while (my $data = $json_stream->parse($json_fh)) {
    # Process each data item
    print "Processing item: ", encode_json($data), "\n";
}
close $json_fh;

Explanation:

• JSON::Stream allows you to process JSON data in a streaming fashion, reducing memory usage for large files.

Conclusion

Perl provides powerful tools for parsing and processing CSV and JSON files, making it a versatile choice for handling various data formats. By using modules like Text::CSV and JSON, you can efficiently read, write, and convert data between these formats. With practical examples and advanced techniques for managing large datasets, Perl helps streamline data handling and automation tasks in any data-driven workflow.