Data compression

Just as we have shown IO with nek type data. We can also employ ADIOS2 to, for example, compress data.

In this example we show a minimun case.

Import general modules

mpi4py is always required when using these tools. Numpy is always good to have if any manipulation is to be done.

# Import required modules
from mpi4py import MPI #equivalent to the use of MPI_init() in C
import matplotlib.pyplot as plt
import numpy as np

# Get mpi info
comm = MPI.COMM_WORLD

Import modules from pysemtools

# Data types
from pysemtools.datatypes.msh import Mesh
from pysemtools.datatypes.coef import Coef
from pysemtools.datatypes.field import FieldRegistry

# Readers
from pysemtools.io.ppymech.neksuite import pynekread

fname = '../data/rbc0.f00001'

Data compression with ADIOS2

To compress data, we only need to pass a mesh and field object to the adios2 wrapper function. The inputs are somewhat similar to those of other IO functions used in the examples so far.

Writing

In this case we write the data in single precision, by indicating a wdsz of 4 bytes.

# Adios2 wrappers
from pysemtools.io.adios2.compress import write_field, read_field

# Instance the empty objects
msh = Mesh(comm, create_connectivity=False)
fld = FieldRegistry(comm)

# Read the data
pynekread(fname, comm, data_dtype=np.single, msh=msh, fld = fld)

# Write the data in a subdomain and with a different order than what was read
fout = 'compressed_rbc0.f00001'
wrd_size = 4

write_field(comm, msh=msh, fld=fld, fname=fout, wrd_size=wrd_size, write_mesh=True)

comm.Barrier() # This is not needed, in general. Here just ensure we don't overlap the read and write operations

2024-08-25 20:01:50,581 - Mesh - INFO - Initializing empty Mesh object.
2024-08-25 20:01:50,582 - Field - INFO - Initializing empty Field object
2024-08-25 20:01:50,583 - pynekread - INFO - Reading file: ../data/rbc0.f00001
2024-08-25 20:01:50,593 - Mesh - INFO - Initializing Mesh object from x,y,z ndarrays.
2024-08-25 20:01:50,594 - Mesh - INFO - Initializing common attributes.
2024-08-25 20:01:50,595 - Mesh - INFO - Mesh object initialized.
2024-08-25 20:01:50,595 - Mesh - INFO - Mesh data is of type: float32
2024-08-25 20:01:50,596 - Mesh - INFO - Elapsed time: 0.0023964810000000007s
2024-08-25 20:01:50,596 - pynekread - INFO - Reading field data
2024-08-25 20:01:50,603 - pynekread - INFO - File read
2024-08-25 20:01:50,604 - pynekread - INFO - Elapsed time: 0.021368915s

Reading

Great! Now if you wish to read the data, simply use the corresponding function.

msh2, fld2 = read_field(comm, fname=fout)

# Check if the data written and read are the same!
print(np.allclose(msh.x, msh2.x))
print(np.allclose(fld.fields['temp'][0], fld2.fields['temp'][0]))

Updated header from file to:: glb_nlv: 600, lxyz: 512,  gdim: 3, dtype: <class 'numpy.float32'>
2024-08-25 20:01:51,809 - Mesh - INFO - Initializing Mesh object from x,y,z ndarrays.
2024-08-25 20:01:51,810 - Mesh - INFO - Initializing common attributes.
2024-08-25 20:01:51,811 - Mesh - INFO - Creating connectivity
2024-08-25 20:01:52,043 - Mesh - INFO - Mesh object initialized.
2024-08-25 20:01:52,043 - Mesh - INFO - Mesh data is of type: float32
2024-08-25 20:01:52,044 - Mesh - INFO - Elapsed time: 0.23442057199999988s
2024-08-25 20:01:52,044 - Field - INFO - Initializing empty Field object
True
True