|
MetaAMR − Metadata format for Adaptive Mesh Refinement datasets |
|
The MetaAMR format is an XML based format for
storing information about AMR datasets. The XML file
contains a description of the subgrid bounding boxes, their
nesting level, their lifetime, as well as any optional
attributes of the grid (such as scalar min/max). The actual
field data is not stored in XML format (One should hope
not!) however a reference to any field data can be stored in
the grid element. |
|
The root element must be named MetaAMR, and should consist of one or more dataset elements (though no tools so far support multiple datasets). The dataset element contains a single header element, which contains information about the fields, attributes, data locations and other global information. The header is followed by a list of grid elements and possibly grid_group elements. Thus, the basic file looks like: |
|
<?xml version="1.0"
encoding="ISO-8859-1"?> |
|
<dataset> |
|
<header> |
|
</header> |
|
<grid ...> ... </grid> |
|
</grid_group> |
|
</dataset> |
|
</MetaAMR> |
|
Grids need not be stored in any particular order. If they are sorted, then one can add a sort_method attribute to the dataset element to indicate the sorting keys. For instance: |
|
<dataset sort_method="time,level" > |
|
indicates that the individual grids are sorted in increasing order with time as the primary key, and refinement level as the secondary key. The sort_order attribute is entirely optional, however. |
|
The format described here is imperfectly handled by the libc_amr library. Several options (such as integer data support) are not yet implemented by libc_amr. These options will be marked with an asterix (*) where they appear in the format description. |
|
The header section describes the field data and attributes, the location of the field data, how to access it, and spacing information for the levels. All of it is optional, though if spacing information is not provided here, then it must be provided on a per-grid basis. |
|
FIELD_DESCRIPTION |
|
The field_description element is used to describe a per-grid data field. For each subgrid, a field is a uniform grid. A field_description element is empty in that it has no sub-elements, but it must contain the following attributes: |
|
name |
A name for the field, such as "BaryonDensity" or "Temperature". This name will be used as an element tag inside each grid element, so it must be a legal XML name. |
|
field_type |
|
The centering for the field. Valid values are cell for cell-centered (body-centered) data, vertex for vertex centered data, face-xy, face-xz, face-yz (*) for data centered on one of the faces, or edge-x, edge-y, edge-z (*) for data centered on an edge. |
|
datum_type |
|
The type associated with each data point. Valid data types are listed in the section DATA TYPES below. |
|
Element Example: |
|
<field_description |
|
name="BaryonDensity" |
|
ATTRIBUTE_DESCRIPTION |
|
The attribute_description element is used to describe a per-grid attribute, useful for storing such things as scalar minima/maxima or the like. It is also an empty element, and must contain the following attributes: |
|
name |
A name for the attribute, such as "ScalarMin" or "AverageError". This name will be used as an element tag inside each grid element, so it must be a legal XML name. |
||
|
type |
The type of the attribute value. Valid types are listed in the section DATA TYPES. |
|
Element Example: |
|
<attribute_description |
|
name="Redshift" |
|
DATA_ACCESS_METHOD |
|
The data access method is a flexible way to describe the location of field data without rigidly tying the MetaAMR format to the data storage format. Each data_access_method element describes a method for accessing data, such as raw access to a file, or a database to query. To associate field data with a subgrid, each subgrid element contains a reference to an access method, and any additional information needed to locate that particular grid’s data. Each data_access_method must contain at least two attributes: |
|
method |
A name for the method of accessing the data. |
||
|
index |
An integer index, to be used later to refer back to this method |
|
Other attributes will most likely be required, depending on the access method used. See the section THE BRICK ACCESS METHOD below for an example. Element Example: |
|
<data_access_method |
|
method="brick" |
|
SPACING_TABLE |
|
For AMR datasets, not all the subgrids have the same spacing. In general, there are different levels of refinement, from coarse to fine. In the Berger-Collela scheme of AMR, there are a finite number of refinement steps, where the grid spacing for a coarse step is some integer multiple of the next finest step. Often, this refinement factor is 2, so that each level has twice the ’resolving power’ of the previous level. For better or worse, the MetaAMR format allows for non-integral refinement by specifying the spacing for each level as a real-valued vector (grid spacing in the X, Y, and Z directions). The benefit of this is that it may allow for representation of a wider number of multigrid formats. The downside is that it is harder to represent very deep hierarchies due to issues of floating-point roundoff and running out of precision. Those who use the MetaAMR format should expect to use at least double precision floats, at least while reading in spacings and coordinates. While spacing can be specified on a per-grid basis (and must be, if a spacing_table is omitted from the header), it is more efficient to store a table relating grid-spacing to refinement levels. The spacing_table element is used for this. The element takes no attributes, and the element body should consist of entry elements, where each entry element has two attributes: |
|
level |
A non-negative integer indicating the level of refinement, where level 0 is the coarsest level |
|
spacing |
|
A space-separated list of three floating point values, giving the grid spacing in each of the X, Y, and Z directions. |
|
Element Example: |
|
<spacing_table> |
|
<entry level="0" spacing="1 1
1"/> |
|
</spacing_table> |
|
Attributes |
|
Subgrids are described with the grid element. A grid element consists of a number of attributes which describe the grid’s location, extents, spacing, and lifetime. The element body consists of references to field data as well as attributes. The required attributes and their types are as follows: |
|
level integer |
|
The refinement level of the grid |
|
origin 3 space-delimited real-valued numbers |
|
The origin of the grid. |
|
extent 3 space-delimited integers |
|
The number of grid-cells, in each direction. Note that this is independent of how any field data is centered (vertex. cell, face or edge centering), allowing for different fields to have different centerings. |
|
spacing 3 space-delimited real-value numbers |
|
The spacing for the grid. This may be omitted if a spacing_table was included in the header, and has an entry for this grid’s level. |
|
time real-valued number |
|
The creation time for the grid. |
|
delta_t real-valued number |
|
The life-span of the grid. |
|
Element Example: |
|
<grid |
|
level="2" |
|
... |
|
</grid> |
|
Inside the grid element, there may be one or more field or attribute elements. |
|
Field Elements |
|
A field element ties together a grid, a data field, and a method to access that data. The name of a field element must have been defined by a field_descriptor in the header section of the file. It must also have a method_index attribute. The method_index attribute must either be the special value none or it must be an integer which was declared as the index of some data_access_method that was declared in the header section. A value of none is equivalent to omitting the field element. Otherwise, the method_index attribute refers to some data_access_method and other attributes may be required, depending on the specific access method involved. See THE BRICK ACCESS METHOD for an example of an access method. Element Example: |
|
<field_description |
|
name="BaryonDensity" |
|
appeared in the header section, then a typical field element might look like: |
|
<BaryonDensity |
|
method_index="4" |
|
Here, method_index refers to a particular data access method (also defined in the header). The offset attribute is specific to the access method used; different access methods might have different attributes. |
|
Attribute Elements |
|
An attribute element is an element which has a name defined by a attribute_descriptor in the header section. It should consist of a single attribute, value which has a value of the type specified by the attribute descriptor Element Example: |
|
<Redshift value="97.52761840820312"/> |
|
To cut down a bit on the size of the XML files, we can observe that for most datasets, there is a large amount of redundant information. For instance, there are likely to be many grids that have the same lifetime, or level. To take advantage of that, we have the grid_group element. A grid_group consists of one or more grid attributes, and encloses one or more grid or grid_group elements. If a grid attribute such as "level" is present in the grid_group attribute list, then it may be omitted for any grid sub elements, and the value provided in the grid_group attribute list will be used instead. If a grid sub-element does contain a "level" attribute, however, that value will override the value provided by the grid_group. Attributes defined in a nested grid_group will override those in any enclosing grid_group. A short example may be in order. Assume that a spacing table has been provided and that there is no field or attribute data. Then : |
|
<grid_group time="1.0" delta_t="0.125" > |
|
<grid_group level="0" > |
|
<grid |
|
origin="0.0 0.0 0.0" |
|
<grid |
|
origin="0.0 0.0 0.5" |
|
</grid_group> |
|
</grid_group> |
|
is equivalent to: |
|
<grid |
|
level="0" |
|
<grid |
|
level="0" |
|
Any attribute that can appear as a grid element attribute can be used as a grid_group attribute. In practice, the most likely candidates will be level, time, and delta_t. Using grid_groups is optional; however it can significantly reduce file size and processing time. |
|
Only two types of data are currently availible in libc_amr (because no others have been needed yet!) However, this listing will include types not yet availible. |
|
Float32, Float64 are the types implemented so far. |
|
Types yet to be implemented: |
|
Int8, Int16, Int32, Int64 (*) |
|
If the maintainer of the libc_amr library is feeling sufficiently ambitious, vector types may be allowed, in which case the syntax for a vector type would be Typename[N], where N is a small integer. For example, Float32[3] would describe a vector of 3 32-bit floats. |
|
The brick access method describes how to locate data when it is stored as a basic brick-of-data; that is, each 3-D volume of data is stored as a contiguous run of elements. Some allowance is made for storing the data in different ordering (z-fastest or x-fastest) and for different architectures (big or little endian). However, there are lots of ways to store arrays of floats, and so no guarantee is made that data stored on one architecture will be readable on another. |
|
The brick data_access_method attributes |
|
Here are the attributes that must be passed to the data_access_method element: |
|
method brick | enzo_amr |
|
Required by all data_access_method elements, a method of either brick or enzo_amr signifies that the brick access method is to be used. |
|
index <integer> |
|
Required by all data_access_method elements, and must be unique. |
|
filename <string> |
|
A path to a data file containing bricks of data. The filename may be relative to the location of the MetaAMR file, or it may be an absolute path. |
|
datum_type <datatype> |
|
The type of data stored in the bricks. Valid values for this attribute can be found in the section DATA TYPES above. |
|
endian big | little |
|
Byte order of the data in the bricks. Yes, there are other possible byte orders (not to mention other floating point or even integer representations), but this should take care of many cases. |
|
array_ordering z_fastest | x_fastest |
|
The ordering which an array is stored in memory, corresponding to column major and row major, respectively. |
|
Element Example: |
|
<data_access_method |
|
method="brick" |
|
Accessing data from a grid element |
|
To use the brick access method to refer to a grid’s data, all that needs to be provided is a byte offset into the file, which we can specify with the offset attribute to a field element. Example: |
|
<field_description |
|
name="BaryonDensity" |
|
then we can locate data associated with a grid by the following: |
|
<grid |
|
level="3" |
|
<BaryonDensity |
|
method_index="4" |
|
</grid> |
|
This indicates that a block of 8*18*32 doubles is stored in the file "MoviePack000.mdat.0_0004", at an offset of 350136 bytes from the beginning. Note that if the field_description element had a field_type of "vertex", then the above would refer to a block of 9*19*33 doubles, since a grid’s extent is always measured in cells. |
|
<?xml version="1.0"
encoding="ISO-8859-1"?> |
|
<dataset sort_method="time,level" > |
|
<header> |
|
<field_description |
|
name="BaryonDensity" |
|
<data_access_method |
|
method="brick" |
|
<data_access_method |
|
method="brick" |
|
<spacing_table> |
|
<entry |
|
level="0" |
|
<entry |
|
level="1" |
|
<entry |
|
level="2" |
|
</spacing_table> |
|
</header> |
|
level="0" |
|
<grid |
|
origin="0 0 0" |
|
<BaryonDensity method_index="0" offset="0" /> |
|
</grid> |
|
origin="0.5 0 0" |
|
<BaryonDensity method_index="1" offset="0" /> |
|
</grid> |
|
</grid_group> |
|
level="1" |
|
<BaryonDensity method_index="0" offset="524288" /> |
|
</grid> |
|
</dataset> |
|
Matthew Hall <mahall at ncsa dot uiuc dot edu> |