Basic concepts and Terminology adopted by RasterLite2

Objects Hierarchy: Coverages, Sections and Tiles

the Coverage is the topmost level object.
A Coverage is a very generic container and is internally articulated into many different Sections (may be, hundredth or even thousand Sections); as a limit case, a Coverage could just contain a single Section.
All Sections belonging to the same Coverage must always share many relevant properties, i.e.:
- same SRID (aka Spatial Reference System).
- same Pixel format (i.e. RGB, or GRAYSCALE or DATAGRID), same Sample type (i.e. UINT8 or UINT16 or DOUBLE) and same Bands Configuration.
- same Compression and same optional Policies.
- all these critical parameters will be defined while creating a Coverage.
  Any subsequent attempt to modify such parameters will always be considered a forbidden operation if the Coverage already contains any Tile.
- a Coverage always has its own Bounding Box (aka BBOX) which simply corresponds to the overall BBOX for all subordinated Sections.
the Section is an intermediate level object.
A Section usually corresponds to a single image file initially used to populate the Coverage.
Each Section has its own private BBOX and there is no special constraint about mutual BBOX relationships of different Sections belonging to the same Coverage:
- Sections aren't required to be strictly adjacent and contiguous; a Coverage is absolutely free to include empty / uncovered areas; in this case they'll be simply considered as full Transparent areas.
- two or more Sections are absolutely free to partially or completely overlap (please see S1 and S4 in the above figure).
  In this case the Section identified by the higher ID value will always cover any other Section of lower ID value.
- Sections in the same Coverage aren't required to have the same Width and Height dimensions.
the Tile is the lowermost level object.
Each Section is always internally stored as a collection of many individual Tiles:
- All Tiles belonging to the same Coverage must always share the same identical layout: same Pixel Type, Sample Type and Band Configuration; and identical Compression.
- All Tiles in the same Coverage must always have the same identical Width and Height dimensions:
  - Tiles belonging to the same Section will always form a strictly packed regular grid.
  - in the above figure Sections S1 and S3 are rather exceptional, because their Width and Height dimensions corresponds to an exact multiple of the Tile dimensions.
  - Sections S2 and S4 instead are more realistic examples; when the Section's Width and Height doesn't exactly match the Tile dimension all rightmost and lowermost Tiles will always be only partially filled. The unused portion of such Tiles will simply correspond to a full Transparent area.
- all Tiles have their individual BBOX; partially populated Tiles will have a reduced BBOX simply representing the portion being effectively populated.
  So the overall extent of all Tiles belonging to the same Section will always exactly match the Section's BBOX.
- as a general design principle any completely empty Tile (i.e. exactly corresponding to a full Transparent area) will never be physically inserted into the DBMS, thus avoiding to waste unnecessary storage.
- Tiles will be mostly considered as read only objects: Tiles can be freely inserted (while importing the corresponding Section) but can never be updated or altered in any way once inserted. Anyway this rule has its exceptions (we'll examine this topic in more depth in a following paragraph: Coverage Policies).

Multi-Resolution Pyramid

Coverages and / or Sections could eventually be supported by a Multi-Resolution Pyramid: creating a Pyramid isn't a strictly required operation, anyway this is a very useful and highly recommended option.

the Multi-Resolution Pyramid will permanently store reduced size images (aka rescaled images, aka thumbnails) accordingly to a scale based on the powers of 2:
- if the Pyramid's support is available then creating rescaled images at any arbitrary resolution will become a realistic option.
  An appropriate Level already supporting a very close resolution will surely be immediately available from the Pyramid; then rescaling to the required arbitrary resolution will be a reasonably fast process and will require a very limited RAM allocations.
- building the Pyramid is an iterative process creating a new half scale Level during each iteration.
Pyramid Levels can be either physical or virtual:
- a physical Level implies consuming some extra storage space.
- on the other hand virtual Levels don't require any storage space at all.
  This is thanks to the systematic adoption of the Adam7 encoding schema supported by all Tiles codecs internally adopted by RasterLite2.
Pyramids extensively based on Virtual Levels will consume a surprisingly low storage amount, but could sometimes suffer from slightly worst performances.
Pyramids using no Virtual Levels at all are always expected to perform better, but can easily consume a noticeable storage amount.
it's always a trade-off between space and time; anyway many Tile codes supported by RasterLite2 will significantly benefit from massive adoption of Virtual Levels without necessarily imposing a noticeable performance loss.

Different Physical / Virtual Levels supported by RasterLite2 Pyramids
Interleaving 3 This is the standard default setting (except in the case of MONOCHROME 1-BIT Coverages). One Pyramid Level every three is physical and the following two are purely virtual. e.g. resolution 1:1 is physical then 1:2 and 1:4 are virtual. Resolution 1:8 is physical then 1:16 and 1:32 are virtual and so on.
Interleaving 2 Pyramid Levels are alternatively one physical and one purely virtual. e.g. resolution 1:1 is physical then 1:2 is virtual. Resolution 1:4 is physical then 1:8 is virtual and so on.
Interleaving 1 This is the standard default setting for MONOCHROME 1-BIT Coverages. All Pyramid Levels are physical; virtual levels are never used.

Please note very well: thanks to the Adam7 Tile encoding extensively adopted by RasterLite2, even the base-resolution Level alone will automatically support the first four resolutions (i.e. 1:1, 1:2, 1:8 and 1:16) even when absolutely no Pyramid support is available.
Anyway this doesn't applies to MONOCHROME 1-BIT Coverages; in this specific case the base-level will simply support 1:1 alone and no other further resolution.

Section-based Pyramids

RasterLite2 can handle Pyramids in two different flavours: the first one in based on many individual and independent Pyramids each one strictly supporting just a single Section. Section-based Pyramids are supported for all Coverage types with no exception.

pyramid base-level

The above figure represents the overall layout of a Coverage articulated into several Sections (A, B, C, D, E, F, G and H).

pyramid by section

This second figure shows the corresponding layout of Section-based Pyramids:

a distinct Pyramid has been created for each single Section.
there is no overall Pyramid supporting the Coverage as a whole.
empty / unused areas aren't covered by any Pyramid. This surely isn't a problem because they are always assumed to represent full Transparency, and a Full Transparent pixel will always remain the same at any possible rescaled resolution.

Monolithic Pyramid

This third figure (same Coverage / Sections layout as is the previous case) represents the alternative case of a Monolithic Pyramid:

a single integrated Pyramid covers the Coverage as a whole.
any distinction between individual Sections is completely ignored.
empty unused areas are now included into the Pyramid; anyway all full Transparent Tiles will never be physically inserted into the DBMS, so only a very negligible extra storage will be required.
not all Coverages are able to support a Monolithic Pyramid; it depends on the specific Coverage type (we'll examine this topic in more depth in the next paragraph: Coverage Policies).

Main differences between Section-based and Monolithic Pyramids

Section-based Pyramids are more flexible:
- removing an already existing Section from a Coverage automatically implies removing the corresponding Pyramid, so such an operation will not affect any other different Section and the Coverage itself will always remain in a full consistent state.
- inserting a further new Section simply requires building a Pyramid supporting that Section; in this case too such an operation will not affect any other different Section and the Coverage itself will always remain in a full consistent state.
Monolithic Pyramids are rather rigid; any change affecting the underlaying Sections (e.g. removing and existing Section or inserting a new Section) will automatically invalidate the whole Pyramid. And rebuilding from scratch a Monolithic Pyramid supporting a really huge Coverage surely isn't a quick and fast process.
Section-based Pyramids are expected to require slightly less storage space than Monolithic Pyramids, but the difference is never expected to be a really striking one.
Monolithic Pyramids are always expected to perform slightly better than Section-based Pyramids.
This is because a Monolithic Pyramid will surely have more resolution Levels than the corresponding Section-based Pyramids, and will efficiently cover even the most reduced (topmost) resolution Levels.

Coverage Policies

Each single Coverage has its own Policies; this is a set of boolean options intended to support a very flexible configuration accordingly to specific requirements.
The current version supports the following Policy Options (future versions of RasterLite2 could easily extend the list by introducing further Options):

strictResolution: if set to true then the strict resolution check mode will always be enforced when importing any data.
mixedResolutions: if set to true then the mixed resolutions mode will be applied.
Please note: strictResolution and mixedResolutions are mutually excluding Options.
immutable: if set to true then any attempt to update any Section and any Tile after the initial insertion will be forbidden.
sectionPaths: if set to true then the input Paths recording mode will be enabled.
sectionMD5: if set to true then the input MD5 checksum mode will be enabled.
sectionSummary: if set to true then the input images summary mode will be enabled.

the strictResolution Policy Option

When this Option Policy is enabled all Sections being imported into the Coverage mist exactly correspond to the Horizontal and Vertical Pixel Resolution declared when creating the Coverage.
The Resolution check will be intended in the most pedantic and fussy mode: even the slightest difference found will inexorably cause a failure to load.

the mixedResolutions Policy Option

This is exactly the opposite than before: in this case the Coverage itself has no intrinsic Horizontal and Vertical Pixel Resolution, and each single Section being imported will then be absolutely free to declare any possible Resolution.
Please note well: any Coverage declaring the mixedResolutions Option will be always forbidden to support a Pyramid of the Monolithic type; anyways it will still able to support a Pyramid of the alternative Section-based type.

Fallback case: what happens when a Coverage doesn't declare neither strictResolution nor mixedResolutions ?
In the case the permissive resolution check mode will be enabled, substantially corresponding to a relaxed equivalent of strictResolution.
The resolution declared by any Section being imported will still continue to be checked for compatibility, but will evaluated in a permissive way thus allowing to consider small differences (<= 1%) as acceptable.
This will certainly cause some occasional pixel misalignment here and there, but this shouldn't be a real problem for many simple, unsophisticated and not too much demanding applications.

the immutable Policy Option

When this Option Policy is enabled then any Section and any Tile can never be altered in any way after its initial insertion into the DBMS.
Anyway a Section could still be completely removed (this including all subordinated Tiles), and may be re-imported yet another time (e.g. because an updated version of the corresponding input image is now available).

the sectionPaths Policy Option

When this Option Policy is enabled then all Sections will permanently record the input path used to initially import the corresponding image from the filesystem.
Please note: activating this Option may probably be useful on typical standalone Personal / Desktop environments, but will surely imply introducing many dangerous security concerns in server environments, because it will externally disclose many critical details about the layout adopted by the internal filesystem.

the sectionMD5 Policy Option

When this Option Policy is enabled then for each Section being imported the MD5 checksum will be computed on behalf of the input file and will then be permanently recorded as a Section attribute.
A short rationale for using MD5 checksums: file paths may eventually change during the time, because rearranging the directories tree is a very frequent maintenance operation. Even individual file names are potentially subject to suddenly change for same reason.
Anyway an MD5 checksum is an absolute invariant; if the internal file content never changes, the MD5 checksum will always remain exactly the same.
So using MD5 checksums as file identifiers surely is an effective mean allowing to keep full trace of the original files initially imported, even when the corresponding paths could crazily change; and without externally disclose any critical information about the internal filesystem.

the sectionSummary Policy Option

When this Option Policy is enabled then for each Section being imported an XML summary describing the input file will be permanently recorded as a Section attribute.

<?xml version="1.0" encoding="UTF-8"?>
<ImportedRaster>
    <RasterFormat>GeoTIFF</RasterFormat>
    <RasterWidth>6181</RasterWidth>
    <RasterHeight>7659</RasterHeight>
    <RowsPerStrip>1</RowsPerStrip>
    <BitsPerSample>8</BitsPerSample>
    <SamplesPerPixel>4</SamplesPerPixel>
    <PhotometricInterpretation>RGB</PhotometricInterpretation>
    <Compression>none</Compression>
    <SampleFormat>unsigned integer</SampleFormat>
    <PlanarConfiguration>single Raster plane</PlanarConfiguration>
    <NoDataPixel>unknown</NoDataPixel>
    <GeoReferencing>
        <SpatialReferenceSystem>
            <SRID>26916</SRID>
            <RefSysName>NAD83 / UTM zone 16N</RefSysName>
        </SpatialReferenceSystem>
        <SpatialResolution>
            <HorizontalResolution>1.0000000000</HorizontalResolution>
            <VerticalResolution>1.0000000000</VerticalResolution>
        </SpatialResolution>
        <BoundingBox>
            <MinX>673195.0000000000</MinX>
            <MinY>4297568.0000000000</MinY>
            <MaxX>679376.0000000000</MaxX>
            <MaxY>4305227.0000000000</MaxY>
        </BoundingBox>
        <Extent>
            <HorizontalExtent>6181.0000000000</HorizontalExtent>
            <VerticalExtent>7659.0000000000</VerticalExtent>
        </Extent>
    </GeoReferencing>
</ImportedRaster>

The above sample is an example of an XML Summary produced by RasterLite2.

Annex A: the Adam7 multi-resolution Tile encoding

Many Tile codecs supported by RasterLite2 are completely based on the Adam7 encoding schema; other are not (e.g. the JPEG, PNG and WebP based codecs) simply because an intrinsic rescaling capability is already supplied by the underlying codec.

How Adam7 works

All pixels require to be are arranged in a more convenient way, so to be immediately accessible in the fastest way accordingly to the Adam7 schema.
Recreating the original image at full resolution (1:1) will require 7 subsequent passes (and this explains why it's called Adam7); more limited resolutions will require less passes, accordingly to the following schema:
- the first pass alone will recreate a reduced image corresponding to the 1:8 resolution.
- the first 3 passes will recreate a reduced image corresponding to the 1:4 resolution.
- the first 5 passes will recreate a reduced image corresponding to the 1:2 resolution.
- and finally, as previously stated, all 7 passes rill recreate the 1:1 image at full resolution.

The following figures will probably help to understand better:

rescaled resolution	Adam7 pass	affected pixels	output image
1:8	#1
1:4	#2
1:4	#3
1:2	#4
1:2	#5
1:1	#6
1:1	#7

Further Adam7 optimizations applied by RasterLite2

All pixels in any Adam7 encoded tile will be split in two distinct blocks of equal size:
- the first block will contain all pixels belonging to an odd numbered scanline (the first scanline is assumed to be numbered starting by 1).
  These pixels are the ones required by the first six Adam7 passes (corresponding to rescaled resolutions 1:8, 1:4 and 1:2).
- the second block will contain all pixels belonging to ab even numbered scanline.
  These pixels are only required by the final seventh Adan7 pass in order to reconstruct the 1:1 full resolution Tile.
splitting the pixels in two equally sized odd and even blocks practically means that in many cases the I/O requirements can be significantly reduced, and this will obviously introduce a noticeable performance boost.
only requests referencing full resolution Tiles impose to read all pixels; in any other case just reading the odd pixels (a 50% cut) is all that is effectively needed.

mapping the conceptual Objects Model into a DBMS layout

Object	DBMS table	Remarks
Coverage	raster_coverages	Main Coverages Catalog; this table is the symmetric equivalent of what geometry_columns is in the case of Vector data. All critical configuration parameters for each Coverage are stored into this table: pixel type, sample type, compression, tile size, SRID and so on.
Coverage	coverage_policies	An auxiliary table storing individual Policy Options for each Coverage. The main rationale suggesting to place all Policy Options in a separate table is in that future versions of RasterLite2 will presumably add many further supported Policies.
Section	<coverage_name>_sections	All Sections being loaded into the Coverage will have a corresponding entry into this table. This including any optional information such as input path, MD5 checksum and section Summary accordingly to Policy Options settings. This one is a geometry table (Section BBOX) and is supported by a Spatial Index.
Tile	<coverage_name>_tiles	Both base-level (aka full resolution) and Pyramid Tiles being loaded into the Coverage will have a corresponding entry into this table. This one is a geometry table (Tile BBOX) and is supported by a Spatial Index.
Tile	<coverage_name>_tile_data	all odd and even BLOB encoded Tile pixels will be stored into this table. The main rationale suggesting to adopt a separate table for storing BLOB encoded data is in that this way the heavy BLOBs have their own distinct storage space; and this allows many internal SQL queries to be executed in a noticeably faster way.
Pyramid (alternative and mutually exclusive layouts)	<coverage_name>_levels	This configuration is adopted by any Coverage declaring the strictResolution or permissiveResolution Policy Option. It's compatible with both Section-based and Monolithic Pyramid.
Pyramid (alternative and mutually exclusive layouts)	<coverage_name>_section_levels	This second configuration is adopted by any Coverage declaring the mixedResolution Policy Option. It's only compatible with a Pyramid of the Section-based type, and completely forbids building a Monolithic Pyramid.

Entity-Relation diagram: strictResolution / permissiveResolution

Entity-Relation diagram: mixedResolutions

Back to RasterLite2 doc index