View Ticket
Not logged in
Ticket Hash: 3743a244cf30c0e424e660af3034bbfbdd4febac
Title: Concept new tool 'spatialite_updatez'
Status: Closed Type: Feature_Request
Severity: Minor Priority: Immediate
Subsystem: Resolution: Fixed
Last Modified: 2019-05-15 08:39:19
Version Found In: development
User Comments:
anonymous added on 2017-09-13 06:17:17:

Goal of the new tool would be:

    to allow a User to update an existing geometry field that contains a Z Value
    by searching for the nearest POINTZ (or POINTZM) of a Dem-Database

Preconditions:
    an existing Database (Source) Geometry field has been converted to a Dimension Z (or ZM)
      the TABLE must be a SpatialTable, since ROWID will be used to update
    An Dem-Database (Dem) exists that contains a POINTZ or POINTZM Geometry
    The (Dem) contains a SpatialIndex
    Both Geometries contain a valid Srid

Input Parameters:

For both Source and Dem
    Path to Database
    Table
    Geometry-Column

Optional Parameter:
    resolution of Dem
      which by default can be calculated by
        (extent_maxx_dem-extent_minx_dem)*(extent_maxy_dem-extent_miny_dem)/(double)rows_count_dem
        
        assuming that the Dem has been completely filled with a Geometry
      this parameter can be used to override this value if not correct.

Commands:
    sniff
      which would allow the User to prepare the command syntax
      receiving a message that all preconditions are fulfilled.
      Source and Dem can be done separately or together
      Goal is to ease the creation of the parameters and receiving information about what is to happen

    update
      everything done during a sniff, will be done here
      if both Source and Dem are correct, the update will start
      All of the (not NULL) geometries will be selected
        if the Srid of the two are different, a transformed geometry will also be created
      A function called gaiaUpdateZ (based on gg_transform.c gaiaTransformCommon) is called with both geometries and the extent of Dem
        only Geometries completely contained inside the Dem will be processed (returning NULL)
        a Geometry will be returned when any of the Z-Values (that are not 0.0) have changed (otherwise returning NULL)
      if the result gaiaUpdateZ returns NOT NULL, the Geometry will be updated

retrieve_dem_points:
is called during gaiaUpdateZ (4 times, when needed) to retrieve the nearest point from a sent array of x/y points
    gaiaUpdateZ will collect and send (when needed the transformed) points to retrieve_dem_points
A query is called to the ATTACHed Database, using the SpatialIndex
    The set resolution is used with ST_Buffer for the search_frame
Checking is done if the returned z (or possibly, yet unlikely, a m) value is different from the sent value.
Counters exist in all functions to insure that update will only be done when needed
    but also to inform the user the amount of geometries/points and how many of those have been changed

At present the 'Help' looks like this:
usage: spatialite_updatez ARGLIST
==============================================================
-h or --help                    print this help message
========================== Parameters ========================
  -- -- ---------------- Dem-Data Database ---------------- --
-ddem or --dem-path  pathname to the SpatiaLite Dem DB 
-tdem or --table-dem table_name [SpatialTable or SpatialView]
-gdem or --geometry-dem-column col_name the Geometry column
	 must be a POINT Z or a POINT ZM type
-rdem or --dem-resolution of the dem points while searching
	 the automatic resolution calculation is based on the row_count
	 within the extent, which may not be correct!
	 Use '-rdem' to set a realistic value

  -- -- ----------------- Source Database ----------------- --
-d or --db-path pathname to the SpatiaLite DB
-t or --table table_name,  must be a SpatialTable
-g or --geometry-column the Geometry column to update
	 must  be a Z or a ZM Dimension type
	 use CastToXYZ(geom) or CastToXYZM(geom) to convert 
  -- -- --------------- General Parameters ---------------- --
-mdem or --copy-m [0=no, 1= yes [default] if exists]

  -- -- -------------------- Notes:  ---------------------- --
-I-> the Z value will be copied from the nearest point found
-I-> the Srid of the source Geometry and the Dem-POINT can be different

=========================== Commands ===========================
-sniff   [default] analyse settings without UPDATE of z-values 
-update Perform UPDATE of z-values 

Tests:
    Valgrind reports only the 2 well known GEOS leaks

berlin_dhh92.db
Was build from 279 .xyz files and combined and sorted,
each containing 4000.000 records (1.116 billion)
    sort -k2 -n -k1 3902_581820.txt -o 3902_581820.dhhn92.txt
Was the imported into a sqlite3 Database
.mode csv
.separator " "
.import 2007.berlin.dhh92.xyz berlin_dhh92_import
A Spatialite script created a copy, preparing 2 Geometries.
As of date only 114 million geometries, covering an area of 12*12 Km have been created.

berlin_street_segments
    contains 42716 LINESTRING's, many of which are outside the Dem area (that contain geometries)
SQLite version: 3.17.0
SpatiaLite version: 4.5.0-devel

Source: srid 3068
Source: extent min x/y(3584.6332144,1452.2834924)
	       mix x/y(48941.5043681,37058.9517384)
Source: rows_count(soldner_segment) 42720
Source: geometry_type(1002) has_z[1]
Source: spatial_index_enabled[1]
Source 'berlin_admin_geometries.db'
will convert TABLE[berlin_street_segments] with GEOMETRY-Column[soldner_segment] Z-Value
	from nearest POINT found in
Source Database: has passed all checks.

Dem: srid 25833
Dem: extent min x/y(385000.0000000,5814000.0000000)
	    max x/y(397000.0000000,5826000.0000000)
Dem: rows_count(utm_point) 1116000000
Dem: resolution(utm_point) 0.1290323
Dem: geometry_type(1001) has_z[1]
Dem: spatial_index_enabled[1]
Source 'berlin_admin_geometries.db'
will convert TABLE[berlin_street_segments] or GEOMETRY-Column[soldner_segment] Z-Value
	from nearest POINT found in
Dem 'berlin_dhh92.db'
 TABLE[berlin_dhh92_2007] with GEOMETRY-Column[utm_point]
-W-> -rdem was  set. Using; resolution(0.9990000), overriding the calculated value: 0.1290323
Dem srid[25833]: is different from the Source srid[3068].
	 When searching for the nearest point, the Source points will be transformed to srid[25833].
Dem Database: has passed all checks.
	DB-file successfully converted !!!

-I-> geometries total[42716] changed[11808] ; points total[41010] changed z[41010] changed m[0]
Which took about 30 minutes to complete.

The results show: that of the 42716 geometries, 11808 were updated containing 41010 points - all of which were updated.

A sample LINESTRING that was updated:
SRID=3068;LINESTRING(24910.98653436587 21463.5093835775 35.39999999999999,24948.09927146823 21394.95263390163 35.60999999999999)
shows correct results.

A second run took about 2 minutes:
-I-> geometries total[42716] changed[0] ; points total[41010] changed z[0] changed m[0]
showing that no updates were made, since nothing has changed.

Although not yet tested, it should mean that if new points are added that may not contain reliable results
    a (more or less) swift update can be made from the Dem-Database.

Conclusion:
To create an SQL-Command for the same task is unrealistic, since it would be very time consuming.

So adding this as a new tool would be justified.

It could also be adapted with a -retrieve or -fetch command,
    together with a config-file where the Dem information is stored,
    could return the Z-value from a given srid,x,y position (or EWKT).

Considering the size of these Databases, a semi-automated use of them, would be a useful extra tool.


sandro added on 2019-05-15 08:39:19:
implemented in spatialite-tools-5.0.0 as spatialite_dem