1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Featured TUTORIAL Race Track Builder - Inject elevation data and terrain texture

Discussion in 'Tracks' started by Wurstkoffer, Sep 17, 2017.

  1. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    1. About
    This workaround allows RTB to obtain the terrain texture and elevation data from another source, which ideally provides more detail than Google. This source can be anything in theory. A GeoTiff, a database or file containing data in XYZ format, another mapping service, such as the many regional WMS[1] servers. It can be used for both public and local services. And yes, it can also be a LIDAR based source. An example is provided which retrieves data from a regional public WMS server. Programming skills may be required to access other sources.



    2. Principle
    The interface that RTB uses to retrieve terrain texture and elevation data is the Google Maps API[2]. The use of the API is very well described[3]. RTB sends requests for locations to Google via HTTPS and receives replies of type PNG and JSON[4]. This is where the Inject takes place. RTB's requests to Google are redirected to a local service that generates replies based on an alternative source. The redirection is easily done in the hosts file[5]. Since RTB communicates via HTTPS, the local service must pretend to be Google. This is achieved via a self-signed SSL certificate for the address maps.google.com. The certificate must be imported into the Windows Certificate Store as a trusted root certificate authority.



    3. Instruction & Example
    In this example, a web server is used as the local service that mediates between RTB and the alternative data source. On this web server runs a PHP-based interface that behaves like the Google Maps API to RTB and retrieves the alternative data from a regional public WMS server.

    Since some editing is done in files, a good text editor is recommended. Of course the Windows Editor can be used, but this manual always refers to Notepad++.

    3.1. The Webserver
    • Download XAMPP for Windows
    • Install it:
      • The required components are Apache and PHP. All others can be deactivated.
      • The installation directory used in this example is C:\xampp. Make sure there is enough space on the drive. The example script allows you to save all map tiles to create a high-resolution terrain texture later on. This can generate 3-5GB of image files.
      • Further installation offers, such as Bitnami, are adware and should be rejected.
      • When the Windows Firewall logs on, allow access to all networks.
      • At installation completion, allow the Control Panel to start.
      • For faster access to it in the future, anchor the Control Panel in the Windows Start Bar.
    • Test it:
      • Click the start button of the Apache module and make sure it listens on ports 80 and 443.
        [​IMG]
      • If everything's ok, stop Apache again. If not, ask the web for help.

    3.2. SSL Certificate
    • Before the certificate can be generated with the batch file makecert.bat located in C:\xampp\apache\, the batch must be modified.
    • Rightclick makecert.bat and choose "Edit with Notepad++". Change the line "bin\openssl req -new -out server.csr" to "bin\openssl req -nodes -new -out server.csr". This prevents from stating a password for the certificate. Save it and close Notepadd++.
    • run makecert.bat by double click.
    • A black window appears and you will be asked to enter some information. The first five can be left blank (just hit return). The sixth asks for "Common Name (e. g. server FQDN or YOUR name)". Enter "maps.google.com" there (without quotation marks, of course). Proceed with blank inputs until "The certificate was provided" appears and complete.
      [​IMG]
    • Go to Internet Options (hit Windows Key, so the Startmenu comes up, type "internet options")
    • Switch to "Content" tab and click on "Certificates".
      [​IMG]
    • Switch to "Trusted Root Certification Authorities" tab and click "Import".
      [​IMG]
    • On import hit "browse" and choose server.crt from folder C:\xampp\apache\conf\ssl.crt\
    • Proceed to the end. A security warning will appear. Just hit "Yes" to install the certificate.
    • After successful import, "maps.google.com" should appear in the list of certificates.
      [​IMG]

    3.3. Hosts File
    • Run Notepad++ as Administrator
    • Open "hosts" file from folder C:\Windows\System32\drivers\etc
    • Add a line without a leading "#": "127.0.0.1 maps.google.com".
      [​IMG]
    • Save the file and exit.
    • From now on it is no longer possible to reach "https://maps.google.com/" in your web browser. If it becomes necessary to use Google Maps, alternatively use "https://maps.google.co.uk/" or temporarily deactivate the redirection in the Hosts file by inserting a "#" at the beginning of the previously added line and save the file.

    3.4. Test redirection
    • Start Apache Module in XAMPP Control Panel
    • Open your web browser and visit "http://maps.google.com/". The browser may refuse to call it via HTTPS, so just use HTTP.
      • If you still reach the original Google Maps Website try the following:
        • restart your brower
        • use another browser
        • flush dns in command prompt (hit windows key, type cmd, hit enter, type "ipconfig /flushdns").
          [​IMG]
      • If everything's working, a welcome page from XAMPP should be served.
        [​IMG]
    • Start RTB (if it's already running, close it and restart)
    • At "Create New or Open Venue" window click Google button. An error should appear: "Can't load Google Maps image. The remote server returned an error: (404) Not Found.". That's fine. It means the redirection works.
    • Let's take a look if the server received a request. Open "ssl_request.log" file at folder C:\xampp\apache\logs\. It should look like this:
      [​IMG]

    3.5. The PHP example scripts
    • Extract the content from RTB_WMS_Inject.zip to C:\xampp\htdocs\ so it looks like this:
      [​IMG]
    • Let's see if the scripts work as expected with default settings.
    • Open config.php from folder C:\xampp\htdocs\maps in Notepad++. Scroll down to bottom and add your Google API key by replacing the fake value at $key variable.
    • Visit "http://maps.google.com/maps" in your web browser. If you still got problems to reach it, try "http://localhost/maps". A Config Test site should load. If error messages appear, try to understand and solve them ;). If everything went well, you'll see 3 map tiles and 3 elevation responses from google, wms and a merged version from localhost:
      [​IMG]
      In the WMS pictures you will spot a white border in terrain. This appears, because the WMS server contains map tiles with bad alpha channel.

    3.6. Test fetching in RTB
    Since the example Config uses a regional public WMS server from Saxony, Germany, we should choose a location from this region for testing. It delivers laser scanned elevation data in 2m resolution.
    • So, switch to RTB (or restart), paste "50.842566,14.7440267" at the "Set Latidude/Longitude" input box and hit the Google button again.
    • Zoom in a bit.
      [​IMG]
    • Hit OK, and OK again to let RTB start the project and receive data from WMS.
    • While RTB is fetching, take a look at the log file at C:\xampp\htdocs\maps\logs. Every block starts with the request URL that was sent from RTB, followed by some details from Googles and WMS's responses for every single location from the request.
      [​IMG]
    • Back in RTB, when Terrain is loaded, start to sub divide a bit. But be warned. You have to be really patient. The WMS server delivers responses much slower as Google. A way to improve that drastically is to find download resources and use it with a local geoserver.
    • If you sub divide till 9 or 10, you will see some ugly level steps. Unfortunately, the WMS simply does not interpolate elevation between measuring points... or rather: a measuring point is a 2x2 meter pixel.
      [​IMG]
      There is nothing we can do about that at the moment, except manual flattening. You can certainly achieve much better results with higher resolution material, which may be available as XYZ points or vector data.


    4. Finding WMS Servers
    Finding other sources is a demanding task. There are a lot of map services out there but you have to find one that delivers elevation data via GetFeatureInfo. Search the web for following keywords: WMS, elevation, LIDAR, GIS, DEM/DSM/DTM and the region you're interested in. A good starting point for Europe is https://www.europeandataportal.eu. If you found a resource, take a look in config.php and enter the appropiate URLs, Layername and Reference System supported by the WMS you found. Everything is described in detail in config.php.



    5. Other kind of sources
    If you want to use some downloaded data like LIDAR point clouds you need to install and set up a local geoserver and load your data into it. I'm pretty sure that's possible, but I haven't done that myself yet. So you have to do some research. If it's loaded and running you just need to change some settings in config.php.



    6. Code using / improvements
    Do whatever you want with the provided PHP code. If you can enhance it further, do it! If you want to rewrite it in proper object oriented code, you're welcome! There are also some frameworks out there for using WMS with PHP. I'm sure there are much better ways to communicate with WMS, for instance evaluating the GetCapatabilities request. It would only be nice, if you share your improvements.



    7. Footnotes
    [1]
    "A Web Map Service (WMS) is a standard protocol for serving (over the Internet) georeferenced map images which a map server generates using data from a GIS [remark: Geo Information Service] database. [...] GetFeatureInfo - if a layer is marked as 'queryable' then you can request data about a coordinate of the map image." (for example elevation data) - https://en.wikipedia.org/wiki/Web_Map_Service

    [2]
    "[...] a service for retrieving static map images, [...] and obtaining elevation profiles." - https://en.wikipedia.org/wiki/Google_Maps#Google_Maps_API

    [3]

    [4]
    When creating the terrain texture, RTB requests multiple map tiles via Google Static Map. It sends coordinates via Center Lat/Lon, in conjunction with zoom level, format, width & height of the tiles and the API key in this format:
    Code:
    https://maps.google.com/maps/api/staticmap?center=50.8355467,14.7050059&zoom=18&size=512x512&scale=2&format=png&maptype=satellite&sensor=false&key=
    A tile will then be received:
    [​IMG]
    A similar request is made to receive elevation data. Here, however, several locations are transmitted, separated by pipe:
    Code:
    https://maps.google.com/maps/api/elevation/json?locations=50.8457672356934,14.7382761104377|50.8477157928223,14.7382759988057|50.8477157928223,14.7409487&key=
    The response comes in JSON format:
    Code:
    {
       "results" : [
          {
             "elevation" : 450.2297668457031,
             "location" : {
                "lat" : 50.8457672356934,
                "lng" : 14.7382761104377
             },
             "resolution" : 152.7032318115234
          },
          {
             "elevation" : 494.1800537109375,
             "location" : {
                "lat" : 50.8477157928223,
                "lng" : 14.7382759988057
             },
             "resolution" : 152.7032318115234
          },
          {
             "elevation" : 485.4239807128906,
             "location" : {
                "lat" : 50.8477157928223,
                "lng" : 14.7409487
             },
             "resolution" : 152.7032318115234
          }
       ],
       "status" : "OK"
    }
    These requests could be confirmed with the help of mitmproxy.

    [5]
    "The computer file hosts is an operating system file that maps hostnames to IP addresses." - https://en.wikipedia.org/wiki/Hosts_(file)
     

    Attached Files:

    Last edited: Sep 17, 2017
  2. luchian

    luchian Administrator ACM Racing Team ACM Club Driver

    Joined:
    Jun 3, 2014
    Likes Received:
    560
    You're a bloody genius ! Not only for implementation but for actually thinking of this trick.
    I don't even own RTB, but I might just buy it to try your little sorcery. Thank you for sharing, I find this inspirational.
    Love the presentation also. Programming style ftw ! :)

    Sent from my phone using Tapatalk
     
  3. Capzor

    Capzor New Member

    Joined:
    Sep 17, 2017
    Likes Received:
    0
    Really cool!

    Thanks for sharing :)

    Any hints on how to use hoydedata.no with this?

    I tried GeoServer and am utterly lost :banghead:
     
  4. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    That's a nice mapping service. Unfortunately i can't find any hint if it's WMS. I think it's not. At WMS servers you always find descriptions on how to access the data.

    It seems you can download point clouds there. But, yeah, combining them with geoserver seems to be not a simple task. :\
     
  5. Capzor

    Capzor New Member

    Joined:
    Sep 17, 2017
    Likes Received:
    0
  6. Pixelchaser

    Pixelchaser Well-Known Member ACM Club Driver

    Joined:
    Jan 4, 2015
    Likes Received:
    796
    :lol:. its all way over my head too. I was to shy to say earlier. haha, I do want to see it working though.
     
  7. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    Oh nice, it is a WMS server! But the link contains just some of its data. There has to be some more to find.

    Referring to "over my head", i feel you bro's. The whole topic about mapping, spatial reference systems, WMS servers 'n shit was completely new to me too. I researched and did a lot of try & error for weeks. :>

    What helped me a lot was installing a local WMS Viewer (QGis). If you find a GetCapabilities URL, load it into QGis. This way you can see, what's contained and the layer name.

    Plugin Tipps:
    • OpenLayers - it loads Google or Bing maps - you'll need it as a background/base map
    • ValueTool - displays values at current mouse position - makes use of GetFeatureInfo, which can deliver height data, and many more stuff
     
    luchian likes this.
  8. linamza

    linamza New Member

    Joined:
    Oct 9, 2017
    Likes Received:
    0
    Great work man!!

    I check about the QGis (WMS Viewer), and i use the Value Tool, as you mention using the mouse, but i didn't get any information on the Value Tools, what i'm doing wrong?
     
  9. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    Use either Current Layer Mode and select one layer ...
    QGis Identify Feature #2.png

    or Layer selection mode and select multiple layers
    QGis Identify Feature #3.png

    You have to find a layer which provides values via GetFeatureInfo.
    QGis Identify Feature #1.png
     
    luchian likes this.
  10. paolosu83

    paolosu83 New Member

    Joined:
    Oct 30, 2017
    Likes Received:
    0
    Hi... i foolow your guide step by step. I find a lot of DEM data of my island (1 mt ) but i don't know what i have to paste on config php.... Take a look here please
    Here there is all the data http://www.sardegnageoportale.it/webgis2/sardegnamappe/?map=download_raster
    I find the ASC file to import with the elevation data (500120_12_WGS.ASC). I import it on qgis but i cannot find the info's that i need to compile the config.php

    I'm glad to pay a drink to you if you can help me to compile elevation layer and orto
     
    Last edited: Oct 30, 2017
  11. paolosu83

    paolosu83 New Member

    Joined:
    Oct 30, 2017
    Likes Received:
    0
    Last edited: Oct 30, 2017
  12. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    Hmm can't find it either. In my example the WMS server delivers the elevation layer as RGB png tiles plus additional info via GetFeatureInfo in html/txt format, which the script uses to retrieve elevation value at a certain point.

    There is a similar layer at that Sardegna WMS server called Altimetria (10m), but it doesn't deliver additional infos and i can't find any metadata or info on how to access it. :(

    These 1m DTM/DSM layers doesn't cover the whole island and i don't now yet how to access it's data.
     
    luchian likes this.
  13. paolosu83

    paolosu83 New Member

    Joined:
    Oct 30, 2017
    Likes Received:
    0
    i saw. No elevation on wms... There is only the map for downloading the asc files with elevation data at 1 mt for all the coast zone.. THe only way i think is to create a local wms server with geoserver and load the asc file as layer with the elevation data, but i don't know if i able to do this. Any suggestion is appreciate!!
     
  14. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    If there is no way to access the elevation data through their WMS server, downloading ASC and setup a local geoserver might be a possibility. It's no simple task, though. And the PHP script has to be adapted to receive elevation data via this other kind of layer.
     
    luchian likes this.
  15. Les Neilson

    Les Neilson Member

    Joined:
    Sep 21, 2016
    Likes Received:
    23
    I have a couple of questions:
    If I download the data and therefore need to set up a geoserver to use it, do I still need xampp as well?
    How do I go about resetting things back to normal if I have problems setting things up?
     
  16. Wurstkoffer

    Wurstkoffer Member

    Joined:
    Sep 6, 2017
    Likes Received:
    27
    Yeah of course you still need xampp. Don't know what you mean with resetting things to normal. The only mayor change you have to make for the work-around is the redirection in hosts file. So resetting that thing to normal is either delete the redirection line or deactivate it by adding a "#" at front of the line.
     
    luchian likes this.
  17. Les Neilson

    Les Neilson Member

    Joined:
    Sep 21, 2016
    Likes Received:
    23
    Okay, thanks, that's as I thought then. When I have time I'll give this a try... Great idea!
     
  18. Les Neilson

    Les Neilson Member

    Joined:
    Sep 21, 2016
    Likes Received:
    23
    Ummm, where do I find RTB_WMS_Inject.zip?

    Thanks!
     
  19. Les Neilson

    Les Neilson Member

    Joined:
    Sep 21, 2016
    Likes Received:
    23
  20. Les Neilson

    Les Neilson Member

    Joined:
    Sep 21, 2016
    Likes Received:
    23
    I'm at step 3.6 but it seems RTB isn't getting any elevation data. I've tried opening localhost/maps to see what happens there, and it seems to be receiving a valid number. In the log file I'm getting the following:

    21:34:54 lat: 50.8355467 Elevation Google: https://maps.googleapis.com/maps/api/elevation/json?locations=50.8355467,14.7050059&key=MYAPIKEY
    21:34:54 lng: 14.7050059 Elevation WMS: 517.59 https://geodienste.sachsen.de/wms_g...text/plain&QUERY_LAYERS=Gelaendehoehe&I=1&J=1

    21:35:05 https://maps.googleapis.com/maps/ap....8815371425781,14.6263425054718&key= MYAPIKEY
    21:35:05 --------------------------------------------------------------------------------------------------------------------------------------------
    21:35:08 Error: on: json_obj[results][0][elevation]
    21:35:23



    For every successful request, I get a hundred zeros... Is this what you meant when you said the response would be slow?