Featured TUTORIAL LIDAR (Point cloud) to mesh tutorial

Discussion in 'Tracks' started by DanTDBV, Jan 8, 2016.

  1. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Updated 31/1-2017. Comments about experience's with exported mesh added. Tutorial is finished.

    This is to keep others from having as bumpy, a ride into this as i have had. Many frustrations.

    Finding the right way into LIDAR point clouds, is like navigating a dense jungle.

    There are so many purposes and ways to use LIDAR data, that we have acces to a forest full of free and paid for software, plus lots of file formats. Ad to that, all the things you can do to those files.

    Created with CloudCompare 2.6.2 and Blender 2.76b

    A heartfelt thankyou to Daniel Girardeau-Montaut, the creator of Cloudcompare. For creating the software and the pointers, that made this tutorial better.

    What is LIDAR?
    Lidar
    (also written LIDAR, LiDAR or LADAR) is a remote sensing technology that measures distance by illuminating a target with a laser and analyzing the reflected light. Although thought by some to be an acronym of Light Detection And Ranging, the term lidar was actually created as a portmanteau of "light" and "radar". Lidar is popularly used as a technology to make high-resolution maps, with applications in geodesy, geomatics,archaeology, geography, geology, geomorphology, seismology, forestry, remote sensing, atmospheric physics, airborne laser swath mapping (ALSM), laser altimetry, and contour mapping. (From Wikipedia).

    Getting LIDAR data.
    This is somewhat beyond the scope off this tutorial, but searching for geodata or height model, together with the country, should be a good starting point.

    Software needed
    CloudCompare
    It is Open Source and can be downloaded here:
    http://www.danielgm.net/cc/

    CloudCompare supports import of 27 different file formats. There is a pretty fair chance, it will eat your file.

    Before we start, a copyright notice needed for the data used.
    Copyright notice Bellahøj Park Point cloud and Ortho photo
    As required, according to this site: (In Danish)
    http://download.kortforsyningen.dk/content/vilk%C3%A5r-og-betingelser
    Contains data from Geodatastyrelsen and Danske kommuner Fot-data
    Ekstracted lokation : Bellahøj Park Copenhagen
    DHM/Point cloud
    DLD 30/10-2015 Punktsky_617_72_LAZ_UTM32-ETRS89
    Ekstracted lokation : Nisseringen Næstved
    DHM/Point cloud
    DLD 14/11-2015 Punktsky_612_68_LAZ_UTM32-ETRS89
    http://download.kortforsyningen.dk/content/geodataprodukter
    DLD = Download date
    DanTDBV D@nN (My photo and film signature)

    My experience's with exported mesh.

    I have found that Blender, does not like the mesh, that comes out from CloudCompare.

    I am now using MeshMixer to reduce the mesh. Erase & Fill, Reduce and Smooth are the tools I use.

    Retopoflow and the Knive tool, are not working as expected. The creator of Retopoflow, has looked at it and he says, that it is behaving strangely.

    Upon import in Blender, I am now moving the Origin, to somewhere inside the model. It does give a better behavier, but not much. I have decided, that I am capable of lining up parts manually. When using obj, it needs a 90 degree flip on the X axis.

    Conclusion:
    If stl or obj is the right choise, I really don't know, but Retopoflow is behaving a little better with obj.

    Current way I do it.
    Apply Global shift/scale upon import
    Export full area, reduce in MeshMixer, that does not worry about reference/anchor/pivot point. Import in Blender, move origin to model, center it, then cut it to pieces.

    On to the regular show

    How do you do, you do, you do


    I use LAZ files in this case. LAZ is a compressed LAS file.
    • Open CloudCompare
    • Open the file you wish to use. Select multiple files with Shift and Control
    • Open LAS File Chose Apply all
    • Global shift/scale Yes to All
    • Navigation: Left mousebutton Rotate, Scroll wheel Zoom, Right mouse button Move
    • You always need to select the file/s You want to work with.
      Select each file in turn (In DB Tree chose the filename)
    • Change active Scalar Fields to Classification (This is the one I prefer to use)
    2 fields have been changed here
    CC Classification.jpg

    Merge and select

    I usually do a fast selection, of a wider area than needed and save that, before refining the area further.
    • Control select all files (Same as above)
    CC Selection.jpg

    • Click Merge multiple clouds button in Top ToolBar (TTB)
    CC Top Toolbar.jpg
    • Adjust the view, until the desired area is visible
    • Click Segment button in TTB (scissors)
    • (Opens Segment toolbar. Hover mouse over tools for tool tips)
    Segment selection.jpg
    • Select area of cloud to keep. For an irregular area, use Polygonal selection
    • If you are satisfied Hit Confirm segmentation. If you are not hit Clear segmentation. Cancel to quit
      In DB Tree chose the parts, of the cloud you wish to discard. (I clear the checkmarks, from the parts I want to keep) Hit Delete
    • Select resulting file and save a backup. I have good experience with exporting as LAZ. I chose Highest resolution. LAS/LAZ comparison: LAS 287 MB, LAZ 40,6 MB.
    Further area refining.

    CC pre clean.jpg
    • Now use the Segment tool, to remove the areas not visible from the track.
    • Polygon selection makes it possible to do complex selections.
    Polygon selection.jpg
    After Segment out selection
    CC Segment out selection.jpg

    CC cleaned.jpg
    • Save another backup

    Go and have a break. Do what you do to relax and refresh.


    Welcome back. Are you ready?

    Reference/anchor/pivot point.

    Remember all the big numbers in the Global shift/scale dialog?

    The numbers in the big box, is the placement of the reference/anchor/pivot point, in the file you open.

    The numbers in the small box, is the placement of the reference/anchor/pivot point, in the file while you work.

    The Global shift information is discarded, when exporting to .stl, so the reference/anchor/pivot point, coordinates will be the numbers from the little box.

    Abandonned earlier conclusion.
    For landscapes I prefer to have the pivot point, in the center of the model. It makes it easier to cut the model into managable pieces and reassemble them.

    Moving reference/anchor/pivot point.
    • Select the file
    • Chose Edit
    CC Edit menu.jpg
    • Chose Edit global shift and scale
    • Change X, Y, Z values to zero. Keep the scale.
    • Hit yes
    • Click Point picking in TTB
    • Click on the point in the cloud, where you want the center to be.
    • Write down X, Y, Z values. Click discard
    • Select the file
    • Chose Edit
    • Chose Apply transformation
    • Go to Axis, Angle pane
    • Enter your written values, in the Translation section. You may need to check Apply inverse transformation.
    • Hit Ok
    Export test mesh
    • Select the file
    • Chose Edit
    • Chose Mesh
    • Chose Delaunay 2.5D (XY plane) (Until now I have left Max egde length at default)
    • Select the file
    • Save as Binary .stl file.
    Open the exported .stl in Blender and check, if it is where you want it.

    When you have it right, do another backup.

    Very important!
    Whenever you reload your centered cloud, do NOT! Shift the global scale!


    • Your choise should be:
    • Global shift/scale No
    Chosing the level of the cloud, to convert to mesh
    • Select the file
    • In the DB Tree Properties SF display params window, You can play around with the 2 circles until you only have the ground. (With the files I have, 2 is the ground)
    CC Scalar Fields parameters 2.jpg
    • I find it easier to uncheck, show NaN/out of range values in grey, on the parameters pane.
    • Chose Filter points by value in TTB
    • Change both fields to your SF display params ground value
    • Hit Ok
    • Select Mesh and save. (I usually ad the levels to the name)
    Creating a full resolution Mesh (This works for me)
    • Select the file
    • Chose Edit
    CC Edit menu.jpg
    • Chose Mesh
    • Chose Delaunay 2.5D (XY plane) (Until now I have left Max egde length at default)
    • Select the file
    • Save as Binary .stl file.
    PC level 2 full resolution .jpg

    Nisseringen mesh example from cloud full density.jpg

    Abandonned earlier conclusion.
    I found that in cleaning a full res mesh in Meshmixer, I was reducing the areas I worked with about 70-90%. It is much easier to reduce the cloud, before creating the mesh.
    This however have the disadvantage, that the terrain details is washed out. (Not as sharp as with the full resolution cloud).
    I have decided to keep the full poly mesh and creating a low poly mesh over that.
    It does however slows down the PC.
    My solution is to cut the full poly mesh, into chunks that I work on seperatly. This is way more easier in than in Blender. At least from what I know. More about this below.

    How I do it now
    I export the full res mesh. Using Meshmixer's Erase & Fill, Reduce and Smooth tools as approriate.

    Creating a reduced resolution Mesh
    • Select the file
    • Chose Edit
    • Chose Subsample
    • Move the slider from small towards large, remember your setting, hit ok
    • You can delete the resulting layer, without touching the base layer.
    • Play around with this until you, are sattisfied with the cloud density.
    • Select the file
    • Chose Mesh
    • Chose Delaunay 2.5D (XY plane) (Until now I have left Max egde length at default)
    • Select the file
    • Save as Binary .stl file.
    PC level 2 reduced resolution .jpg

    Nisseringen mesh example from cloud reduced density.jpg

    Cutting a mesh into chunks

    Delete everything in CloudCompare. Or as I do, close and reopen CloudCompare. (You did remember to save your progress?)

    Open the .stl file you created above.
    • Select the file
    • Now use the Segment tool, to cut the mesh into managable chunks
    • Keep only the parts you are working with visible
    • If you have cut something, that should not have been, use the merge tool
    • Save each part as you go
    • Always uncheck the parts, you have exported
    When you import the chunks, they are in the right places.

    I have used the resulting files in Meshlab, Meshmixer and Blender.

    I would advise to lock the Location:, Rotation: and Scale: settings in Blender.

    You are should now be ready, to continue in your favorite 3D program.

    Some Blender information

    Moving view close to the 3D cursor
    • Hit Alt/Home
    Moving reference/anchor/pivot point in Blender.

    I have kept this as it is usefull for other offcenter models.

    When importing the mesh in Blender, you might find that the placement is off, to way off.
    Chosing Yes to All for Global shift/scale, have seriosly reduced, the placement being off problem.

    The following example, was with No to Global shift/scale.

    This is not happening in Meshlab or Meshmixer, only in Blender.

    One example is the model in the picture. Blenderunits from scene center:
    Transform
    Location
    X=684203.25000
    Y=6123295.5000
    Z=55.56239

    Nisseringen Full Pivot.jpg

    There are 6 cars in this picture, can you spot them?

    The reason for this, is the Anchor/Pivot point of the model.

    The Location coordinates tells, where in the world the Anchor/Pivot point is.

    Move the pivot point into the model

    The first step, is to get the 3D cursor, out to the model.
    I had some trouble doing this. The 3D cursor would just not appear, when clicking close to the mesh. With the view back, on the center of the world, the cursor worked as advertised.

    This is how I got it to work:
    • Make sure that the model is selected
    • Chose View Selected
    • Change to Edit mode
    • Hit A to deselect all
    • Select a Vertex
    • Change to Object mode
    Now you should get the 3D cursor, when you click. Well at least I do.
    • Place the 3D cursor, where you want the Pivot point to be.
    • Chose Object/Transform/Origin to 3D Cursor.
    Your model is no longer off the mark.

    A fast way to move the model, is to change the Transform/Location values.
    Transform
    Location
    X:684203.25000
    Y:6123295.5000
    Z:55.56239

    Center of the world
    X:0
    Y:0
    Z:0

    To the best of my knowledge, I consider this tutorial finished.

    Thank you for your patience. ;-)

    Good luck.

    If you have any questions, then do not hesitate to ask and I will do my best to answer.

    Keep in mind that English, is not my first language.

    Update history
    Updated 31/1-2017. Comments about experience's with exported mesh added. Tutorial is finished.
    27/1-2016. Software versions added. Tutorial is finished.
    24/1-2016. Important note added to, Move reference/anchor/pivot point section.
    23/1-2016. Move reference/anchor/pivot point + splitting mesh in CloudCompare sections added.
    18/1-2016. Found a quicker way to do it.
     
    Last edited: Jan 31, 2017
  2. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Update:
    I have changed the last picture to the correct one.
     
  3. luchian

    luchian Administrator Staff Member

    Joined:
    Jun 3, 2014
    Messages:
    3,007
    Likes Received:
    1,491
    WOW @DanTDBV , this is gold !

    Thank you very much for sharing this - explanations are to the point and I am sure it will be useful to whomever is serious about track modding. Cheers !
     
  4. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    A lot of information here.

    looks like you take your time to explain it properly, I will read it until I understand everything... thanks a lot for share this value information.

    cheers.
     
  5. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Update:

    Added some Blender import help information.
     
    liquido likes this.
  6. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    Hi Dan, I'm from Spain and I'm tryning to learn your method... sadly I can't found LIDAR data in LAS format, your link is in your native lenguaje I supose... I need to use google translate to understand something inside but it dosen't make a good work and I get lost.

    I can found LIDAR data in other formats... ASC for example, in this web (assetto corsa mods) there is a blog article http://assettocorsamods.net/threads/lidar-data-available-for-free-for-england.378/ with links to english elevation data.... I get data from Outlon Park Circuit by this link http://environment.data.gov.uk/ds/survey#/download

    Did you ever load one of those files in the sofware cloud compare? couse I get error when I try to load those files.

    [​IMG]

    [​IMG]

    you can read line corrupted in the console... the software seems to load that kind of LIDAR data, There is data from others country in the link your share?

    I have LIDAR data from others places and cloud compare dosen't open any of them without some error.

    cheers.


    EDIT:

    http://rapidlasso.com/2016/01/10/create-proper-las-1-4-files-with-lastools-for-free/

    I just find that software... looks like you can create you onw LAS files from other sources... I will try to change the format with that.... I will tell you how it goes.
     
    Last edited: Jan 14, 2016
  7. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Hi liquido.

    I am doing a writeup on the Danish LIDAR data, so that will come at a later time.

    Yes, sometimes Google translate sucks.

    I generally only translate 1 or 2 sentences at a time.

    I would keep the translator window open, next to the original language window.


    Regarding the English data

    After several failed download attempts, I finally got a dataset downloaded.

    I have tried to open the files in the set, without any real succes.

    From what I have been able to, find on the net, the red boxes in the opening dialog, is because something CloudCompare does not like.
    I have tried to delete those lines, in Notepad+ and now the files do load, but I am not getting anything usefull from them.
    Apart from the fact, that you need to do something, I have no idea what to do with the sets.


    My suggestion would be, to go to the CloudCompare forum questions section and ask there.

    http://www.danielgm.net/cc/forum/viewforum.php?f=9&sid=d989168378b606fa93ad5d3396df7ed1

    Remember to tell, what you are attempting to do and what you have done so far.

    I would tell where I got the data from.

    You are welcome to quote me, when you ask.

    When I am this far, from having a clue, I would ask this:
    Is there anything more, that you need from me, to be able to help?


    I am sorry that I have not, been able to do more, but I do hope that, it is a step in the right direction.

    Good luck
     
    liquido likes this.
  8. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    I really apreciate your time and your tips, thanks a lot to try open ASC... I'm searching for a software to make the conversion, I hope I found something that I know how to use, is matter of time... I hope.

    I also have Notepad+ (great to open any kind of file and look inside) but seems to me I don't have the knowlegde to remove parts from archives and make them works.

    I'm a new member of the cloud compare forum, thanks for the advice.

    I will tell you any news. Thanks a lot :)
     
  9. Mr Whippy

    Mr Whippy Active Member

    Joined:
    Jan 18, 2016
    Messages:
    199
    Likes Received:
    142
    Liquido, what program are you wanting to get the meshes into?

    In 3DS Max for example you can just write a simple script to achieve the building of the mesh from the ASCII formatted ASC files.
    The data is simply a grid field the size as specified in the header, with Z axis data listed in the rest of the file.


    It's a shame that there isn't raw point cloud data available for the UK LiDAR, I'm still downloading some data to check out.


    But to clarify, the data you're accessing is *already* a mesh. It's in ASCII format.

    If you find actual point cloud data (arbitrary XYZ point data + colours) let me know :D



    Also great this data has finally arrived for 'free'. A few years back you had to sign up and sign papers etc to get access to this, great it's now open for free use like the OS open data sets are!

    Cheers

    Dave
     
    Last edited: Jan 18, 2016
    liquido likes this.
  10. Mr Whippy

    Mr Whippy Active Member

    Joined:
    Jan 18, 2016
    Messages:
    199
    Likes Received:
    142
    OK more checking the data types.

    There is some LiDAR tiles, and LiDAR composite data.

    Tiles seems to be mm scaled (Z axis) with more rows and columns but the same 'cellsize', so I can only assume the grids are twice as big as the 'composite' data, which use metre scaling (Z axis), but the mm scale is no higher resolution than the metres scaled data.


    I've got this feeling the environment gov uk data isn't available as a point cloud, just grid data as either DTM or DSM.



    Either way, even the 1m DSM are massive.

    1 million polygons per 1km x 1km block!

    The 0.25m DSM will be 4 million polygons. Ooodles of data :D

    Dave
     
  11. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Tutorial has been updated. A major rewrite.
     
  12. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Hi Dave.

    Or any of the rest of you.

    Do you have any idea, how to get something usefull out the British data, with freeware/opensource software.

    3DS Max and the rest of the proffessional software, is way beyond my payscale.
     
  13. Mr Whippy

    Mr Whippy Active Member

    Joined:
    Jan 18, 2016
    Messages:
    199
    Likes Received:
    142
    Well, the files are just ASCII, with the XY data inferred from the coordinates/metadata at the top of the ASCII files.

    So you could in theory use some moderately powerful scripting language to turn the meta data about XY, and the long list of Z data, into vert/face data.



    In Max what I'm doing now is generating a plane the correct size, then loading in the Z data, then just appending the plane's Z data for each vert to the Z data from the ASCII file.

    That works great for the 50m DTM at 200 x 200 size (10km x 10km), but Max is just dying on me trying to do the 1000 x 1000 size (25 times more points!)

    At this point I'm not sure why (I'm moderately good in Maxscript, but not super-programmer level) it's not happy, but I'm sure I'll figure it out.


    I just tried 40 x 1000 and it worked fine. Now trying 500 x 1000 to see if it completes :)

    I may end up just doing it in 10 blocks of 100 x 1000 then combining at the end.


    As soon as I get some decent looking meshes put together I'll put some pics up :D

    Dave
     
    liquido likes this.
  14. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    I'm working on Blender, honestly... I have a 3dmax version to install, but I try it two times in the past and I belive that software is too much for me, I should go to learn in academy/school... tutorials from youtube needs free time, maybe someday I finally go to learn 3dmax in academy.

    As you said Dave, I can't find point cloud elevation data from anywhere.... all "LIDAR" data seems to be grids in diferents formats... thanks for the clarification about asc data = mesh.

    I found elevation data from USA on that link,

    http://eros.usgs.gov/elevation-products

    there is LIDAR data but never try to work with... I know how to use ASC format and SRTM1/2

    LIDAR data is a new world for me, and geo spatial vocabulary and concepts is also not my playground, but really interested in learn how to.

    Until now I follow one method to build scketchs with BTB (Bob's Track Builder) and diferents elevation data, but I'm open to learn new methods to work with new formats of elevation data.
     
  15. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    To get LIDAR data from USGS you need to use the earth explorer http://earthexplorer.usgs.gov/

    [​IMG]



    EDIT:

    I didn't remember... you need to have an account in USGS to donwload data... it is free, I have one... only need to registrate and provide an email.
     
    Last edited: Jan 18, 2016
  16. liquido

    liquido Active Member

    Joined:
    Apr 7, 2015
    Messages:
    166
    Likes Received:
    205
    Location:
    Málaga, Spain
    Yes mate!!!!!

    finally I was able to export a mesh from the LIDAR data, thanks to your tips... We acctually have a surface that could be used for physic layer... amazing detail, We are thinking about take parts of that info to replicate this surface in our future projects.

    [​IMG]

    My new machine is powerfull but she don't have enough memory ram to work with the full scene, so I decided to delete all the points wich aren't on surface road and then join all in one and start to simplify with subsample tool.

    Once I have a 2gb point cloud file to work, my computer was able to finish the subsample process, after play a little with that, we allready have a 15Mb point cloud file that you can load on almost every computer.

    Thanks you sir.
     
    Pixelchaser likes this.
  17. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Updated 23/1-2016. Move reference/anchor/pivot point + splitting mesh sections added.
     
  18. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Wonderfull liquido!
    That is very great to hear.

    My approach is this:
    • Trim and merge the cloud/s, to the area that i need.
    • Move the reference/anchor/pivot point to the center of the model.
    • Create and save an .stl mesh, from the full resolution cloud.
    • Cut the .stl mesh into managable pieces.
    • You can do this in CloudCompare and I have just updated the tutorial, with how to do it information.
    • Now you can import the chunks into blender and work with them seperatly.
     
    liquido likes this.
  19. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    24/1-2016. Important note added to, Move reference/anchor/pivot point section.
     
    liquido likes this.
  20. DanTDBV

    DanTDBV Member

    Joined:
    Jan 1, 2016
    Messages:
    84
    Likes Received:
    80
    Update 27/1-2016. Software versions added. Tutorial is now finished.
     
    luchian likes this.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice