Spatial operations - Large shapefiles and 32-bit/64-bit operations

Nov 14, 2011 at 2:15 PM

Presently, when dealing with spatial operations, Sergei has written into the MW library, the Clipper engine, which is much quicker than the GEOS. However, both fail when using very large shapefiles and performing back to back spatial operations e.g. A unioned with B, then the result unioned with C, then the result unioned with D etc. Memory is used and not being released, which appears to be occuring on the editing of the shapefile when it is loaded into memory. When large shapefiles are used, threading needs to be built in as well as possibly the selecting before hand only those shapes that are relevant to the operation being performed using a spatial extent search etc.

Large files can also have x number of shapes written away to a number of temporary shapefiles, which can be used to perform the spatial operations. This same principle can be used when creating the result, where if the number of shapes exceed x, then the result is separated into y number of parts until after the final operation, and then the whole is reconstructed at the end.