Integrating Deep Learning with GIS
The field of Artificial Intelligence has made rapid progress in recent years, matching or in some cases, even surpassing human accuracy at tasks such as computer vision, natural language processing and machine translation. The intersection of artificial intelligence (AI) and GIS is creating massive opportunities that weren’t possible before. AI, machine learning and deep learning are helping us make a better world by helping increase crop yield through precision agriculture, to fighting crime by deploying predictive policing models, to predicting when the next big storm will hit and being better equipped to handle it.
Broadly speaking, AI is the ability of computers to perform a task that typically requires some level of human intelligence. Machine learning is one type of engine that makes this possible. It uses data driven algorithms to learn from data to give you the answers that you need. One type of machine learning that has emerged in recent years is deep learning and it refers to deep neural networks, that are inspired from and loosely resemble the human brain.
Machine Learning in ArcGIS
Machine learning has been a core component of spatial analysis in GIS. These tools and algorithms have been applied to geoprocessing tools to solve problems in three broad categories. With classification you can use support vector machine algorithms to create land cover classification layers. Another example is clustering that lets you process large quantities of input point data, identify the meaningful clusters within them, and separate them from the sparse noise. Prediction algorithms such as geographically weighted regression allows you to use geography to calibrate the factors that help you predict. These methods work well in several areas and their results are interpretable, but they need experts to identify or feed in those factors (or features) that affect the outcome that we’re trying to predict.
The rise of Deep Learning
Wouldn’t it be great if the machine figured out what those factors/features should be just by looking at the data? That’s where deep learning comes in. It’s inspired by and loosely resembles the human brain. In a deep neural network, there are neurons that respond to stimulus and they are connected with each other in layers. Neural networks have been around for decades but it has been a challenge to train them.
Integrating Deep Learning with ArcGIS — Webinar
So what’s changed now, and made them so successful at what they do? The advent of deep learning can be attributed to three primary developments in recent years — Data, Compute and Algorithmic improvements:
Data: We now have vast quantities of data thanks to the Internet, the sensors all around us and the numerous satellites that are imaging the whole world, every day.
Compute: We have powerful compute, thanks to cloud computing and GPUs that have become more powerful than ever, and gone down in price thanks to the gaming industry.
Algorithmic improvements: Finally, researchers have now cracked some of the most challenging aspects of training the deep neural networks through algorithmic improvements and network architectures.
Applying Computer Vision to geospatial analysis
One area of AI where deep learning has done exceedingly well is computer vision, or the ability for computers to see. This is particularly useful for GIS, as satellite, aerial and drone imagery is being produced at a rate that makes it impossible to analyse and derive insight from through traditional means.
The figure below shows some of the most important computer vision tasks or use cases and how they can be applied to GIS:
Fig 1. Important Computer Vision tasks applied to GIS
The simplest is Image Classification, in which the computer assigns a label, such as ‘cat’ or ‘dog’ to an image. This can be used in GIS to categorize geotagged photos. In the example above, we have an image that’s been classified as having a ‘dense crowd’ and such pedestrian activity classification can be used for pedestrian and traffic management planning during public events. An example of this was demonstrated at the Esri User Conference 2018 plenary session by Cobb County.
Traffic and pedestrian movement planning by Cobb County
Next, we have Object Detection in which the computer needs to find objects within an image as well as their location . This is a very important task in GIS — finding what is in satellite, aerial or drone imagery, and where, and plotting it on a map can be used for infrastructure mapping, anomaly detection and feature extraction.
Detected swimming pools within residential parcels
Another important task in computer vision is Semantic Segmentation — in which we classify each pixel of an image as belonging to a particular class. For instance, in the image in Fig 1, the cat is in the yellow pixels, and the green pixels belong to the ground class and the sky is in blue. In GIS, semantic segmentation can be used for Land Cover Classification or to extract road networks from satellite imagery.
Land cover classification using deep learning
A nice early example of this work and its impact is the success the Chesapeake Conservancy has had in combining Esri’s GIS technology with Microsoft’s AI toolkit (CNTK) and cloud solutions to produce the first high-resolution land cover map of the Chesapeake watershed. This work is now also available as a tutorial that can be deployed on an Azure Geo AI Data Science virtual machine.
Another type of segmentation is Instance Segmentation, that you can think of as a more precise object detection in which the precise boundary of each object instance is marked out. This can be used for tasks like improving basemaps by adding building footprints or reconstructing 3D buildings from LiDAR data.
Building reconstructed in 3D using aerial LiDAR. The same building reconstructed in 3D from the masks digitized by human editors (left), and semantic segmentation masks produced by the Mask R-CNN (right)
Esri recently collaborated with NVIDIA to use deep learning and automate the manually-intensive process of creating complex 3D building models from aerial LiDAR data for Miami-Dade county. This task used this data to create segmentation masks for roof segments that were then used for 3D reconstruction of the buildings. See this detailed blog post on Medium on how this was done, or the companion post at ArcGIS Blogs.
Deep learning for mapping
An important application of deep learning for satellite imagery is to create digital maps by automatically extracting road networks and building footprints.
Imagine applying a trained deep learning model on a large geographic area and arriving at a map containing all the roads in the region, as well as the ability to create driving directions using this detected road network. This can be particularly useful for developing countries that do not have high quality digital maps or in areas where newer development have taken place.
Roads detected using deep learning and converted to geographical features
Good maps need more than just roads though — they need buildings. Recent developments of instance segmentation models like Mask-RCNN are particularly useful for building footprint segmentation, and can help create building footprints without any need of manual digitizing. However, these networks result in masks that look more like Antonio Gaudi masterpieces than regular buildings with straight edges and right angles. ArcGIS includes tools such as Regularize Building Footprints just for this.
Building footprints extracted out of satellite imagery and regularized using the Regularize Building Footprintstool in ArcGIS Pro.
Integrating ArcGIS with AI
ArcGIS has tools to help with every step of the data science workflow: from data preparation and exploratory data analysis, to training the model and to performing spatial analysis and finally disseminating results using web layers and maps and driving field activity. The Living Atlas provides access to a large collection of Esri curated and partner provided imagery that can be critical to a deep learning workflow.
Export Training Data for Deep Learning tool in ArcGIS Pro
ArcGIS Pro includes tools for helping with data preparation for deep learning workflows and has being enhanced for deploying trained models for feature extraction or classification. The ArcGIS Image Server in the ArcGIS Enterprise 10.7 release has similar capabilities and allow deploying deep learning models at scale by leveraging distributed computing. ArcGIS includes built in Python raster functions for object detection and classification workflows using CNTK, Keras, PyTorch, fast.ai and TensorFlow. Additionally, you can write your own python raster function that use your deep learning library of choice or specific deep learning model/architecture. See this handy guide to get started!
Deep learning is a rapidly evolving field and using a code first approach allows data scientists to leverage cutting edge research while taking advantage of an industrial strength GIS. Python has emerged as the lingua franca of the deep learning world with popular libraries like Tensorflow, PyTorch and CNTK choosing it is their primary programming language. ArcGIS API for Python andArcPy are a natural fit for integrating with these deep learning libraries and allow your capabilities to expand.
The Road Ahead
The field of deep learning is nothing short of fascinating. This is a rapidly evolving field with state of the art results and new research papers coming out every week or month.
An example of superresolution — increasing the resolution of satellite imagery. Left — lower resolution, Right — increased resolution using deep learning
Some innovative uses of deep learning are for enhancing imagery such as by increasing zoom levels through the use of ‘Superresolution networks’. This technique can be used to increase clarity of satellite images and even go beyond the resolution of the sensors used.
Map art generated by applying the style of a painting to the satellite imagery of Eye of Sahara, a geographical feature in northern Africa.
Another innovative use of deep learning is in the field of ‘Creative AI’. Neural style transfer techniques can be used for generating ‘map art’ and can find practical use in GIS through cartographic style transfer.
Catrographic style transfer: Applying the style of Ortelius from the first world atlas in the 15th century(top right) to a map of New York city (bottom right) shows New York as a swamp(left) which it apparently was at that time!
Generative Adversarial Networks (GANs) are an active area of research and can be used for generating map tiles straight out of imagery.
While the examples above have focused on imagery and computer vision, deep learning can also be used equally well for processing large volumes of structured data such as observations from sensors, or attributes from a feature layer. Applications of such techniques to structured data include predicting the probability of accidents to sales forecasting, and natural language routing and geocoding.
Esri is investing heavily in these emerging technologies and has started a new R&D center in New Delhi, focused on AI and Deep Learning on satellite imagery and location data. Check out https://newdelhi.esri.com to learn more about our work, see the open positions for data scientists, developers and engineers and apply online.