Grid Workflow Management for Satellite Data Processing within UN-SPIDER Program

One of the most important problems associated with satellite data processing for disaster management is a timely delivery of information to end-users. To enable such capabilities, an appropriate infrastructure is necessary to allow for rapid and efficient access to, processing and delivery of geospatial information that is further used for damage assessment and risk management. In this paper, we will describe the use of Grid technologies [1, 2] for automated acquisition, processing and visualization of satellite synthetic-aperture radar (SAR) and optical data for rapid flood mapping. The developed services are used within the UN-SPIDER Regional Support Office in Ukraine that was established in February of 2010.
Within the infrastructure we developed an automated workflow of satellite SAR data acquisition, processing and visualization, and corresponding geospatial services for flood mapping from SAR imagery. The data are automatically downloaded from ESA rolling archives where satellite images are available within 2-4 hours after their acquisition. We implemented both programming and graphical interfaces to enable search, discovery and acquisition of data. Through the portal the user can select interested geographical region and time range, and find the data that match the search request. After the user selects a file to be processed, it is transferred to the resources of the Grid system at the Space Research Institute NASU-NSAU. After that, a workflow is automatically generated and executed on the resources of the Grid infrastructure.
To enable such a workflow in the Grid system a set of services has been implemented. We followed an approach that is used in Earth System Grid [3]. The four major components of the system are as follows:
1. Client applications. Web portal is a main entry point, and provides interfaces to communicate with system services.
2. High-level services. This level includes security subsystem, catalogue services, metadata services (description and access), automatic workflow generation services, and data aggregation, sub-setting & visualisation services. These services are connected to Grid services at the lower level.
3. Grid services. These services provide access to the shared resources of the Grid system, access to credentials, file transfer, job submission and management.
4. Database and application services. This level provides physical data and computational resources of the system.
We developed a neural network approach to SAR image segmentation and classification [4]. The workflow of data processing is as follows:
1. Data calibration. Transformation of pixel values (in digital numbers) to to backscatter coefficient (in dB).
2. Orthorectification and geocoding. This step is intended to remove geometrical and radiometric corrections associated by SAR imaging technology, and provide precise georeferencing of data.
3. Image processing. Segmentation and classification of the image using neural network.
4. Topographic effects removal. Using digital elevation model (DEM), such effects as shadows are removed from the image. The output of this step is a binary image classified into two classes: “Water” and “No water”.
4. Transformation to geographic projection. The image is transformed to the projection for further visualisation via Internet using OGC-compliant standards (KML or WMS) or desktop Geographic Information Systems (GIS) using shape file.
After processing the user request from the portal, such a workflow is automatically generated using Karajan engine and is executed on the resources of the Grid system. Through the portal the user can monitor the status of each step of the workflow. After the workflow is completed, flood map is delivered to the user via OGC-compliant standards.
In order to benefit from data of different nature (for example, optical and radar) and provide integration of different products in case of emergency, our flood mapping service was integrated with flood mapping services provided by the Center of Earth Observation ad Digital Earth of the Chinese Academy of Sciences (CEODE-CAS). This service is based on the use of optical data acquired by MODIS instrument onboard Terra and Aqua satellites.
The integration of the Ukrainian and Chinese systems is done at the level of services. Portals of SRI and CEODE are operated independently and communicate with corresponding brokers that provide interfaces to flood mapping services. These brokers process requests from both local and trusted remote sites. For example, to provide flood mapping products using SAR data CEODE portal generates corresponding request to the broker at the SRI side. This request is processed by the broker and the results are displayed at CEODE portal. The user selects required data to be processed, and request again goes to the SRI broker which generates and executes workflow, and delivers the results to CEODE portal. The same applies to the broker operated at the CEODE side that provides flood mapping services using optical data. In order to get access to the portal user must obtain a certificate. SRI runs Virtual Organisation Management Server (VOMS) to manage with this issue.
References.
1. Shelestov A., Kussul N., Skakun S. Grid Technologies in Monitoring Systems Based on Satellite Data. J of Automation and Inf Sci, 2006, 38(3), pp. 69–80.
2. Fusco L., Cossu R., Retscher C. Open Grid Services for Envisat and Earth Observation Applications. In: Plaza AJ, Chang C-I (ed) High performance computing in remote sensing, 1st edn. Taylor & Francis Group, New York, 2007, pp. 237–280.
3. Williams D.N. et al. Data management and analysis for the Earth System Grid. J. Phys.: Conf. Ser., 2008, 125, 012072. doi: 10.1088/1742-6596/125/1/012072.
4. Kussul N., Shelestov A., Skakun S. Grid System for Flood Extent Extraction from Satellite Images. Earth Science Informatics, 2008, 1(3-4), pp. 105–117.