Cloud mask
Dataset description
The cloud mask was derived from radiance measurements of the SWIR camera of the specMACS instrument. To overcome the effects of the sunglint, spectral water vapor absorption was used to identify clouds in front of a bright background. The specMACS cloud mask is created by the combination of a mask based on the brightness of the measurements (brightness_cloud_mask) and another based on the measured water vapor absorption (combined_brightness_watervapor_mask). The following flowchart illustrates the different steps of the cloud detection algorithm.
Flowchart of cloud detection algorithm; taken from Master's Thesis of Felix Gödde (Figure 3.16).
The cloud mask for each research flight is stored in a separate NETCDF4 file. Each file contains the following variables:
- ”cloud_mask”: It is generally recommended to work with this cloud mask! This cloud mask is the result of merging the brightness cloud mask and the combined brightness water vapor mask. It is created in the following way: In the presence of sunglint the combined brightness water vapor cloud mask is used, while in scenes without sunglint the pure brightness mask is preferred. To decide whether sunglint is present in the scene or not the ocean reflectance was simulated with libRadtran. Three classes are discriminated: 0 indicates probably clear, 1 probably cloudy pixels and 2 most likely cloudy pixels. This mask is located at the bottom of the flowchart and is called "best estimate cloud mask".
In some cases (depending on the specific application), the combined mask may not give the best result. In such cases it is possible to analyse the two masks (brightness and combined brightness water vapor mask) separately. This is why the two variables ”brightness_cloud_mask” and "combined_brightness_watervapor_mask" are added to the files:
”brightness_cloud_mask”: cloud mask which uses a brightness threshold to distinguish (bright) cloudy from (dark) cloud-free pixels. This mask should be handled carefully: In the presence of (bright) sunglint, using this mask the sunglint region is incorrectly identified as cloudy. Three classes are discriminated: 0 indicates probably clear pixels, 1 probably cloudy or sunglint contaminated pixels, and 2 most likely cloudy or sunglint contaminated pixels. This mask is located to the right, in the middle of the flowchart.
”combined_brightness_watervapor_mask”: starting from the brightness cloud mask, this mask identifies cloudy pixels by the absorption of water vapor. Highest absorption occurs if no cloud is present. Using this mask it is possible to identify clouds in the presence of sunglint. Three classes are discriminated: 0 indicates probably clear pixels, 1 probably cloudy pixels, and 2 most likely cloudy pixels. Class “1” filters the sunglint less strictly, which is why this should be used with caution. In the flowchart this mask is called "water vapor cloud mask" but we decided to rename it to "combined brightness water vapor mask" to make clear that this mask is based on the brightness mask.
In addition to the cloud mask, there are additional variables in the files which simplify the work with the cloud mask:
- ”swir_radiance”: radiance measured by the SWIR camera at a wavelength of around 1601 nm in units of [mW m-2 nm-1 sr-1] (data was smoothed by a gaussian 3x3 kernel to minimize inconsistencies in the measurements and obtain a decent result for the cloud mask)
- ”vza”: angle between the local vertical axis with respect to the WGS 84 ellipsoid and the line of sight to each pixel of the sensor. The angle is given in [degree]
- ”vaz”: horizontal angle between the line of sight to each pixel of the sensor and due north. The angle is given in [degree]
- ”sza”: angle between the local vertical axis with respect to the WGS 84 ellipsoid and the line of sight to the sun. The angle is given in [degree]
- ”saz”: horizontal angle between the line of sight to the sun and due north. The angle is given in [degree]
On certain flight dates one additional variable is added:
- ”valid”: On 2020-01-19 and 2020-02-18 the window of the instrument housing was iced and on 2020-01-30 during the first 1.5 hours of the flight, no dark current was measured. These circumstances influence the measurements which is why for these days a binary mask is added. Values of 0 mean that the data is not valid (due to icing or missing dark current measurements) and values of 1 mean that the data is valid.
- Important note: the icing of the window or missing dark current measurements can introduce a bias in the cloud mask (more/less clouds are detected). Ice affected parts or parts with missing dark current measurements should therefore be handled carefully.
Important notes and known problems:
- In general, the cloud_mask should give the best results. If you are interested in specific scenes we recommend having a look at the brightness or the combined brightness water vapor mask separately to decide which mask fits best to your application.
- For measurements with large solar zenith angle the cloud mask only identifies the brightest pixels as cloudy which results in an underestimation of the total cloud fraction.
- In class 2 (“most likely cloudy”) small and thin clouds are identified only poorly but the sunglint is rarely identified as cloud.
- In class 1 (“probably cloudy”) also small and thin clouds are detected but in some cases sunglint contaminated pixels are identified as cloud.
- Clouds that are covered by a shadow are sometimes not identified as cloud.
- The cloud mask only discriminates between probably clear, probably cloudy and most likely cloudy pixels. It should be noted that there is no class which contains the most likely clear sky pixels!
- Even if the thresholds are chosen carefully there are certain situations where the cloud mask fails. In cases with sunglint accompanied by high aerosol concentrations it occurred that small areas were incorrectly identified as clouds. Furthermore, especially in scenes with low sun where only little light was reaching the sensor, artifacts of the sensor occured in the measurements. This results in the incorrect classification of single pixels as clouds. Therefore it was decided to apply a binary opening operation on the cloud mask with a 3x3 kernel. This operation was already carried out and should not be repeated. It removes all clouds which are smaller than 3x3 pixels. So if you are interested in analysing the cloud size distribution, you must be aware that very small clouds are not detected.
Having these notes in mind you should be able to decide which cloud mask you should take. Here are some examples of possible applications and recommendations:
- "I need a cloud mask and I want to be sure that the pixels which are classified as cloudy are indeed clouds." Recommendation: Use all values = 2 of cloud_mask (= most likely cloudy)
- "I need a cloud mask and it is important that also small or thin clouds are correctly classified as cloudy but it is okay for me if some clear sky pixels are incorrectly classified as clouds (often in the sunglint region)." Recommendation: Use all values = 1 together with all values = 2 of cloud_mask (= probably cloud + most likely cloudy)
- "I am interested in clear sky pixels and I am aware of the problem that with the specMACS cloud mask I can never be totally sure that all clear sky pixels are really clear sky." Recommendation: Use all values = 0 of cloud_mask (= probably clear sky)
- "I am analysing a short sequence and already had a look at the cloud_mask variable and want to know whether this is the best cloud mask." Recommendation: In special cases it could happen that the brightness_cloud_mask or the combined_brightness_watervapor_mask yield a better result than the cloud_mask. You can have a look at those two masks and then decide which one to take.
- "I want to use the cloud mask product but I am not sure which cloud mask I should take." Recommendation: First, always use the cloud_mask variable. If you are not happy with it or want to better understand the product, contact Veronika Pörtge.
Changes
The algorithm developed to extract the cloud mask from the specMACS data in the presence of sunglint is described in the Master's Thesis of Felix Gödde. The Pdf document of the thesis can be downloaded here. For the EUREC4A cloud mask product some minor modifications of the algorithm were done:
- In the original cloud detection algorithm three water vapor absorption bands were used (total spectral range: 1015 nm - 1910 nm). For the EUREC4A data we decided to reduce the spectral range of the fitting algorithm to 1015 nm - 1300 nm. This only includes the first absorption band which has the advantage of not being affected by liquid water absorption. In the other two bands liquid water absorption occurs.
- The brightness threshold is no longer constant but a function of the solar zenith angle. This was done by obtaining the best matching brightness threshold for different scenes with varying solar zenith angles and fitting a function through these datapoints.
- The cloud mask is no longer a binary cloud mask but is made up of three different classes (probably clear, probably cloudy and most likely cloudy pixels). Technically this was done by applying two different brightness thresholds to the fitting results. One being much lower and therefore allowing much darker regions to be identified as cloud.
- The meteorological data for the water vapor profiles now comes from ERA-5 data (previously: ERA-Interim).
Download
Example plot
Here you can find a simple python code showing how to plot the cloudmasks. The resulting image should look like the image below. In the upper panel the cloud_mask is shown. In the middle the brightness_cloud_mask and in the lower panel the combined_brightness_watervapor_mask. In the background of all three panels the swir_radiance is plotted. The violet lines surround all pixels which are either probably cloudy or most likely cloudy. The orange lines define the contourlines of the pixels which are most likely cloudy. One can clearly see the bright sunglint in the upper part of the radiance images. Having a look at the brightness mask it can be seen that the classification of the "most likely cloudy pixels" seems correct (orange contourlines), while the classification of the "probably cloudy pixels" is wrong (violet contourlines): Here, large parts of the sunglint are identified as clouds (basically the whole upper half of the image). This problem is removed in the combined brightness water vapor mask (lower panel). Since sunglint is present in this scene, the combined brightness water vapor mask is used for the final mask (upper panel).
It should be noted that the y-axis of the image is inverted in the script. This is done because the SWIR measurements are mirrored compared to the measurements of the 2D (polarization) cameras of specMACS. Inverting the y-axis simplifies a visual comparison of the two measurements. Have a look at the videos of the 2D camera measurements for a better understanding of the cloud situation! 
Help needed?
If you have any questions concerning the cloud mask please feel free to contact us! This could be, e.g., by sending an e-mail to Veronika Pörtge.
Versions
- 2020-07-22: V1.1 uploaded. Changes: renaming variable "water_vapor_cloud_mask" to "combined_brightness_watervapor_mask"; Bug fix of binary opening operation -> now the single pixels which were identified as clouds are correctly removed.
- 2020-07-14: V1.0 uploaded
Literature
- The algorithm developed to extract the cloud mask from the specMACS data in the presence of sunglint is described in the Master's Thesis of Felix Gödde. The Pdf document of the thesis can be downloaded here.
- A comparison of cloud masks derived from different instruments during NARVAL-II is shown in the Bachelor's Thesis of Sabrina Pavicic. The Pdf document of the thesis can be downloaded here.
- The specMACS cloudmask which was produced for the NARVAL-II measurements was used in the following paper: Höppler, L., Gödde, F., Gutleben, M., Kölling, T., Mayer, B., and Zinner, T.: Synergy of Active- and Passive Remote Sensing: An Approach to Reconstruct Three-Dimensional Cloud Macro- and Microphysics, Atmos. Meas. Tech. Discuss., https://doi.org/10.5194/amt-2020-49, in review, 2020.