Strips still exist in viirs dnb after swatTogrid operation

How do I...?
Post Reply
User avatar
onion5376
Posts: 4
Joined: Fri Nov 29, 2019 1:49 pm

Strips still exist in viirs dnb after swatTogrid operation

Post by onion5376 »

(1) I tried to remove bowtie effects in raw viirs dnb using ‘swatTogrid’ in Formulas, but there were still strips in the processed output, as showed below.


(2) how to choose and do further operations on one output from one formulas in McIDAS
In general, multi-step operations were needed to process raw satellite images, for example, raw viirs dnb (GDNBO-SVDNB_npp_d20190425_t1753089_e1758493_b38822_c20190815141605982802_noac_ops.h5).

step 1: remove bowtie in viirs dnb
The “swathTogrid” in formulas was used to remove bowtie in viirs dnb data.
[img]
2.jpg
[/img]

step2: filter the no bowtie data from step 1
The “Gradient Image Filter” in formulas was used to filter the data without bowtie from step1. However, the output data from "step 1" did not showed in the “Field Selector”. Only the raw data could be selected to process, I did not know how to process the intermediate file outputed from one formulas.
[img]
3.jpg
[/img]

(3) the radiance unit for the VIIRS DNB was identified incorrectly by McIDAS-V
The unit for radiance in viirs dnb was showed as “10000.0 kg.s-3”, but in fact, the right unit was W/(cm2·sr). Though “Change display unit” in Edit menu was provided in “Layer Controls”, but it got a error message by entering the correct unit manually.
[img]
5.jpg
[/img]

Did Anyone also meet these problems, and give some help. Thanks.
Attachments
output from swathTogrid operation
output from swathTogrid operation
select swathTogrid in formulas and field selector
select swathTogrid in formulas and field selector
select "Gradient Image Filter" in formulas and field selector
select "Gradient Image Filter" in formulas and field selector
wrong unit shown in "Layer Controls"
wrong unit shown in "Layer Controls"
change unit with the manual operation and error message
change unit with the manual operation and error message
User avatar
bobc
Posts: 938
Joined: Mon Nov 15, 2010 5:57 pm

Re: Strips still exist in viirs dnb after swatTogrid operation

Post by bobc »

Hello,

The swathToGrid formula is intended to remove the bow-tie effect, which are the black stripes you see on the edges of SVM/SVI granules. However, the DNB (Day/Night Band) doesn't actually have these black stripes due to the nearly-constant resolution across the entire swath, so there's no need to use swathToGrid. For more information, see the "VIIRS and Bow-Tie Effect" section of this CIRA Guide.

To see what swathToGrid does, you can display SVM16 data at full resolution without swathToGrid and with swathToGrid. See the image below, where the left image is the original data, and the right is with swathToGrid removing the bowtie lines. This is close to the eastern edge of the granule, and both images are centered at the same location.

Bowtie lines on left, not on right
Bowtie lines on left, not on right

What I believe you are seeing in this image (without having the actual data) are lightning strikes. Lightning creates these bright stripes across the scene. You can see several examples of this in the CIMSS blog.

Granted that swathToGrid doesn't need to be used with DNB data, you can pass a data source through multiple formulas at once. For example, You could:
  1. Choose the Gradient image filter formula.
  2. In the Field Selector window that appears, choose Formulas > swathToGrid.
  3. Choose your DNB field, subset a region, and click OK.

The Change Unit window can be a little picky about how you define units. Could you try:

W/cm**2/sr

The actual data values in the display won't change, but the probe output will show the new unit. Additionally, if you add a color bar to the display and choose to show the unit, the new unit will be displayed.

Thanks,
Bob Carp
McIDAS User Services
User avatar
onion5376
Posts: 4
Joined: Fri Nov 29, 2019 1:49 pm

Re: Strips still exist in viirs dnb after swatTogrid operation

Post by onion5376 »

Thanks Bob. I was still confused about the intermediate file output from formulas. Is it possible that I could process the raw data step by step? In most cases, I maybe do not know how many processing steps for one raw data, so setting up multiple formulas at once for this data was difficult, and the step-by-step operation may be more convenient .

The other key point was that how to export the intermediate file from formulas into my local hard disks. Sometimes, data processing involves more softwares, some special tools in other softwares would be applied to further process the intermediate file from McIDAS-V. I could not find any option menu in McIDAS-V 1.8 or functions in Jython shell. Is it possible to implement this?

Best wishes, onion
User avatar
bobc
Posts: 938
Joined: Mon Nov 15, 2010 5:57 pm

Re: Strips still exist in viirs dnb after swatTogrid operation

Post by bobc »

Hello,

Point 1:
Through the UI you can export displayed data to cache and then operate on it further. For example, if you run swathToGrid on a SVM14 granule and then perform a mask, you could:

  1. Load your data, choose the swathToGrid formula, and click Create Display.
  2. In the Field Selector window, choose the band 14 > Temperature field, subset a region, and click OK.
  3. From the Layer Controls, choose "File > Save > Save Data in Cache". Enter a name, such as "swathToGrid band 14" and click OK. This adds a new data source to the Field Selector called "Cached data". If you click on "Cached data" you'll see all of the cached data sources from your current session.
  4. From the Field Selector, choose the "Imagery > Mask Value" formula and click Create Display.
  5. In the window that appears, enter your desired values. For example "200" "<" "1" to only show values less than 200K in the display. Click OK.
  6. In the Field Selector window that appears, choose "Cached Data > swathToGrid band 14" for both inputFieldForMask and displayFieldToBeMasked. Click OK.

Note that it's possible to pass data through multiple formulas through a script, as well. I find that using a script makes things a little easier to follow than using the UI when using multiple formulas, but that's just a personal preference. If you want an example of how you can load and operate on JPSS data through the Jython Shell (or a background script), see the "Create VIIRS RGB images entirely from background script" forum post.

Point 2:
With most data sources in McIDAS-V, you can export them to netCDF. This can be done with the "Export > Export Grid/Image to netCDF" formula, or by selecting "File > Save > Export Displayed Data to NetCDF" from the Layer Controls of a displayed data source. However, this doesn't currently work with JPSS data. This is written up as inquiry 1433, though I don't have an estimate on when this will be complete.

Thanks,
Bob
User avatar
onion5376
Posts: 4
Joined: Fri Nov 29, 2019 1:49 pm

Re: Strips still exist in viirs dnb after swatTogrid operation

Post by onion5376 »

Thank you very much Bop, I will try these. I think that some common raster file type (such as, esri grid, geotiff) could be added in the output menu in furture development of McIDAS-V. This functionality may make it better collaboration with other softwares.
User avatar
bobc
Posts: 938
Joined: Mon Nov 15, 2010 5:57 pm

Re: Strips still exist in viirs dnb after swatTogrid operation

Post by bobc »

Thanks for your suggestion. We have inquiry 1806 written up to add the ability to capture GeoTIFFs. I added a note to the inquiry to contact you when this functionality is added, though I can't give an estimate on when this will be done.

Please let me know if you have any other questions or new feature requests!

Thanks again,
Bob
Post Reply