Scatter Analysis with simple difference and mask

[joleenf]

Hi,

I have observed the following difficulty twice in the past few weeks working with two different research groups:

When performing a scatter analysis of channel differences, the process becomes cumbersome. Both groups were examining current H-8 data on the server. I am not sure of the exact channel combination the first group was using, but in a recent example, the researcher was looking at 11-12 and 10-8.5 channel differences. In her case, the problem became even more complicated, because there was some bad data that obscured what she was trying to see, so we tried applying a maskWithinRange.

Steps (without creating a user formula):
1.) Formulas->Imagery->Mask Values Within Range display as Scatter Analysis
2.) In Mask Value Within Range popup window, use a range of -5 to 5, and use NaN
3.) In the MaskWithinRange field selectors. For inputFieldForMask, select Formulas->Imagery->Simple Difference a-b
4.) For displayFieldForMask, select Formulas->Imagery->Simple Difference a-b
5.) In the secondary field selector, select 11.2µm Temperature for Field: a, and 12.4 µm Temperature for Field:b (select a small region for this example)
6.) In the tertiary(?) field selector, select the same two fields for Field:a and Field: b as in step 5.
7.) In the Field Selector for "select Y Axis Field", select Formulas->Imagery->Mask Values Within Range
8.) Use same as step 2 for range.
9.) Same as Step 3
10.) Same as Step 4
11.) In the secondary field selector, select 12.4µm Temperature for Field:a, and 9.6 µm field selector for field: b.
12.) In the tertiary field selector, select same two field for Field:a and Field: b as in step 11.

If the user is lucky, no mistakes are made. I have tried this numerous times, and been successful none of the
times.

The next step was to create a user formula in the jython library.
The formula is somewhat redundant to what already exists,
but takes a few steps out of the click and point dilema:

def diffMask(a,b,min,max):

diff=sub(a,b)
diffMasked=maskWithinRange(diff,min,max,1)

return diffMasked


The user could then connect to this formula in the jython library.

but I could not get this to work.

It would also be better, if a user needed to repeat this task for the same channel combinations
over and over, to be able to hardwire a process somewhere and the only changing piece would be the
region selected or the date/time of the data. Towards that end, would there be a way to
code that into a formula or is there a way to create a Scatter Analysis layer using the
McIDAS-V scripting API?

Thanks,
Joleen

[bobc]

Hi Joleen -

I'm not seeing any easy way of creating a scatter analysis display through scripting. I wrote up adding this functionality as Inquiry 2164.

I came up with a somewhat shorter way of creating the scatter analysis display than your original list. It makes use of the diffMask function from your post. I created two different functions, one for the x-axis field and one for the y-axis field:

Code: Select all

def himaX(band14,band15,minvalue,maxvalue):
    xScat = sub(band14,band15)
    xScatMask = maskWithinRange(xScat,minvalue,maxvalue,1)
    xScatResult = mul(xScatMask,xScat)
    return xScatResult

def himaY(band13,band11,minvalue,maxvalue):
    yScat = sub(band13,band11)
    yScatMask = maskWithinRange(yScat,minvalue,maxvalue,1)
    yScatResult = mul(yScatMask,yScat)
    return yScatResult

I then created two different formulas to access these functions:

Formula 1
Description: Himawari Scatter Plot X
Name: himawariX
Formula: himaX(band14,band15,float(minvalue[isuser=true,default=-5]),float(maxvalue[isuser=true,default=5]))

Formula 2
Description: Himawari Scatter Plot Y
Name: himawariY
Formula: himaY(band13,band11,float(minvalue[isuser=true,default=-5]),float(maxvalue[isuser=true,default=5]))

Using the above, here is how I created the scatter analysis display:

1. Load in the data for each band.
2. Select the 'Himawari Scatter Plot X' formula, the Scatter Analysis display, and click Create Display.
3. Choose bands 14 and 15 and click OK.
4. In the new Field Selector window, select Formulas>'Himawari Scatter Plot Y'.
5. Choose bands 13 and 11 and click OK.

Does this work for you?

I'm not thinking of any way to edit the formula so McIDAS-V remembers the last field you selected. It seems like what you want to do is create a scatter analysis display then change the domain/timesteps without having to go and re-select anything. This is possible with other types of displays (like the Image Display) where you can display the data and then choose Edit>Properties. From here, there are two tabs for Times and Spatial Subset. I've written up inquiries to get this functionality added and working with the Scatter Analysis display.

Thanks -
Bob Carp

[bobc]

Hi Joleen -

Following up with your phone call, it is possible to create derived parameters for the x and y axes of the scatter plot. Here is how I went about doing this and an example of the output. This was working with Himawari FD data off the real-time server.

1. In setting up for the formulas, I came up with parameter groups for both the x and y axes of the scatter plot. The x-axis uses band 14 temperature - band 15 temperature. The y-axis uses band 13 - band 11:
   

   

   Parameter Groups

2. Set up a the formulas for the x and y axes. The formula uses D1 and D2 for the parameter (D meaning 'derived'). In the Settings tab of the Formula Editor window (not shown), I selected only the Scatter Analysis display to be available. In the Derived tab, uncheck 'For end user' so the formula doesn't list in the Field Selector when you select Formulas under Data Sources. Use the Parameter Groups dropdown to select scatPlotX (or whatever you called it in #1 above). Do the same thing for y-axis:
   

   

   x-axis formula

   
   

   

   y-axis formula

3. From here, reload the Himawari data source, and you should see your two derived fields for the x and y axes. Choose the x-axis derived field, make sure that Scatter Analysis is selected as the display and click Create Display. Enter your desired min/max values and click OK. In the new Field Selector window, choose the derived field for the y-axis. Enter your min/max range and click OK. From here, you should get a scatter analysis display like this:
   

   

   scatter analysis

I was able to get this working in both 1.5 and the current 1.6beta1 nightly. If you have any questions, or if you are still seeing a 1 to 1 relationship between the x and y axes, please let me know.

Thanks -
Bob

[joleenf]

Bob,

This looks like a nice possibility, but I am getting strange results.

Since the description in the parameter group definition is what shows up for selection in the "Derived Fields" tab, that is probably a good place to define exactly which fields will be involved in the derived field calculation:

However, when I run a x versus y scatter plot with the customized formula for the x axis and for the y access, following the Fomulas->Simple Difference approach, I get very strange results. I don't get a 1 to 1 correspondence as expected and I get a difference region for the x and y fields. I have verified when I run through the process that the region is set to the same size. I also can't tell what I am actually plotting in the scatter analysis because the labels make it seem that I am plotting bands 13 and 15, when in fact I am pretty certain I was checking the Band13-Band12 difference.

Thanks,
Joleen

[joleenf]

Hi Bob,

I thought that the result above had to be due to a mistake on my part. One clarification, I was not having any luck applying the mask, so my x and y formulas just use the sub(a,b) function. I reran the test on H8 data from 2015-Sept-29 at 22:00 UTC. The region was centered at lat/lon of 31.8, 127.7. The size of the region was 250x252. The Mag=1,1.

Testing the formula created (automated difference of 12.4-9.6) versus using Formulas->Simple Difference a-b, I get

Still not what is expected, but closer than example above.

Testing the automated x versus y formulas, I get

This looks possible. What is interesting is that the region remains consistent in the second test, but not the first.

Joleen

[joleenf]

Hi,

Sorry for another addition. It seems that the first formula I created (11-12) using this method works consistently. However, I still see the strange difference in region selection.

In addition, I learned that I should not use "µ" in the formula title, because each time I start a new session of McV, another Å symbol gets added. For example, if I used the title: "11µm-12µm," on first new session of McV, the title would now be "11 Å µm - 12 Å µm." On the next new session of view, the title would become "11 ÅÅ µm - 12 ÅÅ µm." I also for a brief period had replication of the formula for each restart. This seems to have cleared after removing all the formulas and starting over.

Joleen

[bobc]

Hi Joleen -

I wrote up three inquiries:

1. Inquiry 2179 - The x and y-axes of a scatter analysis should be labeled appropriately if a band subtraction is done for the axis. Currently, it just lists out the first band used in the subtraction for each axis, so you see Band14_TEMP on the x-axis and Band13_TEMP on the y-axis.
2. Inquiry 2180 - Special characters may not work well in the Description of a formula between sessions. I verified that this is a bug on our OS X testing machine.
3. Inquiry 2181 - The sizing issue of a derived field of 2km data.

For number 3 above, here is an explanation (as I see it) as to what is going on:

Both of the bands used in the derived field are 2km resolution, but once the derived field is created, it is lists as 1km resolution in the Advanced tab of the Field Selector. When evaluating the scatter analysis, if you select a size of 250x252 for the derived field with 1x1 res this same size wants to be carried over to the y-axis field. The y-axis field, where you evaluate the misc>simple difference formula, does treat each field in the subtraction as 2km. Since McIDAS-V wants to retain the geographical domain used for the x-axis, and it thinks that this x-axis is 1km res, McIDAS-V cuts the image size in half for the simple difference formula (2 km is double 1 km) so the image size listed out in the Advanced tab of the Field Selector window when choosing the simple difference fields is 125x128. Once the display is created, McIDAS-V must recognize that the derived field actually is 2km resolution, so it ends up double the size of the simple difference field. A workaround for this is that once you are selecting your first simple difference field, change the image size in the Advanced tab from 125x128 to 250x252. This size will be carried over to the remaining fields you select for simple difference.

An additional problem with this derived field (also mentioned in the inquiry) is that the image size isn't correct for the bands used to create the field. It shows as having a size of 11,000 x 11,000 (the resolution of band 1 of the data), when it should be 5,500 x 5,500 to match the resolution of the IR bands used in the derived field.

If you have any questions about this, let me know.

Thanks -
Bob

[joleenf]

Hi Bob,

I had an error in the Himawari Channel Diff group. The Params should be 86_Band15_TEMP rather than 86_Band13_TEMP. The odd error of the first field returning the whole selection and getting a subset of the selection for the second field is still happening in McV 1.6 Nightly. It is interesting though, if I select the X axis of the scatter analysis as the formula (see image 1), and keep the region as is, the mag is -11, -11.

image 1

Next, if for the y-axis, I select Formulas-->Simple Difference a-b and do nothing with the region selection for the a and b fields, for whatever reason, the mag is already reduced to -6,-6. (see image 2)

image 2

I tested this for GOES, and I don't see the same problem. However, I did notice a different bug with GOES, which I will put in a different bug report...

Thanks,
Joleen

[bobc]

Hi Joleen -

I think this MAG=-11,11 vs MAG=-6,-6 is related to Inquiry 2181.

The derived field for the x-axis of the scatter analysis is generated from two fields with 2km resolution. However, the derived field itself says that it is 1km resolution. This derived field defaults to a mag of -11, and 11km resolution. When you are setting your y-axis, instead of using a derived field, you are using one of the native bands included with the Himawari data. These bands, having not been run through a formula, still have their native resolution of 2km. To try matching the resolution of the derived field, it goes to a mag of -6.

I'd have to check with a programmer, but I'd guess that it tries setting these bands at 11km resolution. To get 11km resolution from this 2km data, it would have to use a mag of -5.5, and only integers are allowed to be passed through mag, so it rounds to -6 to get 12km resolution.

- Bob