Difference between raw data values and scaled data values

Post any questions, ideas, or topics related to Jython and Python scripting.
Post Reply
User avatar
notion21223
Posts: 3
Joined: Wed Mar 16, 2016 7:17 pm

Difference between raw data values and scaled data values

Post by notion21223 »

Dear All,
I have extracted scaled data values and raw data values of Aerosol Optical Depth Scientific Data Set Optical_Depth_Land_And_Ocean and Corrected_Optical_Depth_Land at 550 nm from MODIS Terra, Aqua aerosol product and several variables from MODIS Atmospheric profile product. I used the formula,
Real_value= scale_factor*(stored_value – add_offset)
to convert raw data values to physical values in R.
However, I noticed that the converted physical value is not similar to most of the extracted scaled data values on the similar hour and date. So, I am wondering how the scaled data values are extracted from McIDAS-V as I didn’t use any formula in McIDAS-V to scale data? I just used an option to extract scaled data values.
So, shouldn’t it be the same values when raw data values are converted to physical value using above formula with the extracted scale values? What is the difference between raw data values and scaled data values in McIDAS when formula is not used?
Also, when I used the same formula [Real_value= scale_factor*(stored_value – add_offset)] with extracted scaled data values in place of stored_value, the data shows somewhat positive results in my research. However, I need to understand how McIDAS is extracting scaled data?
So, I look forward to the answer. Please, respond to it.
Thank you.
User avatar
tommy.jasmin
Posts: 35
Joined: Tue Aug 17, 2010 6:56 pm

Re: Difference between raw data values and scaled data values

Post by tommy.jasmin »

Hi notion21223,

If a remote sensing product has scale and offset information in the associated metadata, McIDAS-V should automatically apply it for you. However, you may get different results depending on what you select as the Data Type. Try using "HYDRA Multi/Hyper-Spectral Data Source" and see if you get satisfactory results.
User avatar
mhiley
Posts: 90
Joined: Mon Jan 23, 2012 10:22 pm

Re: Difference between raw data values and scaled data values

Post by mhiley »

Hello notion21223,

Also, I believe there is an error in your extraction formula. It should be (parentheses added for clarity):

Real_value= (scale_factor*stored_value) – add_offset

instead of

Real_value= scale_factor*(stored_value – add_offset)

-Mike
User avatar
notion21223
Posts: 3
Joined: Wed Mar 16, 2016 7:17 pm

Re: Difference between raw data values and scaled data values

Post by notion21223 »

Hi Mike,

Thank you for your reply.

For AOD, as there is no offset value, I believe the formula gives the same result. However, for MODIS atmospheric profile product, I used the same formula,

Real_value= scale_factor*(stored_value – add_offset)

which is given in NASA MOD07_L2 format and content section in MODIS atmospheric product webpage (http://modis-atmos.gsfc.nasa.gov/MOD07_L2/format.html).

Is it that the data extracted using McIDAS-V for MOD07, MYD07 product should be used with,

Real_value= (scale_factor*stored_value) – add_offset

formula to covert the stored data values to real physical meaningful values?

I will check with the results using the formula you provided as well to see the changes it brings!

Thank you again.

Sincerely,
Noshan
User avatar
notion21223
Posts: 3
Joined: Wed Mar 16, 2016 7:17 pm

Re: Difference between raw data values and scaled data values

Post by notion21223 »

Hi tommy.jasmin,

Thank you for your reply.

It is good to know that McIDAS-V automatically applies the associated metadata of a remote sensing product for users.

I am also particularly interested to know how McIDAS-V gives different results when I select different data type for a SDS such as raw data values and scaled data values for MODIS aerosol product (a screenshot of a McIDAS-V data explorer for MODIS Aerosol product is attached herewith~file name: MODIS aerosol product SDS) . And coded parameter and temperature, SDS-Surface Temperature, which is highlighted in a separate attachment as well ~file name: MODIS atmospheric product SDS.

Is there any particular scaling factor used by McIDAS-V when it provides scaled data values for MODIS aerosol product compared to raw data values? Because, in my study, the results were better for scaled data values! so, I am curious know how it works in McIDAS-V.

Thank you.

Sincerely,
Noshan
Attachments
MODIS_atmospheric_profiles_SDS.JPG
MODIS_aerosol_product_SDS.JPG
User avatar
tommy.jasmin
Posts: 35
Joined: Tue Aug 17, 2010 6:56 pm

Re: Difference between raw data values and scaled data values

Post by tommy.jasmin »

Hi Noshan - If you are loading the data via the HYDRA Data Source, McIDAS-V
will be applying the rule:

Physical Value = Scale Factor * (Stored Value - Offset)

Using values in the HDF metadata for that MODIS granule.

However, if you are loading data from an ADDE server, calibration will be applied
in the server code.
Post Reply