Hi,
I was helping a co-worker display H8 data as an rgb in McIDAS-V. He had a machine with 25 GB of memory and we could not display a H8image, even at reduced size from the geoarc server. First, the data selection was painfully slow, then the actual image creation failed. It seems that data from the real-time H8 data server is slightly faster. I wrote a script for him so that he could at least get an image created, at the same size we were requesting through the GUI. He had NPOT checked in the Advanced Preferences. This will be a big problem when GOES-R is launched if the speed of data access is the same.
Second, the weighted average as default confused again. It might be a good idea to create a checkbox for some of these features at install so the user knows that default is "Weighted Average." I am not sure if NPOT is still by default "off", but a prompt at install might be appropriate for this as well if it is still default "off."
Joleen
geoarc very slow for h8 data
Re: geoarc very slow for h8 data
Hi Joleen -
A few notes:
1. It is a known issue that the archive server on geoarc for Himawari is slow. This is likely due to the way that the JMA code works on the server. There are a lot of files with this data, with there being 10 minute imagery, 16 bands per image, and 10 segments per band at each timestep. There will be work to improve the performance of the server, but it may take some time.
One workaround that you can use for now is the QHIM08 dataset on geoarc. With this dataset, there are the following Image Type items available:
ALL ALL Himawari8 data
FD01 Channel 01 Full Disk Himawari8 data
FD02 Channel 02 Full Disk Himawari8 data
FD03 Channel 03 Full Disk Himawari8 data
FD04 Channel 04 Full Disk Himawari8 data
FD05 Channel 05 Full Disk Himawari8 data
FD06 Channel 06 Full Disk Himawari8 data
FD07 Channel 07 Full Disk Himawari8 data
FD08 Channel 08 Full Disk Himawari8 data
FD09 Channel 09 Full Disk Himawari8 data
FD10 Channel 10 Full Disk Himawari8 data
FD11 Channel 11 Full Disk Himawari8 data
FD12 Channel 12 Full Disk Himawari8 data
FD13 Channel 13 Full Disk Himawari8 data
FD14 Channel 14 Full Disk Himawari8 data
FD15 Channel 15 Full Disk Himawari8 data
FD16 Channel 16 Full Disk Himawari8 data
JAPAN Japan Sector Himawari8 data
TARGET Target Sector Himawari8 data
I ran a test where I loaded in a 06:00Z time off AHIM08/FD and QHIM08/FD01 and displayed band 1 full res. There was quite an improvement when using QHIM08. Here are the results of my tests. This is a list ordered by the step completed; the time for AHIM08 to complete the step; the time for QHIM08 to complete the step:
* Time from date selected until Absolute times tab populated; 56 seconds; 17 seconds
* Time from Add Source until Field Selector comes up and is populated; 13 seconds; 2 seconds
* Time from selecting a band/calibration until Create Display button available; 57 seconds, 9 seconds
* Time to display band 1 full resolution; 1 min 1 second; 24 seconds
* Total time; 3 min 7 seconds; 52 seconds
For creating a RGB from QHIM08, this would mean that you would need a separate data source for each band you are working with, but it should still be a quicker process than AHIM08. If you want to, please give QHIM08 a try and let us know how it works for you.
2. You mentioned that the image creation failed. Did you see any errors, or did the image just not display? What band were you working with? Were you displaying full resolution and how large of a geographical area were you displaying?
3. NPOT is disabled by default.
4. I wrote up Inquiry 2280 to cover prompting the user for certain User Preference settings at the installation of McIDAS-V. I think this could be a good idea because it will let users know more about the state of their session and let them know that they have control over those settings.
Thanks -
Bob Carp
A few notes:
1. It is a known issue that the archive server on geoarc for Himawari is slow. This is likely due to the way that the JMA code works on the server. There are a lot of files with this data, with there being 10 minute imagery, 16 bands per image, and 10 segments per band at each timestep. There will be work to improve the performance of the server, but it may take some time.
One workaround that you can use for now is the QHIM08 dataset on geoarc. With this dataset, there are the following Image Type items available:
ALL ALL Himawari8 data
FD01 Channel 01 Full Disk Himawari8 data
FD02 Channel 02 Full Disk Himawari8 data
FD03 Channel 03 Full Disk Himawari8 data
FD04 Channel 04 Full Disk Himawari8 data
FD05 Channel 05 Full Disk Himawari8 data
FD06 Channel 06 Full Disk Himawari8 data
FD07 Channel 07 Full Disk Himawari8 data
FD08 Channel 08 Full Disk Himawari8 data
FD09 Channel 09 Full Disk Himawari8 data
FD10 Channel 10 Full Disk Himawari8 data
FD11 Channel 11 Full Disk Himawari8 data
FD12 Channel 12 Full Disk Himawari8 data
FD13 Channel 13 Full Disk Himawari8 data
FD14 Channel 14 Full Disk Himawari8 data
FD15 Channel 15 Full Disk Himawari8 data
FD16 Channel 16 Full Disk Himawari8 data
JAPAN Japan Sector Himawari8 data
TARGET Target Sector Himawari8 data
I ran a test where I loaded in a 06:00Z time off AHIM08/FD and QHIM08/FD01 and displayed band 1 full res. There was quite an improvement when using QHIM08. Here are the results of my tests. This is a list ordered by the step completed; the time for AHIM08 to complete the step; the time for QHIM08 to complete the step:
* Time from date selected until Absolute times tab populated; 56 seconds; 17 seconds
* Time from Add Source until Field Selector comes up and is populated; 13 seconds; 2 seconds
* Time from selecting a band/calibration until Create Display button available; 57 seconds, 9 seconds
* Time to display band 1 full resolution; 1 min 1 second; 24 seconds
* Total time; 3 min 7 seconds; 52 seconds
For creating a RGB from QHIM08, this would mean that you would need a separate data source for each band you are working with, but it should still be a quicker process than AHIM08. If you want to, please give QHIM08 a try and let us know how it works for you.
2. You mentioned that the image creation failed. Did you see any errors, or did the image just not display? What band were you working with? Were you displaying full resolution and how large of a geographical area were you displaying?
3. NPOT is disabled by default.
4. I wrote up Inquiry 2280 to cover prompting the user for certain User Preference settings at the installation of McIDAS-V. I think this could be a good idea because it will let users know more about the state of their session and let them know that they have control over those settings.
Thanks -
Bob Carp
Re: geoarc very slow for h8 data
2. You mentioned that the image creation failed. Did you see any errors, or did the image just not display? What band were you working with? Were you displaying full resolution and how large of a geographical area were you displaying?
No errors, but we gave up because it took too long.
The region selection was the same as the attached script
h8forForum.py- (1.18 KiB) Downloaded 326 times
Will there be a performance improvement in the Imagery Chooser before the launch of GOES-R?
Joleen
Re: geoarc very slow for h8 data
Correction to the question:
I am not sure if the slow performance is in the geoarc server, Imagery Chooser, the Field Selector, the rgb combination, the display of the layer, or all them. It seems that the major problems were at the time of region selection to data display when processed through the Field selector in the GUI rather than in the script.
Also, do you know if I select the "green" band region in the rgb combine, will this scale down the "red" band to 1km from 0.5 km when this is done through the gui? If this the GUI tries force the two 1km bands to 0.5 km, computing time wasted.
Thanks,
Joleen
I am not sure if the slow performance is in the geoarc server, Imagery Chooser, the Field Selector, the rgb combination, the display of the layer, or all them. It seems that the major problems were at the time of region selection to data display when processed through the Field selector in the GUI rather than in the script.
Also, do you know if I select the "green" band region in the rgb combine, will this scale down the "red" band to 1km from 0.5 km when this is done through the gui? If this the GUI tries force the two 1km bands to 0.5 km, computing time wasted.
Thanks,
Joleen
Re: geoarc very slow for h8 data
Sorry about the multiple comments,
I ran through the gui to create an rgb on geoarc with preview off and using the adaptive resolution chooser in the Data Sources Tab:Under Development-->Imagery - Satellite
The data selection took a little over 1 min. waiting for the Field Selector to load and update the tabs. Then, I was able to get an rgb in about 3 minutes, but unfortunately, adaptive resolution does not work with RGB data, or at least I can't figure out how to zoom in and get better resolution data.
Joleen
I ran through the gui to create an rgb on geoarc with preview off and using the adaptive resolution chooser in the Data Sources Tab:Under Development-->Imagery - Satellite
The data selection took a little over 1 min. waiting for the Field Selector to load and update the tabs. Then, I was able to get an rgb in about 3 minutes, but unfortunately, adaptive resolution does not work with RGB data, or at least I can't figure out how to zoom in and get better resolution data.
Joleen
Re: geoarc very slow for h8 data
Hi Bob,
It is much faster in the field selection portion for the QHIM08 data. However, the time lag is transferred to waiting for an absolute time list in the layer controls. This probably could be sped up by allowing a time selection, or time selection range before the data is loaded into the absolute times tab...I am pretty sure that is already an inquiry.
However, it is nice that QHIM08 also seems faster in a script and that it is pretty simple to grab three different bands from three different descriptors in a script.
Is there a way to use adaptive resolution with an RGB? There is a coworker who wants to create FD RGBs of H8. It really would make sense for him to be using adaptive resolution in that case.
Joleen
It is much faster in the field selection portion for the QHIM08 data. However, the time lag is transferred to waiting for an absolute time list in the layer controls. This probably could be sped up by allowing a time selection, or time selection range before the data is loaded into the absolute times tab...I am pretty sure that is already an inquiry.
However, it is nice that QHIM08 also seems faster in a script and that it is pretty simple to grab three different bands from three different descriptors in a script.
Is there a way to use adaptive resolution with an RGB? There is a coworker who wants to create FD RGBs of H8. It really would make sense for him to be using adaptive resolution in that case.
Joleen
Re: geoarc very slow for h8 data
Hi Joleen -
It looks like you are correct that the "RGB Composite" display type from the "Three Color (RGB) Image (Auto-scale)" formula doesn't currently allow for Adaptive Resolution (AR). I wrote this up as Inquiry 2281. The other RGB display type, "3 Color (RGB) Image" from the "Three Color (RGB) Image" formula does allow for AR, but you don't have all of the extra options in the Layer Controls for setting the red, green, and blue ranges or the gamma values.
You are correct that there is already an inquiry to allow the user to specify dates/times without having to wait for the entire Absolute times tab to populate. This is inquiry 1798.
As for the RGBs, I found that creating a RGB display using bands 3, 2, and 1 (0.5km, 1km, and 1km res), the resultant display is in 0.5km resolution. I have to do more testing with this, but you might be correct that computing time is being wasted.
Thanks for reporting these problems -
Bob
It looks like you are correct that the "RGB Composite" display type from the "Three Color (RGB) Image (Auto-scale)" formula doesn't currently allow for Adaptive Resolution (AR). I wrote this up as Inquiry 2281. The other RGB display type, "3 Color (RGB) Image" from the "Three Color (RGB) Image" formula does allow for AR, but you don't have all of the extra options in the Layer Controls for setting the red, green, and blue ranges or the gamma values.
You are correct that there is already an inquiry to allow the user to specify dates/times without having to wait for the entire Absolute times tab to populate. This is inquiry 1798.
As for the RGBs, I found that creating a RGB display using bands 3, 2, and 1 (0.5km, 1km, and 1km res), the resultant display is in 0.5km resolution. I have to do more testing with this, but you might be correct that computing time is being wasted.
Thanks for reporting these problems -
Bob