Himawari server very slow

[kbedka1]

I have loaded in a single 1000x1000 Himawari-8 image from the geoarc archive server. I am finding that the channel and dataset selection is VERY slow. See attached screenshot of an RGB I am trying to do. It has taken me several minutes just to be able to select the data fields I want to use. For example, if I select a channel, and then the parameter (i.e. "Brightness"), the software hangs for at least a minute to get the domain information (I'm guessing that is what it's doing). This is not a problem for other satellites I've worked with, especially if only for a single relatively small image. Has anyone else noticed this? I am on a very fast network connection so that could not be the problem.

[bobc]

Hi Kris -

I've noticed the slowness of this archive Himawari dataset as well. I ran a comparison between times working with this archive Himawari data and AGOES15 (both on geoarc) and AGOES15 is a good bit faster. I verified with the SSEC data center that the reason for this is the amount of files on the server for the datasets. For example, AGOES15/FD has only 8 times for one day, whereas AHIM08/FD has 142 times. In addition to this, the Himawari data has 10 segments per band at each timestep, so there are a lot more files in this AHIM08 dataset compared to AGOES15.

Please let me know if you have any questions.

Thanks -
Bob Carp

[kbedka1]

If the number of files per day were an issue, then AGOES14 with 1-min data would be the slowest of all, but it is quite fast if a person is only dealing with 1 file at a time as I am here with AHIM. The performance I'm seeing with 1 file from AHIM is comparable to an aggregated 100+ GOES-14 1-min files. Now, the 10 segments per file is probably the key difference between Himawari and GOES and perhaps efforts could be focused on speeding up how the segments are handled. With the current setup with AHIM, building a multichannel RGB over maybe a 2 hour period (the typical lifetime of an individual thunderstorm for example) would be so frustrating that I probably wouldn't take the time to bother with it. This is not a very good situation given the level of excitement for Himawari and GOES-R.

[hproe]

Hi Bob and Kris -

Not sure whether my recent exprience is useful information.

Kris, bringing up the slow AHIM08/FD server, reminds me that around 30 Nov / 1st Dec I noted the slowness as well. But during these days also other servers, like AMET10/FD, were slow, actuially so slow I could not go reasonably beyond loading single time slots. I started to look into my machine (having just come back from a week in another network I suspected some local setup problem), but while checking the machine speeds went back to normal again. Rigth now I cannot check as my workhorse is suffering from a hardware problem.

best wishes for the festivities, HP

[bobc]

Hi Kris -

I've sent your concerns about the speed of the geoarc/AHIM08 dataset to a couple programmers. I'll let you know when I hear anything back from them.

Thanks -
Bob

[beckys]

Hi Kris,

In your example, I see that you are accessing data that is only a couple of days old. If you're accessing data on the AHIM08 archive server that is still available on the real-time server, you will get much better performance using the real-time server (HIMAWARI at himawari.ssec.wisc.edu). We have noticed that the Himawari archive server is a lot slower than the real-time server, both with ADDE and with moving files around using unix commands. The Data Center is aware of the problem and looking into it, but they have not yet found a solution.

Becky

[bobc]

Hi Kris -

If you are still trying to work with archived Himawari data, one workaround that you can use for now is the QHIM08 dataset on geoarc. With this dataset, there are the following Image Type items available:

ALL ALL Himawari8 data
FD01 Channel 01 Full Disk Himawari8 data
FD02 Channel 02 Full Disk Himawari8 data
FD03 Channel 03 Full Disk Himawari8 data
FD04 Channel 04 Full Disk Himawari8 data
FD05 Channel 05 Full Disk Himawari8 data
FD06 Channel 06 Full Disk Himawari8 data
FD07 Channel 07 Full Disk Himawari8 data
FD08 Channel 08 Full Disk Himawari8 data
FD09 Channel 09 Full Disk Himawari8 data
FD10 Channel 10 Full Disk Himawari8 data
FD11 Channel 11 Full Disk Himawari8 data
FD12 Channel 12 Full Disk Himawari8 data
FD13 Channel 13 Full Disk Himawari8 data
FD14 Channel 14 Full Disk Himawari8 data
FD15 Channel 15 Full Disk Himawari8 data
FD16 Channel 16 Full Disk Himawari8 data
JAPAN Japan Sector Himawari8 data
TARGET Target Sector Himawari8 data

I ran a test where I loaded in a 06:00Z time off AHIM08/FD and QHIM08/FD01 and displayed band 1 full res. There was quite an improvement when using QHIM08. Here are the results of my tests. This is a list ordered by the step completed; the time for AHIM08 to complete the step; the time for QHIM08 to complete the step:

* Time from date selected until Absolute times tab populated; 56 seconds; 17 seconds
* Time from Add Source until Field Selector comes up and is populated; 13 seconds; 2 seconds
* Time from selecting a band/calibration until Create Display button available; 57 seconds, 9 seconds
* Time to display band 1 full resolution; 1 min 1 second; 24 seconds
* Total time; 3 min 7 seconds; 52 seconds

Using QHIM08 would mean that you would need an individual data source for each of the 6 bands you are using in your formula, but there should still be a good improvement in the overall time that it takes to create your display.

If you give QHIM08 a try please let us know how it goes.

Thanks -
Bob Carp