unpacked hdf in hydra

[kbedka1]

I recently had submitted a bug report via the mug email list about destriped, decompressed MODIS Level 1B data not working in McIDAS-V. This data when ordered from NASA in internally compressed format works fine. Liam Gumley's destriping algorithm only works on the decompressed data. I really need to view and make a graphic with the destriped decompressed data for a publication I am working on. Do you have an estimate on when a fix for this might be ready?

[tdrink]

So you start with the DAAC files, and the destriper spits out another HDF file?
In general arbitrary HDF files aren't supported, but we're going to do some
development to help with this. If you send me the a destriped file, I'll see
if we can get an update in the nightly - I'd have to see the file first.

[kbedka1]

I have to unpack first before destriping. Liam's destriper spits out a nearly identical file to the original. It definitely conforms to EOS-HDF. I put a sample file at: ftp://ftp.ssec.wisc.edu/pub/incoming/MY ... f_unpacked

[tdrink]

Hi Kris,

When you say 'de-compressed', I think you're really saying that the radiance
scales and offsets have been applied before the de-striping algorithm. However,
the destriped data is internally compressed, and this is causing a problem. Do
you have the option to just write-out floats for the destriped arrays? Otherwise,
it looks like we can't handle hdf4 compression right now.

Tom

[tdrink]

Here's a thought:

Maybe the destriped data should be re-scaled using the same scale/offsets as the
original file into the new file? That way, all the software in the world made to
work with MO/MYD02*hdf would not have to be modified?

If you can not hdf compress the data, the HYDRA data should work as it would
not apply any scales and offsets that don't exist.

[kbedka1]

no, the destriped data is not internally compressed. The file I passed to you was already uncompressed and destriped. I used the hrepack software to unpack the data:

http://nsidc.org/data/hdfeos/hrepack.html

Mc-V is handling the "compressed" versions of the file just fine, when I unpack it though, this is where the problems start.

[tdrink]

Now I'm totally confused: the files you put on the ftp server don't work
with Java-NetCDF library, with errors complaining about unrecognized
compression, and errors with tiling. Java-NetCDF is how we access
HDF4/5, so they aren't going to work in their current form real soon.
Maybe the problem is that hrepack uses szip, which is proprietary,
and thus not part of the Java-NetCDF library? Maybe just skip the
unpacking? Like I said, it would be best if the destriper just made
the hdf file the same way, except destriped.

[kbedka1]

I think it would be helpful to have both files for comparison:

Packed (i.e. compressed version)
ftp://ftp.ssec.wisc.edu/pub/incoming/MY ... 151020.hdf

Unpacked via hrepack and destriped
ftp://ftp.ssec.wisc.edu/pub/incoming/MY ... f_unpacked

The destriping program is distributed via the IMAPP package and is designed to deal with direct broadcast data. I think that package is pretty fixed in its configuration since it has already been adopted worldwide and developers are likely not interested in dealing with decompression -> destripe -> compression. The file must be decompressed before destriping or it will not work within the software. I've spoken with Liam about the details of this package, some feedback is provided below. The bottom line is that data from the water vapor channels is unpresentable, destriped data has to be supported by Mc-V somehow. I've created all figures for a journal article in Mc-V except one that uses the WV channel, so this hiccup is holding me up. Maybe someone can take the destriping code and turn it into a formula?

***************
There is no difference in the HDF structure of a destriped file L1B 1KM file vs. a non-destriped file, other than

(a) the scaled integer values in the destriped bands are different
(b) there is one extra global attribute to identify the file as having been destriped

You could ask Tom to run a HDF4 'ncdump -h' on a pristine LAADS L1B 1KM file and the uncompressed destriped version of the same file and compare the results; I believe it will show that the HDF structure is identical except for the one additional global attribute in the destriped file.

I have never heard of any software system (IDL, Matlab, C, Fortran, Python, McIDAS-X, Hydra standalone) that could not read the uncompressed destriped files (other than McV).
***************

[tdrink]

I've sent a note to the Java-NetCDF developers. I'll keep you posted, I think
this is really just a dumb bug in their code. When we get the fix, we can decide
the best way for you to use it.

[tdrink]

There's a fix in the works for the Java-NetCDF library, should be ready
in a day or two.