building new color tables from ascii text?

Post ideas for new functionality you'd like to see in McIDAS-V or ideas for new tutorials.
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

building new color tables from ascii text?

Post by joleenf »

Hi,

I was recently doing some work in McIDAS-V and wanted to use a formula to grab some color levels from a set color table and apply it to different levels in a new color table. Since the new table was not linear, I would not be able to do this easi

1.) A way to import a color table that is built as ascii text format? I would not find it easy to create an xml or binary file that matches the current expected formats.
2.) A way to set breakpoints and construct a color table in the jython shell?

Probably it would be easier to achieve option 1. I used a temporary solution which was to import an old McX color table into McIDAS-V. Use that old color table to pick BRIT values that matched a certain temperature based on an equation, and created a McIDAS-X batch file which would replicate this old BRIT table at color levels which would match my new temperature endpoints. I brought the modified color table into McX using the batch file. Fixed the endpoint colors. Saved that new color table. Imported that back into McIDAS-V. Perhaps there should be a more fluid way of doing this. I am sure there is, but I could not figure out how to build a color table directly in McV using the java docs without messing up my current set of color tables.

Code: Select all


def convertColorTable(inputCT=None, output_file=None, category='User'):
    """
        Usage:  convertColorTable(inputCT='WVJL2', outputCT='WVJL2_163_330', category='User')
 
        inputCT:  this is a color table which is scaling well for pixel "Brightness
                  Values" (UNIT=BRIT or brightness in field selector) values but not for
                  "Brightness Temperature" Values (UNIT=TEMP or Temperature in field selector)
        output_file: desired name and path of output color table as ascii file name.
                  (outputfile will be of form filename.ascii)
                  If no name is provided, the code skips writing any information to
                  a file and does not import a new color table.

        category:  Category name for fileing in color table
                 
    """

    import os
    import ucar.unidata.util.ColorTable as ColorTable

    rgb_list=getColorTableColorList(inputCT)

    if (output_file is not None):
        # create ascii extension for corresponding filename of outputCT (needed for import)
        filename, file_extension = os.path.splitext(output_file)
        if (file_extension != '.ascii'):
            output_file='{}.{}'.format(filename,'ascii')

        outputCT = os.path.splitext(os.path.basename(output_file))[0]
        # Importing twice creates different names, so at least warn that
        # this will happend
        # check color table manager to see if ct in already in manager
        ctm=_mcv.getColorTableManager()
        ct_checkifnew = ctm.getColorTable(outputCT)
        if (ct_checkifnew is not None):
            raise RuntimeError('Color table name already in use: {}'.format(outputCT))

    breakpoints = bilinear(rgb_list)

    if (output_file is not None):
        with open(output_file, 'w+') as fh:
           for breakpoint in breakpoints:
               red=breakpoint[2].getRed()
               grn=breakpoint[2].getGreen()
               blu=breakpoint[2].getBlue()
               rgbLine='{} {} {}\n'.format(red,grn,blu)
               fh.write(rgbLine)
        # import new color table
        test=importEnhancement(filename=output_file, category=category)
    return breakpoints

def getColorTableColorList(color_table):
    """
        Returns a list of RGB colors from an input color table
        Raises Error if color table is not already loaded in McIDAS-V
       
        color_table - color table already in McIDAS-V. 
    """

    import ucar.unidata.util.ColorTable as ColorTable
   
    # load color table
    ctm=_mcv.getColorTableManager()
    ct = ctm.getColorTable(color_table)
    # don't continue if color table is not in list
    if (ct is None):
        raise NameError ('Color Table Not Found In Color Table Manager: {}'.format(color_table))
    else:
        table = ct.getColorTable()
        rgb_list=ct.getColorList()

    return rgb_list

def bilinear(rgb_list):
    """
        Return a zipped array of brightness values corresponding to color level
        after a bi-linear stretch has been applied.
 
        color_list of current McIDAS-V color table (Retrieved by getColorTableList)
                 
    """
    # get scale of newCT from temp to BRIT  (assuming bi-linear scaling applied to 163-330 K)
    minTempScale = 163     # this should remain fixed!!!!!
    maxTempScale = 330     # this should remain fixed!!!!!
    scale = float(maxTempScale - minTempScale)/float(255)
   
    brightness_values=[]
    temperatures=[]
    new_rgb_list=[]

    ct_len=len(rgb_list)
    if (ct_len < 256):
       raise RuntimeError('bilinear stretch expects a color table of 256 colors, this is {}'.format(ct_len))
   
    for temp in range(minTempScale,maxTempScale+1):
        if (temp < 242) or (temp == 330):
            temperatures.append(temp)
        else:
            temperatures.extend([temp,(temp+0.5)])

    for temp in temperatures:
        if (temp < 242):
            # increment by one degree until 242K
            BV = int(418 - temp)
        else:
            BV = int(660 - (2.*float(temp)))

        # these tables came from fortran and I believe that the first index of these tables
        # was actually 1, but here it is 0
        brightness_values.append(BV)

        new_rgb_list.append(rgb_list[BV])
       
    breakpoints=list(zip(brightness_values,temperatures,new_rgb_list))

    return breakpoints


Perhaps there are other ways to approach this that I have not thought about.

Thanks for your help,
Joleen
Last edited by joleenf on Mon Sep 25, 2017 2:25 pm, edited 1 time in total.
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: building new color tables from ascii text?

Post by bobc »

Hi Joleen -

Regarding creating enhancements with Jython (not the Color Table Editor), I've written up McIDAS-V Inquiry 2383. I'm not finding the ability to do something like this currently.

As for the ascii format, you can import 3-column RGB color tables in the ascii format through File>Import in the Color Table Editor. The User's Guide has the following example:

180 0 10
140 0 50
140 0 140
80 0 180
0 120 60
0 140 80

A file containing these lines would be a color table with 6 evenly-sized blocks of color (colored by the RGB values in each line). I'm assuming your case is an ascii enhancement that isn't quite as simple? If that is the case, can you please post your enhancement somewhere for us to investigate?

Thanks -
Bob Carp
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: building new color tables from ascii text?

Post by joleenf »

Hi Bob,

They are not necessarily evenly spaced, but I probably could create a text file that has evenly spaced intervals, with some of the colors repeating over several intervals.

Joleen
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: building new color tables from ascii text?

Post by joleenf »

Hi Bob,

Attached is code which works to convert an old McIDAS-X BRIT color table to brightness temperature scale. This color table should have endpoints of minTemp and maxTemp as built in the jython code. The formula used was pulled from the bi-linear stretchhttp://www.goes-r.gov/products/ATBDs/baseline/Imagery_v2.0_no_color.pdf#page=25. This is the same formula which has been used in McIDAS-X to interpret a color level in BRIT space as a brightness temperature. I found out from TJS, that McX had always stretched these color tables from 163-330K. Using this information, I was able to write the code to convert the old color table and apply it to a Brightness temperature image in McV. I have a few requests to add to the inquiry

1.) While I can bring an ascii table in via importEnhancement, I can't set the min/max range on that import. Therefore, the ascii color table has min/max values of 0-0 upon import. It would be nice to specify 163-330 K.

2.) Considering the difference in how old software packages and new software packages handle color tables, it would be nice to specify if the color table was created for BRIT scaling or temperature scaling. The BRIT version of GA20P would be inverted and linear. The Brightness Temperature version will have the bi-linear scaling as seen in the code. (Add a unit or a descriptor field beyond the name?) Alternately, maybe a color table that is a BRIT space color table should be dynamically converted to brightness temperature space rather than carrying along two color tables.

3.) The color table in McIDAS-V could be more dynamic. There should be one color level per temperature in the new color tables. I probably won't need all 163-330K level, unless I start working at half degrees, but I would like to keep the colors assigned to the temperatures the same while adjusting the range. I realized that I did not want to do this with transparency, but by slicing the section of the color table I need to display (manually, this would be eliminating colors and setting a new range). In this way, I could have one traditional color table, but apply it dynamically to any IR band.

I have attached a few files for testing. There is a file that pending a long development period, we would like to see available to others for use (see comment for CIMSS_GOES_IR.xml).


See http://mcidasv.ssec.wisc.edu/forums/viewtopic.php?f=14&t=1849&sid=6c3e0b104ca3fd12b6fed6218aa19cd1

rescaleCT.py
(4.28 KiB) Downloaded 431 times


Thanks,
Joleen
Attachments
WVJL2.ET
I have imported this one with ranges such as 180-310K, but could foresee needing a different range. This is where the question of dynamic slicing or even dynamic BRIT-Brightness Temperature becomes pertinent.
(3.19 KiB) Downloaded 434 times
CIMSS_GOES_IR.xml
The converted table, the code above should reproduce this with GA20P imported (barring the range issue). TJS has requested that this version be included in McV 1.7 (assuming we are too late for 1.6) under the name CIMSS_GOES_IR. This is the color table which spawned the questions in this post.
(5.98 KiB) Downloaded 531 times
GA20P.ET
Traditional CIMSS GOES IR color table. (20 P stands for 20 panel...).
(3.19 KiB) Downloaded 455 times
Last edited by joleenf on Mon Sep 25, 2017 2:27 pm, edited 3 times in total.
User avatar
ghansham
Posts: 175
Joined: Thu Nov 11, 2010 5:40 pm

Re: building new color tables from ascii text?

Post by ghansham »

Hello everybody
I have been facing these kind of issues and have tried to address these issues in this way.
I have written a java program which converts a given ascii file describing break point color table list into idv colortable xml. converting to jython should not be a big issuen This program takes care of irregularly placed breakpoints by calculating the number of colors based on the minimum breakpoint spacing. I will share that program on this forum. But the basic issue with this approach is that if that if minimum interval is too small for a large range colortable the number of colors in the colortable increase like anything. This has something to do with the approach the color tables are built in idv. The older way of building colortables seem more relevant in case of ascii input. Another issue is open ended color tables. Although visad internally assigns same color for all values below the minima of colortable and similarly for maxima (same color to values above maxima) but they always need min/max. Here we can actually see how GrADS type of software handle colortables.

The other set of modifications I have done is related to the way the colorbar plotting that has been done in ImageGenerator class by adding another tag which is passed from writeImage function.
Have a look at the following link:
http://satellite.imd.gov.in/img/3Dhalfhr_he.jpg

See the colorbar it has lot more than 256 colors and each color has been provided same width although the inrervals are not regularly placed. This to avoid cluttering of labels on colorbar. Even I have done alternative up/down of colorbar labels to avoid mixing up of labels if their width becomes bigger.


Regards
Ghansham
User avatar
TJS
Posts: 90
Joined: Thu Mar 05, 2009 7:31 pm

Re: building new color tables from ascii text?

Post by TJS »

Agreed that this capability is needed, often folks will apply / transfer a given enhancement table and expect to see the same results. Although you know it depends on the ranges (min / max values) and if it's linear, bi-linear, etc. Given that McIDAS-X did a number of maps 'under the hood', folks may not have known that in building the BRIT values, a square-root function was applied for visible bands and a bi-linear (at 242K) for the IR bands. So, these should be able to be replicated in the McIDAS-V.

Thanks,
TJS
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: building new color tables from ascii text?

Post by bobc »

Hi all -

Thanks for noting your concerns and making suggestions about the Color Table Editor in McV. From Joleen's 07/08 post, here are my comments:

  1. I wrote up Inquiry 2385 for adding this functionality. If you are importing an XML enhancement that was originally exported from McV, then the min/max range is saved within the file and applied when the import is done. However, as you have noted, this information is not contained within a 3-column RGB ascii file. This inquiry is for having McV prompt the user for min/max enhancement range values if none are found within the file.
  2. This functionality has been requested in the past and it is written up as Inquiry 1732. I've added a note to the inquiry that the three of you brought it up again.
  3. I wrote up Inquiry 2387 for adding this functionality. The closest request I could find for this in the past was Inquiry 802, which is basically what you are asking for but only for breakpoint colors, not the colors of all points on the enhancement.

Please let me know if you have any more comments, and thanks for your input.

- Bob Carp
User avatar
spipouh
Posts: 2
Joined: Thu Mar 29, 2012 9:21 pm
Location: algerie

Re: building new color tables from ascii text?

Post by spipouh »

Thanks for you a have not a suggestions
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: building new color tables from ascii text?

Post by joleenf »

An updated rescaleCT.py has been posted. There was an error at line 24 of the old code. In the old code, the line was:

newInd = int((temp - 180)/scale)

which should have been

newInd = int((temp - minTemp)/scale)

180 was a carry over from when I had set the minTemp of the color table to 180 instead of using the entire range of the original McIDAS-X color table which was 163-330.

Joleen
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: building new color tables from ascii text?

Post by joleenf »

Hi,

Has there been any progress on Inquiry 2385? I am wondering because the clavrx color tables import as range 0-0 and I could provide both a range and suggested breakpoints if there is a format that would be easier to import.

Thanks,
Joleen
Post Reply