aircraft point data vertical profiles

[kbedka1]

We have a aircraft-based system that can measure vertical profiles of wind at high vertical/spatial resolution. I want to visualize these data as the aircraft is moving around a domain. So we would have aircraft lat/lon/time and then a vertical profile of wind components with altitude at each level. The levels are not yet constant (i.e. 100 m, 200 m, etc..) but they could be in the future if this facilitates visualization. I'd like to view this data in Mc-V. Does Mc-V support such a data model, i.e. a moving point data vertical profile? Ultimately it would be great to see this information in a curtain of user-specified duration. To turn perhaps 100 sets of wind components per profile into a vector or barb, then visualize across an hour could be quite memory intensive. What are your thoughts on all this?

[bobc]

Hi Kris -

The closest thing I can think of to this would be upsonde/dropsonde data, which is loaded through the local files chooser and allows for plotting a vertical profile of vectors as well as a skew-t sounding and hodograph. The files I've tested with have also only included one timestep so it isn't quite the same as what you're asking for.

What is the format of the data? For example: point netCDF, grid netCDF, text, CSV. I'd be interested in seeing a sample of this data if you have any that you can make available.

Thanks -
Bob Carp

[kbedka1]

The file is in csv format, see the attached file. This is one profile of data from a single time. Here is the remainder of the profiles for this flight: ftp://mwsci.jpl.nasa.gov/outgoing/cpex/ ... 1117_PROF/

Filenames are structured YYYYMMDD_HHMMSS1_HHMMSS2_profnum_prof_quick_ALL_SORT.csv

With HHMMSS1=time of instrument power on
HHMMSS2=time of wind profile

[bobc]

Hi Kris -

Thanks for the data URL. I picked up these 12 files:

• 20170611_133843_144946_100_prof_quick_ALL_SORT.csv
• 20170611_133843_145029_101_prof_quick_ALL_SORT.csv
• 20170611_133843_145111_102_prof_quick_ALL_SORT.csv
• 20170611_133843_145402_106_prof_quick_ALL_SORT.csv
• 20170611_133843_145445_107_prof_quick_ALL_SORT.csv
• 20170611_133843_145528_108_prof_quick_ALL_SORT.csv
• 20170611_133843_145611_109_prof_quick_ALL_SORT.csv
• 20170611_133843_145653_110_prof_quick_ALL_SORT.csv
• 20170611_133843_145736_111_prof_quick_ALL_SORT.csv
• 20170611_133843_150109_116_prof_quick_ALL_SORT.csv
• 20170611_133843_150152_117_prof_quick_ALL_SORT.csv
• 20170611_133843_150235_118_prof_quick_ALL_SORT.csv

I saw that these files only have time in the filename, not in the actual data. For McIDAS-V to create a loop of the data, it needs to have a time for each file. There are certain cases with netCDF files where you can write a NcML wrapper file to extract the date/time information from the filename, but I'm not aware of a way to do this with CSV. To get around this, I added a new column to each of the above files for time. I used '20170611' as the year/month/day, and the 2nd block of times as the actual time of the observed data in each file. For example, in the first file above, I added a column of "20170611144946" (yyyyMMddHHmmSS) values for each row.

Once I was done with editing the files, I went to the General>Files/Directories chooser, chose the "Text Point Data files" data type, used Shift+click to select all of the CSV files, and clicked Add Source. This brings up a Point Data window where you tell McIDAS-V what each column of data is.

Point Data window

I didn't go about defining every parameter in the window above, but you could define everything if you want to. All that really has to be defined to plot the barbs/vectors would be Height, Direction, Speed, Latitude, Longitude, and Time. The Missing Value column tells McIDAS-V to treat -999.99 values as missing and not use them. Once you are done defining this, you could use the Preferences button to either write out a header to keep it so you don't have to use this Point Data window next time, or save your entered data by clicking Preferences>Save Current. If you do this, enter a name and click OK. Next time you load in one of these files, in this Point Data window you'll be able to click the Preferences button, choose the name you saved previously, and the window will be populated with the correct values.

Once in the Field Selector, choose the top level Point Data field:

Field Selector

... the Point Data Plot display, and hit Create Display

At this point, there are a couple of things you'll have to do to see all of the data:

1. Uncheck Declutter in the Layer Controls. Note that there does appear to be an issue with decluttering in this case where all of the points are at the same lat/lon, but different altitudes. The only way to view multiple data points at each timestep is to disable decluttering completely, which shows all of the points. I'll write this up as an inquiry if one doesn't already exist.
2. By default, with auto-set projection enabled, McIDAS-V will zoom in to the domain of your displayed data. In my case where I only have 12 files, this is a pretty small domain, so the data points are packed very close together. You can get around this by adjusting the height of the wireframe box, or change your projection to World (which is what I did).

I then went on and created a layout model for wind vectors using the speed/direction fields, and I colored the barbs by speed to get the following animation:

Animation

The first part of this animation is an overhead view, and the second part is a partially rotated view to see the vertical component of the display. If you have any questions about creating the layout model (or any other part of this process), please let me know.

Does it seem like this does what you were looking to accomplish? Are you okay with having to modify these CSV files to add a column for date/time?

Thanks -
Bob

[kbedka1]

I've gotten back to this now. I've created a date/time column for these files and have appended a text header to each file. See example below. I've created a station model for this data ad have linked to the wind speed and direction columns to create the wind barb. I have two observations from what was displayed:

1) Mc-V doesn't seem to recognize the Altitude column. It puts all barbs at the ground
2) It is interpreting -999 as a valid speed and direction. How do I tell Mc-V that -999 is missing?


(index) -> (Time,Altitude,wspd,wdir,junk1,junk2,junk3,junk4,junk5,junk6,junk7,Latitude,Longitude,junk8,junk9,junk10,junk11)
Time[fmt="yyyy-MM-dd HHmm"],wspd,wdir,Altitude[unit="meters"],junk1,junk2,junk3,junk4,junk5,junk6,junk7,Latitude[unit="deg"],Longitude[unit="degrees east"],junk8,junk9,junk10,junk11
2017-06-10 212705,8932.6,139.82,9.0298,-5.8261,6.8989,0,3.9362,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8932.6,257.84,27.896,27.271,5.8762,-999.99,5.8844,1,2,2,26.014,-73.117,93.061,9030.2,263.19,1
2017-06-10 212705,8867.4,257.84,27.896,27.271,5.8762,0,9.8409,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8802.3,259.51,28.372,27.897,5.1672,0,9.1913,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8802.3,234.6,9.5346,7.7715,5.5239,-999.99,29.046,1,2,2,26.014,-73.117,93.061,9030.2,526.39,2
2017-06-10 212705,8737.2,228.66,10.409,7.8146,6.8752,0,29.77,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8672.1,235.5,8.6174,7.1014,4.8815,0,35.764,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8672.1,235.5,8.6174,7.1014,4.8815,-999.99,32.262,1,2,2,26.014,-73.117,93.061,9030.2,789.58,3
2017-06-10 212705,8672.1,234.6,9.5346,7.7715,5.5239,-999.99,35.697,1,2,2,26.014,-73.117,93.061,9030.2,263.19,1

[bobc]

Hi Kris,

A few points about your file:

1. It looks like you have things in the wrong order in your header. In the first line, Altitude is the 2nd parameter, however when you are defining the units, altitude is the 4th parameter. I think this is why all of your data values are plotting at the surface
2. Just from looking at the data values, I believe you have wspd and wdir in the incorrect order in the header.
3. I took your data lines (without the header lines) and used the Point Data window to define the column values, where you can define the a missing value by entering the value in the "Missing Value" column. I entered a value of -999 as a missing value for wspd and wdir in this window, and the header written out from the window defined the missing value like it is in the example below (for example: wspd[unit="m/s" missing="-999"]).
4. For columns of data to skip over, in the Point Data window I didn't enter anything for them. In my header file, the lines to skip over are denoted by "skip[]", whereas yours are defined as "junk1", "junk2", "junk3", etc.

To the lines of data you posted, I added a couple additional lines that included -999 for speed and direction. If the lines below are all saved in the a text file, it loads in as a "Text Point Data files" data source. The data plots in 3D, and the lower vertical values with -999 for speed or direction are ignored when plotting wind barbs.

Code: Select all

(index)->(Time,Altitude,wdir,wspd,Latitude,Longitude)
Time[fmt="yyyy-MM-dd HHmm"],Altitude[unit="meters"],wdir[unit="degrees" missing="-999"],wspd[unit="m/s" missing="-999"],skip[],skip[],skip[],skip[],skip[],skip[],skip[],Latitude[unit="degree"],Longitude[unit="degree"],skip[],skip[],skip[],skip[]
2017-06-10 212705,8932.6,139.82,9.0298,-5.8261,6.8989,0,3.9362,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8932.6,257.84,27.896,27.271,5.8762,-999.99,5.8844,1,2,2,26.014,-73.117,93.061,9030.2,263.19,1
2017-06-10 212705,8867.4,257.84,27.896,27.271,5.8762,0,9.8409,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8802.3,259.51,28.372,27.897,5.1672,0,9.1913,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8802.3,234.6,9.5346,7.7715,5.5239,-999.99,29.046,1,2,2,26.014,-73.117,93.061,9030.2,526.39,2
2017-06-10 212705,8737.2,228.66,10.409,7.8146,6.8752,0,29.77,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8672.1,235.5,8.6174,7.1014,4.8815,0,35.764,1,2,2,26.014,-73.117,93.061,9030.2,131.6,0
2017-06-10 212705,8672.1,235.5,8.6174,7.1014,4.8815,-999.99,32.262,1,2,2,26.014,-73.117,93.061,9030.2,789.58,3
2017-06-10 212705,8672.1,234.6,9.5346,7.7715,5.5239,-999.99,35.697,1,2,2,26.014,-73.117,93.061,9030.2,263.19,1
2017-06-10 212705,4672.1,235.5,-999,7.1014,4.8815,-999.99,32.262,1,2,2,26.014,-73.117,93.061,9030.2,789.58,3
2017-06-10 212705,2672.1,-999,9.5346,7.7715,5.5239,-999.99,35.697,1,2,2,26.014,-73.117,93.061,9030.2,263.19,1

If you have any questions about this, please let me know.

Thanks,
Bob

[kbedka1]

Thank you Bob, sorry for the errant header. The unit= and missing= was what I was looking for and I am now able to get my desired wind barbs. Question: Is it possible to do some sort of gridded "Barnes-scheme" like analysis of point data in a vertical cross section. Imagine I fly a plane around a region and collect wind profiles every X kilometers, can Mc-V have these profiles see each other and create a curtain plot of wind speed throughout the depth of the atmosphere? The profiles will not always have the same number of levels nor will their vertical resolution be the same and we cannot count on every level having valid data, i.e. the sensor can't see through clouds.

[bobc]

Hi Kris,

I've been testing this out with the *prof_quick_ALL_SORT.csv files you posted back in March. What you're asking for currently doesn't seem to be possible, so I wrote adding this functionality as Inquiry 2734.

I was able to get something fairly close:

But it's not exactly what you're looking for. The data on the left are from 5 individual files. If you wanted to take a look at how I created this display, here are some steps:

1. Load in the data the same way as described in my Wed Mar 28, 2018 9:11 am post earlier.
2. By default, creating loop will plot each profile in its own timestep. However, you can adjust the Time Binning settings (through the Properties window of the Data Source) to group all of the data into one timestep.
3. In the Field Selector, I chose the "Point Cloud > Speed" field, the Point Cloud display, and clicked Create Display.
4. By default the size of the points in the display is pretty small. You can enter a larger value into the Point Size text entry field of the Layer Controls and hit Enter to make the points larger.

This isn't the same as a cross section display for several reasons including:

• There's no analysis done to generate the display, it's just the data values in each profile
• You don't have the same controls with the display as you would with a contour display, such as shading colors in a color-shaded display

However, if your data points are pretty close to each other, perhaps something like this could be used to create these displays until the functionality from inquiry 2734 is added.

Please let me know if you have any questions about this.

Thanks,
Bob