Latitude and longitude of globir area data

How do I...?
Post Reply
User avatar
jconforte
Posts: 8
Joined: Wed Apr 20, 2022 11:36 am

Latitude and longitude of globir area data

Post by jconforte »

Hi,
After several years I am using Mcidas V again. I already asked some questions how to get a latitude and longitude file of my image. In Mcidas V window I clicked on the Field Selector icon, then on the General, Grid Table and Create Display icons. And get a new window with the latitude and longitude values ​​and the values ​​of my image. It is a file of 5500 rows by 9900 columns. My question is is it possible to save this file. Attached is a print screen of these values and the figure of my data ​​that we got with Mcidas V.
Attachments
Image_data_set-Grid_Table.png
globir_05_11_2022_1200.png
User avatar
bobc
Posts: 987
Joined: Mon Nov 15, 2010 5:57 pm

Re: Latitude and longitude of globir area data

Post by bobc »

Hello,

We have an inquiry written up to able to export the output of a Grid Table display to CSV/Excel format. However, this functionality is not currently available. In the past, we've suggested that users click in one of the cells of the Grid Table display, do Ctrl+A to select all of the cells, then Ctrl+C to copy the table. This table can then be pasted into Excel or another application. You can give this a try, though some tables may be too large to do this copy/paste.

If you are working with gridded data, there is a formula "Export > Export Grid to Excel" that you can try.

If you are using satellite imagery from an ADDE server, you can try using something like the following script to write a CSV file with rows for Latitude, Longitude, and Data Value. You will need to update the directory path to the CSV file. I was able to run this in the Jython Shell and write out a CSV file with more rows/columns than I could copy/paste from the Grid Table display:

Code: Select all

# load in some satellite data
addeParms = dict(
    server = 'adde.ucar.edu',
    dataset = 'RTGOESR',
    descriptor = 'CONUS',
    band = 13,
    unit = 'TEMP',
    size = 'ALL',
    )
data = loadADDEImage(**addeParms)

# get at the lat/lon values of the data
domainSet = getDomainSet(data)
dataLatLons = getLatLons(domainSet)
lats = dataLatLons[0]
lons = dataLatLons[1]

# get all of the data values
dataVals = getValues(data)

# initialize empty lists for lat, lon, values
lat = []
lon = []
val = []

# create individual lists for lat, lon, and data values
for x in range(0, len(lats)):
    lat.append(lats[x])
    lon.append(lons[x])
    val.append(dataVals[0][x])
    
# write lat, lon, and data values to csv file
import csv
header = ['Latitude', 'Longitude', 'Value']
with open('C:/Users/rcarp/sat_data_conus.csv', 'wb') as csv_out:
    mywriter = csv.writer(csv_out)
    mywriter.writerow(header)
    rows = zip(lat, lon, val)
    mywriter.writerows(rows)

# print 'done' so the user knows the script is complete
print('done')
There are several resources available to learn more about scripting in McIDAS-V, including the User's Guide and the Scripting tutorials on our Documentation page. The Advanced Scripting tutorial covers a good bit of the script I posted above outside of writing the data lists to a CSV file.

Thanks,
Bob Carp
McIDAS User Services
User avatar
jconforte
Posts: 8
Joined: Wed Apr 20, 2022 11:36 am

Re: Latitude and longitude of globir area data

Post by jconforte »

Mr. Bob Carp,

I did that you wrote Grid Table display, do Ctrl+A to select all of the cells, then Ctrl+C to copy the table. But, I didn't had success, the tables may be too large to do this copy/paste. Then I use the option:

formula "Export > Export Grid to Excel" and I save it the name globir.xlm, attached it is a copy of this file. How can you see, I don't have the latitude and longitude grid.

The data that I'm using I downloade from NASA's Earthdata, This data are in Area format, it is a mosaic of several geostarionary satellites.
I already use a python script to read this data. IN McIDAS-V - Data Explorer -> File -> Save I save the data displayed as Netcdf and I have these two mcidas_area in my file,

mcidas_area:AreaDirectory = 0., 4., 11., 122132., 0., 7162., 5051., 0., 5400., 9900., 1., 1., 1., 1., 0., 0., 122132., 1854., 8., 0., 0., 0., 0., 0., 538976288., 538976288., 538976288., 538976288., 538976288., 538976288., 538976288., 538976288., 9076., 768., 256., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 122132., 0., 0., 0., 0., 0., 1447646034., 1112688980., 0., 0., 0., 1447646034., 538976288., 1., 0., 0., 0., 0., 2. ;
mcidas_area:NavBlock = 1296388675., 10000., 10000., 0., 4000., 1100000., 6378137., 81819., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ;

I would like know how the McIDAS V use the mcidas_area to navigate the image to get the latitude and longitude values for each pixel.

I'm not using using satellite imagery from an ADDE server, then I don't know how can I use the script tthat you send me

Thanks for your help,

Conrado
Attachments
globir.xml
(435 Bytes) Downloaded 49 times
User avatar
bobc
Posts: 987
Joined: Mon Nov 15, 2010 5:57 pm

Re: Latitude and longitude of globir area data

Post by bobc »

Hello,

It would be helpful to have one of your AREA files for me to test with. Would you mind posting one to our ftp? Here are instructions for doing that through a Terminal window (from the directory containing the AREA file):

ftp ftp.ssec.wisc.edu
user: anonymous
password: email
bin
prom
cd pub/incoming
put filename
bye

Please let me know the file's name once it's posted and I'll give it a look.

Thanks,
Bob
User avatar
jconforte
Posts: 8
Joined: Wed Apr 20, 2022 11:36 am

Re: Latitude and longitude of globir area data

Post by jconforte »

Mr Bob,

I already put the file in the directory and the name of file is globir.22132.0000.

Thanks for all help,

Conrado
User avatar
bobc
Posts: 987
Joined: Mon Nov 15, 2010 5:57 pm

Re: Latitude and longitude of globir area data

Post by bobc »

Hello,

Thanks for the file. I wrote the following script that I believe writes out all of the lat, lon, and data points. Here is the script:

Code: Select all

# specify data and output directories
dataDir = 'C:/Users/rcarp/conrado/data'
outDir = 'C:/Users/rcarp/conrado/output'

# create local dataset
localParms = dict(
    dataset = 'AREA',
	imageType = 'Conrado',
	save = True,
	mask = dataDir,
	format = 'McIDAS Area',
	)
makeLocalADDEEntry(**localParms)

# get local entry
desc = getLocalADDEEntry(dataset = 'AREA', imageType ='Conrado')

# load in some satellite data
addeParms = dict(
    localEntry = desc,
	size = 'ALL',
    )
data = loadADDEImage(**addeParms)

# get at the lat/lon values of the data
domainSet = getDomainSet(data)
dataLatLons = getLatLons(domainSet)
lats = dataLatLons[0]
lons = dataLatLons[1]

# get all of the data values
dataVals = getValues(data)

# initialize empty lists for lat, lon, values
lat = []
lon = []
val = []

# create individual lists for lat, lon, and data values
for x in range(0, len(lats)):
    lat.append(lats[x])
    lon.append(lons[x])
    val.append(dataVals[0][x])
print('lists created')

# break up lat, lon, val lists into smaller chunked lists
chunk_size = 1000
chunked_lat_list = []
for i in range(0, len(lat), chunk_size):
    chunked_lat_list.append(lat[i:i+chunk_size])
chunked_lon_list = []
for i in range(0, len(lon), chunk_size):
    chunked_lon_list.append(lon[i:i+chunk_size])
chunked_val_list = []
for i in range(0, len(val), chunk_size):
    chunked_val_list.append(val[i:i+chunk_size])

# write lat, lon, and data values to csv file
import csv
header = ['Latitude', 'Longitude', 'Value']
with open('%s/sat_data.csv' % outDir, 'wb') as csv_out:
    mywriter = csv.writer(csv_out)
    mywriter.writerow(header)

print('Done with header')

for x in range(0, len(chunked_lat_list)):
    with open('%s/sat_data.csv' % outDir, 'a') as csv_out:
        mywriter = csv.writer(csv_out)
        rows = zip(chunked_lat_list[x], chunked_lon_list[x], chunked_val_list[x])
        mywriter.writerows(rows)

print('Script done')
You'll want to change a couple things:
  • Change dataDir to point at your data, and outDir to point to where you want to write the CSV
  • If you want to, you can change the "dataset" and "imageType" variables in the localParms dictionary. If you do this, you'll need to change those variables in getLocalADDEEntry as well
This script is pretty similar to the last script I sent you, except for:
  • Instead of going to a remote dataset, I created a local dataset and called it through loadADDEImage
  • Writing this many lines (more than 53 million) to a CSV isn't very efficient. To make things a bit quicker, I broke up these very long lat/lon/val lists into smaller lists of 1,000. This was done in the section with the comment "break up lat, lon, val lists into smaller chunked lists"
I ran this script in the background and it took ~8 minutes to complete. Here was the command I ran:
runMcV -script C:/Users/rcarp/conrado/script.py

The output file was around 2.36GB, which is too large for Excel to open on my machine. A quick Google search showed a couple suggested other applications to view such a large file, but I haven't investigated them yet.

If you don't need every data point, there are a couple things you can try to reduce the time it takes to write the CSV, as well as the size of the CSV. For example:
  • If you are only interested in a particular geographic area, you can specify a center lat/lon and x/y size in loadADDEImage
  • If you want data over the entire scene, but could use every 2nd, 3rd, etc point, you can specify MAG= and SIZE= in loadADDEImage
Thanks,
Bob
User avatar
jconforte
Posts: 8
Joined: Wed Apr 20, 2022 11:36 am

Re: Latitude and longitude of globir area data

Post by jconforte »

Mr. Bob Carp,

I have a copy your script.py in /home/conrado and I would like to ask you. Where in this script I set the name of my file to be read. The file name is globir.22133.0000. My data globir.22133.0000 is in my /home/conrado/.

To run the script I did:

./runMcV -script /home/conrado/script.py

But, I didn't have no message for the print commands in this script and my .csv files were not saved.

my data globir.22133.0000 is in my /home/conrado/.

I'm sorry for this, I don't have much experience with running python scripts in Mcidas V.

Thanks,

Conrado
User avatar
bobc
Posts: 987
Joined: Mon Nov 15, 2010 5:57 pm

Re: Latitude and longitude of globir area data

Post by bobc »

Hello,

No worries. You specify your directory paths in lines 2 and 3 of the script. Note that this just points to a directory, not the actual filename. Here is how I have it set:

Code: Select all

# specify data and output directories
dataDir = 'C:/Users/rcarp/conrado/data'
outDir = 'C:/Users/rcarp/conrado/output'
I'd suggest you make a new directory for the data and output. For example, you could create a directory '/home/conrado/mosaic'. In this directory, you could have directories for 'area' and 'output'. Move your globir.22133.0000 file to /home/conrado/mosaic/area, and set the dataDir/outDir to:

Code: Select all

# specify data and output directories
dataDir = '/home/conrado/mosaic/area'
outDir = '/home/conrado/mosaic/output'
Give this a try and let me know how it goes. Note that it may take several minutes to finish writing the CSV file. If the runMcV process completes and you don't see a CSV file, you could go to /home/conrado/McIDAS-V and look at the mcidasv.log file.

One specific thing to check is that the indentations are all done by spaces instead of tabs. I ran into a problem with copying/pasting where some lines were tab-indented and some were space indented. I would make sure that the single indentations are done with 4 individual spaces instead of 1 tab, and the double indentations (in the "with open" loop near the end of the script) are done with 8 individual spaces instead of 2 tabs.

Thanks,
Bob
User avatar
jconforte
Posts: 8
Joined: Wed Apr 20, 2022 11:36 am

Re: Latitude and longitude of globir area data

Post by jconforte »

Mr. Bob Carp,

I used the script that you send me,is is attached. In my directory:

/home/conrado/McIDAS-V-System

I did ;

./runMcV -script /home/conrado/McIDAS-V-System/scriptrc1.py

and I had this message:

Reading system configuration...
sys mem: 7743
"/home/conrado/McIDAS-V-System/jre/bin/java" -Xmx6194M -Djava.ext.dirs="/home/conrado/McIDAS-V-System/jre/lib/ext" -Djava.library.path="/home/conrado/McIDAS-V-System/jre/lib/ext" -Dloglevel="INFO" -Dlogback.configurationFile="/home/conrado/McIDAS-V/logback.xml" -Dmcv.userpath="/home/conrado/McIDAS-V" -Dmcv.logpath="/home/conrado/McIDAS-V/mcidasv.log" -Dfile.encoding=UTF-8 -Dpython.security.respectJavaAccessibility=false -jar ./mcidasv-1.8-SNAPSHOT.jar -Dvisad.java3d.noerasebackground=true -Dvisad.java3d.textureNpot=false -Dvisad.java3d.imageByRef=true -Dvisad.java3d.geometryByRef=true -Didv.3d=true -DtextureWidthMax=4096 -userpath "/home/conrado/McIDAS-V" -bundle "0;" -script /home/conrado/McIDAS-V-System/scriptrc1.py

After some time the script ended and the .csv files were not created. In this script there are some print commands and I didn't have the messages of this prints commands.

the image is in the directory: /home/conrado/mosaic/area/ ----> globir.22133.0000

Why i was not successful in running this script.

Thanks,

Conrado
Attachments
scriptrc1.py
(2.11 KiB) Downloaded 53 times
User avatar
bobc
Posts: 987
Joined: Mon Nov 15, 2010 5:57 pm

Re: Latitude and longitude of globir area data

Post by bobc »

Hello,

There are a couple things I see wrong with your script when writing all of the rows to the CSV file. You should make a couple changes to line 100 in your script:

Code: Select all

with open('/home/conrado/mosaic/sat_data1.csv', 'w') as csv_out:
  1. You want to write to the same file that the header was written to above. The header just gives labels to the columns, and then the above line writes out the actual data values.
  2. You are using "w" mode (after the CSV file path). You want to change 'w' to 'a'. 'a' appends data to an existing file.
The changes above would make line 100 in your script:

Code: Select all

with open('/home/conrado/mosaic/sat_data.csv', 'a') as csv_out:
Outside of this change, and changing the directory paths to point to where the data lives on my machine, I was able to run your script in core McIDAS-V 1.8. A couple things to check/try include:
  1. Look in your /home/conrado/McIDAS-V (not McIDAS-V-System) directory for a mcidasv.log file. After running the script and not getting any output, look at this file and see if there are any error messages to help diagnose the problem. Note that print lines in the script don't go to the Terminal window, but they will show up in mcidasv.log.
  2. You can try running your script in the foreground through the Jython Shell. The script will likely still fail if it fails from the background, but you may be able to see where it fails in realtime if you add print lines throughout your script. You will also see the output of your print lines in the Jython Shell.
  3. If you aren't seeing any CSV files written, it's possible that the local dataset isn't being set up correctly. You could test this local dataset in the foreground by going to the Satellite>Imagery chooser, connecting to <LOCAL-DATA> / AREA and seeing if you can access the local dataset this way.
Note that when running through the Jython Shell, there are several times early on in the script where McIDAS-V becomes unresponsive on my machine due to the amount of data it is processing through to create the lat/lon/data lists. However, after a bit of time, the script does run fine on my machine in the Jython Shell. There's a chance this is a resource issue if you don't have enough memory on your machine, but error messages in mcidasv.log or the Jython Shell should indicate this.

Thanks,
Bob
Post Reply