usevortex
— Proxies to reach resources using Vortex¶
Contains proxies to reach resources using Vortex.
Of course, this module need to have a proper Vortex installation !
Functions¶
-
usevortex.
quiet_get
(*args, **kwds)[source]¶ Shut off some loggers (set level to ERROR) while executing action, then restore initial level.
Example of use:
with quiet_get(): get_resources(...)
-
usevortex.
get_resources
(getmode=u'epygram', uselocalcache=False, **description)[source]¶ Get resources, given their description.
Parameters: - getmode –
controls the action and returned object:
- ‘epygram’ return the epygram resources
- ‘locate’ return the physical resolved location of the resource
- ‘exist’ return the physical resolved location of the resource and its existence
- ‘fetch’ fetches the resource in local, as filename local
- ‘vortex’ return the vortex data handler of the resource
- ‘prestaging’ puts a prestaging request for the resolved resources, on the archive system
- uselocalcache – if True, store resources in a local cache (not automatically cleaned, take care) defined either (and by priority order) in $MTOOL_STEP_CACHE, $MTOOLDIR, $FTDIR, $WORKDIR, $TMPDIR.
- meta_rtype – if not None and getmode is ‘epygram’, return the resource(s) as meta_resource. Cf. epygram.resources.meta_resources() for documentation. [Not implemented yet]
Examples:
- for the analysis of AROME-france from experiment 864G on 2015/08/15/00,
description will look like:
- experiment=‘864G’, # the experiment id
- model=’arome’,
- block=’analysis’, # the OLIVE block
- kind=’analysis’, # the kind of resource
- date=‘2015081500’, # the initial date and time
- geometry=’franmgsp’, # the name of the model domain
- local=’analysis_[experiment]’, # the local filename of the resource, once fetched.
- for the model state at term 18h of ALADIN-reunion oper, production
cutoff, on 2015/08/15/00,
description will look like:
- suite=’oper’, # the suite // ‘dble’ = e-suite
- kind=’historic’, # model state
- date=‘2015081500’, # the initial date and time
- term=18, # the forecast term
- geometry=’reunionsp’, # the name of the model domain
- local=’ICMSHALAD_[term]’, # the local filename of the resource, once fetched.
- cutoff=’prod’, # type of cutoff
- vapp=’aladin’, # type of application in operations namespace
- vconf=’reunion’, # name of config in operation namespace
- model=’aladin’, # name of the model
- for the GRIB post-processed output on MASCA025 BDAP domain at terms
22->24h of ALADIN-reunion oper, production cutoff, on 2015/08/15/00,
description will look like:
- suite=’oper’, # the suite // ‘dble’ = e-suite
- kind=’gridpoint’, # model state
- date=‘2015081500’, # the initial date and time
- term=[22,23,24], # the forecast terms
- geometry=’masca025’, # the name of the post-processing domain
- local=’[geometry]_[term].grb’, # the local filename of the resource, once fetched.
- nativefmt=’grib’ # to fetch the gribs and not the FA post-processed files
- cutoff=’prod’, # type of cutoff
- origin=’hst’, # origin of post-processed files: historic
- vapp=’aladin’, # type of application in operations namespace
- vconf=’reunion’, # name of config in operation namespace
- model=’aladin’, # name of the model
- getmode –
-
usevortex.
extractor
(vortex_description, coords, start_cutoff, end_cutoff, start_term, end_term, points_fields=None, profiles_FAfields=None, everycutoff_in_hours=24, everyterm_in_hours=1, vertical_coordinate=None, extraction_options=None, use_local_cache=False, prestaging=False, outdir='/sauvegarde/mary/git/EPyGrAM/epygram/doc_sphinx/source', progressmode=None, nc_global_attributes={'institution': u'M\xe9t\xe9o France', 'history': u'Extracted on 2018-03-06 using epygram.'})[source]¶ Extract temporal series of points (and profiles for FA historical files) to netCDF files.
- Series are either:
- one series == one file per cutoff if start_term* != **end_term
- one series == one file containing all cutoffs if start_term* == **end_term
Shall be extended to series of profiles for other formats later on.
Parameters: - vortex_description –
vortex description of the resources, incl. providers, excepted dates/terms Example: {‘model’:’arome’,
‘vapp’:’arome’, ‘vconf’:‘3dvarfr’, ‘kind’:’historic’, ‘suite’:’oper’, ‘geometry’:’franmgsp’, ‘cutoff’:’prod’, #’namespace’:’oper.archive.fr’, #’experiment’:‘86U8’, #’block’:’forecast’, #’nativefmt’:’fa’, #’member’:8,} - coords – either a list of lon/lat coordinates or a dict of, in which case key would be a “name of the point”, used in output filename
- start_cutoff – first cutoff of the series (datetime.datetime instance)
- end_cutoff – last cutoff of the series
- start_term – first term of the series (datetime.datetime instance)
- end_term – last term of the series
- points_fields –
dictionnary of fields to extract with options, e.g.:
- {fid1:{‘nc_name’:nc_fid1,
- ‘nc_attributes’:{‘unit’:...,
- ‘comment’:..., },
‘decumulate’:True, ‘operation’:(‘-‘,273.15) or (‘exp’,), ‘reproject_wind_on_lonlat’:False, },
fid2:... }
- ‘decumulate’ can be absent==False, True, or ‘center’ for centered decumulation (cf. D3Field.decumulate for more details);
- ‘operation’ is done after decumulation, if both present
- ‘reproject_wind_on_lonlat: FA only: reproject winds on lonlat axes (True by default)
- profiles_FAfields – like points_fields but for profiles, but only for FA historic resources ! fids supposed to be ‘S*PARAMETER’
- everycutoff_in_hours – step between 2 cutoffs
- everyterm_in_hours – step between 2 terms
- vertical_coordinate – vertical coordinate requested for profiles, among (‘pressure’, ‘height’, ‘altitude’, None)
- extraction_options – options to FA.extractprofile() and H2DField.extract_point() concerning interpolation, as dict...
- use_local_cache – use a local Vortex cache, in which files are stored
- prestaging – triggers prestaging for the requested resources (Hendrix)
- outdir – directory in which to store output files
- progressmode – sets the verbosity of the task among (None, ‘percentage’, ‘verbose’)
- nc_global_attributes – given as a dict. Defaults are above in module variable default_nc_global_attributes
Classes¶
-
class
usevortex.
Extractor
(*args, **kwargs)[source]¶ Bases:
taylorism.Worker
Independant extractor.
Note
This class is managed by footprint.
- info: Run extractor().
- priority: PriorityLevel::DEFAULT (rank=1)
Automatic parameters from the footprint:
- directives (
footprints.stdtypes.FPDict
) - rxx - Contains all arguments to extractor(...). - expected_time (
float
) - rxx - How long the worker is expected to run (in s).- Optional. Default is 0.0.
- memory (
float
) - rxx - Memory that should be used by the worker (in MiB).- Optional. Default is 0.0.
- name (
str
) - rwx - Name of the worker.- Optional. Default is None.
- scheduler_hooks (
footprints.stdtypes.FPList
) - rxx - List of callbacks before starting effective task work.- Optional. Default is FPList::<<as_list:: []>>.
- scheduler_ticket (
int
) - rxx - The slot number given by the scheduler (optional).- Optional. Default is None.
Shortcuts¶
-
usevortex.
get_gribfc_arome_oper
(date, term, geometry=u'FRANGP0025', **others)[source]¶ Proxy for AROME oper GRIBs.
-
usevortex.
get_gribfc_arome_xp
(xp, date, term, geometry=u'FRANGP0025', **others)[source]¶ Proxy for AROME experiment GRIBs.
-
usevortex.
get_histfc_arome_xp
(xp, date, term, **others)[source]¶ Proxy for AROME experiment historic FAs.
-
usevortex.
get_gribfc_arpege_oper
(date, term, geometry=u'FRANX01', **others)[source]¶ Proxy for ARPEGE oper GRIBs.