usevortex — Proxies to reach resources using Vortex

Contains proxies to reach resources using Vortex.

Of course, this module need to have a proper Vortex installation !

---

Functions

usevortex.set_defaults(**defaults)

Set defaults key/value pairs for get_resource().

usevortex.quiet_get(loggers=['vortex.data.stores', 'vortex.data.handlers', 'vortex.tools.net'])

Shut off some loggers (set level to ERROR) while executing action, then restore initial level.

Example of use:

with quiet_get():
    get_resources(...)

usevortex.get_resources(getmode='epygram', uselocalcache=False, **description)

Get resources, given their description.

Parameters:

• getmode –

  controls the action and returned object:

  • ’check’ return True only if the description is complete
  • ’epygram’ return the epygram resources
  • ’locate’ return the physical resolved location of the resource
  • ’exist’ return the physical resolved location of the resource and its existence
  • ’fetch’ fetches the resource in local, as filename local
  • ’vortex’ return the vortex data handler of the resource
  • ’prestaging’ puts a prestaging request for the resolved resources, on the archive system
• uselocalcache – if True, store resources in a local cache (not automatically cleaned, take care) defined either (and by priority order) in $MTOOL_STEP_CACHE, $MTOOLDIR, $FTDIR, $WORKDIR, $TMPDIR.
• meta_rtype – if not None and getmode is ‘epygram’, return the resource(s) as meta_resource. Cf. epygram.resources.meta_resources() for documentation. [Not implemented yet]

Examples:

• for the analysis of AROME-france from experiment 864G on 2015/08/15/00, description will look like:
  • experiment=‘864G’, # the experiment id
  • model=’arome’,
  • block=’analysis’, # the OLIVE block
  • kind=’analysis’, # the kind of resource
  • date=‘2015081500’, # the initial date and time
  • geometry=’franmgsp’, # the name of the model domain
  • local=’analysis_[experiment]’, # the local filename of the resource, once fetched.
• for the model state at term 18h of ALADIN-reunion oper, production cutoff, on 2015/08/15/00, description will look like:
  • suite=’oper’, # the suite // ‘dble’ = e-suite
  • kind=’historic’, # model state
  • date=‘2015081500’, # the initial date and time
  • term=18, # the forecast term
  • geometry=’reunionsp’, # the name of the model domain
  • local=’ICMSHALAD_[term]’, # the local filename of the resource, once fetched.
  • cutoff=’prod’, # type of cutoff
  • vapp=’aladin’, # type of application in operations namespace
  • vconf=’reunion’, # name of config in operation namespace
  • model=’aladin’, # name of the model
• for the GRIB post-processed output on MASCA025 BDAP domain at terms 22->24h of ALADIN-reunion oper, production cutoff, on 2015/08/15/00, description will look like:
  • suite=’oper’, # the suite // ‘dble’ = e-suite
  • kind=’gridpoint’, # model state
  • date=‘2015081500’, # the initial date and time
  • term=[22,23,24], # the forecast terms
  • geometry=’masca025’, # the name of the post-processing domain
  • local=’[geometry]_[term].grb’, # the local filename of the resource, once fetched.
  • nativefmt=’grib’ # to fetch the gribs and not the FA post-processed files
  • cutoff=’prod’, # type of cutoff
  • origin=’hst’, # origin of post-processed files: historic
  • vapp=’aladin’, # type of application in operations namespace
  • vconf=’reunion’, # name of config in operation namespace
  • model=’aladin’, # name of the model

usevortex.extractor(vortex_description, coords, start_cutoff, end_cutoff, start_term, end_term, points_fields=None, profiles_FAfields=None, everycutoff_in_hours=24, everyterm_in_hours=1, vertical_coordinate=None, extraction_options=None, use_local_cache=False, prestaging=False, outdir='/sauvegarde/mary/git/EPyGrAM/epygram/doc_sphinx/source', progressmode=None, nc_global_attributes={'history': 'Extracted on 2020-02-05 using epygram.', 'institution': 'Météo France'})

Extract temporal series of points (and profiles for FA historical files) to netCDF files.

Series are either:

• one series == one file per cutoff if start_term* != **end_term
• one series == one file containing all cutoffs if start_term* == **end_term

Shall be extended to series of profiles for other formats later on.

Parameters:

• vortex_description –

  vortex description of the resources, incl. providers, excepted dates/terms Example: {‘model’:’arome’,

  ’vapp’:’arome’, ‘vconf’:‘3dvarfr’, ‘kind’:’historic’, ‘suite’:’oper’, ‘geometry’:’franmgsp’, ‘cutoff’:’prod’, #’namespace’:’oper.archive.fr’, #’experiment’:‘86U8’, #’block’:’forecast’, #’nativefmt’:’fa’, #’member’:8,}

• coords – either a list of lon/lat coordinates or a dict of, in which case key would be a “name of the point”, used in output filename
• start_cutoff – first cutoff of the series (datetime.datetime instance)
• end_cutoff – last cutoff of the series
• start_term – first term of the series (datetime.datetime instance)
• end_term – last term of the series
• points_fields –

  dictionnary of fields to extract with options, e.g.:

  {fid1:{‘nc_name’:nc_fid1,

  ’nc_attributes’:{‘unit’:…,

  ’comment’:…, },

  ’decumulate’:True, ‘operation’:(‘-‘,273.15) or (‘exp’,), ‘reproject_wind_on_lonlat’:False, },

  fid2:… }

  • ’decumulate’ can be absent==False, True, or ‘center’ for centered decumulation (cf. D3Field.decumulate for more details);
  • ’operation’ is done after decumulation, if both present
  • ’reproject_wind_on_lonlat: FA only: reproject winds on lonlat axes (True by default)
• profiles_FAfields – like points_fields but for profiles, but only for FA historic resources ! fids supposed to be ‘S*PARAMETER’
• everycutoff_in_hours – step between 2 cutoffs
• everyterm_in_hours – step between 2 terms
• vertical_coordinate – vertical coordinate requested for profiles, among (‘pressure’, ‘height’, ‘altitude’, None)
• extraction_options – options to FA.extractprofile() and H2DField.extract_point() concerning interpolation, as dict…
• use_local_cache – use a local Vortex cache, in which files are stored
• prestaging – triggers prestaging for the requested resources (Hendrix)
• outdir – directory in which to store output files
• progressmode – sets the verbosity of the task among (None, ‘percentage’, ‘verbose’)
• nc_global_attributes – given as a dict. Defaults are above in module variable default_nc_global_attributes

---

Classes

class usevortex.Extractor(*args, **kwargs)

Bases: taylorism.Worker

Independant extractor.

Note

This class is managed by footprint.

• info: Run extractor().
• priority: PriorityLevel::DEFAULT (rank=1)

Automatic parameters from the footprint:

• directives (footprints.stdtypes.FPDict) - rxx - Contains all arguments to extractor(…).
• expected_time (builtins.float) - rxx - How long the worker is expected to run (in s).
  • Optional. Default is 0.0.
• memory (builtins.float) - rxx - Memory that should be used by the worker (in MiB).
  • Optional. Default is 0.0.
• name (builtins.str) - rwx - Name of the worker.
  • Optional. Default is None.
• scheduler_hooks (footprints.stdtypes.FPList) - rxx - List of callbacks before starting effective task work.
  • Optional. Default is FPList::<<as_list:: []>>.
• scheduler_ticket (builtins.int) - rxx - The slot number given by the scheduler (optional).
  • Optional. Default is None.

directives

Undocumented footprint attribute

expected_time

Undocumented footprint attribute

memory

Undocumented footprint attribute

name

Undocumented footprint attribute

scheduler_hooks

Undocumented footprint attribute

scheduler_ticket

Undocumented footprint attribute

---

Shortcuts

usevortex.get_gribfc_arome_oper(date, term, geometry='FRANGP0025', **others)

Proxy for AROME oper GRIBs.

usevortex.get_histfc_arome_oper(date, term, **others)

Proxy for AROME oper historic FAs.

usevortex.get_gribfc_arome_xp(xp, date, term, geometry='FRANGP0025', **others)

Proxy for AROME experiment GRIBs.

usevortex.get_histfc_arome_xp(xp, date, term, **others)

Proxy for AROME experiment historic FAs.

usevortex.get_gribfc_arpege_oper(date, term, geometry='FRANX01', **others)

Proxy for ARPEGE oper GRIBs.

usevortex.get_histfc_arpege_oper(date, term, **others)

Proxy for ARPEGE oper historic FAs.

usevortex.get_gribfc_arpege_xp(xp, date, term, geometry='FRANX01', **others)

Proxy for ARPEGE experiment GRIBs.

usevortex.get_histfc_arpege_xp(xp, date, term, **others)

Proxy for ARPEGE experiment historic FAs.