Python modules

This modules are provided with the library to ease the interoperability between I/O of the Fortran library with python scripting

Utils module

This file contains some low-level useful functions

class futile.Utils.Node(obj, valid_if=None)

Bases: object

An object that is associated to a queue. Has a requirement and a validity function as well as a generation function that is triggered in case the node is not valid.

Parameters

obj (object) – a generic object

requires(node, generator)

Set the dependency between two nodes.

Parameters

• node (Node) – the node from which this one depends. If this is a valid node, and the present is not, it should employ the generator function for creating the node.

• generator (func) – function which should be called to make the node valid. Should have as arguments the two objects associated the node (the current one is the first argument)

property valid

valid_if(func)

Set the function as callback to check the validity of the node.

Parameters

func (func) – function that has the node object as a the first argument and returns a boolean which assess the validity of the function.

validate()

This function makes the node valid by validating its dependency and by calling the generator function if this is not the case.

Returns

the value of self.valid. Should be true.

Return type

bool

class futile.Utils.ObjectSerialization(obj=None, files=None, version=None)

Bases: object

Serialization of a class into an archive.

This class can be employed each time that we would like to identify the minimal set of files needed to instantiate an object. Files and objects are put in an archive that can in this way be used to retrieve and reinstantiate the object.

Parameters

• obj (object) – a instance of the object to be serailized. Necessary in case some attributes of it have to be included in the serialization.

• version (str) – the version of the desired serialization. If absent, the default is considered.

• files (dict) – dictionary of the files to be included. The dictionary should be of the form {<filename>: <abspath>} or rather {<filename>: {‘archive’:<archive_path>, ‘file’:<member>}} in which case the file <member> of the archive <archive_path> will be included. The file will be included in the serialization as <filename>.

cached_attributes = {'1.0': {}}

dump(archive, extra_encoder_functions=[])

Create an archive with the entire set of information of the Serialization. Such a tarfile should be such that the same analysis of the object is possible

Parameters

• archive (str) – path of the archive to serialize the object to.

• extra_encoder_functions (list) – see :py:func:serialize_objects.

classmethod load(archive, init_function, tmpdir_prefix='', load_functions={}, serialization_version=None, **kwargs)

Create a class instance from a serialized archive.

Parameters

• archive (str) – the path of the archive

• init_function (func) – function that should be called at the class instantiation

• load_functions (dict) – dictionary of load functions per cached attribute. The function signature is the class instance, the files list, and the attribute.

• serialization_version (str) – version of the load

• tmpdir_prefix (str) – prefix to be added to the temporary directory

• **kwargs – other arguments that have to be passed to the

• init_function –

Returns

Instance of the class

version = '1.0'

version of the serialization

futile.Utils.create_tarball(filename, files, objects={})

Assemble files and objects in a tarball

Parameters

• filename (str) – the name of the archive. Determine the tarball compression method from its extension.

• files (dict,set) – file paths that have to be included in the tarball. If it is a dict, it should be in the form “{arcname : file}”, where file is the path of the file to be put, and arcname is the name of thefile that would be used in the archive. If it is a set, the file will preserve its name in the archive

• objects (dict) – dictionary ‘{arcname: buffer}’ of the buffers that will have to be serialized in the arcname position the buffers are given as class::io.BytesIO instance, following specification of the func:serialize_objects function.

futile.Utils.data_path(archive, dest='.', path='datalake', branch='main', github_repo='BigDFT-group/resources')

futile.Utils.dict_get(inp, *subfields)

Find the value of the provided sequence of keys in the dictionary, if available.

Retrieve the value of the dictionary in a sequence of keys if it is available. Otherwise it provides as default value the last item of the sequence subfields.

Parameters

• inp (dict) – the top-level dictionary. Unchanged on exit.

• subfields (str,object) – keys, ordered by level, that have to be retrieved from topmost level of inp. The last item correspond to the value to be set.

Returns

The value provided by the sequence of subfields if available, otherwise the default value given as the last item of the subfields sequence.

futile.Utils.dict_merge(dest, src)

Recursive dict merge. Inspired by dict.update(), instead of updating only top-level keys, dict_merge recurses down into dicts nested to an arbitrary depth, updating keys. The src is merged into dest. From angstwad/dict-merge.py

Parameters

• dest (dict) – dict onto which the merge is executed

• src (dict) – dict merged into dest

futile.Utils.dict_set(inp, *subfields)

Ensure the provided fields and set the value

Provide a entry point to the dictionary. Useful to define a key in a dictionary that may not have the previous keys already defined.

Parameters

• inp (dict) – the top-level dictionary

• subfields (str,object) – keys, ordered by level, that have to be retrieved from topmost level of inp. The last item correspond to the value to be set .

Example

>>> inp={}
>>> dict_set(inp,'dft','nspin','mpol',2)
>>> print (inp)
{'dft': {'nspin': {'mpol': 2}}}

futile.Utils.ensure_copy(src, dest)

Copy src into dest.

Guarantees that the file dest is a copy of the file src.

Parameters

• src (str) – path of the source file. Should be valid.

• dest (src) – path of the destination file

Returns

True if the file needed to be copied, False if src

and dest are identical

Return type

bool

futile.Utils.ensure_dir(file_path)

Guarantees the existance on the directory given by the (relative) file_path

Parameters

file_path (str) – path of the directory to be created

Returns

True if the directory needed to be created,

False if it existed already or if an error happened during the creation.

Return type

bool

futile.Utils.ensure_object(filename, code_if_obj_not_found=None, glob=None, loc=None)

Identify a pickle file to save a given object on it. In case this file is present, read the object from it. Otherwise, assume that the object is ready to be dumped and write it in the file.

Parameters

• filename (str) – the path of the file in which the object is saved/loaded.

• code_if_obj_not_found (str) – the code to be evaluated if the object has

• found. (not been) –

• globals (dict) – the global variables dictionary

• locals (dict) – the local variables dictionary

Returns

The result from the pickle file or from the code to be executed

Return type

object

futile.Utils.execute(*args)

futile.Utils.execute_code_if(condition, code, glob=None, loc=None)

Execute code if condition is true

Parameters

• condition (bool) – if true the code is executed

• code_if_obj_not_found (str) – the code to be evaluated if the object has not been found.

• globals (dict) – the global variables dictionary

• locals (dict) – the local variables dictionary

Returns

the object returned from the code executed, None otherwise

futile.Utils.file_list(directory='.', suffix=None, prefix=None, exclude=None, include_directory_path=False)

Return the list of the files inside a given directory

Parameters

• directory (str) – path of the directory to search into

• suffix (str) – the suffix that the files should have

• prefix (str) – the prefix that the files should have

• exclude (str) – exclude the files which matches this string from the list

• include_directory_path (bool) – if True includes the path of the directory in the list.

Returns

list of the files that matches the requirements.

Return type

list

futile.Utils.file_time(filename)

Determine the time of the last modification of a file.

Parameters

filename (str) – path of the file to inspect.

Returns

time of the modified file. Returns 0 if the file does not exist.

Return type

float

futile.Utils.fill_dictionary_in_parallel(nthreads, keys, func, **kwargs)

Fill a dictionary of a given set of keys with the return value of a function which accept this key as a first argument

Parameters

• nthreads (int) – the number of threads of the pool

• keys (list) – the arguments of the function. Will be the list of the dictionary

• func (func) – the python function that has the key as a last argument

• **kwargs – further arguments of the function, if needed

Returns

the key-> obj dictionary with the obj the return value of func

Return type

dict

futile.Utils.find_files(regexp, archive=None)

Returns a list of the paths to the files that follow the regular expression regexp. They are searched from the current working directory or from an archive given as optional argument.

Parameters

• regexp (string) – A regular expression

• archive – an opened tarfile archive (optional)

Returns

a list of all the paths that agree with the regexp

Return type

list of strings

Raises

ValueError if the regexp does not find a single path.

Example:

#Find all python files in the current working directory
find_files('*py')

#An exmple outside of the current working directory
find_files('*/log-*.yaml')

#Example using a tarfile
import tarfile
my_archive = tarfile.open('archive.tar.gz')
find_files('*/*/log-*.yaml', archive=my_archive)

futile.Utils.floatify(scalar)

Useful to make float from strings compatible from fortran

Parameters

scalar (str, float) – When string representing a float that might be given in fortran notation, otherwise it might be a floating point

Returns

float. The value associated to scalar as a floating point number

Example

>>> # this would be the same with "1.e-4" or with 0.0001
>>> floatify('1.d-4')
1.e-4

futile.Utils.function_signature_regenerator(target_kwargs_function, fun_name='', fun_docstring='', **kwargs)

Generate the function of the name provided by fun_name, with signature provided by the kwargs dictionary.

Parameters

• target_kwargs_function (func) – keyword arguments function that will be used for the generated function.

• fun_name (str) – name of the regenerated function. If empty it will be the target_kwargs_functon.__name__ prefixed by regenerated, which will be copied in the docstring of the regenerated function.

• fun_docstring (str) – docstring of the generated function, if empty it will take the docstring from target_kwargs_function.

• **kwargs – keyword arguments which will represent the signature of the generated function.

Example

>>> def write_kwargs(**kwargs):
>>>     """
>>>     Convert keyword arguments into a string
>>>     """
>>>     return str(kwargs)
>>> write_opts=function_signature_regenerator(write_kwargs,
>>>                                           fun_name='write_opts',
>>>                                           opt1='default1',
>>>                                           opt2='default2')
>>> help(write_opts)
>>> print (write_opts())
Help on function write_opts:

write_opts(opt1=’default1’, opt2=’default2’)

Convert keyword arguments into a string

{‘opt1’: ‘default1’, ‘opt2’: ‘default2’}

futile.Utils.get_curl_command(sha, size, repo)

futile.Utils.get_sha_and_size(filename)

futile.Utils.kw_pop(*args, **kwargs)

Treatment of kwargs. Eliminate from kwargs the tuple in args.

Example

>>> kwargs = {'one': 1, 'two': 2, 'three': 3}
>>> # Existing value, default ignored
>>> kw2, two_maybe = kw_pop('two', 100, **kwargs)
>>> print (kw2, two_maybe)
{'one': 1, 'three': 3}, 2
>>> # Not, existing value, default considered
>>> kw2, four_maybe = kw_pop('four', 4, **kwargs)
>>> print (kw2, four_maybe)
{'one': 1, 'two': 2, 'three': 3}, 4

futile.Utils.make_dict(inp)

Transform the instance inp into a python dictionary. If inp is already a dictionary, it performs a copy.

Parameters

inp (dict) – a instance of a Class which inherits from dict

Returns

the copy of the class, converted as a dictionary

Return type

dict

futile.Utils.merge_two_dicts(x, y)

futile.Utils.more_recent_than_parent(filename, parent)

Filename should be more recent than parent

futile.Utils.non_null_size(filename)

Control if the file has nonzero size and exists

futile.Utils.option_line_generator(separator='--', **kwargs)

Associate to each of the keyword arguments a command line argument.

Parameters

• separator (str) – The string needed to separate the options.

• arguments (Might be '--' for command-line) –

• ' (but also) –

• ' –

• signatures. (for function) –

Warning

The separator comes before the first argument therefore pay attention to lstrip it in case you want to use it as a function signature string.

Example

>>> option_line_generator(arg1='val1',arg2='val2')
'--arg1=val1 --arg2=val2'

futile.Utils.property_attribute(self, attribute, code_if_not_found)

futile.Utils.push_path(inp, *keys)

Follow in the dictionary inp the path indicated by the keys. If this path does not exists creates it.

Parameters

• inp (dict) – dictionary

• keys (str) – keys of the path to follow

Returns

(branch,``key``) tuple, where

• branch (dict): the dictionary of the second-last item of the path

• key (str): the last item of the path

Example

>>> inp={}
>>> d,key=push_path(inp,'dft','nspin','mpol')
>>> print (d,key)
>>> print (inp)
{},'mpol'
{'dft': {'nspin': {}}}

>>> inp={'dft': {'nspin': {'mpol': 2}}}
>>> d,key=push_path(inp,'dft','nspin','mpol')
>>> print (d,key)
>>> print (inp)
{'mpol': 2},'mpol'
{'dft': {'nspin': {'mpol': 2}}}

futile.Utils.serialize_objects(objects, extra_encoder_functions=[])

Convert a dictionary of objects into buffers. Employs json serialization into StringIO instances

Parameters

• objects (dict) – dictionary of key/value pair of objects to be serialized

• extra_encoder_functions (list)) – list of dictionaries of the format {‘cls’: Class, ‘func’: function} which is employed in the serialization

Returns

dictionary of key/buffer pairs

Return type

dict

futile.Utils.sort_lists(sort_by, ascending, *lists)

Sort lists altogether following the lists indicated by the sort_by index.

Parameters

• sort_by (int) – the index of the list which has to be taken as reference for sorting

• ascending (bool) – Sort is performed in ascending order if True

• *lists – sequence of lists to be mutually sorted. They have to be of the same length.

Returns

tuple of sorted lists

Example: >>> l1=[5,3,4] >>> l2=[‘c’,’t’,’q’] >>> l3=[6,3,7] >>> print (sort_lists(0,True,l1,l2,l3)) >>> print (sort_lists(2,True,l1,l2,l3)) [(3, 4, 5), (‘t’, ‘q’, ‘c’), (3, 7, 6)] [(3, 5, 4), (‘t’, ‘c’, ‘q’), (3, 6, 7)]

futile.Utils.split_kwargs(kwargs, sub_kwargs)

Split a dictionary of kwargs from a subset.

Parameters

• kwargs (dict) – the original dictionary.

• sub_kwargs (dict) – dictionary providing the {‘key’: default} set of arguments which should be splitted from kwargs.

Returns

(new_kw, sub_kw) the splitted dictionaries

Return type

tuple

futile.Utils.tarfile_is_coherent(filename)

Checks the coherency of the tarfile.

Parameters

filename (str) – path of the tarfile.

Returns

True if the tarfile is good to go.

Return type

bool

Raises

Exception if the tarfile is not valid. –

futile.Utils.unique_filename(prefix)

Provides a filename with a unique id appended

Parameters

prefix (str) – the prefix of the file

Returns

filename

Return type

str

futile.Utils.unpack_tarball(archive, tmpdir_prefix='tmp_')

Open an archive in a temporary directory

Parameters

• archive (str) – the path of th archive to open

• tmpdir_prefix (str) – prefix of the temporary directory to untar the archive to.

Returns

tmpdir, files path of the temporary directory and names of

the files extracted by the tarfile

Return type

tuple

futile.Utils.untar_archive(archive, dest='.')

Untar the archive in the destination directory.

Parameters

• archive (str) – path of the file to untar.

• dest (str) – destination directory. Create if not exists.

Returns

list of the files contained in the tarball.

Return type

list

futile.Utils.version_is_compatible(desired_version, present_version)

Assess the compatibility of a version id with a reference.

Parameters

• desired_version (str) – the version which is used as the reference, in the format “x.y”, “x.y.z”, “x.y.z.w”, …

• present_version (str) – the version to be tested against.

Returns

True if all the version numbers of the present version are

of lower number than the desired one.

Return type

bool

futile.Utils.write(*args, **kwargs)

Wrapper for print function or print to ensure compatibility with python 2 The arguments are used similarly as the print_function They can also be generalized to python 2 cases

YamlIO module

Inputvars module

Handle the input variable specifications.

This module is the python complement to the specification of the input variables of a file as provided by the f_input_file module. It uses the same syntax as defined there and make possible the interplay between a python-based serialization of the input dictionary and the construction of

class futile.Inputvars.InputVariable(name, spec)

Bases: object

Define a input variable of a library. Such object can be initialized and inspected from the dictionary used by futile and then initialized according to the provided specification.

is_valid()

set(val)

Set the value of the input variable

set_dependent_variable(var)

Set the dependent variable from which impose the profile

set_master_variable(var)

Set the variable which is activated when the present has suitable values or profiles

ArgParse module

Handle and automatize the parsing of input arguments

This module uses the same convention of the yaml_parse fortran module to define the command line arguments. Such module can be used to define command line arguments of python scripts that follows the same conventions or to generate python functions that have the same signature than the provided command arguments.

futile.YamlArgparse.get_python_function(target_kwargs_function, func_name, func_spec)

Convert a argparse spec into a python function

This function provides a python function with a signature indicated by the fun_spec dictionary With the conventions of the yaml_argparse modules. The futile.Utils.function_signature_regenerator() function is used for the conversion

Parameters

• target_kwargs_function (func) – the keyword arguments function we want to give the signature to.

• func_name (str) – Name of the function, usually the key of the dictionary whose func_spec is the value

• func_spec (dict) – dictionary of the function specifications to be provided to the futile.Utils.function_signature_regenerator() function.

Returns

the genreated function with signature given by the arguments of func_spec

defaulting to their default value.

Return type

func

Todo

Create the docstring of the generated function by also including the docstring of the arguments

Figures module

class futile.Figures.AxisSet(fig, rect, *, facecolor=None, frameon=True, sharex=None, sharey=None, label='', xscale=None, yscale=None, box_aspect=None, **kwargs)

Bases: matplotlib.axes._axes.Axes

cla()

Clear the Axes.

create_twin()

set(*, adjustable=<UNSET>, agg_filter=<UNSET>, alpha=<UNSET>, anchor=<UNSET>, animated=<UNSET>, aspect=<UNSET>, autoscale_on=<UNSET>, autoscalex_on=<UNSET>, autoscaley_on=<UNSET>, axes_locator=<UNSET>, axisbelow=<UNSET>, box_aspect=<UNSET>, clip_box=<UNSET>, clip_on=<UNSET>, clip_path=<UNSET>, facecolor=<UNSET>, frame_on=<UNSET>, gid=<UNSET>, in_layout=<UNSET>, label=<UNSET>, navigate=<UNSET>, path_effects=<UNSET>, picker=<UNSET>, position=<UNSET>, prop_cycle=<UNSET>, rasterization_zorder=<UNSET>, rasterized=<UNSET>, sketch_params=<UNSET>, snap=<UNSET>, title=<UNSET>, transform=<UNSET>, url=<UNSET>, visible=<UNSET>, xbound=<UNSET>, xlabel=<UNSET>, xlim=<UNSET>, xmargin=<UNSET>, xscale=<UNSET>, xticklabels=<UNSET>, xticks=<UNSET>, ybound=<UNSET>, ylabel=<UNSET>, ylim=<UNSET>, ymargin=<UNSET>, yscale=<UNSET>, yticklabels=<UNSET>, yticks=<UNSET>, zorder=<UNSET>)

Set multiple properties at once.

Supported properties are

Properties:

adjustable: {‘box’, ‘datalim’} agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array alpha: scalar or None anchor: (float, float) or {‘C’, ‘SW’, ‘S’, ‘SE’, ‘E’, ‘NE’, …} animated: bool aspect: {‘auto’, ‘equal’} or float autoscale_on: bool autoscalex_on: bool autoscaley_on: bool axes_locator: Callable[[Axes, Renderer], Bbox] axisbelow: bool or ‘line’ box_aspect: float or None clip_box: .Bbox clip_on: bool clip_path: Patch or (Path, Transform) or None facecolor or fc: color figure: .Figure frame_on: bool gid: str in_layout: bool label: object navigate: bool navigate_mode: unknown path_effects: .AbstractPathEffect picker: None or bool or float or callable position: [left, bottom, width, height] or ~matplotlib.transforms.Bbox prop_cycle: unknown rasterization_zorder: float or None rasterized: bool sketch_params: (scale: float, length: float, randomness: float) snap: bool or None title: str transform: .Transform url: str visible: bool xbound: unknown xlabel: str xlim: (bottom: float, top: float) xmargin: float greater than -0.5 xscale: {“linear”, “log”, “symlog”, “logit”, …} or .ScaleBase xticklabels: unknown xticks: unknown ybound: unknown ylabel: str ylim: (bottom: float, top: float) ymargin: float greater than -0.5 yscale: {“linear”, “log”, “symlog”, “logit”, …} or .ScaleBase yticklabels: unknown yticks: unknown zorder: float

classmethod twinify(ax)

Include the axis provided as a radix of the set

class futile.Figures.FigureSet(**kwargs)

Bases: object

Container for multiple figures.

Define a container for a plot with the possiblity to switch between simple and gnuplot plotting

Arguments: title: The title of the master figure **kwargs: arguments for the axis instance

add(**kwargs)

exists(figname)

True if the Figure exists in the Set

invoke(idx)

show(figname=None)

class futile.Figures.VertSlider(ax, label, valmin, valmax, valinit=0.5, valfmt='%1.2f', closedmin=True, closedmax=True, slidermin=None, slidermax=None, dragging=True, **kwargs)

Bases: matplotlib.widgets.AxesWidget

A slider representing a floating point range.

For the slider to remain responsive you must maintain a reference to it.

\*ax*

the slider matplotlib.axes.Axes instance

\*val*

the current slider value

\*hline*

a matplotlib.lines.Line2D instance representing the initial value of the slider

\*poly*

A matplotlib.patches.Polygon instance which is the slider knob

\*valfmt*

the format string for formatting the slider text

\*label*

a matplotlib.text.Text instance for the slider label

\*closedmin*

whether the slider is closed on the minimum

\*closedmax*

whether the slider is closed on the maximum

\*slidermin*

another slider - if not None, this slider must be greater than slidermin

\*slidermax*

another slider - if not None, this slider must be less than slidermax

\*dragging*

allow for mouse dragging on slider

Call on_changed() to connect to the slider event

disconnect(cid)

remove the observer with connection id cid

on_changed(func)

When the slider value is changed, call func with the new slider position

A connection id is returned which can be used to disconnect

reset()

reset the slider to the initial value if needed

set_val(val)

futile.Figures.axis_from_data(fig, ax, data)

Transform a data tuple into axis coordinates

futile.Figures.data_from_data(fig, dst, src, data)

Transform a data tuple of anothe axis in the figure into data of another axis

futile.Figures.show_image(imgfile, title=None)

Show image file using matplotlib imgread. Useful to bypass the Jupyter bug for converting a notebook into a pdf file

Time module

class futile.Time.TimeData(*filenames, **kwargs)

Bases: object

barwidth = 0.9

collect_categories(dict_list, vals)

Collect all the categories which belong to all the dictionaries

counters()

Inspect the available counters

draw_barfigure(fig, axis, data, title)

draw_lineplot(ax, data, label)

find_items(category, dict_list)

For a given category find the items which have them

find_unbalanced(data)

Determine lookup array of unbalanced categories

gnuplot_figure(lookup=None, epsfile=None, aggregate=None, select_category=None)

Create a figure to be plotted with gnuplot

Create a gnuplot histogram that can be plotted for production results.

Parameters

• lookup (list) – A list of the items of the TimeData instance that will be considered for the histogram

• epsfile (str) – Name of the eps file in which the data will be plot

• aggregate (list) – list of tuples of the categories to aggregate, accoding to the convention (newkey ,[list of oldkeys]) between the original keys and the desired keys

• select_category (str) – Name of the category to be plot, if specified the internal items of the category will be plot.

ignored_counters = ['CPU parallelism', 'Routines timing and number of calls', 'SUMMARY', 'Report timestamp', 'Hostnames']

inspect_category(cat)

inspect_counter(counter, unit=None)

load_unbalancing(ax, dict, hosts)

Extract the data for plotting the hostname balancings between different categories in bar chart

replot(label)

routines_plot(index, event=None)

Draw the plot of the routines level for the run identified by index

show()

unbalanced(val)

Criterion for unbalancing

workload_plot(index, event=None)

Draw the plot of the workload of different classes for the run identified by index

futile.Time.aggregate_names(data, list_agg)

Aggregate the names of the plot in some different categories the structure of dict_agg should ba a list of tuples (newkey ,[list of oldkeys]) between the original keys and the desired keys

class futile.Time.polar_axis(fig, ax, data)

Bases: object

draw_polarplot()

dump_timing_level(level, starting_point=None, ilev=0, theta=0, data=None)

Inspect the first level of the given dictionary and dump the profile subroutines at this level

step = 5