Using Configurator¶
This document goes into more detail than the quickstart and should cover enough functionality for most use cases. For examples of how to use this functionality, see Patterns of Use.
Installation¶
Configurator is available on the Python Package Index and can be installed with any tools for managing Python environments. The package has no hard dependencies beyond the standard library, but you will need extra libraries for most file formats from which you may want to read configuration information. As a result, you may wish to install Configurator with the appropriate extra requirement to meet your needs:
pip install configurator[toml]
pip install configurator[yaml]
Getting configuration information¶
The most common source of configuration information is reading from files. Given a file such as this:
>>> print(open('/etc/myapp.yml').read())
myapp:
cache:
location: /var/my_app/
A Config
object can be obtained as follows:
>>> from configurator import Config
>>> Config.from_path('/etc/myapp.yml')
configurator.config.Config({'myapp': {'cache': {'location': '/var/my_app/'}}})
If you already have an open stream, it woud be this instead:
>>> with open('/etc/myapp.yml') as source:
... Config.from_stream(source)
configurator.config.Config({'myapp': {'cache': {'location': '/var/my_app/'}}})
Finally, if you have the text source in memory, you would do the following:
>>> text = """
... cache:
... location: /var/my_app/
... """
>>> Config.from_text(text, 'yaml')
configurator.config.Config({'cache': {'location': '/var/my_app/'}})
Note that because we have no way of guessing, the parser must be specified.
The parser
parameter can also be used with Config.from_path()
and
Config.from_stream()
to explicitly specify a parser, regardless of the name of
the file.
The parser can also be specified as a callable, if you have one-off unusual parsing needs:
>>> text = """
... {'format': 'not json'}
... """
>>> import ast
>>> def python(stream):
... return ast.literal_eval(stream.read())
>>> Config.from_text(text, python)
configurator.config.Config({'format': 'not json'})
If you need to add support for a new config file format, see Adding new parsers.
It is also quite normal to instantiate a Config
and then merge
configuration into it from several other sources:
>>> Config()
configurator.config.Config({})
If you already have a deserialized source of configuration information, you can
wrap a Config
around it and use it from that point onwards:
>>> import requests
>>> import requests
>>> Config(requests.get('http://config-store/myapp.json').json())
configurator.config.Config({'cache': {'location': '/var/my_app/'}})
Accessing configuration information¶
Configurator aims to provide access to configuration information in a simple and natural way, similar to the underlying python data structures but allowing both item and attribute access to be used interchangeably.
So, with a config such as this:
>>> config = Config({'logs': '/var/my_app/',
... 'sources': [{'url': 'http://example.com/1',
... 'username': 'user1',
... 'password': 'p1'},
... {'url': 'http://example.com/2',
... 'username': 'user2',
... 'password': 'p2'}]})
The various parts can be accessed as follows:
>>> config.logs
'/var/my_app/'
>>> for source in config.sources:
... print(source.url, source.username, source.password)
http://example.com/1 user1 p1
http://example.com/2 user2 p2
Item access can also be used, if preferred:
>>> config['sources'][1]['url']
'http://example.com/2'
Where it’s more natural, configuration can also be treated like a dictionary. For example, with this config:
>>> config = Config({'databases': {'main': 'mysql://foo@bar/main',
... 'backup': 'mysql://baz@bob/backup'},
... 'priority': ['main', 'backup']})
You could iterate through the databases as follows:
>>> for name, url in sorted(config.databases.items()):
... print(name, url)
backup mysql://baz@bob/backup
main mysql://foo@bar/main
Likewise, if a key may not be present:
>>> config.databases.get('read_only', default=config.databases.get('backup'))
'mysql://baz@bob/backup'
As a fallback, every node in the config will have a data
attribute
that can be used to get hold of the underlying configuration information:
>>> config.priority.data
['main', 'backup']
Combining sources of configuration¶
It’s rare that configuration for an application will come from a single source and so configurator makes it easy to combine them.
The simplest way is by adding two Config
instances. This will recursively
merge the underlying configuration data, unioning dictionary items and concatenating
sequences:
>>> config1 = Config({'mapping': {'a': 1, 'b': 2}, 'sequence': ['a']})
>>> config2 = Config({'mapping': {'b': 3, 'c': 4}, 'sequence': ['b']})
>>> config1 + config2
configurator.config.Config({'mapping': {'a': 1, 'b': 3, 'c': 4}, 'sequence': ['a', 'b']})
If you need to have more control over this process, Config.merge()
allows
you to specify how merging will be performed per python object type:
>>> config1 = Config([1, 2, 3, 4, 5])
>>> config2 = Config([6, 7, 8, 9, 10])
>>> from configurator import default_mergers
>>> from itertools import chain, zip_longest
>>> def alternate(context, source, target):
... return [i for i in chain.from_iterable(zip_longest(target, source)) if i]
>>> config1.merge(config2, mergers=default_mergers+{list: alternate})
>>> config1
configurator.config.Config([1, 6, 2, 7, 3, 8, 4, 9, 5, 10])
Note
merge()
mutates the Config
on which it is called
while addition leaves both of the source configs unmodified and returns a
new Config
.
If you need more flexibility in how parts of the configuration source are mapped in, or if the source data structure is not compatible with merging, you can use a mapping:
>>> source = Mock()
>>> source.foo.bar = 'some_value'
>>> config = Config({'bar': {'type': 'foo'}, 'baz': False})
>>> config.merge(source, {'foo.bar': 'bar.name'})
>>> from configurator.mapping import convert
>>> from ast import literal_eval
>>> config.merge(os.environ, {convert('BAZ', literal_eval): 'baz'})
>>> config
configurator.config.Config({'bar': {'name': 'some_value', 'type': 'foo'}, 'baz': True})
There is a lot of flexibility in how mapping and merging can be performed. For detailed documentation on this see Mapping and Merging.
One other form of manipulation that’s worth mentioning is when incoming data isn’t quite the right shape. Take this YAML:
>>> print(open('/etc/my_app/config.yaml').read())
actions:
- checkout:
repo: git@github.com:Simplistix/configurator.git
branch: master
- run: "cat /foo/bar"
The actions, while easy to read, aren’t homogeneous or easy for the application to use. It might be easier if they were something like:
{'actions': [{'type': 'checkout', 'kw': {'repo': '...', 'branch': 'master'}},
{'type': 'run', 'args': ('cat /foo/var',)}]}
We can achieve this by modifying the data in the Config
programmatically
with a function such as this:
def normalise(data):
actions = []
for action_data in data:
(type_, params), = action_data.items()
if isinstance(params, dict):
actions.append({'type': type_, 'args': (), 'kw': params})
else:
actions.append({'type': type_, 'args': (params,), 'kw': {}})
data[:] = actions
This can be applied to the raw config as follows:
>>> config = Config.from_path('/etc/my_app/config.yaml')
>>> normalise(config.actions.data)
Now, the application code can use the config in a uniform way:
>>> for action in config.actions:
... output = action_handlers[action.type](*action.args, **action.kw.data)