Commit 9db00d70 authored by Phyks (Lucas Verney)'s avatar Phyks (Lucas Verney) Committed by Florent Fourcot
Browse files

Rewrite Weboob doc guides according to latest changes in CI and modules.

parent b78131f2
Pipeline #278635050 failed with stages
in 27 minutes and 50 seconds
......@@ -4,52 +4,42 @@ How to contribute
Write a patch
-------------
Help yourself with the [documentation](http://dev.weboob.org).
Help yourself with the `documentation <http://docs.weboob.org/>`_.
Find an [opened issue on gitlab](https://git.weboob.org/weboob/devel/issues),
or write you own bugfix or feature. Then, once it is necessary, commit with:
Find an opened issue on `this website <https://git.weboob.org/weboob/devel/issues>`_, or write your own bugfix or feature.
Then, once it is necessary, commit with::
```
$ git commit -a
```
$ git commit -a
Do not forget to write a helpful commit message.
Check your patch
----------------
You can run these scripts to be sure your patch doesn’t break anything:
```
$ tools/pyflakes.sh
$ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything
```
You can run these scripts to be sure your patch doesn't break anything::
Perhaps you should also write or fix tests.
$ tools/pyflakes.sh
$ tools/weboob_lint.sh
$ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything
Send a patch
------------
Perhaps you should also write or fix tests. These tests are automatically run by
`Gitlab CI <https://git.weboob.org/weboob/devel/pipelines>`_ at each commit and merge requests.
```
$ git format-patch -n -s origin
```
Create a merge request or send a patch
--------------------------------------
Then, send them with this command:
The easiest way to send your patch is to create a fork on `the Weboob Gitlab <https://git.weboob.org>`_ and create a merge
request from there. This way, the code review process is easier and continuous integration is run automatically (see
previous section).
```
$ git send-email --to=weboob@weboob.org *.patch
```
If you prefer good old email patches, just use
You can also send the files by yourself if you haven’t any configured MTA on
your system.
::
Create a fork on git.weboob.org
-------------------------------
$ git format-patch -n -s origin
If you think you’ll contribute to Weboob regularly, you can create a repository
on [git.weboob.org](git.weboob.org). To do so,
[signup](https://git.weboob.org/users/sign_in) and click on _Fork_ on the weboob
repository's page.
Then, send them with this command::
Then, you can push your commits, and ask for merge requests.
$ git send-email --to=weboob@weboob.org *.patch
All git branch are listed here: https://git.weboob.org/explore/projects
\ No newline at end of file
You can also send the files by yourself if you haven't any configured MTA on your system.
......@@ -9,7 +9,8 @@ Write a patch
Help yourself with the `documentation <http://docs.weboob.org/>`_.
Find an opened issue on this website, or write you own bugfix or feature. Then, once it is necessary, commit with::
Find an opened issue on `this website <https://git.weboob.org/weboob/devel/issues>`_, or write your own bugfix or feature.
Then, once it is necessary, commit with::
$ git commit -a
......@@ -21,12 +22,20 @@ Check your patch
You can run these scripts to be sure your patch doesn't break anything::
$ tools/pyflakes.sh
$ tools/weboob_lint.sh
$ tools/run_tests.sh yourmodulename # or without yourmodulename to test everything
Perhaps you should also write or fix tests.
Perhaps you should also write or fix tests. These tests are automatically run by
`Gitlab CI <https://git.weboob.org/weboob/devel/pipelines>`_ at each commit and merge requests.
Send a patch
------------
Create a merge request or send a patch
--------------------------------------
The easiest way to send your patch is to create a fork on `the Weboob Gitlab <https://git.weboob.org>`_ and create a merge
request from there. This way, the code review process is easier and continuous integration is run automatically (see
previous section).
If you prefer good old email patches, just use
::
......@@ -38,62 +47,17 @@ Then, send them with this command::
You can also send the files by yourself if you haven't any configured MTA on your system.
Ask for a public repository on git.symlink.me
---------------------------------------------
If you think you'll contribute to Weboob regularly, you can ask for a public repository. You'll also be able to push your commits in, and they'll be merged into the main repository easily.
All git branch are listed here: http://git.symlink.me/
By hosting a buildbot slave
***************************
To be sure weboob works fine on lot of architectures, OS and configurations, but also that websites haven't changed and backends still support them, it's important to have enough buildbot slaves.
If you are interested by hosting a buildbot slave, follow these instructions:
Create a slave
--------------
Firstly, you have to install ``pyflakes``, ``nose`` and `buildbot <http://buildbot.net>`_.
Run::
$ buildslave create-slave <dirname> buildbot.weboob.org:9080 <name> <password>
.. note::
if you use an old version of buildbot, run ``buildbot`` instead of ``buildslave``.
Parameters are:
* **dirname** — the path where you want to setup your slave on your host.
* **name** — the name of your slave. It would be for example your name, your nickname, your hostname. Check on http://buildbot.weboob.org the name you want to use isn't already taken.
* **password** — choose a password to login on the master.
For example::
$ buildslave create-slave /home/me/buildbot buildbot.weboob.org:9080 me secret123
Then, edit files in ``/home/me/buildbot/info/`` and run the slave::
$ buildslave start /home/me/buildbot
Contact us
----------
To connect your slave to our master, you can send us an email on admin@weboob.org with the following information:
* The name of your slave;
* The IP address of the host;
* The password of your slave;
* Indicate if you want to run tests for every merges (three times a day) or only do a nightly build.
When your slave will be accepted, you will see it on http://buildbot.weboob.org/waterfall.
By hosting a Gitlab CI runner
*****************************
How it works
------------
To be sure weboob works fine on lot of architectures, OS and configurations, but also that websites haven't changed and
backends still support them, it's important to have enough runners with different configurations, especially since
running some tests requires a working backend.
When a build is requested by master, your slave updates its local git repository, and run ``tools/run_tests.sh``.
If you are interested by hosting a Gitlab-CI runner, follow these instructions:
To work correctly, we suggest you to add as many as possible backends with the user of the slave. No private information will be sent to master, and it's better to have tests on backends which need authentication, because not every developers have accounts on them.
You can `install a Gitlab runner <https://docs.gitlab.com/runner/install/>`_ and make it use a specific backend file (be
it either by creating a dedicated Docker image with your credentials or running it in ``shell`` mode and making the
backend file available to it).
Then, you should contact us at admin@weboob.org so that we could help you register your runner with our Gitlab.
......@@ -31,7 +31,7 @@ For example, there is the :class:`CapMessages <weboob.capabilities.messages.CapM
Pick an existing capability
---------------------------
When you want to create a new module, you may have a look on existing capabilities to decide which one can be
When you want to create a new module, you may have a look at existing capabilities to decide which one can be
implemented. It is quite important, because each already existing capability is supported by at least one application.
So if your new module implements an existing capability, it will be usable from the existing applications right now.
......@@ -70,6 +70,12 @@ In a module directory, there are commonly these files:
* **test.py** - functional tests
* **favicon.png** - a 64x64 transparent PNG icon
.. note::
A module can implement multiple capabilities, even though the ``tools/boilerplate.py`` script can only generate a
template for a single capability. You can freely add inheritance from other capabilities afterwards in
``module.py``.
Update modules list
-------------------
......@@ -120,11 +126,12 @@ class is used to define a value.
Available parameters of :class:`Value <weboob.tools.value.Value>` are:
* **label** - human readable description of a value
* **required** - if ``True``, the backend can't loaded if the key isn't found in its configuration
* **required** - if ``True``, the backend can't be loaded if the key isn't found in its configuration
* **default** - an optional default value, used when the key is not in config. If there is no default value and the key
is not found in configuration, the **required** parameter is implicitly set
* **masked** - if ``True``, the value is masked. It is useful for applications to know if this key is a password
* **regexp** - if specified, on load the specified value is checked against this regexp, and an error is raised if it doesn't match
* **regexp** - if specified, the specified value is checked against this regexp upon loading, and an error is raised if
it doesn't match
* **choices** - if this parameter is set, the value must be in the list
.. note::
......@@ -132,6 +139,11 @@ Available parameters of :class:`Value <weboob.tools.value.Value>` are:
There is a special class, :class:`ValueBackendPassword <weboob.tools.value.ValueBackendPassword>`, which is used to manage
private parameters of the config (like passwords or sensible information).
.. note::
Other classes are available to store specific types of configuration options. See :mod:`weboob.tools.value
<weboob.tools.value>` for a full list of them.
For example::
from weboob.tools.value import Value, ValueBool, ValueInt, ValueBackendPassword
......@@ -173,14 +185,18 @@ You need to implement each method of all of the capabilities your module impleme
If you ran the ``boilerplate`` script command ``cap``, every methods are already in ``module.py`` and documented.
Read :class:`documentation of the capability <weboob.capabilities.bank.CapBank>` to know what are types of arguments,
what are expected returned objects, and what exceptions it may raises.
what are expected returned objects, and what exceptions it may raise.
When you are done writing your module, you should remove all the not implemented methods from your module, as the base
capability code will anyway ``raise NotImplementedError()``.
Browser
*******
Most of modules use a class derived from :class:`PagesBrowser <weboob.browser.browsers.PagesBrowser>` or
:class:`LoginBrowser <weboob.browser.browsers.LoginBrowser>` (for authenticated websites) to interact with a website.
:class:`LoginBrowser <weboob.browser.browsers.LoginBrowser>` (for authenticated websites) to interact with a website or
:class:`APIBrowser <weboob.browser.browsers.APIBrowser>` to interact with an API.
Edit ``browser.py``::
......@@ -242,6 +258,11 @@ Then, you have to declare them in your browser, with the :class:`URL <weboob.bro
Easy, isn't it? The first parameters are regexps of the urls (if you give only a path, it uses the ``BASEURL`` class attribute), and the last one is the class used to handle the response.
.. note::
You can handle parameters in the URL using ``(?P<someName>)``. You can then use a keyword argument `someName` to
bind a value to this parameter in :func:`stay_or_go() <weboob.browser.url.URL.stay_or_go>`.
Each time you will go on the home page, ``IndexPage`` will be instanced and set as the ``page`` attribute.
For example, we can now implement some methods in ``ExampleBrowser``::
......@@ -278,7 +299,7 @@ Now you have a functional browser, you can use it in your class ``ExampleModule`
You can now access it with member ``browser``. The class is instanced at the first call to this attribute.
For example, we can now implement :func:`CapBank.iter_accounts <weboob.capabilities.bank.CapBank.iter_accounts`::
For example, we can now implement :func:`CapBank.iter_accounts <weboob.capabilities.bank.CapBank.iter_accounts>`::
def iter_accounts(self):
return self.browser.iter_accounts_list()
......@@ -339,7 +360,8 @@ Parsing of pages
****************
.. note::
Depending of the base class you use for your page, it will parse html, json, csv, etc. In our case, it will be only html documents.
Depending of the base class you use for your page, it will parse html, json, csv, etc. In this section, we will
describe the case of HTML documents.
When your browser locates on a page, an instance of the class related to the
......@@ -349,7 +371,7 @@ interact with it.
The first thing to know is that page parsing is done in a descriptive way. You
don't have to loop on HTML elements to construct the object. Just describe how
to get correct data to construct it. It is the Browser class work to actually
to get correct data to construct it. It is the ``Browser`` class work to actually
construct the object.
For example::
......@@ -370,10 +392,10 @@ For example::
obj_label = CleanText('./td[@class="name"]')
obj_balance = CleanDecimal('./td[@class="balance"]')
As you see, we first set ``item_xpath`` which is the xpath string used to
iterate over elements to access data. In a second time we define ``klass`` which
is the real class of our object. And then we describe how to fill each object's
attribute using what we call filters.
As you can see, we first set ``item_xpath`` which is the xpath string used to iterate over elements to access data. In a
second time we define ``klass`` which is the real class of our object. And then we describe how to fill each object's
attribute using what we call filters. To set an attribute `foobar` of the object, we should fill `obj_foobar`. It can
either be a filter, a constant or a function.
Some example of filters:
......@@ -381,11 +403,19 @@ Some example of filters:
* :class:`CleanText <weboob.browser.filters.standard.CleanText>`: get a cleaned text from an element
* :class:`CleanDecimal <weboob.browser.filters.standard.CleanDecimal>`: get a cleaned Decimal value from an element
* :class:`Date <weboob.browser.filters.standard.Date>`: read common date formats
* :class:`DateTime <weboob.browser.filters.standard.Date>`: read common datetime formats
* :class:`Env <weboob.browser.filters.standard.Env>`: typically useful to get a named parameter in the URL (passed as a
keyword argument to :func:`stay_or_go() <weboob.browser.url.URL.stay_or_go>`)
* :class:`Eval <weboob.browser.filters.standard.Eval>`: evaluate a lambda on the given value
* :class:`Format <weboob.browser.filters.standard.Format>`: a formatting filter, uses the standard Python format string
notations.
* :class:`Link <weboob.browser.filters.html.Link>`: get the link uri of an element
* :class:`Regexp <weboob.browser.filters.standard.Regexp>`: apply a regex
* :class:`Time <weboob.browser.filters.standard.Time>`: read common time formats
* :class:`Type <weboob.browser.filters.standard.Type>`: get a cleaned value of any type from an element text
The full list of filters can be found in :doc:`weboob.browser.filters </api/browser/filters/index>`.
Filters can be combined. For example::
obj_id = Link('./a[1]') & Regexp(r'id=(\d+)') & Type(type=int)
......@@ -396,6 +426,23 @@ This code do several things, in order:
#) apply a regex to extract a value
#) convert this value to int type
When you want to access some attributes of your :class:`HTMLPage <weboob.browser.pages.HTMLPage>` object to fill an
attribute in a Filter, you should use the function construction for this attribute. For example::
def obj_url(self):
return (
u'%s%s' % (
self.page.browser.BASEURL,
Link(
u'//a[1]'
)(self)
)
)
which will return a full URL, concatenating the ``BASEURL`` from the browser
with the (relative) link uri of the first ``a`` tag child.
.. note::
All objects ID must be unique, and useful to get more information later
......@@ -437,6 +484,12 @@ Advanced topics
Filling objects
---------------
.. note::
Filling objects using ``fillobj`` should be used whenever you need to fill some fields automatically based on data
fetched from the scraping. If you only want to fill some fields automatically based on some static data, you should
just inherit the base object class and set these fields.
An object returned by a method of a capability can be not fully completed.
The class :class:`Module <weboob.tools.backend.Module>` provides a method named
......@@ -445,9 +498,9 @@ fill some unloaded fields of a specific object, for example with::
backend.fillobj(video, ['url', 'author'])
The ``fillobj`` method will check on the object what fields, in the ones given in list, which are
not loaded (equal to ``NotLoaded``, which is the default value), to reduce the list to the real
uncompleted fields, and call the method associated to the type of the object.
The ``fillobj`` method will check on the object which fields (in the ones given in the list argument) are not loaded
(equal to ``NotLoaded``, which is the default value), to reduce the list to the real uncompleted fields, and call the
method associated to the type of the object.
To define what objects are supported to be filled, and what method to call, define the ``OBJECTS``
class attribute in your ``ExampleModule``::
......
......@@ -19,6 +19,10 @@ It looks for every files named ``test.py``, and find classes derivated from ``Te
Then, it run every method which name starts with ``test_``.
.. note::
Some environment variables are available, to use specific backend file or send the test results. Refer to `the
comments in the script <https://git.weboob.org/weboob/devel/blob/master/tools/run_tests.sh#L4-8>`_ for more infos on this.
Write a test case
*****************
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment