Commit bac6ad28 authored by Erik Kalkoken's avatar Erik Kalkoken
Browse files

Update documentation

parent 8b462d4e
Pipeline #176679968 passed with stage
in 2 minutes and 8 seconds
......@@ -7,12 +7,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
## [Unreleased] - yyyy-mm-dd
## [0.3.0] - 2020-08-11
### Added
- New management command `eveuniverse_load_types` making it easier for apps to preload the eve objects they need
- New manager method `bulk_get_or_create_esi()` for bulk loading of new eve objects.
- Type hints for all methods
- Spinx docs
- Improved documentation
### Changed
......
......@@ -2,11 +2,11 @@
Complete set of Eve Online Universe models in Django with on-demand loading from ESI
![release](https://img.shields.io/pypi/v/django-eveuniverse?label=release) ![python](https://img.shields.io/pypi/pyversions/django-eveuniverse) ![django](https://img.shields.io/pypi/djversions/django-eveuniverse?label=django) ![pipeline](https://gitlab.com/ErikKalkoken/django-eveuniverse/badges/master/pipeline.svg) ![coverage](https://gitlab.com/ErikKalkoken/django-eveuniverse/badges/master/coverage.svg) ![license](https://img.shields.io/badge/license-MIT-green) ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)
![release](https://img.shields.io/pypi/v/django-eveuniverse?label=release) ![python](https://img.shields.io/pypi/pyversions/django-eveuniverse) ![django](https://img.shields.io/pypi/djversions/django-eveuniverse?label=django) ![pipeline](https://gitlab.com/ErikKalkoken/django-eveuniverse/badges/master/pipeline.svg) ![coverage](https://gitlab.com/ErikKalkoken/django-eveuniverse/badges/master/coverage.svg) ![Documentation Status](https://readthedocs.org/projects/django-eveuniverse/badge/?version=latest)![license](https://img.shields.io/badge/license-MIT-green) ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)
## Overview
*django-eveuniverse* is a foundation app meant to help speed up the development of Django apps that are using data from ESI. It provides all Eve Universe classes as Django models, including all relationships between then, ready to be used in your project. Furthermore, all Eve models have an on-demand loading mechanism that allows you to load and store every eve objects ad-hoc.
*django-eveuniverse* is a foundation app meant to help speed up the development of Django apps that are using data from ESI. It provides all Eve classes from the Universe category in ESI as Django models, including all relationships between then, ready to be used in your project. Furthermore, all Eve models have an on-demand loading mechanism with database caching that will load eve objects from ESI as needed.
Here is an overview of the main features:
......@@ -19,4 +19,4 @@ Here is an overview of the main features:
## Documentation
For details on how to install and use *django-eveuniverse* please see the documentation.
For details on how to install and use *django-eveuniverse* please see the [documentation](https://dhooks-lite.readthedocs.io/en/latest/?badge=latest).
......@@ -4,13 +4,9 @@
API
===============
Relationship diagramm
=====================
This chapter contains the developer reference documentation of the public API for *django-eveuniverse* consisting of all models, their manager methods and helpers.
The following graph shows all models and how they are interrelated:
.. image:: https://i.imgur.com/FYYihzt.png
:target: https://i.imgur.com/FYYihzt.png
.. _api-eve-models:
Eve Models
==========
......@@ -51,6 +47,8 @@ EveDogmaEffect
.. autoclass:: eveuniverse.models.EveDogmaEffect
:members:
.. _api-models-eve-entity:
Eve Entity
--------------
......@@ -130,19 +128,23 @@ EveUnit
.. autoclass:: eveuniverse.models.EveUnit
:members:
.. _api-manager-methods:
Manager methods
====================
Eve Models
----------
Default manager methods
-------------------------
All eve models have the following manager methods:
.. autoclass:: eveuniverse.managers.EveUniverseEntityModelManager
:members:
EveEntity
----------
.. _api-managers-eve-entity:
EveEntity manager methods
-------------------------
EveEntity comes with some additional manager methods.
......@@ -161,4 +163,12 @@ Helpers
.. autofunction:: eveuniverse.helpers.meters_to_au
.. autofunction:: eveuniverse.helpers.meters_to_ly
\ No newline at end of file
Tools
====================
.. automodule:: eveuniverse.tools.testdata
:members:
.. seealso::
Please also see :ref:`developer-testdata` on how to create test data for your app.
# Developer Guide
## Basics
The developer guide describes how to develop apps with *django-eveuniverse*.
*django-eveuniverse* implements Django models for all eve objects from the ESI Universe category plus some related objects. These models can then be used in queries or included as related models in your app's own models.
## Models
The name of all models start with `Eve` and then the name of the object class. For example the model for solar systems is called `EveSolarSystem`.
*django-eveuniverse* provides you with ready-made Django models for all Eve Universe classes. These models can be used like any other Django model in queries or included as related models in your app's own models.
Please see the API for a list of all implemented Eve models.
```eval_rst
.. note::
The "Eve Universe" classes are the classes from the Universe category in ESI plus the related classes for dogma and market groups. The objects of those classes change rarely and most changes are just adding new objects (e.g. new types). They are therefore well suited to be stored and cached locally for a longer period of time.
The Eve Universe classes consist mostly of the same objects as the `Static Data Export <https://wiki.eveuniversity.org/Static_Data_Export>`_ (SDE).
```
```eval_rst
.. seealso::
Please see :ref:`api-eve-models` for the full documentation of all available models.
```
### Properties
### Relationship diagram
The following graph shows all models and how they are interrelated:
[![Models](https://i.imgur.com/FYYihzt.png)](https://i.imgur.com/FYYihzt.png)
### Naming of models and properties
The name of all models start with `Eve`. For example the model for solar systems is called `EveSolarSystem`.
All Eve model share the following basic properties:
......@@ -16,29 +34,58 @@ All Eve model share the following basic properties:
- `name`: The name of the eve objects
- `last_updated`: The date & time this object was last updated from ESI
Properties that reference other Eve models always have the name of the referenced model, just in snake case. e.g. a property references a EveSolarSystem object would be called `eve_solar_system`.
Property names are mostly the same as in the ESI specification. The exceptions are:
- The common properties `id` and `name` as described above
- Boolean fields start with `is_`
- Properties that reference other Eve models always have the name of the referenced model, just in snake case. e.g. a property references a EveSolarSystem object would be called `eve_solar_system`.
### Magic Methods
All Eve models have the following magic methods implemented:
- `__str__()`: returns the name of an object
- `__repr__()`: returns the complete object with all properties as model instance
- `__repr__()`: returns the complete object with all properties as model instance.
Examples:
```Python
>>> str(EveSolarSystem.objects.get(id=30000142))
'Jita'
```
```Python
>>> repr(EveSolarSystem.objects.get(id=30000142))
"EveSolarSystem(eve_constellation_id=20000020, eve_star_id=None, id=30000142, name='Jita', position_x=-1.2906486173487826e+17, position_y=6.075530690996363e+16, position_z=1.1746922706009029e+17, security_status=0.9459131360054016)"
```
### Additional functionality in Eve Models
Some Eve models provide additional useful functionality, e.g. icon image URLs.
For example the `EveSolarSystem` comes with a lot of additional features incl. a route finder:
```Python
>>> jita = EveSolarSystem.objects.get(id=30000142)
>>> akidagi = EveSolarSystem.objects.get(id=30045342)
>>> jita.jumps_to(akidagi)
10
```
## Fetching eve objects from ESI
### Fetching eve objects
### Fetching eve objects on-demand
All Eve models support on-demand loading of eve objects from ESI. This functionality is available through manager methods. One of these methods is `get_or_create_esi()`, which works similar to Django's `get_or_create()` method and will return the requested object along with a boolean flag showing if the object was created or not.
To fetch an eve object you can simply call it's manager method `get_or_create_esi()`. This will return the requested eve objects from the database if it exists, or else automatically load it from ESI:
For example for getting the solar system of Jita you could do the following:
For example for getting the solar system of Jita on-demand you could do the following:
```python
>>> EveSolarSystem.objects.get_or_create_esi(id=30000142)
(EveSolarSystem(eve_constellation_id=20000020, eve_star_id=None, id=30000142, name='Jita', position_x=-1.2906486173487826e+17, position_y=6.075530690996363e+16, position_z=1.1746922706009029e+17, security_status=0.9459131360054016), True)
```
Once loaded the object will be automatically stored in the database and the next time that same command would return the local copy.
Sometimes you may want to always fetch a fresh Eve objects from Esi. For that you can call `update_or_create_esi()`, which will always retrieve a new Eve objects and update the local copy.
Or if want to fetch a fresh Eve object from ESI you can call the manager method `update_or_create_esi()`. This which will always retrieve a new Eve objects from ESI and update the local copy.
Our example for Jita would then look like this:
......@@ -47,13 +94,32 @@ Our example for Jita would then look like this:
(EveSolarSystem(eve_constellation_id=20000020, eve_star_id=None, id=30000142, name='Jita', position_x=-1.2906486173487826e+17, position_y=6.075530690996363e+16, position_z=1.1746922706009029e+17, security_status=0.9459131360054016), False)
```
Alternatively, a set of eve objects can be preloaded, e.g. during installation of an app. For details please see [Preloading data](#preloading-data)
```eval_rst
.. hint::
Please see :ref:`api-manager-methods` for an overview of all available methods.
```
### Fetching parent and child objects
Many Eve models have parent and child models. For example a `EveSolarSystem` has `EveConstellation` as parent, and `EvePlanet` as one of its many children. When fetching an Eve objects for the first time from ESI, the related parent objects will automatically be loaded to preserve the integrity of the database. For example if you are fetching Jita for the first time, the objects for Jita's constellation (parent of solar system) and Jita's region (parent of constellation) will be fetched too.
Many Eve models have parent and child models. For example `EveSolarSystem` has `EveConstellation` as parent model, and `EvePlanet` is one of its child models. When fetching an Eve objects for the first time from ESI, the related parent objects will automatically be loaded to preserve the integrity of the database.
For example if you are fetching Jita for the first time, the objects for Jita's constellation (parent of solar system) and Jita's region (parent of constellation) will be fetched too.
In addition it is possible to automatically fetch all children of an object. This can be very useful for loading larger sets of data. For example, if you want to load all ship types, you can just fetch the inventory category for ships and include children by setting `include_children` to `True`.
Example:
In addition it is possible to automatically fetch all children of an object. This can be very useful for loading larger sets of data. For example, if you want to load all ship types, you can just fetch the inventory category for ships (id = 6) and include children. (Please see the method's API for all arguments.)
```python
>>> EveCategory.objects.get_or_create_esi(id=6, include_children=True)
(EveCategory(id=6, name='Ship', published=True), False)
```
This will load all children blocking, which can take quite some time. For large sets of data it is often is better to load children async (via Celery). This can be done by setting `wait_for_children` to `False`.
```python
>>> EveCategory.objects.get_or_create_esi(id=6, include_children=True, wait_for_children=False)
(EveCategory(id=6, name='Ship', published=True), False)
```
### Selecting which related models are loaded
......@@ -84,21 +150,59 @@ Our solution here is to offer developers control over which related models are l
When turning on loading of related models you usually want to reload related eve objects that already exist in the database to make sure all relations are created correctly. e.g. after turning on ``EveStargate`` you want to reload all solar systems.
```
### Additional functionality
### Preloading data
Some Eve models provide additional useful functionality, e.g. icon image URLs. Especially `EveSolarSystem` comes with a lot of additional features incl. a route finder. Please see the API for details.
While all models support loading eve objects on demand from ESI, some apps might need specific data sets to be preloaded. For example an app might want to provide a drop down list of all structure types, and loading that list on demand would not be fast enough to guarantee acceptable UI response times.
### Name resolution
The solution is to provide the user with a management command, so he an preload the needed data sets - for example all ship types - during app installation. Since this is a command use case *django-eveuniverse* offers a management helper command with all the needed functionality for loading data and which can be easily utilized with just a very small and simple management command in your own app.
A common problem when working with data from ESI is the need to resolve IDs to names / objects. To make this easier *django-eveuniverse* provides a dedicated model called `EveEntity`. `EveEntity` allows you to quickly resolve large amounts of IDs to objects, which include their respective names and categories in bulk. Resolved objects are automatically stored locally and used to speed up subsequent ID resolving.
Here is an example for creating such a management command. We want to load all kinds of structures to show to the user in a drop down list. We therefore want to preload all structure types (`category_id = 65`), all control towers (`group_id = 365`) and the customs office (`type_id = 2233`):
Here is a simple example for resolving one ID:
```Python
from django.core.management import call_command
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = "Preloads data required for this app from ESI"
def handle(self, *args, **options):
call_command(
"eveuniverse_load_types",
__title__,
"--category_id",
"65",
"--group_id",
"365",
"--type_id",
"2233",
)
```
For more details on how to use `eveuniverse_load_types` just call it with `--help` from a console.
```eval_rst
.. seealso::
For an overview of all management commands please see :ref:`operations-management-commands`.
```
## Eve ID to name resolution
A common problem when working with data from ESI is the need to resolve the ID of an Eve object to it's name. To make this easier *django-eveuniverse* provides a dedicated model called `EveEntity`. `EveEntity` allows you to quickly resolve large amounts of Eve IDs to objects in bulk, with every object having it's name and entity category. Resolved objects are stored locally and automatically used to speed up subsequent ID resolving.
Here is a simple example for resolving the ID of the Jita solar system:
```python
>>> EveEntity.objects.resolve_name(30000142)
'Jita'
```
```eval_rst
.. note::
Eve IDs have unique ranged for the supported categories, which means they can be safely resolved without having to specify a category.
```
This examples show how to resolve a list of IDs in bulk and using a resolver object to access the results:
```python
......@@ -115,7 +219,14 @@ This examples show how to resolve a list of IDs in bulk and using a resolver obj
Another approach is to bulk create EveEntity objects with the ID only and then resolve all "new" objects with `EveEntity.objects.bulk_update_new_esi()`. This approach works well when using EveEntity objects as property in you app's models.
For more features and details please see the API of `EveEntity`.
```eval_rst
.. seealso::
For more features and details please see :ref:`api-models-eve-entity` and :ref:`api-managers-eve-entity`.
```
```eval_rst
.. _developer-testdata:
```
## Test data
......@@ -157,7 +268,7 @@ class CreateEveUniverseTestData(TestCase):
### Using generated testdata in your tests
To user the generated testdata file in your test you need another script that creates objects from your generated JSON file.
To utilize the generated testdata file in your test you need another script that creates objects from your generated JSON file.
#### load_eveuniverse.py
......@@ -205,34 +316,3 @@ class MyTest(TestCase):
svipul = EveType.objects.get(id=34562)
# ...
```
## Preloading data
While all models support loading eve objects on demand from ESI, some apps might need specific data sets to be preloaded. For example an app might want to provide a drop down list of all structure types, and loading that list on demand would not be fast enough to guarantee acceptable UI response times.
The solution is to provide the user with a management command, so he an preload the needed data sets - for example all ship types - during app installation. Since this is a command use case *django-eveuniverse* offers a management helper command with all the needed functionality for loading data and which can be easily utilized with just a very small and simple management command ine the app.
Here is an example for creating such a management command in the app. We want to load all kinds of structures to show to the user in a drop down list. We therefore want to preload all structure types (category_id = 65), all control towers (group_id = 365) and the customs office (type_id = 2233):
```Python
from django.core.management import call_command
from django.core.management.base import BaseCommand
class Command(BaseCommand):
help = "Preloads data required for this app from ESI"
def handle(self, *args, **options):
call_command(
"eveuniverse_load_types",
__title__,
"--category_id",
"65",
"--group_id",
"365",
"--type_id",
"2233",
)
```
For more details on how to use `eveuniverse_load_types` just call it with `--help` from a console.
......@@ -6,13 +6,13 @@
Welcome to django-eveuniverse's documentation!
==============================================
*django-eveuniverse* is a foundation app meant to help speed up the development of Django apps that are using data from ESI. It provides all Eve Universe classes as Django models, including all relationships between then, ready to be used in your project. Furthermore, all Eve models have an on-demand loading mechanism that allows you to load and store every eve objects ad-hoc.
*django-eveuniverse* is a foundation app meant to help speed up the development of Django apps that are using data from ESI. It provides all Eve classes from the Universe category in ESI as Django models, including all relationships between then, ready to be used in your project. Furthermore, all Eve models have an on-demand loading mechanism with database caching that will load eve objects from ESI as needed.
.. toctree::
:maxdepth: 2
:caption: Contents:
installation
operations
developer
api
......
# Operations Guide
The operations guide describes how to install, configure and maintain *django-eveuniverse*.
## Installation
To install django-eveuniverse into your Django project please follow the these steps:
......@@ -20,15 +22,15 @@ By default only the core models are automatically loaded on-demand. If you want
### Setup celery
This app uses celery for loading large sets of data, e.g. with the load commands. Please make sure celery is setup and working for your Django project.
This app uses [Celery](https://docs.celeryproject.org/en/stable/index.html) for loading large sets of data, e.g. with the load commands. Please make sure celery is setup and working for your Django project.
```eval_rst
.. note::
Note on celery worker setup
**Note on celery worker setup**
For an efficient loading of large amounts of data from ESI we recommend a thread based setup of celery workers with at least 10 concurrent workers.
For example on our test system with 20 gevent threads the loading of the complete Eve Online map (with the command: **eveuniverse_load_data map**) consisting of all regions, constellation and solar systems took only about 5 minutes.
For example on our test system with 20 `gevent <http://www.gevent.org/>`_ threads the loading of the complete Eve Online map (with the command: **eveuniverse_load_data map**) consisting of all regions, constellation and solar systems took only about 15 minutes.
```
### Finalize installation
......@@ -107,6 +109,10 @@ Most settings will enable the automatic loading of related models. Note that thi
Note that all settings are optional and the app will use the documented default settings if they are not used.
```eval_rst
.. _operations-management-commands:
```
## Management commands
The following management commands are available:
......@@ -122,7 +128,7 @@ The following management commands are available:
On some DBMS like MySQL it is not possible to reset the database and remove all eveuniverse tables with the standard "migrate zero" command. The reason is that eveuniverse is using composite primary keys and Django seams to have problems dealing with that correctly, when trying to roll back migrations.
As workaround you will need remove all tables with SQL commands. To make this easier we are providing a SQL script that contains all commands to drop the tables. The full process for "migrating to zero" is as follows:
As workaround you will need remove all tables with SQL commands. To make this easier we are providing a SQL script that contains all commands to drop the tables. The process for "migrating to zero" is then as follows:
1. Run SQL script `drop_tables.sql` on your database
2. Run `python manage.py migrate eveuniverse zero --fake`
......@@ -591,7 +591,7 @@ class EveEntityManager(EveUniverseEntityModelManager):
raise NotImplementedError()
def bulk_update_new_esi(self) -> int:
"""updates all unresolved EveEntity objectrs in the database from ESI.
"""updates all unresolved EveEntity objects in the database from ESI.
Returns:
Count of updated entities.
......
......@@ -2,6 +2,7 @@ from copy import deepcopy
from collections import OrderedDict, namedtuple
import logging
import json
from typing import Dict
from django.core.serializers.json import DjangoJSONEncoder
......@@ -17,8 +18,15 @@ logger = LoggerAddTag(logging.getLogger(__name__), __title__)
ModelSpec = namedtuple("ModelSpec", ["ids", "include_children"])
def create_testdata(spec: dict, filepath: str) -> None:
"""Loads eve data from ESI as defined by spec and dumps it to file as JSON"""
def create_testdata(
spec: Dict[EveUniverseEntityModel, ModelSpec], filepath: str
) -> None:
"""Loads eve data from ESI as defined by spec and dumps it to file as JSON
Args:
spec: Specification of which Eve objects to load
filepath: absolute path of where to store the resulting JSON file
"""
# clear database
for MyModel in EveUniverseEntityModel.all_models():
......@@ -53,8 +61,12 @@ def create_testdata(spec: dict, filepath: str) -> None:
json.dump(data, f, cls=DjangoJSONEncoder, indent=4, sort_keys=True)
def load_testdata_from_dict(testdata: dict):
"""loads testdata from a dict"""
def load_testdata_from_dict(testdata: dict) -> None:
"""creates eve objects in the database from testdata dump given as dict
Args:
testdata: The dict containing the testdata as created by `create_testdata()`
"""
for MyModel in EveUniverseEntityModel.all_models():
model_name = MyModel.__name__
if model_name in testdata:
......@@ -85,8 +97,12 @@ def load_testdata_from_dict(testdata: dict):
MyModel.objects.create(**obj)
def load_testdata_from_file(filepath: str):
"""loads testdata from a JSON file"""
def load_testdata_from_file(filepath: str) -> None:
"""creates eve objects in the database from testdata dump given as JSON file
Args:
filepath: Absolute path to the JSON file containing the testdata created by `create_testdata()`
"""
with open(filepath, "r", encoding="utf-8") as f:
testdata = json.load(f)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment