Commit e83ccb69 authored by Julien Cloud's avatar Julien Cloud
Browse files

Cleanup and add sequences

parent 194ddb10
# Andalucia <-> FaST login flow
This document outlines the proposed authentication/authorization flows between the FaST ecosystem and the Andalucia DAT.
It builds upon the generic `iacs_apis.md` flow document.
The proposed flows include:
1. the user installs the FaST mobile app, starts it and logs in using DAT *from within the app*. The DAT authentication is used to create the corresponding FaST user and retrieve the necessary data.
2. the user navigates to the desktop portal of FaST and logs in using the DAT using a redirect or an iframe
3. the user logs in to the DAT using his mobile browser and selects the FaST service from there. This will in turn redirect him to the FaST App (if it is installed) or to a redirect mobile page where he will be given the choice to either install the app or use the desktop version.
4. the user logs in to the DAT using his desktop browser and selects the FaST service from there. This will redirect him to the FaST desktop portal and his DAT login token will be used to log him into the portal.
## From FaST App
```mermaid
sequenceDiagram
Note over FaST App: User install FaST<br>mobile app from<br>App Store or Google<br>Play
Note over FaST App: User starts FaST App<br>and select Andalucia<br>region
FaST App ->> FaST Backend: Request user authentication
FaST Backend ->> DAT: Redirect to DAT login screen
Note over DAT: User logs in to DAT
DAT ->> FaST Backend: Callback with JWT + user info
FaST Backend ->> FaST Backend: Get or create user
FaST Backend ->> DAT: Request authorizations for user
Note over DAT: API GetFarms
Note right of DAT: [note: the list of<br>farms/roles could<br>also be returned as<br>a claim of the login<br>JWT]
DAT ->> FaST Backend: List of farms (F1, F2..) and role for each farm (owner, advisor...)
loop For each farm
FaST Backend ->> REAFA: Get farm F details from user (using JWT)
Note over REAFA: API GetFarmDetails
REAFA ->> FaST Backend: Farm details
end
FaST Backend ->> FaST App: List of farms with details
Note over FaST App: Display farms
```
## From FaST desktop portal
```mermaid
sequenceDiagram
Note over Browser: User navigates to<br>FaST desktop portal
Note over Browser: User selects<br>Andalucia region
Browser ->> FaST Backend: Request user authentication
FaST Backend ->> DAT: Redirect to DAT login screen
Note over DAT: User logs in to DAT
DAT ->> FaST Backend: Callback with JWT + user info
FaST Backend ->> FaST Backend: Get or create user
FaST Backend ->> FaST Backend: [... get farms details (see above)]
FaST Backend ->> Browser: List of farms with details
Note over Browser: Display farms
```
## From DAT on mobile
```mermaid
sequenceDiagram
Note over DAT: User navigates to<br>DAT on his browser
DAT ->> DAT: User logs in to DAT
Note over DAT: List of farms/roles
DAT ->> DAT: User selects farm + role
Note over DAT: List of services
DAT ->> DAT: User clicks on "FaST"
DAT ->> Mobile browser: Open FaST redirect web page (universal link) with JWT as parameter
alt FaST App is already installed
Mobile browser ->> FaST App: Open FaST App through<br>deeplinking and pass JWT
FaST App ->> FaST Backend: Get FaST user from JWT
FaST Backend ->> FaST Backend: Get or create user
FaST Backend ->> FaST Backend: [... get farms details (see above)]
FaST Backend ->> FaST App: Return user and farms
Note over FaST App: User now logged in<br>to FaST App
else FaST App not installed
Note over Mobile browser: Display redirect<br>page with links to<br>App Store and FaST<br>desktop portal
alt User chooses App Store
Mobile browser ->> FaST App: Navigate to App store to install FaST App
Note over Mobile browser: User installs the<br>FaST App
Note over Mobile browser: User installs the<br>FaST App
else User chooses desktop portal
Mobile browser ->> FaST Backend: Navigate to FaST desktop portal with JWT
FaST Backend ->> FaST Backend: Get or create user
FaST Backend ->> FaST Backend: [... get farms details (see above)]
FaST Backend ->> Mobile browser: Desktop portal with user, farms, etc
Note over Mobile browser: User is now on FaST <br>desktop portal.
end
end
```
## From DAT on desktop
```mermaid
sequenceDiagram
Note over DAT: User navigates to<br>DAT on his mobile <br>browser (or uses<br>DAT PWA)
DAT ->> DAT: User logs in to DAT
Note over DAT: List of farms/roles
DAT ->> DAT: User selects farm + role
Note over DAT: List of services
DAT ->> DAT: User clicks on "FaST"
DAT ->> Browser: Open FaST redirect web page with JWT as parameter
Browser ->> FaST Backend: Navigate to FaST desktop portal with JWT
FaST Backend ->> FaST Backend: Get or create user
FaST Backend ->> FaST Backend: [... get farms details (see above)]
FaST Backend ->> Browser: Desktop portal with user, farms, etc
Note over Browser: User is now on FaST <br>desktop portal.
```
\ No newline at end of file
# Required APIs from IACS systems
The purpose of these APIs is for the FaST to be able to pull the farmer's farm data and present it to him/her automaticallly, without needing any input. This mechanism also lowers the risk of errors (wrong input from the farmer), simplifies and speeds up the process and the adoption of the FaST tool.
The general sequence of usage of the main APIs is as follows:
```mermaid
sequenceDiagram
FaST App ->> FaST Backend: User logs first time
FaST Backend ->> Identity Provider: Delegate authentication
Identity Provider ->> FaST Backend: Authenticated user U
FaST Backend ->> IACS API: Get farms and roles for user U
Note over IACS API: API GetFarms
IACS API ->> FaST Backend: List of farms (F1, F2..) and role for each farm (owner, advisor...)
Note over FaST Backend: Get or create<br/>corresponding farms
loop For each farm
FaST Backend ->> IACS API: Get farm F details from user U
Note over IACS API: API GetFarmDetails
rect rgba(255, 0, 0, 0.2)
IACS API ->> IACS API: Check user U<br>permission for<br/>farm F
end
IACS API ->> FaST Backend: Farm details
end
FaST Backend ->> FaST App: List of farms with details
Note over FaST App: Display farms
Note over FaST App: [later...]<br>User requests<br>farm F1 update
FaST App ->> FaST Backend: User U requests update farm F1
FaST Backend ->> IACS API: Get farm F1 details from user U
Note over IACS API: API GetFarmDetails
rect rgba(255, 0, 0, 0.2)
IACS API ->> IACS API: Check user U<br>permission for<br/>farm F1
end
IACS API ->> FaST Backend: Farm details
Note over FaST Backend: Update farm F1
FaST Backend ->> FaST App: Updated farm F1
```
Therefore, 2 APIs are necessary: one to get the list of farms a user can "see" (with a role) and one to get the details of a given farm (parcels, crops and possibly animals).
The 2 APIs can be regrouped into a single API, which always returns all the details about each farm a user can see.
## API GetFarms
**Inputs**
{
"user_id": ID of the user as returned from the identity provider
}
**Expected output**
List of farms that the user is allowed to query data for, as per the permissions of the IACS system.
HTTP 200
{
"farms": [
{
"farm_id": unique ID of farm,
"role": "advisor", "owner", etc,
"name": [optional] name of the farm,
"address": [optional] address of the farm
},
...
]
}
## API GetFarmDetails
**Inputs**
{
"user_id": ID of the user as returned from the identity provider,
"farm_id": ID of the farm
}
**Expected output**
Details of a specific farm, if the user is allowed to request such data, or an error if not.
HTTP 200
{
"farm_id": ID of the farm,
"agricultural_parcels": [
{
"parcel_id": ID of parcel from the agricultural parcel registry (GIS),
"parcel_block": block identifier, if the parcels are grouped in blocks,
"crops": [
{
"crop_id": ID of crop from national nomenclature,
"date_start": start date of crop,
"date_end": end date of crop,
},
...
],
"geometry": [optional] can be retrieved from the parcel registry GIS
}
],
"livestock": [
{
"building_id": ID of animal building,
"species_id": ID of livestock species from national nomenclature,
"headcount": number of animals of this species in this building,
"weight": [optional] average weight of animals,
"storage_days": [optional] number of days in the building/stable,
"latitude": [optional] can be retrieved from building GIS layer if provided,
"longitude": [optional] can be retrieved from building GIS layer if provided
},
...
]
}
The example above is a "conceptual" example, ie the formats, labels and arrangements can be different if, in the end, the necessary information is provided. Especially, the labels of the fields can be in the national language and do not need to be in English.
In the case where the provided user does not have the access rights to request data for this farm (for example, an advisor that has been removed from a farm), the API should return an error. For example:
HTTP 401 Unauthorized
{
"message": [optional] "User ... does not have the necessary permissions to request data for farm ..."
}
If the requested `farm_id` does not exist, the response should clearly state it. For example:
HTTP 404 Not found
{
"message": [optional] "Farm ... does not exist"
}
## API GetAllParcels
The purpose of this API is to be able to retrieve the *complete* agricultural parcel layer for the region. This layer will be used as a background, and when the farmer wants to add a parcel to his farm in the FaST (or for demo users to build a fake farm using existing agricultural parcels). The IDs of the parcels/features should match the IDs used in the `GetFarmDetails` API.
This API can be provided as a WFS layer, a flat shapefile, or any other static format suitable for sharing a high volume of georeferenced polygons.
## API GetAllBuildings
If the building/stable coordinates are not provided in the *livestock* part of the `GetFarmDetails` API, then a layer of georeferenced points should be provided to retrieve the latitude and longitude for a given `building_id`
## Reference tables
The following reference nomenclature / tables must be provided, at a minimum:
- list of crops: the identifier (ID) column should match the `crop_id` provided in the `GetFarmDetails` API.
- list of livestock species: the identifier (ID) should match the `species_id` provided in the `GetFarmDetails` API.
## Formats and technologies
The APIs should be developped as HTTP or WFS services (or a combination). In case of HTTP, the input and output formats can be selected from standard web formats such as JSON, XML, or other.
## Security considerations
The developed IACS APIs should:
- authenticate the FaST backend on every request: this can be proposed through a signed token (API key), a certificate, HTTP basic authentication, IP whitelisting, or a combination of factors
- transmit data in a secure way: SSL is required
- **it is up to the implemented APIs to check that the provided user still has the access rights to request data for a given farm. The permissions in the FaST database might be stale.**
In the case where FaST backend authentication fails, a self-explanatory error code should be returned, for example:
HTTP 401 Unauthorized or 403 Access forbidden
Note: the requests to the IACS APIs will *always* be performed from the FaST backend, and *never* from the FaST applications (mobile app, portal).
The FaST platform will adapt to the security requirements requested by the Paying Agency.
# Authentication and authorization flows
## Definitions
- Farmer: the legal representative of a farm (physical person)
- Advisor to a farm: a physical person that has been granted the right to view the data of a farm in the FaST
- Fast App: the FaST mobile app or the FaST laptop app for farmers and advisors
- Fast Backend: the FaST cluster or services running on the cluster
- Identity Provider: either a national identity provider (eg TARA for Estonia, SPID for Italy) or an identity provider provided by the Paying Agency or region (eg DAT for Andalucia or the system being developed by the CyL Paying Agency)
- Authorization Provider: a service that stores and serves access rights (which person can see/write which farm data). Usually provided for by the Paying Agency.
## App start and login
This sequence describes the steps between the moment the application starts and the moment the user actually sees his home page. The flow depends on whether the user is already logged in (ie has some token already locally stored in the application) or not.
If the user is not yet logged in (does not have a FaST token locally stored), he is presented a list of regions and applicable identity providers.
If he is logged in (has a FaST token locally), he is immediately redirected to the home page of the app (optimistic navigation), while the token is being validated. If the token is deemed not valid or not refreshable, the user force logged out and redirected to the login page.
```mermaid
sequenceDiagram
Note over FaST App: App starts
alt If no region & token stored in app
FaST App->>FaST Backend: Get list of regions/endpoints
FaST Backend->>FaST App:
Note over FaST App: User selects region
FaST App->>FaST Backend: Get list of identity providers for region
FaST Backend->>FaST App:
Note over FaST App: User selects IdP
FaST App->>Identity Provider: Redirect to identity provider login page
Note over Identity Provider: User logs in with IdP
Identity Provider->>FaST Backend: Redirect with user IdP ID, info and authorization code
FaST Backend->>FaST Backend: Get FaST user for user IdP ID
alt User does not exist
FaST Backend->>FaST Backend: Create user
FaST Backend->>Authorization Provider: Request initial access rights for user
Authorization Provider->>FaST Backend:
FaST Backend->>FaST Backend: Assign rights/roles
end
FaST Backend->>FaST App: Return user details and FaST token
FaST App->>FaST App: Persist region & FaST token in app
else Else if region & token already in app
FaST App->>+FaST Backend: Check token valid
FaST Backend->>FaST Backend: Refresh token if necessary/allowed
alt If token is valid
FaST Backend->>-FaST App: Return (refreshed) token
FaST App->>FaST App: Persist token in app
else If token not valid
FaST Backend->>FaST App: Return login error
FaST App->>FaST App: Delete local token
Note over FaST App: Redirect to login page
end
end
```
## Farmer data requests from government systems
To avoid double inputs by the farmer, the FaST will pull the necessary data from the government systems (usually GSAA systems from the Paying Agency). This data includes mainly the description of parcels, crops, livestock. No financial/subsidy information is required.
For the FaST to query this data, the following conditions are necessary:
- the user is authenticated through an official authentication service (eg TARA, SPID, DAT, etc)
- the user has given his explicit consent for pulling this data from the government system (GDPR)
- the user has the necessary access rights within the government system (ie if the user is an advisor, he/she needs to be declared *in the government system*)
- the FaST backend is authenticated by the government system
```mermaid
sequenceDiagram
Note over FaST App: User is logged in
FaST App->>FaST Backend: Request farm data update from gov data provider
FaST Backend->>FaST Backend: Check user has given GDPR consent
alt User has not given (or has removed) content
FaST Backend->>FaST App: Consent required
Note over FaST App: Redirect to consent<br>form
alt User does not consent
Note over FaST App: Farm data not<br>updated
else User consents
FaST App->>FaST Backend: Persist consentment
end
end
FaST Backend->>Gov Data Provider: Request data from gov systems
Gov Data Provider->>Authorization Provider: Check access rights for user
Authorization Provider->>Gov Data Provider: Return access rights
alt Access rights ok
Gov Data Provider->>FaST Backend: Return data
FaST Backend->>FaST Backend: Persist data
FaST Backend->>FaST App: Return data
Note over FaST App: Farm data updated
else Access rights not ok
Gov Data Provider->>FaST Backend: Return error
FaST Backend->>FaST App: Return error
Note over FaST App: Farm data not<br>updated
Note over FaST App: Explain to user the<br>process to get<br>acccess rights in the<br>gov systems
end
```
## Farmer user adds advisor to his farm
The farmer can add an advisor to one of his farms.
```mermaid
sequenceDiagram
Note over FaST App (farmer): Farmer searches for<br>advisor in FaST<br>directory
FaST App (farmer)->>FaST Backend: Farmer adds advisor to his farm
FaST Backend->>FaST App (advisor): Notify advisor
alt If advisor accepts
Note over FaST App (advisor): Advisor accepts
FaST App (advisor)->>FaST Backend: Accept & add relationship
rect rgb(128, 128, 128)
FaST Backend->>FaST Backend: The relationship is <br>persisted in FaST only
end
FaST Backend->>FaST App (farmer): Notify farmer
Note over FaST App (farmer),FaST App (advisor): At this point, both the farmer and the advisor see the same farm<br>data
else If advisor refuses
Note over FaST App (advisor): Advisor refuses
FaST App (advisor)->>FaST Backend: Refuse
FaST Backend->>FaST App (farmer): Notify farmer
end
```
## Advisor requests access to a farmer's farm
This is the reverse process of the previous paragraph. The creation of a relationship between a farm and an advisor can be initiated by the advisor. In this case, we try to cover the case where the farmer does not yet have the FaST App installed: however, it means that the FaST needs to have access to either the farmer's email address (could be obtained through the IdP or through the gov data provider) or his mobile phone number (same).
```mermaid
sequenceDiagram
Note over FaST App (advisor): Advisor searches for<br>farmer/farm in FaST<br>directory
FaST App (advisor)->>FaST Backend: Advisor adds farm/farmer to his FaST environment
FaST Backend->>FaST App (farmer): Notify farmer
Note over FaST Backend,FaST App (farmer): The notification can be either:<br>- a push notification (if the FaST<br>App is installed)<br>- an email with a link to accept<br>(with a time-limited token)<br>- an SMS with a link to accept<br>(with a time-limited token)<br>- other mechanism?
alt If farmer accepts
FaST App (farmer)->>FaST Backend: Accept & add relationship
rect rgb(128, 128, 128)
FaST Backend->>FaST Backend: The relationship is <br>persisted in FaST only
end
FaST Backend->>FaST App (advisor): Notify advisor
Note over FaST App (farmer),FaST App (advisor): At this point, both the farmer and the advisor see the same farm<br>data.
else If farmer refuses
FaST App (farmer)->>FaST Backend: Refuse
FaST Backend->>FaST App (advisor): Notify farmer
Note over FaST App (advisor): Farm data access<br>request is canceled
end
```
# WORKSHOP: BAZEL + CI/CD
Duration: ~ 1h
## Bazel
### **Why ?**
**Speed up your builds and tests**
Bazel only rebuilds what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.
**One tool, multiple languages**
Build and test Java, C++, Android, iOS, Go and a wide variety of other language platforms. Bazel runs on Windows, macOS, and Linux.
**Scalable**
Bazel helps you scale your organization, codebase and Continuous Integration system. It handles codebases of any size, in multiple repositories or a huge monorepo.
**Extensibleto your needs**
Easily add support for new languages and platforms with Bazel's familiar extension language. Share and re-use language rules written by the growing Bazel community.
### **Basics**
#### How do I use Bazel ?
To build or test a project with Bazel, you typically do the following:
1. **Set up Bazel**. Download and install Bazel. [Linux](https://docs.bazel.build/versions/3.1.0/install-ubuntu.html) [OSX](https://docs.bazel.build/versions/3.1.0/install-os-x.html)
2. **Set up a project** workspace, which is a directory where Bazel looks for build inputs and BUILD files, and where it stores build outputs.
3. **Write a BUILD file**, which tells Bazel what to build and how to build it. You write your BUILD file by declaring build targets using Starlark, a domain-specific language. A build target specifies a set of input artifacts that Bazel will build plus their dependencies, the build rule Bazel will use to build it, and options that configure the build rule.
4. **Run Bazel** from the command line. Bazel places your outputs within the workspace.
### **How does Bazel work ?**
When running a build or a test, Bazel does the following:
1. **Loads** the BUILD files relevant to the target.
2. **Analyzes** the inputs and their dependencies, applies the specified build rules, and produces an action graph.
3. **Executes** the build actions on the inputs until the final build outputs are produced.
### **The Fast Plaform case**
#### The Core module
Focus on:
* services/workshop/hello_clement (FastAPI project)
* services/web/backend (Django project)
### Take away
TODO
## Hands-on
### Exercice 1: Update an existing core service
Add a new handler to the core/services/workshop/hello_clement service as described below:
paths:
* / <-- already implemented
* /hello-{my-own-name} <-- to implement (return 'hello Clément it's {my-own-name} !')
tips:
1. Create a feature branch
```bash
git checkout -b feature/workshop-hello-clement-new-handler-{my-own-name}
cd services/workshop/hello_clement
```
2. Start the FastAPI app with Bazel
```bash
bazel run .
```
3. Implement the new handler **/hello-{my-own-name}**
4. Restart the app (cf. 2)
5. Run local tests with Bazel (to avoid unnecessary CI iterations)
```bash
## To test only the service
bazel test //services/workshop/hello_clement
## To test all the core module
bazel test //...
```
6. Push the feature branch
```bash
git push --set-upstream origin feature/workshop-hello-clement-new-handler-{my-own-name}
```
7. If OK open a new merge request on gitlab.com and check the running pipeline
![](merge-request-1.png)
8. Check the running CI pipeline
![](merge-request-2.png)
9. If the pipeline succeeds deploy a review (click on the Deploy button)
![](merge-request-3.png)
10. When ready click on the view app button
![](merge-request-4.png)
11 **Edit the URL** as following:
```
https://workshop-hello-clement.your-mr-slug.reviews.staging.fastplatform.eu/hello-{my-own-name}
```
### Exercice 2: Create a new core service
Add a service to the core in core/services/workshop/hello_{my-own-name}with root handler as described below:
paths:
* / <-- to implement (return 'Hello World by {my-own-name} !')
......@@ -147,9 +147,9 @@ A review workshop consists of three different groups participating in different
**Workshop Organisation**
Each workshop will consist of 1 session per user group. Each session should last 1.5h with 15 to 30 minutes between any back to back sessions. Sessions must be successive but can be organised how you wish, spreading over the 2 days if necessary.
Each workshop will consist of 1 session per user group. Each session should last 1h30 with 15 to 30 minutes between any back to back sessions. Sessions must be successive but can be organised how you wish, spreading over the 2 days if necessary.
- Plan a lunch break for at least 1.5h
- Plan 1.5h at the end of the day to debrief if possible
- Plan 1h30 at the end of the day to debrief if possible
**Example Agenda**
......@@ -364,6 +364,37 @@ The data that FAST will need to be able to propose services to the farmer only i
This data is "pulled" into FaST but never pushed back into the IACS system ; the process is purely one way. Once the data is inside FaST, the farmer has the possibility to edit it in FaST without any impact his IACS/GSAA data.
```mermaid
sequenceDiagram
FaST App ->> FaST Backend: User logs first time
FaST Backend ->> Identity Provider: Delegate authentication
Identity Provider ->> FaST Backend: Authenticated user U
FaST Backend ->> IACS API: Get farms and roles for user U
Note over IACS API: API GetFarms
IACS API ->> FaST Backend: List of farms (F1, F2..) and role for each farm (owner, advisor...)
Note over FaST Backend: Get or create<br/>corresponding farms
loop For each farm
FaST Backend ->> IACS API: Get farm F details from user U
Note over IACS API: API GetFarmDetails
rect rgba(255, 0, 0, 0.2)
IACS API ->> IACS API: Check user U<br>permission for<br/>farm F
end
IACS API ->> FaST Backend: Farm details
end
FaST Backend ->> FaST App: List of farms with details
Note over FaST App: Display farms
Note over FaST App: [later...]<br>User requests<br>farm F1 update
FaST App ->> FaST Backend: User U requests update farm F1
FaST Backend ->> IACS API: Get farm F1 details from user U
Note over IACS API: API GetFarmDetails
rect rgba(255, 0, 0, 0.2)
IACS API ->> IACS API: Check user U<br>permission for<br/>farm F1
end
IACS API ->> FaST Backend: Farm details
Note over FaST Backend: Update farm F1
FaST Backend ->> FaST App: Updated farm F1
```
> **Activities:**
>
> - Identify which IT system contains the relevant farmer's data
......@@ -541,6 +572,42 @@ Security-wise, the delivery team will adapt to your requirements, using for exam
- Custom network (e.g. X-Road)
- ...
```mermaid
sequenceDiagram
Note over FaST App: App starts
alt If no region & token stored in app
FaST App->>FaST Backend: Get list of regions/endpoints
FaST Backend->>FaST App:
Note over FaST App: User selects region
FaST App->>FaST Backend: Get list of identity providers for region
FaST Backend->>FaST App:
Note over FaST App: User selects IdP
FaST App->>Identity Provider: Redirect to identity provider login page
Note over Identity Provider: User logs in with IdP
Identity Provider->>FaST Backend: Redirect with user IdP ID, info and authorization code
FaST Backend->>FaST Backend: Get FaST user for user IdP ID
alt User does not exist
FaST Backend->>FaST Backend: Create user
FaST Backend->>Authorization Provider: Request initial access rights for user
Authorization Provider->>FaST Backend:
FaST Backend->>FaST Backend: Assign rights/roles
end
FaST Backend->>FaST App: Return user details and FaST token
FaST App->>FaST App: Persist region & FaST token in app
else Else if region & token already in app
FaST App->>+FaST Backend: Check token valid
FaST Backend->>FaST Backend: Refresh token if necessary/allowed
alt If token is valid
FaST Backend->>-FaST App: Return (refreshed) token
FaST App->>FaST App: Persist token in app
else If token not valid
FaST Backend->>FaST App: Return login error
FaST App->>FaST App: Delete local token
Note over FaST App: Redirect to login page
end
end