Skip to content

Operation functions: revamp, first draft

Rémy El Sibaïe requested to merge new-operation-functions into master

The script is currently a test-script, not mergeable in this shape. But I'll amend my commit or add commits if you have remarks. Also I'm not really an indexer developer, don't hesitate to give me advices on how to organize my file/code/build process.

The purpose of this MR is to improve/rewrite the operations fetching functions located in mezos.sql.

Why ? Problems and needs

  • We need all operations including mempool grouped/sortable/filterable in one query result
  • The current implementation is very verbose and relies on code generation of sql on OCaml side where it should be full sql if possible. (easier to filter/sort etc)
  • In the current implementation the limit parameter is passed to all union operand. It is valid only if the order is given. In the current implementation, there is only one possible ordering which is on operation_id
  • We need token transactions to be represented as standard transactions.

Variations from current implementation

I used the same query pattern as this file, except for a few variations.

differences:

  • define a common output type to reduce verbosity
  • all specific data lies in a jsonb field called data. It avoid all the null fields in returned value
  • an array of public key hashes as parameter to be able to ask for several keys
  • keys can be both source or destination
  • no more limit in each union operand

Token and mempool

It was more efficient to fetch tokens transfers in a specific get_tx_token select and to filter non-token transfer from the general get_tx (renamed `get_tx_non_token) related select and then union.

TODO: The mempool will be add later when I have a database to test on

Filtering

  • Filtering by kind : the second parameter is an array containing kinds names as a text. If empty, all the operations are fetched. If it contains at least an element, union operand where kind is not contained in this array are filtered out.

Result

I ran those queries to evaluate performance

explain analyze select count(*)
     from get_manager_operations(ARRAY['tz1UBwnNSFrvbPmtVT4vHCsba3mjMGtKhpu2', 'tz1Rrc4XieKdAqhLkYHGJUeJGGjZWMfx7qgC'], ARRAY[]:text);
select count(*)
    from get_manager_operations(ARRAY['tz1UBwnNSFrvbPmtVT4vHCsba3mjMGtKhpu2', 'tz1Rrc4XieKdAqhLkYHGJUeJGGjZWMfx7qgC'], ARRAY[]:text);

explain analyse tries to evaluate the query execution time statically and gives a strategy of this execution, then it executes it. This query fetches operations for two keys, and we are around 10ms/key on my laptop on qa-api, which is pretty good. We should test it on api mainnet now.

                                                           QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=12.75..12.76 rows=1 width=8) (actual time=20.544..20.545 rows=1 loops=1)
   ->  Function Scan on get_manager_operations  (cost=0.25..10.25 rows=1000 width=0) (actual time=20.435..20.514 rows=366 loops=1)
 Planning Time: 0.066 ms
 Execution Time: 20.593 ms
(4 rows)
 count 
-------
   366
(1 row)
Edited by Philippe Wang

Merge request reports