Variable `old` which contains tuple reference wasn't
unrefed at all. Fix this.

Sergey Ostanevich authored Oct 23, 2020 and Vladislav Shpilevoy committed Nov 03, 2020

Static buffer overflow in thread local pool causes random fails on OSX
platform. This was caused by an incorrect use of the allocator result.

Fixes #5312
Co-authored-by: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>

"Too long WAL write" is supposed to warn a user that either the
disk write was too long, or the event loop is too slow, maybe due
to certain fibers not doing yields often enough.

It was printed by the code doing the transaction commit. As a
result, for synchronous transactions the check also included the
replication time, often overflowing a threshold and printing
"too long WAL write" even when it had nothing to do with a WAL
write or the event loop being too slow.

The patch makes so the warning is checked and printed after WAL
write right away, not after commit.

Closes #5139

txn_complete used to handle all the transaction outcomes:
- manual rollback;
- error at WAL write;
- successful WAL write and commit;
- successful WAL write and wait for synchronization with replicas.

The code became a mess after synchronous replication was
introduced. This patch splits txn_complete's code into multiple
pieces.

Now WAL write success and fail are handled by
txn_on_journal_write() exclusively. It also runs the WAL write
triggers. It was very strange to call them from txn_complete().

txn_on_journal_write() also checks if the transaction is
synchronous, and if it is not, it completes it with
txn_complete_success() whose code is simple now, and only works
on committing the changes.

In case of fail the transaction always ends up in
txn_complete_fail().

These success and fail functions are now used by the limbo as
well. It appeared all the places finishing a transaction always
know if they want to fail it or complete successfully.

This should also remove a few ifs from the hot code of transaction
commit.

The patch simplifies the code in order to fix the false warning
about too long WAL write for synchronous transactions, which is
printed not at WAL write now, but at commit. These two events are
far from each other for synchro requests.

Part of #5139

The function is called only by the journal when write is finished.

Besides, it may not complete the transaction. In case of
synchronous replication it is not enough for completion. It means,
it can't have 'complete' in its name.

Also the function is never used out of txn.c, so it is removed
from txn.h and is now static.

The patch is a preparation for not spaming "too long WAL write" on
synchronous transactions, because it is simply misleading.

Part of #5139

Cyrill Gorcunov authored Oct 29, 2020 and Vladislav Shpilevoy committed Nov 02, 2020

We should never trust the request data it can
carry anything, thus lets make sure we're not
decoding some trash into a string.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Cyrill Gorcunov authored Oct 28, 2020 and Vladislav Shpilevoy committed Nov 02, 2020

The size should be matched to the real size of a buffer,
otherwise it is a room for mistake. Same time make sure
we're not overriding the buffer.
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Alexander V. Tikhonov authored Oct 29, 2020 and Kirill Yukhin committed Nov 01, 2020

Found that the previously fixed vinyl/gh.test.lua test in commit:

  94dc5bdd ('test: gh test hangs after gh-4957-too-many-upserts')

with adding fiber.sleep(1) workaround to avoid of raise from the
previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
changed in the other way. The new change from one side will leave
the found issue untouched to be able to resolve it within opened
issue in github. And from the other side it will let the test-run
tool to be able to avoid of this issue using fragile list feature
to save the stability of testing due to found issue is flaky and
can be passed on reruns.

The current fix changes the forever waiting loop to especially
created for such situations test_run:wait_cond() routine which has
timeout in it to avoid of hanging the test till global timeout will
occure. It will let the testing to be continued even after the fail.

Needed for #5141

Alexander V. Tikhonov authored Oct 29, 2020 and Kirill Yukhin committed Nov 01, 2020

Added restart the current server to resolve the issue #5141 which
reproduced in test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.

Part of #5141

Alexander V. Tikhonov authored Oct 29, 2020 and Kirill Yukhin committed Nov 01, 2020

Created the stable reproducer for the issue #5141:

  box.snapshot()
  ---
 -- ok
 +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
  ...

flaky occured in vinyl/ suite tests if running after the test:

  vinyl/gh-4957-too-many-upserts.test.lua

as new standalone test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

based on test:

  vinyl/gh-4957-too-many-upserts.test.lua

Due to issue not reproduced on FreeBSD 12, then test was blocked with:

  vinyl/gh-5141-invalid-vylog-file.skipcond

Needed for #5141

Alexander V. Tikhonov authored Oct 29, 2020 and Kirill Yukhin committed Nov 01, 2020

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.
Also added checksums to fragile list for the following tests:

  vinyl/iterator.test.lua                       gh-5141
  vinyl/snapshot.test.lua                       gh-4984

Needed for #5141
Needed for #4984

Alexander V. Tikhonov authored Oct 08, 2020 and Kirill Yukhin committed Nov 01, 2020

Sometimes it is convenient to use default compiler on CentOS 7.
Added test job which uses for compiling default compiler files:

  CC=/usr/bin/gcc
  CXX=/usr/bin/g++

Closes #4941

Alexander V. Tikhonov authored Oct 14, 2020 and Kirill Yukhin committed Nov 01, 2020

Added ENABLE_WERROR flag to build options to enable Werror.

Part of #4941

Alexander V. Tikhonov authored Oct 20, 2020 and Kirill Yukhin committed Nov 01, 2020

Building with gcc-4.8.5 on CentOS 7 found issue:

cd /source/build/usr/src/debug/tarantool-2.6.0.144/test/unit && /usr/bin/g++ ... -Wp,-D_FORTIFY_SOURCE=2 ... -O2 ... -O0 -o CMakeFiles/fiber.test.dir/fiber.cc.o -c /source/build/usr/src/debug/tarantool-2.6.0.144/test/unit/fiber.cc
In file included from /usr/include/inttypes.h:25:0,
                 from /source/build/usr/src/debug/tarantool-2.6.0.144/src/lib/small/small/region.h:34,
                 from /source/build/usr/src/debug/tarantool-2.6.0.144/src/lib/core/memory.h:33,
                 from /source/build/usr/src/debug/tarantool-2.6.0.144/test/unit/fiber.cc:1:
/usr/include/features.h:330:4: error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Werror=cpp]
 # warning _FORTIFY_SOURCE requires compiling with optimization (-O)

It happened because _FORTIFY_SOURCE=2 flag needed -O[1|2] optimization,
but latest set in command was -O0. To fix it removed not needed '-O0'
optimization from test/unit/CmakeLists.txt file. This optimization
became unneeded after the commit:

  aa78a941 ("test/uint: fiber")

when the test was completely rewritten.

Needed for #4941

Store *.reject files in ${BUILD}/test/var/rejects/<...>/ instead of
${SOURCE}/test/<...>/.

The past approach leads to problems with testing, when the out of source
build is used and sources are on a read-only filesystem. The main
problem is when a test fails, but it is marked as fragile and should be
run again. The test fail assumes storing of the .reject file and the
testing fails on attempt to write to the read-only filesystem. The
re-run is not performed so.

Side effect for the in-source build: since the test/var/ directory is
gitignored, the *.reject files will not shown in `git status` output
anymore.

https://github.com/tarantool/test-run/pull/209

Follows up #4874

Mary Feofanova authored Oct 06, 2020 and Nikita Pettik committed Oct 30, 2020

Closes #5071

@TarantoolBot document
Title: memtx: varbinary supported in bitset indexes
Now it is possible to create bitset indexes on the fields of varbinary type,
e.g.: s:create_index('b', {type = 'bitset', parts = {1, "varbinary"}})

* test: fix warnings spotted by luacheck

Mary Feofanova authored Oct 08, 2020 and Nikita Pettik committed Oct 30, 2020

Previously accepted formats of index parts:
parts = {field1, type1, field2, type2}, or
parts = {{field1, type1, ...}, {field2, type2, ...}}

Now it is allowed to write without extra brace if there is one part only:
parts = {field1, type1, ...}

Closes #2866

Sergey Bronnikov authored Oct 22, 2020 and Kirill Yukhin committed Oct 30, 2020

To run Jepsen tests in different configurations we need to parametrize run
script by options, so lein options and number of nodes passed with environment
variables. By default script runs testing with Tarantool built from latest
commit.

Added these configurations:

- single instance
- single instance with enabled TXM
- cluster with enabled Raft
- cluster with enabled Raft and TXM

Closes #5437

When the connection was not established yet, netbox reported empty error
while executing a remote request.

Closes #4787

Aleksandr Lyapunov authored Oct 22, 2020 and Kirill Yukhin committed Oct 22, 2020

Tx stories must be linked into correct double-linked list.
Preserve it.

Part of #5423

Aleksandr Lyapunov authored Oct 21, 2020 and Kirill Yukhin committed Oct 22, 2020

There was a mess in tuple refernce in TX history.
Now it was remade in the following asumptions:
 * a clean tuple belongs to space, and the space implicitly holds
a reference to the tuple.
 * a dirty tuple belongs to TX manager and a reference is held
in the corresponding story.

Closes #5423

Serge Petrenko authored Oct 22, 2020 and Kirill Yukhin committed Oct 22, 2020

When an instance is configured as candidate, it has a leader death timer
ticking constantly to schedule an election as soon as leader disappears.
When the instance receives the leader's heartbeat, it resets the timer
to its initial value.

When being a voter, the instance ignores heartbeats, since it has
nothing to wait for. So its timer must be stopped. Otherwise it'll try
to schedule a new election and fail.

Stop the timer on transition from candidate to voter.

Vladislav Shpilevoy authored Oct 17, 2020 and Kirill Yukhin committed Oct 22, 2020

When a node becomes a leader, it restarts relay recovery cursors
to re-send all the data since the last acked row.

But during recovery restart the relay lost the trigger, which used
to update GC state in TX thread.

The patch preserves the trigger.

Follow up for #5433

Vladislav Shpilevoy authored Oct 17, 2020 and Kirill Yukhin committed Oct 22, 2020

When a Raft node is elected as a leader, it should resend all its
data to the followers from the last acked vclock. Because while
the node was not a leader, the other instances ignored all the
changes from it.

The resending is done via restart of the recovery cursor in the
relay thread. When the cursor was restarted, it used the last
acked vclock to find the needed xlog file. But it didn't set the
local LSN component, which was 0 (replicas don't send it).

When in reality the component was not zero, the recovery cursor
still tried to find the oldest xlog file having the first local
row. And it couldn't. The first created local row may be gone long
time ago.

The patch makes the restart keep the local LSN component
unchanged, as it was used by the previous recovery cursor, before
the restart.

Closes #5433

Alexander V. Tikhonov authored Oct 16, 2020 and Kirill Yukhin committed Oct 22, 2020

  box/net.box_incorrect_iterator_gh-841.test.lua	gh-5434
  replication/election_basic.test.lua			gh-5368
  replication/election_qsync.test.lua			gh-5430
  replication/election_qsync_stress.test.lua		gh-5395
  replication/gh-4402-info-errno.test.lua		gh-5366
  replication/gh-5426-election-on-off.test.lua		gh-5433
  wal_off/snapshot_stress.test.lua			gh-5431

Alexander V. Tikhonov authored Oct 19, 2020 and Kirill Yukhin committed Oct 22, 2020

Test replication/election_qsync_stress.test.lua flaky fails and even
often on FreeBSD 12. It has stable reproducer shown in the issue. To
stabilize the gitlab testing blocked this test from running on
FreeBSD 12 with skip condition file till it will be fixed.

Part of #5395

Sergey Kaplun authored Oct 16, 2020 and Kirill Yukhin committed Oct 22, 2020

Since LuaJIT provides public C API symbols that are used in the final
executable, the linker may not just throw it away.
Nevertheless for future compatibility all symbols from LuaJIT public API
should be added to exports.h file.

Follows up #5187

Sergey Kaplun authored Oct 16, 2020 and Kirill Yukhin committed Oct 22, 2020

Since LuaJIT provides extended LuaC API introduced in the scope of
5a61e1ab54b5c66bfebd836db1ac47996611e065 ('misc: add C and Lua API for
platform metrics') corresponding header should be provided along with
other Tarantool development files.

Follows up #5187

Aleksandr Lyapunov authored Oct 15, 2020 and Nikita Pettik committed Oct 21, 2020

Since 9fba29ab (memtx: introduce tuple
compare hint) memtx tree key data (indexes) started to contain extra 8
bytes as a hint. Now it is optional and can be turned off in an index
options with "hint = false" entry.

Closes #4927

@TarantoolBot document
Title: memtx: optional tuple compare hints
Update the documentation for an index creation to reflect that there is
now an option to turn off hints for the index.

Ilya Kosarev authored Oct 12, 2020 and Nikita Pettik committed Oct 21, 2020

It is needed for the further c++ implementation in memtx_tree.cc. To
see the file diff properly it should not be renamed and reworked
in one commit. Some not c++ comparable casts were fixed.

Prerequisites: #4927

* misc: add lmisclib.h to lua.hpp header
* build: fix build for Windows-like systems
* test: fix warning for old gcc at testgetmetrics.c

To make vinyl/upgrade tests hierarchy clean and follow more extendible
folder structure (like in xlog/upgrade), let's a bit reorganize upgrade
test.

Follow-up #5107

It was disabled since 9cf66ac0
Let's remove it to avoid test contamination folder.

Kirill Yukhin authored Aug 11, 2020 and Nikita Pettik committed Oct 20, 2020

This patch enables "ephemeral index" optimization (which is called
Automatic Indexes in the original SQLite implementation). The main idea
behind the optimization is to create temporary (ephemeral) space that lasts
only the duration of single SQL statement. Such space stores intermediate
execution results. Since in Tarantool index can't exist without space
(in contrast to SQLite) we call this optimization "ephemeral index".
It may turn out beneficial due to the fact that cost of creating and
filling space with single index is (NlogN). So if we lookup data more
than logN times during the query execution, ephemeral index may give as
better timings (space full-scan is still O(N)).

As a rule "ephemeral index" space relates to one of spaces used in query
and contains all or fewer fields than the original space. The order of
fields in ephemeral space may be different from order in the original space.
In addition to the mentioned fields, ephemeral index has a rowid -
additional unsigned field with non-decreasing values which allows to
store not unique values.
Example:

CREATE TABLE t1 (i INT PRIMARY KEY, a INT, b INT, c INT);
CREATE TABLE t2 (i INT PRIMARY KEY, a INT, b INT, c INT);
...
SELECT t1.a, t2.b FROM t1, t2 WHERE t1.b = t2.c

In case the planner decides to use an ephemeral index and chooses t1 as
the source space, an ephemeral index with three fields will be created
and will contain the values from t1.b, t1.a, and rowid in that order.
If t2 is chosen as the source space, an ephemeral index will contain the
values from t2.c, t2.b, and rowid.

This optimization can also be applied to store results of subquery
evaluation.

Closes #4933
Co-authored-by: Mergen Imeev <imeevma@gmail.com>
Co-authored-by: Timur Safin <tsafin@tarantool.org>
Reviewed-by: Nikita Pettik <korablev@tarantool.org>

Alexander V. Tikhonov authored Oct 16, 2020 and Kirill Yukhin committed Oct 17, 2020

Found that if the previos test like gh-4957-too-many-upserts.test.lua
leaves some garbage on the current Tarantool server, then the next
running gh.test.lua test hangs. To resolve it decided to use workaround
and to set delay before the test start with fiber.sleep(1) to be able
to get out of one loop iteration.

Closes #5141

Timur Safin authored Oct 16, 2020 and Kirill Yukhin committed Oct 17, 2020

Adding forgotten resource cleanup to the
`test_tuple_validate_formatted` test, to make ASAN
happy again.

Closes #5432
Relates to #5384