Rocto issues error when queries are larger than 32KiB (Octo works just fine)
For example, I used the following commands to create a query file table1.sql
that contains a CREATE TABLE
query with 10,000 columns (code pasted from the test_createtable/TC001
bats subtest). The octo -f table1.sql
command at the end works fine.
cat <<OCTO > table1.sql
CREATE TABLE table1 (
id INTEGER PRIMARY KEY
$(for i in $(seq 1 10000); do echo ", column$i VARCHAR"; done)
);
OCTO
# Load the table schema into octo
octo -f table1.sql >& output.txt
Issue 1
But if I try the same with Rocto, I get the following error in the rocto server log.
[127.0.0.1:57156] [ERROR] src/rocto/read_bytes.c:39 2020-09-07 14:55:58 : Read size 208946 greater than buffer size 32768
Issue 2
I also see the following code in rocto which issues an error if we find a query greater than MAX_STR_CONST
.
src/rocto/handle_query.c
49 // If the query is bigger than the buffer, we would need to copy data from
50 // the query to the buffer after each block is consumed, but right now
51 // this buffer is large enough that this won't happen
52 // Enforced by this check
53 if (query_length > cur_input_max) {
54 ERROR(ERR_ROCTO_QUERY_TOO_LONG, "");
55 return 1;
56 }
Whereas in octo, we have code to handle resizing.
./src/readline_get_more.c
75 /* if input line is too long resize buffer
76 * by min(cur_input_max * 2, line_length) + 2 (for the \n\0)
77 */
78 if (line_length >= cur_input_max - cur_input_index - 2) {
79 int resize_amt = line_length > (cur_input_max * 2) ? line_length : (cur_input_max * 2);
80 char *tmp = malloc(resize_amt + 2);
81 memcpy(tmp, input_buffer_combined, cur_input_index);
82 free(input_buffer_combined);
83 input_buffer_combined = tmp;
84 cur_input_max = resize_amt;
85 }
Issue 3
The maximum allowed query size is currently set to MAX_STR_CONST
. This is not a hardcoded constant. It could be set to different values depending on the value the cmake variable STRING_BUFFER_LENGTH
is set to before running the cmake command as part of building Octo from source. Therefore, one build of Octo would behave differently than another build of Octo with a different value of STRING_BUFFER_LENGTH
. This is not desirable. Code like the below is best changed to use a hardcoded constant like 32KiB instead of a variable constant like MAX_STR_CONST
. All usages of MAX_STR_CONST
need to be examined to see if they fall in this category and if so fixed to use a hardcoded constant instead.
src/octo_init.c
707 // Leave space for null terminator
708 // `read` takes the number of bytes to read _excluding_ the null terminator,
709 // and we pass cur_input_max directly into read in `readline_get_more`.
710 cur_input_max = MAX_STR_CONST - 1;
711 input_buffer_combined = malloc(MAX_STR_CONST);
712 memset(input_buffer_combined, 0, MAX_STR_CONST);