[#860] Fix buffer overflow in OPEN of SOCKET device when LISTEN device parameter is too long

Background

• This is a longstanding issue (in YottaDB and the upstream GT.M) identified by fuzz testing.

• Below is a simple test case demonstrating the issue.

  Debug build

  YDB>set sf="local.socketxxxxxxxxxxxxe xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  YDB>open "s":(LISTEN=sf_":LOCALxDELIMITER=$c(10):ATTACHxistener")::"SOCKET"
  %YDB-F-ASSERT, Assert failed in sr_port/gtm_memcpy_validate_and_execute.c line 36 for expression ((GDL_AllowLargeMemcpy & ydbDebugLevel) || ((0 <= (signed)len) && (MAXPOSINT4 >= len)))

  Release build

  YDB>set sf="local.socketxxxxxxxxxxxxe xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  YDB>open "s":(LISTEN=sf_":LOCALxDELIMITER=$c(10):ATTACHxistener")::"SOCKET"*** buffer overflow detected ***: terminated
  Abort (core dumped)

Issue

• Below is a debugger analysis of the generated core file.

  (gdb) where
  .
  .
  #8  gtm_memcpy_validate_and_execute (target=0x7ffca0dc5410, src=0x1523605, len=18446744073709551493) at sr_port/gtm_memcpy_validate_and_execute.c:36
  #9  iosocket_open (dev=0x14daa40, pp=0x1522e28, file_des=-2, mspace=0x7f2cd1bde498, timepar=9223372036854775800) at sr_port/iosocket_open.c:309
  #10 io_open_try (naml=0x14daa40, tl=0x14daa40, pp=0x1522e28, nsec_timeout=9223372036854775800, mspace=0x7f2cd1bde498) at sr_unix/io_open_try.c:568
  #11 op_open (device=0x7f2cd1bde538, devparms=0x1522e28, timeout=0x7f2cd1bde4b8, mspace=0x7f2cd1bde498) at sr_port/op_open.c:160
  
  (gdb) f 9
  #9  iosocket_open (dev=0x14daa40, pp=0x1522e28, file_des=-2, mspace=0x7f2cd1bde498, timepar=9223372036854775800) at sr_port/iosocket_open.c:309
  309                       memcpy(sockaddr, pp->str.addr + p_offset + 1, len);
  
  (gdb) list
       304          listen_specified = TRUE;
   --> 305          len = (int)(*(pp->str.addr + p_offset));
       306          if (len < SA_MAXLITLEN)
       307          {
       308                  memset(sockaddr, 0, SIZEOF(sockaddr));
       309                  memcpy(sockaddr, pp->str.addr + p_offset + 1, len);
       310          } else
       311                  rts_error_csa(CSA_ARG(NULL) VARLSTCNT(6) ERR_ADDRTOOLONG, 4, len,
       312                                   pp->str.addr + p_offset + 1, len, SA_MAXLITLEN);
       313          break;
  
  (gdb) p len
  $1 = -123
  
  (gdb) p *(pp->str.addr + p_offset)
  $2 = -123 '\205'
  
  (gdb) p (unsigned char)*(pp->str.addr + p_offset)
  $3 = 133 '\205'

• The issue is in line 305 above. We get the 1 byte value (type char which is a signed quantity) and store it in len as an int type. Since int is a signed quantity too, a char value that is greater than 127 (but less than 256) gets stored as a negative value in len and that negative value gets passed to the memcpy() at line 309 and causes the failure.

Fix

• The fix is simple and is to type cast the character value to a unsigned char type before type casting it to a int type. This follows similar type casting done in other places in this file already (see delimiter_len variable for example).

• While at this, noticed a few other places which require similar fixes so made that too.