API improvements

The YDBGo API design leaves much to be desired. This provides a list of improvements. This should be implemented in conjunction with speed improvements issue #54 (closed). Backward compatibility should be maintained, but old function names should be deprecated. This is straightforward because the new API names will seldom overlap the old.

YDBGo documentation has the following potentially misleading statements:

  • BufferT.Alloc "allocates a buffer in YottaDB heap space ..."
  • EasyAPI docs: "As the Go Easy API involves more copying of data between the Go and YottaDB runtime systems, it requires the CPU to perform a little more work than the Go Simple API does. Whether or not this has a measurable impact on performance depends on the application and workload."

These are misleading because:

  • The allocated buffer is in C heap space, not in YottaDB heap space (which the YottaDB C API does not expose).
  • The problem is not "more copying of data between Go and the YottaDB runtime": no more data is transferred to the runtime. Rather, the problem is allocation and constant recreation of data to the EasyAPI wrapper. Specifically, of KeyT, BufferT, and BufferArrayT instances and their subscript arrays on every API invocation.
  • This extra copying occurs because the EasyAPI is not well designed to expose a simple 'node' type like KeyT but with methods that operate simply on that node.
  • The documentation above also downplays the performance problems of EasyAPI which in reality is about 10x slower than SimpleAPI (see YDBLangSpeed benchmark).
  • Instead, a simpler and better API is proposed below that uses 'node' methods to simplify and supercede both the EasyAPI and the SimpleAPI.

Here are issues with the Go documentation section of the MultiLangProgGuide:

  • There should be a section in the README akin to the Intro by example in the YDBLua docs. A Quick Reference Summary table would also be helpful.
  • Reference documentation for each function is slow to grok because:
    • the functions do not have a purpose statement at the start. You have to read through all the notes on exceptional error conditions first. For an example, see BufferT.SetValBAry().
    • SetValBAry() has exactly the same description, verbatim, as SetValStr() so that the reader is left guessing at the difference in functionality based on function name. This is a problem for both BufferT and BufferTArray methods.
    • Only the last bulleted note for many functions (e.g. the above two functions) contains the purpose statement, and even that is worded obscurely as an 'otherwise' clause.
    • There is nothing that tells user that Varnm and SubAry are field names of KeyT instances here.
  • There are separate docs on pkg.go.dev that are auto-generated from the source comments, but they are different than the official docs.
    • This website is the first port of call for Go programmers, so the source comments should be updated to make them the same as the official docs. Then the auto-generated docs should be auto-imported into the official reference manual like was done for the Lua reference docs.
  • Remove "Go" prefix to function names. This is unlikely to cause disambiguation issues and if it does, they should be resolved in the documentation. Anyway, using the auto-generated docs may fix this.
  • Doc for BufferTArray.SetElemLenUsed() references a parameter called buft which is not defined in the prototype.

Issues with the API itself:

  • Function invokation would be a lot tidier if tptoken and errstr were removed. This may be done by storing them in the type instance: either in the KeyT/BufferT instance. Alternatively, and perhaps better, all functions could move to be methods of a master 'connection' type that would store tptoken and errstr. So you would invoke conn.SetValE(...) instead of yottadb.SetValE(yottadb.NOTTP, errstr, ...). Each thread will need its own connection to resolve the threading issues.
  • Method names are systematic but obscurely named. It is not Go-like to append T or ST.
    • For example SetValE() should simply be called Set()
    • Common Go names like NewBuffer, Len, Cap should replace Alloc, LenAlloc and LenUsed.
    • Check whether CGo will allow Alloc() to be incorporated into NewBuffer() and automate defer Free()
      • For example, inBuffer.Str2ZwrST(yottadb.NOTTP, nil, &out) would become the much more readable ydb.Str2Zwr(&out)
  • The DumpToWriter() method on 3 different types seems gratuitous, given the existence of Dump(). It should be renamed to node.GoString() so that Go's fmt.Println("%#v", node) shows dump format. We should define node.String() to work with Go's fmt.Println("%v", node)
  • Dump() would be much more useful if it were extended to output not just metadata but also data contents of the buffer (as ascii), given an optional maxlen parameter to limit each ascii string (0 to suppress contents output like it currently does).
  • BufferTArray.TpST() is a prohibitively complex way to invoke a transaction.
    • A Go programmer should be using Go locals, not M locals, so functionality saving M locals should not be provided.
    • At the very least, a TpST() function should be provided that requires no BufferTArray.
    • Transid should be an optional parameter and default to whatever it does in M.
  • BufferTArray.Alloc() is essentially just to create a subscript array. It will be used frequently and should be convenient. It should accept a vararg list of strings to preload into BufferTArray. There could be a AllocBAry() to match if desired. All other methods to fill its buffer contents should be deprecated. It does not need to be mutable.
  • NodeNext() and SubNext() should be implemented as Go iterators so that they can be used in a FOR loop, like in YDBLua
  • There is no way to simply read KeyT as a string. It has to go into a BufferT first. This is not nice.
  • The API for performing database transactions should be enhanced in two ways:
    • Automatically update the connection's tptoken when inside a transaction so that tptoken is entirely transparent to the user. This has already been implemented in discarded MR !196 (closed) with code here.
    • Provide a wrapping tool which takes a functions and wrap it so that the entire function is automatically performed inside a transaction. This is like what is provided in the YDBLua API here, where transaction(func) returns a new function that runs func within a transaction.
  • The mechanism to call M routines should be greatly simplified for the user:
    • Provide just one API call yottadb.Import(callinTable), analogous to yottadb.require() in YDBLua here. It should return a Go map as a dictionary of M routines defined in the call-in table. Each function in that map should be wrapped to call the M routine as if it were a Go function.
    • Parse the callinTable to extract pre-allocation information so that the Go wrapper knows how much C memory to allocate for each return value, and how many return values to expect. This will remove many error spots for the user.
  • The RegisterSignalNotify function signature could be improved:
    • Rename RegisterSignalNotify to SignalNotify (matches signal.Notify), and UnregisterSignalNotify to SignalReset (matches signal.Reset).
    • panic on error in RegisterSignalNotify (as the code is doing but not the type signature), instead of returning an error, so it's more like signal.Notify which can't fail (and because calling it with an invalid signal is almost certainly a programmer error).
    • Make the notifyChan a chan os.Signal instead of chan bool to mirror signal.Notify (useful if you use the same channel to listen for multiple signals)
    • Make the ackChan a chan struct{} as the bool you send doesn't mean anything, so it's better to send empty struct ("no value")
    • Put the sig arg last and make it varargs, like signal.Notify, where you can listen for multiple signals; so the signature would likely be SignalNotify(notify chan<- os.Signal, ack <-chan os.Signal, when YDBHandlerFlag, signals ...os.Signal)
    • Make the signals arg os.Signal instead of sycall.Signal to match signal.Notify
    • Alternatively consider dropping the when flags: do users actually need that level of control? Or is it acceptable for the library to just choose one (probably "before").

New Syntax Summary

New Types Old Types
Buffer BufferT
Subscripts BufferArrayT
Node KeyT
New Connection Syntax Old Syntax
conn = yottadb.init() yottadb.init()
out = conn.Str2Zwr(inBuffer) inBufferT.Str2ZwrST(yottadb.NOTTP, nil, &out)
conn.call(routine, args...) CallMT(tptoken, &err, retlen, routine, args...)
conn.kill(subscripts...) DeleteE(tptoken, &err, yottadb.YDB_DEL_TREE, subscripts...
New Node Syntax Old Key Syntax
n := Node(varname,sub1,sub2,...) var n yottadb.KeyT
n.Alloc(varSize, numSubs, subSiz)
defer n.Free()
n.Varnm.SetValStr(yottadb.NOTTP, nil, "varname")
n.SubAry.SetValStr(yottadb.NOTTP, nil, "sub1")
n.SubAry.SetValStr(yottadb.NOTTP, nil, "sub2")
n := Node(varname, subsarray...)
n.Set(value) 		SetValE(yottadb.NOTTP, nil, "abc", varname, subsarray)
fmt.Printf("node is: %#v", node) fmt.Print("node is: ")
key.DumpToWriter(os.Stdout)
for subnode := range node {
println(subnode.name)
} 		done := false
subscriptArray = append(subscriptArray, "")
while !done {
subkey, done = NodeNextE(tptoken, &err, varname, subscripArray)
println(subkey)
}
etc...

Notes:

  • The name node rather than 'key' follows YDBLua usage and also follows a poll taken of ANET developers who say node is what they would call a reference into the database.
  • The new syntax will use the C SimpleAPI under the covers rather than the Go SimpleAPI or EasyAPI, and it can be made both faster and easier, to supercede both.
  • The use of the Node type to perform operations on a node can be made both easier than EasyAPI and faster than the SimpleAPI since the node may be created in one function and retained for subsequent operations.
  • Whereas KeyT allocates space for the longest subscript multiplied by the number of subscripts, Node can be made more space-efficient by storing an immutable array of back-to-back subscripts that allocate only the exact space required for each subscript. An efficient way to create a child of a Node with variant subscripts will also be provided. Take YDBLua node syntax and as an example.
Edited by Berwyn Hoyt