1. 10 Jan, 2022 1 commit
  2. 09 Jan, 2022 1 commit
  3. 08 Jan, 2022 1 commit
  4. 06 Jan, 2022 1 commit
  5. 04 Jan, 2022 2 commits
    • Stuart McGraw's avatar
      041-b5d00f.sql: fixed comments. · ad6d9fa0
      Stuart McGraw authored
      ad6d9fa0
    • Stuart McGraw's avatar
      doc: rewrite "install", add "oper", drop "dev" · bb1aeddc
      Stuart McGraw authored
      [Changes noted below for the source .asc filea also apply to
      the corresponding .html product files.]
      dev.asc: Removed pending rewrite.  Most of the info that was in here
        is now in install.asc and oper.asc.
      oper.asc: New document "Operation and Debugging Guide".
      install.asc: Rewritten.  Now includes the "install for development"
        instructions that were formerly in dev.asc.
      schema.dia,.png: Updated to add column .eid to "hist" table (per
       git 211225-3bfc8f17, db-b5d00f).
      index.asc, Makefile: Updated to support the above changes.
      bb1aeddc
  6. 01 Jan, 2022 1 commit
    • Stuart McGraw's avatar
      updates.py: was not showing links for Dec 31 · d085745b
      Stuart McGraw authored
      In the updates.py (daily submissions) web page, a link for Dec 31
      was missing in every year.
      Problem was in a date WHERE clause that I thought was inclusive at
      both ends was not.  Fixed in both the cgi and flask versions.
      
      daily links
      d085745b
  7. 30 Dec, 2021 1 commit
    • Stuart McGraw's avatar
      doc: remove schema.html from index page · 6d07c524
      Stuart McGraw authored
      Building schema.html has been disabled in the Makefile for ages so
      the existing schema.html is out of date.  Attempts to build the html
      from the .odt file still produce poor results.  The XHTML looks better
      at first glance but has spacing problems and footnotes at bottom have
      no numbers.  The native "save as" HTML looks worse (excess space in
      table of contents, no margins, section numbers font too small).
      Seems that HTML is still not viable output from LibreOffice Writer.
      6d07c524
  8. 29 Dec, 2021 3 commits
    • Stuart McGraw's avatar
      doc: major revision of install doc · b6fe462b
      Stuart McGraw authored
      Revised based on a test install in a clean VM.
      - Covers install for both development and production.
      - Requirements updated and verified.
      - CGI instructions removed, WSGI instructions added.
      - Now covers adding JMdictDB users with new bin/user.py.
      - Formatting improvements (table of contents, etc).
      
      Also updated the 2021-11-update doc which information
      learned from the recent EDRDG update.
      b6fe462b
    • Stuart McGraw's avatar
      jmsess: replaced resetpw.py with users.py · 4eed89a6
      Stuart McGraw authored
      resetpw.py was previously used as a backup method of accessing
      the jmess user database if it became inaccessible via the usual
      access method of the users.py web page.  It had very limited
      capabilities though, only allowing changes to passwords or privilge
      level of existing users.
      
      The new program, bin/users.py, allows full management of users
      including listing, adding, deleting and modifying all their settings.
      It is thus a superset of the capabilities of resetpw.py.
      It also eliminates the need for the jmsess creation script to add
      an initial admin user since that can now be done by bin/users.py.
      
      The "list" action of users.py is currently a little crude but
      will be improved in a future revision.
      4eed89a6
    • Stuart McGraw's avatar
      Fixed assorted unrelated errors · 9f2cfb6b
      Stuart McGraw authored
      entrobjs.sql: most serious problem, a syntax error caused script to fail.
      index.asc,index.html: schema.pdf is text, not a diagram.
      fmt.py: was getting a syntax warning on recent Python versions.
      jmdictdb.ini-sample: set the private config file as example.
      9f2cfb6b
  9. 27 Dec, 2021 4 commits
    • Stuart McGraw's avatar
      hist: Set hist.eid in jdb.addentr()/.setkeys() (db-b5d00f) · d6019cbd
      Stuart McGraw authored
      [The db-b5d00f note should have been applied to 211225-3bfc8f17.]
      New column ".eid" in table "hist" was added in revision 211225-3bfc8f17
      and will hold the entry id of the entry the history item was first
      added to.  That revision added the column but did not provide for
      anything to set it.  This revision provides for automatically setting
      a value in that column when an entry is submitted.
      
      The hist.eid value is now set in jdb.addentr() (actually in
      jdb.setkeys() but this is always called by jdb.addentr() before the
      latter writes an entry to the database.  The .eid value is set only
      on the last item in the _hist list which we presume is the hist item
      for the current submission of the entry.  No provision is made for
      using addentr() without making any changes to the last hist item.
      
      Added tests for Entr._hist including the new .eid field in test_submit
      and updated jmtest01 database to db-b5d00f.
      d6019cbd
    • Stuart McGraw's avatar
      Merge branch 'master' into hist · b7d59b6e
      Stuart McGraw authored
      b7d59b6e
    • Stuart McGraw's avatar
      submit.py: fixed get_segment() bug that caused reject() fail · d20aa79b
      Stuart McGraw authored
      jmdictdb/submit.py: get_segment(): This was broken when processing
        a single entry edit on the first of multiple branches.  Caused
        function reject() to fail the assertion that checks for a non-empty
        'rejs' list.  New code is simpler and clearer.
      
      test_submit.py: Added new test Reject.test2000060 to check for the
        problem caused by the get_segment() bug above.  It triggered the
        assertion fail with the previous get_segment() code, passes with
        the current revised code.
        Also inserted new Reject.test2000030 and renumbered old test2000030
        and following tests.
      d20aa79b
    • Stuart McGraw's avatar
      tests: expand locust README.txt · d7a4adf6
      Stuart McGraw authored
      d7a4adf6
  10. 25 Dec, 2021 2 commits
    • Stuart McGraw's avatar
      flaskapp.py: report database version error · 5db25410
      Stuart McGraw authored
      Prior, a database version error was reported with a misleading
      error page that said "Unknown service (jmdict)".  Now says more
      informatively "Database version error (jmdict)" and additionally
      writes debug messages to the log file with details about the
      update version that is missing.
      5db25410
    • Stuart McGraw's avatar
      hist: Add column "eid" to "hist" table · 3bfc8f17
      Stuart McGraw authored
      This branch is to explore recording the entr.id value of the entry
      item a hist item is first attached to, in the hist row itself.  The
      hist.entr column references the entry the history item is currently
      attached to which will be different when current item is the result
      of edits and the original entry item may have been deleted.  Although
      the orignal entry id may not reference any entry currently in the
      database (thus can't be a foreign key) it is still useful since it
      can be matched to entry id values recorded in log files and the log
      file information can provide additional details about the addition
      of the entry the history was first added to.
      
      This revision adds column "eid" to table "hist" and makes the necessary
      collateral changes elsewhere.  It does not yet set or otherwise use
      that column.
      
      The tests in tests/runtests.py all pass but no other testing has been
      done yet.
      3bfc8f17
  11. 23 Dec, 2021 1 commit
  12. 22 Dec, 2021 1 commit
    • Stuart McGraw's avatar
      submit.py: rework database support for submissions (db-972bf3) · 99444c0c
      Stuart McGraw authored
      Replaced the database functions get_edroot and get_edtree with a
      more general and versatile view, "edpaths", with the corresponding
      (substantial) updates to jmdictdb/submit.py.  The edpath view
      provides edit tree information in a form (a set of edit paths,
      from root to each leaf entry) that is simpler to use in the
      submit.py code.  These changes required a update of the database
      schema version number.
      
      The schema update also adds a new contraint to the "entr" table
      that prohibits an approved entr from being an edit of some other
      entry (ie, having a non-null .dfrm value).  This restriction was
      maintained in the JMdictDB software previously.
      
      tests/test_submit.py was updated to add more tests and new helper
      functions were added to make adding tests easier.
      jmtest01.sql: updated to the db-972bf3 schema.
      
      The current tests pass with both the current and the previous
      versions of submit.py.  To test with the previous version of
      submit.py, replace th...
      99444c0c
  13. 21 Dec, 2021 1 commit
    • Stuart McGraw's avatar
      flaskapp.py: svc parameter lost on srchres->entr redirect · c7337c1f
      Stuart McGraw authored
      When a search returned in a single result, the srchres.py route
      handler in flaskapp.py failed to include the "svc" url parameter
      when redirecting to the entr.py page.  If search was done in a
      different svc/database than the default "jmdict", this resulted
      in the wrong entry or an "entry not found" error being displayed
      because the svc database would revert to "jmdict" rather than
      the one the search was done in.
      c7337c1f
  14. 14 Dec, 2021 2 commits
    • Stuart McGraw's avatar
      test_submit.py: simplify adding test entries; new tests · 8efa03bb
      Stuart McGraw authored
      * Added new tests for verifying fail on non-leaf and multiple
        branch edits.
      * Fixed: helper functions mkentr() and addentr() were ignoring
        the 'q' parameter that sets the new entry's seq number.
      * Reorganized helper mkentr() and addentr(), added edentr() and
        addedit().
      * Changed the "u" (unapprove) parameter in the helper functions to
        "a" (approve) with the opposite boolean sense, seems more natural.
      * Removed empty Hist() objects that were added only to satify
        submit.submission() since it no longer requires them.
      8efa03bb
    • Stuart McGraw's avatar
      submit.py: .submission() no longer needs hist · df28fbee
      Stuart McGraw authored
      Prior, submit.submission() would fail with an obscure IndexError
      if given an entry without any history records (i.e., an empty
      entr._hist list), something very convenient to do in tests.  It
      will now accept such entries, supplying an blank Hist() item
      when necessary.
      df28fbee
  15. 13 Dec, 2021 1 commit
    • Stuart McGraw's avatar
      submit.py: Fixed return values in error case · 562f5867
      Stuart McGraw authored
      submission() and some of the functions it calls normally return
      a 3-tuple but when an error is detected, returned None.  This could
      result in an exeption when a caller tried to unpack an expected
      3-tuple return value.  submission() now returns (None,None,None)
      in error cases.
      
      test_submit.py was adjusted correspondingly.  Also added an additional
      test case to check error messages when the parent of an entry
      being edited disappearrs before it is submitted (as can happen when
      another edit of the same entry is approved before the first edit is
      submitted.)
      562f5867
  16. 11 Dec, 2021 1 commit
    • Stuart McGraw's avatar
      cgi: improved robustness of cgiinfo.py · 0722ebf6
      Stuart McGraw authored
      The cgiinfo CGI page failed when a situation arose when it was unable
      to import from the jmdictdb package.  It is intended that it should
      still produce a page reporting the import problem but it instead crashed
      due to a bug (wrong number of arguments when formating the 'Page' string
      in the ImportError exception block.)  Took opportunity to reorganize and
      improve the code a bit to make it present other info (like environment
      variables) that are not dependent on the jmdictdb imports even if imports
      fail.
      Akso made parallel updates to the jmapp version in jmdictdb/views/ but
      checking for import errors there is not very useful because it's called
      by flaskapp.py which had to import the modules to run.
      0722ebf6
  17. 07 Dec, 2021 2 commits
  18. 04 Dec, 2021 2 commits
    • Stuart McGraw's avatar
      jmapp: layout.jinja: improve consistency with cgi version · 728c61ee
      Stuart McGraw authored
      When an editor is logged in, the cgi version of this file would
      show the user's full name.  The jmpp version showed only the short
      userid.  It was changed so it too now shows the full name.
      728c61ee
    • Stuart McGraw's avatar
      jmapp: rename jmapp-related files · 77e7ec3f
      Stuart McGraw authored
      The WSGI/Flask code was developed under the name "jmapp" but that
      name is not particularly informative so this revision renames many
      of the associated files consistent with the plan for the wsgi/flask
      code to eventually be the only supported web back end (ie drop cgi):
        jmdictdb/jmapp.py -> flaskapp.py (can't call it flask.py because
          that conflicts with the flask module.)
        tools/run-jmapp.py -> run-flask.py.
      Documentation (not committed yet) will suggest:
        lib/jmapp.log -> jmdictdb.log (for interim case where both cgi
          and wsgi are available and separate log files are desired, the
          recommentation in jmwsgi.log for the wsgi log file.)
        lib/jmapp.ini -> jmdictdb.ini
        cgi/jmapp.wsgi -> jmdictdb.wsgi
        urlroot: /jmapp -> /jmwsgi
        process group: jmapp -> jmwsgi
      
      Also removed jmdictdb.wsgi from list of files to install; the version
      in the cgi directory is for serving the development files and is
      incorrect for the system install.
      77e7ec3f
  19. 02 Dec, 2021 2 commits
    • Stuart McGraw's avatar
      edform.jinja: remove wrap="physical" on textarea controls. · b30ad884
      Stuart McGraw authored
      This is reported to stop breakage of urls in the "refs" and
      "comments" fields although I did not have time to confirm myself.
      b30ad884
    • Stuart McGraw's avatar
      jelparse.y: fix uncaught KeyError in mk_restrs() · b1902c52
      Stuart McGraw authored
      When JEL input text was parsed that had a restriction (restr/stagr/stagk)
      with an item that did not match anything in the target list (e.g. a kanji
      word in a KR restr that did not exist in the entry's kanji) the call to
      txt2restr() in mk_restrs() would raise a KeyError.  This was not caught
      by anything and bubbled up to the top-level handler.  In the jmapp/WSGI
      version it produced a stack dump to the JMdictDB log file and a generic
      "something's wrong" error page which gave no clue about the source of the
      problem.  This revision fixes it to be a ParseError handled like the other
      conditions that raise that and with a useful error page message.
      
      Also in jelparse.mk_restrs(), removed code that duplicated the target
      lookup check that happens in restr.txt2restr() (which raises the exception
      noted above).
      
      Simplified the error message.  Since we report the restriction text that
      caused the problem and the input text, it doesn't seem necessary to spell
      out if it is ...
      b1902c52
  20. 01 Dec, 2021 2 commits
    • Stuart McGraw's avatar
      jmapp.py: add a catch-all exception handler · 4b6c5625
      Stuart McGraw authored
      Previously the CGI code caught unexpected exceptions via Python's
      sys.excepthook feature which was setup by logger.enable().  Under
      Flask this is no longer an option since Flask co-opts use of it
      for itself.  Flask provides an alternative way of doing something
      similar which this revision implements.  Prior to this exceptions
      that weren't explictly caught would cause Flask to issue a 500-
      Server Fail response and a Python stack trace would be written to
      the web server's log file.  Now, our exception handler writes the
      stack trace to the JMdictDB log file instead and returns a generic
      "something went wrong" page to the user as the previous CGI
      exception handler did.
      
      Also made unrelated cosmetic change to import of and calls to
      srvlib.vLogEntry.
      4b6c5625
    • Stuart McGraw's avatar
      cgi, jmapp: error reporting for jelparse syntax errors · 1c66717a
      Stuart McGraw authored
      Previously, jelparse syntax errors were reported by showing
      the input JEL text with a with a line below containing a caret
      (^) character under the approximate position of the error as
      reported by parser.  This meant the error text had to use
      "pre" formatting in the html to preserve whatspace.  But that
      meant other message did not wrap and, if long, extended off
      the right side of the window.  The was also a problem positioning
      the caret correctly in presence of wider Japanese characters
      despite specifying a mnonspace font.
      
      We now insert a couple of blot (▇) characters into the input
      JEL text at the error position reported by the parser.  This
      allows the lines (and other error message lines) to be wrapped.
      
      An outstanding problem of how to pass prolog and epilog text
      to the error page from edconf.py view was also fixed.
      1c66717a
  21. 29 Nov, 2021 1 commit
  22. 28 Nov, 2021 1 commit
    • Stuart McGraw's avatar
      jmapp: alternative for sys.excepthook · c88ad23e
      Stuart McGraw authored
      The CGI code the the jmapp/WSGI/Flask code will replace used a last-
      chance exception handler (sys.excepthook) to catch all exceptions
      including exceptions in jmdictdb/submit.py that occur when two users
      submit changes to the same entry at the same time.  However, under
      Flask that handler never gets called.
      These changes catch such exceptions in submit.py and report them via
      the 'errs' parameter that same way other problems are reported rather
      letting them bubble up to a now non-existant last-chance error handler.
      c88ad23e
  23. 25 Nov, 2021 1 commit
  24. 23 Nov, 2021 1 commit
  25. 07 Nov, 2021 1 commit
    • Stuart McGraw's avatar
      html: html-escape kwfreq.descr text · cb61bb8e
      Stuart McGraw authored
      Some rows in the kwfreq table contain quote (") characters in the
      .descr field text and that text was making its way into entr.jinja
      unescaped (via jinja.TALfreqs()) where it was used in the "title"
      attribute of two elements, causing the attributes to render incorrectly.
      The escaping (turning into '"') needs to be done in jdb.freq2txt()
      because it returns html which can't be additionally escaped later.
      (Some grepping indicated that those were the only places the html
      output of jdb.freq2txt() was used so nothing else should end up
      double-escaped.)
      cb61bb8e
  26. 04 Nov, 2021 1 commit
    • Stuart McGraw's avatar
      jmapp: srchres url parameter was ignored in srchres.py url · 4db2c850
      Stuart McGraw authored
      Normally the srchres.py (search results) page will redirect to the
      entr.py page if the search resulted in finding only one entry.  However
      it allows a &srchres=1 url parameter to override this behavior and display
      the srchres page with the usual list of results (with only one item in it).
      Processing of the parameter happens in jmapp.py but was overlooked when
      the srchres view was migrated from the cgi version.  Fixed now.
      4db2c850
  27. 01 Nov, 2021 2 commits
    • Stuart McGraw's avatar
      jmsess: Fix timezone confusion (jmsess db-04f177) · 83f01624
      Stuart McGraw authored
      Several of the db* functions in jmcgi.py run sql to compare a user's
      session timestamp in the jmsess database sessions.ts column with the
      current time returned by Postgresql function NOW().  The problem was
      that the time in sessions.ts was set from a default value that was in
      UTC but the time returned by NOW() is localtime resulting in bogus
      comparisions if the server localtime timezone is not UTC.
      
      Fixed by changing the sessions.ts default value to be NOW() rather
      than the former expression that converted NOW() to UTC which required
      update to mksess.sql for a new jmsess database and and new script
      039s-04f177.sql to update an existing jmsess database.
      
      rest.py: unrelated minor typo fix not worth a separate commit.
      83f01624
    • Stuart McGraw's avatar
      cgi: add "SameSite" cookie attribute, lengthen login timeout · 8ced31b2
      Stuart McGraw authored
      In jmcgi.py added the attribute "SameSite=Strict" to the jmdictdb_sid
      cookie to try to diagnose a premature logout problem reported by a
      user.  This will be needed in future anyway as browsers are updated
      to reject insecure cookies without this attribute.
      
      Also lengthened the SESSION_TIMEOUT value from 2 hours to 4 and use
      its value for both the database timeout and the sid cookie' max-age
      attribute.
      
      Added debug logging statements to the some of the db* functions.
      8ced31b2