workbench ingest as repository size increases (archive discussion from Islandora Slack for future)
https://islandora.slack.com/archives/CM5PPAV28/p1701204947976609
Annie Oelschlager 1:55 PM @a Stanley Did you find the culprit for this by chance: https://islandora.slack.com/archives/C01TLCB0EEQ/p1677847623005729 I'm starting to see this during my migration. The cpu is running high for mysqld and when I run SHOW FULL PROCESSLIST; in the mariadb container it is the same select statement you had. I currently have all the derivative contexts disabled so I'm not sure where the slow down is happening. If anyone else has run into this or has any tips on fixing this please let me know.
A Stanley I think this is a workbench question - I'm doing an import of several hundred thousand objects. The imports have slowed down from about 6 a minute to slightly less than two. The following SELECT statement seems to be the culprit it takes about 15 seconds to run. Each of the columns on the numerous inner joins is indexed. Has anyone, anywhere had any success optimizing this query? SELECT base_table.vid AS vid, base_table.mid AS mid Show more Posted in islandoraworkbench | Mar 3rd, 2023 | View message
A Stanley :ambassador: 2:02 PM I was unable to get anywhere with that one Annie. All the tables are already indexed, so that wasn’t the problem. I threw some extra memory at the container, but it made no real difference. I ran htop, and like you saw the CPUs were busy but I always had some capacity left.
Annie Oelschlager 2:17 PM Thank you @a Stanley that is helpful to know. I do think this function might have something to do with it: https://github.com/Islandora/islandora/blob/2.x/src/IslandoraUtils.php#L184C1-L221
A Stanley :ambassador: 2:20 PM I started chasing that one a while ago - I’m not sure it’s necessary all the time. It might be worth seeing if we could toggle that.
Annie Oelschlager 2:23 PM I'm half tempted to change the access check to false to see if that would help any.
1 reply 4 months agoView thread
Jeffery Antoniuk 4:57 PM Here's a related message I've bookmarked from the conversation https://islandora.slack.com/archives/C01TLCB0EEQ/p1677896203533909
Chris Burgess https://github.com/Islandora/islandora/blob/2923a1a8b9303569fdea4bdbf580c1f56e3ed033/src/IslandoraUtils.php#L183-L219 looks like your culprit to me, at a glance Posted in islandoraworkbench | Mar 3rd, 2023 | View message