Pinned symbol libraries are broken on 6.99

Description

When I pin libraries in the symbol editor, the pinned libraries are not saved correctly in the .kicad_pro file. Here is a comparison of what I see in the UI vs. what is actually saved in the .kicad_pro file.

Furthermore, when I fully close the KiCad project manager window, the list of pinned_symbol_libs is changed. Here's an image showing what the list looks like after I quit the KiCad app:

KiCad Version

Application: KiCad

Version: (6.99.0-4200-gde21eb5268), release build

Libraries:
	wxWidgets 3.1.5
	FreeType 2.12.1
	HarfBuzz 5.3.1
	FontConfig 2.14.0
	libcurl/7.84.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.47.0

Platform: macOS Version 13.0 (Build 22A380), 64 bit, Little endian, wxMac

Build Info:
	Date: Nov  4 2022 22:12:21
	wxWidgets: 3.1.5 (wchar_t,wx containers)
	Boost: 1.80.0
	OCC: 7.6.3
	Curl: 7.64.1
	ngspice: 37
	Compiler: Clang 12.0.0 with C++ ABI 1002

Build settings:
	KICAD_SPICE=ON

added 7.0 priorityundecided statusnew labels

The project file only saves pinned project libraries. (Pinned global libraries are saved in preferences.)

Are all the libraries shown above project?

@jeffyoung The opposite actually, all the libraries in the above project are global:

Just checked, it doesn't look like anything is being saved in /Library/Preferences/kicad/6.99/kicad_common.json when I pin global libraries:

Well that's odd. You're even on the same platform I am, so it's not that...

Hmmm... is it possible that you pinned those libraries months ago, and haven't pinned/unpinned them since? (This was a bug fix about 6 weeks ago.)

Try unpinning and re-pinning and see if they appear in kicad_common.json.

I confirm pinned libs list in kicad_common.json is always empty, regardless pinned libs. The .kicad_pro pinned list has also an issue: as soon as you pinned a project lib and closed the project, the pinned list contains only one name: the past pinned.

I am unsure this is a good idea to maintain 2 lists: only the list in project file is suitable. (I guess the pinned libs are the most important libs: the choice is of course related to the project)

@charrasjp that was the original design (& code), but people found it unpredictable. I'll see if I can dig up the issue....

Ha! That's a nice easy number to remember: #12000 (closed).

(There's no discussion in the bug report so it must have been done on Zulip....)

Ahh... more discussion in #11892 (closed)...

It's not quite as I postulated though; anything you pin or unpin will be added-to/removed-from both lists.

(At least in theory. Something appears to be broken.)

I still thinking the first behavior (save only in project) is the right behavior.

We should avoid to try to please everyone.

And if I understand well the 11892, the user did not open a project.

This is now not acceptable for a normal work.

I could see doing some footprint and/or symbol library editing without a project open. And I'd still want my pinned libraries at the top.

That being said, if the way to work around that was to open a project it wouldn't bother me much.

@kicad/code/lead_devs ? (see #11892 (closed) for the background)

the user did not open a project. This is now not acceptable for a normal work.

@charrasjp keep in mind that a user can now (as of 6.0) open the symbol/footprint editors without a project open. I think this was the reason for the change to store pinning in two locations.

I think my idea for how this would work was to stick the library in the global pinned list if it existed in the global library table, and project otherwise. I didn't look at the code to see if this is what we were trying to do...

(I do have a fix for the bug part. But I'll wait until we resolve whether or not we want to change the storage.)

assigned to @jeffyoung

added prioritymedium label and removed priorityundecided label

changed milestone to %7.0

Maybe I'm missing something but wouldn't it make sense to pin the global libraries in appropriate KiCad settings file since they are always available and project libraries in the project file since they are only available for the current project? In stand alone mode, you only have the global library table loaded so the pinning would be preserved. I guess I could see a use case where you might want to pin global libraries in the project file but that breaks in stand alone mode so I don't think this is a good idea. When you have a conflict between the pinning in the kicad settings files and the current project file, how do you resolve the conflict?

closed with commit 165c9bf6

Just a couple thoughts: from reading the comments and other tickets, it sounds like there are a couple different workflows that people are trying to accomplish with pinned libraries, and depending on which workflow they are using, the behavior of pinned libraries can be initially a bit confusing (ignoring the existing bugs and just looking at the desired behavior).

Workflows (from perspective of an end user):

1. For a single project, I have a set of frequently used global & project-local libraries that are specific to this project only. I want this list of libraries to be at the top of the list in all contexts for ONLLY this project (schematic symbol picker, symbol editor, etc.)
2. Without a project open, I have a set of "favorite" global libraries that I want to be easily accessible whenever I open the symbol editor.
3. For all my projects, I ONLY use project-local libraries, and I always want project-local libraries to appear at the top of the list by default. (I think this is what was being described in #11892 (closed) and what was requested in #6342 (closed))

For workflow #1, I think it makes sense for both global & project-local libraries to be stored in the project file (as @charrasjp said), since the list of pinned projects could be totally different for each project.

To support workflow #2, one approach might be to save/read the pinned libraries from the user preferences only if there is no project open. i.e. there would effectively be two independent sets of pinned libraries: one for the project context, and one for the non-project context. However, this might be initially confusing for some people if they pin something in the symbol editor without a project open, and then don't see that same library pinned when they open a project. It makes sense once you understand that KiCad effectively has two different modes of operation.

For workflow #3, it requires re-pinning project-local libraries for each new project. Arguably, this is not a big deal since it only needs to be done once for each project. An alternative approach might be an automatic grouping mechanism like what's shown in #6342 (comment 1033115155)

I think this may be complicating things too much in the interest of flexibility.

I don't think we want to introduce two different modes of operation for this feature as you propose for (2).

If the user only uses project-local libraries, I'm not sure they need the pinning feature at all.

Honestly, I'm not sure that manual library pinning is in fact the best solution to the actual user need (finding certain libraries quickly).

Honestly, I'm not sure that manual library pinning is in fact the best solution to the actual user need

Yeah, that's what I was trying to point out. There are different user needs which are all trying to be solved with pinning, and the behavior is confusing depending on what you're trying to do.

I think there is actually a 4th workflow which is: "I have some favorite global libraries that I use on every project, so I want those global libraries to always be pinned for any project I work on."

I'm not sure how you support workflow #1, #2 and #4 with the existing pinning solution. I think you need to pick one workflow you're trying to solve for. I'd be OK saying that pinning is only solving for #1 (i.e. just stick everything in the project file).

It may be workable to just have a "recent libraries" list instead of / in addition to "recent symbols", and store that globally. Any libraries that aren't found in the tables can just be skipped.

I am starting to think that any solution that uses pinning to let the user manually separate out "favorite libraries" from "other libraries" isn't really the right path. If there are libraries the user rarely uses, why shouldn't we just encourage them to be hidden rather than un-favorited?

I always thought pinning was a short cut for actual library management. The original library table design intent was that user could (would?) prune the global library table according to their needs and then add project libraries as required to minimize the amount of library loading and searching. The default global library includes all of the KiCad libraries because we don't know in advance which libraries will be useful to a designer. I've never thought that pinning was a solution to library management.

I'm mostly a workflow 4 guy: I use a lot of libraries, but there are 4 or 5 that I want to be able to find quickly.

If I wanted only project libraries for a specific project then I'd edit the symbol table to reflect that.

So for me the current solution works well. "Recent Libraries" would not, because every time I used an infrequent library it would get stuck up at the top.

We might have to stick with the status quo until/unless we decide to go a radically different direction with symbol choice UX (palettes or something)

The original library table design intent was that user could (would?) prune the global library table according to their needs and then add project libraries as required to minimize the amount of library loading and searching.

@stambaughw the global libraries that are actually used on a project can vary project-to-project. Since there is only a single global library table, you'd have to edit the global table each time you switched projects. To me, pinning provides a mechanism to filter the global table so that it's more efficient for a specific task (working on a specific project or editing a specific set of symbol libraries).

On the other hand, being able to pin project-local libraries feels like a workaround for the fact that there is no grouping in the UI for global vs. project-local libraries. When a user adds a project-local library to the library tables, they are already effectively saying "I explicitly want to use this library for this project only", but then that library gets lost in a sea of global libraries.

Another alternative approach could be the following:

1. Do not allow pinning project-local libraries. Instead, provide a simple grouping mechanism in the UI for global vs. project-local libraries (e.g. something like #6342 (comment 1033115155)). This would ensure that project-local libraries are always prioritized at the top of the list.
2. Only allow pinning global libraries and save the pins in the user preferences (instead of the project file). This would ensure that pins are consistent whether the user is editing a project or only working on symbols in the editor without a project open.

the global libraries that are actually used on a project can vary project-to-project. Since there is only a single global library table, you'd have to edit the global table each time you switched projects.

Why wouldn't you just add the project specific libraries to the project library tables? Changing the global library table for every project completely defeats the purpose of having project specific libraries.

Why wouldn't you just add the project specific libraries to the project library tables? Changing the global library table for every project completely defeats the purpose of having project specific libraries.

I agree with you.

There are two types of libraries that somebody would want to add to the symbol tables:

1. Globally-installed libraries (like those bundled with KiCad or installed via PCM)
2. Project-local libraries that are specific to the project (often colocated with the project in the project directory)

For project-local libraries, I would always add those to the project sym-lib-table. For globally-installed libraries, I would normally add them to the global sym-lib-table. As you suggested, I would never want to change the global library table for each project (that sounds like a nightmare).

You are correct that it's also possible to add the globally-installed libraries to the project sym-lib-table. Assuming that the global sym-lib-table was empty, you could use a project sym-lib-table as a sort of "filter" to only include libraries specific for that project (containing both globally-installed and project-local libraries). This would allow a different subset of the globally-installed libraries to be used for each project, without requiring pinning.

However, if the global sym-lib-table is not empty, then it contains all the globally-installed libraries used across all projects. Adding globally-installed libraries to the project sym-lib-table (as you suggested) doesn't help in this case because there is no separation of project vs. global sym-lib-table entries in the UI. You're back to a giant list of libraries, and you need some mechanism to "filter" this list (pinning or grouping or some other solution).

if the global sym-lib-table is not empty, then it contains all the globally-installed libraries used across all projects

That's false. The global library tables are seeded with all of the libraries that install with KiCad only once the first time KiCad is run (which you can optionally ignore). After that, you have full control over which libraries are in you global library tables. Just remove the libraries you never use from the table and they will never show up again unless you specifically reinstall the default global library table. In other words, the global library is always the same for every project.

Sorry, imprecise wording on my part.

the global library is always the same for every project

This is exactly what I meant when I said "it contains all the globally-installed libraries used across all projects"

Just remove the libraries you never use from the table

This is where I'm getting hung up. I don't know which of the global libraries I'm going to use on future projects. It varies project-to-project. As a result, my workflow is to simply add every globally-installed library to the global sym-lib-table so that they are always available if I need them. However, for each individual project, there are libraries I use frequently and I'd like some way for those to show up at the top of the list.

We might have to stick with the status quo until/unless we decide to go a radically different direction

@jeffyoung with 165c9bf6 what will the expected behavior be? Pinned global libraries are saved in user preferences and pinned project libraries are saved in the project file? If so, I'm OK with this until you guys want to revisit. The only downside I see is that it would not be possible to have project-specific pinning of global libraries, but that's feels like an OK compromise (changing pins is very quick vs. changing the sym-lib-tables, so I'd just update them as needed when switching between projects).

When you pin a library it's saved to the "pinned" list of both preferences and project settings. When you unpin a library it's removed from both lists.

The list is just a white-list, so a library has to be enabled before we'll ever ask if it's pinned. So if you switch to another project it would have to have said library for it to get pinned. But if you unpin a global library in one project, then it will be unpinned for all projects.

Maybe I'm misunderstanding, but if the "pinned" libraries are added in both user preferences and project settings, which takes precedence when they conflict for the same library?

For example, let's say a user pins global library "A" in Project #1. Now both user preferences and project settings have "pinned_symbol_libs": ["A"].

Now, let's say that the user opens a new Project #2. The new project settings will have "pinned_symbol_libs": [] but the user preferences will have "pinned_symbol_libs": ["A"]. How does this get reconciled for global library "A"? Is "A" pinned in the UI or not?

Neither takes precedence. If the library appears in either list then it's pinned.

(So the answer to your last question is yes, "A" is pinned.)

Expanding the example from above a couple steps further, is this scenario possible?

1. Let's say a user pins global library "A" in Project #1. Now both user preferences and project settings have "pinned_symbol_libs": ["A"].

2. The user opens a new Project #2. The new project settings will have "pinned_symbol_libs": [] but the user preferences will have "pinned_symbol_libs": ["A"]. The UI will show "A" is pinned.

3. The user unpins "A" in Project #2. Now both user preferences and project settings have "pinned_symbol_libs": []. The UI will show "A" is unpinned.

4. The user opens Project #1 again. The project settings have "pinned_symbol_libs": ["A"], but the user preferences will have "pinned_symbol_libs": []. The UI will show "A" is pinned.

Now, the user is confused because they just unpinned "A" in Project #2, but now it's pinned again when they opened Project #1.

Yes, that is possible.

It's a trade-off between predictability and simplicity. Logged issues suggest that most users complain when their pinned libraries are no longer pinned, rather than when a library is unexpectedly pinned. (My guess is that they just un-pin them in that case and continue on without logging a bug.)