Missing loose foreign key cleanup cron on self-managed Free
Summary
On our self-managed/Free GitLab installations (we're using GitLab-Omnibus), a few weeks after migrating from 14.3.x to 14.9.x, we see the loose_foreign_keys_deleted_records
table growing and growing, and no loose foreign key cleanup seem to ever be executed. My understanding is that the required sidekiq cron job is only configured by default for commercial editions, despite LFK being also used in Free.
Steps to reproduce
I've not actually tried with a fresh install, but I assume that would do:
- install self-managed/Free, default settings
- create and then delete some projects and pipelines
- watch deleted records accumulating and never being processed (try the SQL query from this documentation)
- (repeat 2 and 3)
- run the cleanup job from console (same documentation) to see what should have happen automatically
What is the current bug behavior?
No deleted records is ever processed, they are only accumulating in loose_foreign_keys_deleted_records
(with status=1
).
In the /admin/background_jobs
page, in the Cron tab, there is no loose_foreign_keys_cleanup_worker
job.
What is the expected correct behavior?
LooseForeignKeys::CleanupWorker
should be periodically processing some of these deleted records, as described in the documentation. I should find a loose_foreign_keys_cleanup_worker
job in the /admin/background_jobs
page, in the Cron tab.
Relevant logs and/or screenshots
Only a few weeks after upgrading from 14.3 to 14.9, that's what we see on our biggest installation:
gitlab-db=> SELECT fully_qualified_table_name, status, count(*)
FROM loose_foreign_keys_deleted_records
GROUP BY 1, 2
ORDER BY 3 DESC;
fully_qualified_table_name | status | count
----------------------------+--------+--------
public.ci_job_artifacts | 1 | 464954
public.ci_builds | 1 | 68784
public.ci_pipelines | 1 | 11570
public.namespaces | 1 | 4217
public.projects | 1 | 4104
public.merge_requests | 1 | 1110
public.ci_runners | 1 | 449
public.users | 1 | 212
(8 rows)
Output of checks
Results of GitLab environment info
(that's results from a small installation, same version as the big one, and same issue)
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.2.33 Rake Version: 13.0.6 Redis Version: 6.2.6 Sidekiq Version:6.4.0 Go Version: unknown GitLab information Version: 14.9.4 Revision: eca17098bf8 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.6 URL: https://XXXXXXXXXXXXXXX HTTP Clone URL: https://XXXXXXXXXXXXXXX/some-group/some-project.git SSH Clone URL: git@XXXXXXXXXXXXXXX:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 13.24.0 Repository storage paths: - default: /gitlab-nfs/gitlab-data/git-data/repositories - praefect: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
(that's results from a small installation, same version as the big one, and same issue)
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.24.0 ? ... OK (13.24.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK praefect ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 3/2 ... yes 3/3 ... yes 5/4 ... yes 7/5 ... yes 3/6 ... yes 10/10 ... yes 20/61 ... yes 48/1226 ... yes 48/1229 ... yes 48/1230 ... yes 48/1232 ... yes 72/1414 ... yes 31/1415 ... yes 142/1708 ... yes 165/1710 ... yes 165/1711 ... yes 165/1712 ... yes 165/1713 ... yes 165/1714 ... yes 165/1715 ... yes 165/1716 ... yes 165/1717 ... yes 165/1718 ... yes 165/1719 ... yes 165/1720 ... yes 165/1721 ... yes 165/1722 ... yes 165/1723 ... yes 165/1724 ... yes 165/1725 ... yes 165/1726 ... yes 165/1727 ... yes 165/1728 ... yes 165/1729 ... yes 165/1730 ... yes 165/1731 ... yes 165/1732 ... yes 165/1733 ... yes 165/1734 ... yes 165/1735 ... yes 165/1736 ... yes 165/1737 ... yes 165/1738 ... yes 165/1739 ... yes 206/1757 ... yes 214/1767 ... yes 214/1768 ... yes 214/1769 ... yes 214/1770 ... yes 214/1771 ... yes 214/1772 ... yes 214/1773 ... yes 214/1774 ... yes 214/1775 ... yes 214/1776 ... yes 214/1777 ... yes 214/1778 ... yes 214/1779 ... yes 214/1780 ... yes 214/1781 ... yes 214/1782 ... yes 214/1783 ... yes 214/1784 ... yes 542/1790 ... yes 541/1791 ... yes 541/1792 ... yes 543/1793 ... yes 543/1794 ... yes 543/1795 ... yes 543/1796 ... yes 543/1797 ... yes 543/1798 ... yes 543/1799 ... yes 543/1800 ... yes 543/1801 ... yes 543/1802 ... yes 543/1804 ... yes 543/1805 ... yes 544/1806 ... yes 544/1811 ... yes 544/1812 ... yes 547/1813 ... yes 547/1814 ... yes 548/1815 ... yes 547/1816 ... yes 547/1817 ... yes 549/1818 ... yes 550/1819 ... yes 554/1821 ... yes 552/1822 ... yes 123/1823 ... yes 206/1825 ... yes 545/1826 ... yes 206/1827 ... yes 206/1828 ... yes 206/1829 ... yes 539/1830 ... yes 124/1833 ... yes 592/1844 ... yes 598/1845 ... yes 555/1850 ... yes 611/1857 ... yes 611/1866 ... yes 617/1869 ... yes 617/1870 ... yes 611/1876 ... yes 611/1877 ... yes 31/1913 ... yes 620/1914 ... yes 628/1963 ... yes 629/1972 ... yes 629/1973 ... yes 621/2018 ... yes 632/2025 ... yes 633/2026 ... yes 633/2027 ... yes 537/2046 ... yes 642/2076 ... yes 206/2101 ... yes 646/2102 ... yes 642/2103 ... yes 645/2122 ... yes 206/2125 ... yes 206/2126 ... yes 2/2137 ... yes 2/2139 ... yes 645/2141 ... yes 2/2146 ... yes 2/2150 ... yes 645/2151 ... yes 645/2152 ... yes 2/2185 ... yes 535/2210 ... yes 654/2212 ... yes 43/2213 ... yes 43/2214 ... yes 655/2230 ... yes 633/2260 ... yes 685/2341 ... yes 685/2342 ... yes 685/2343 ... yes 685/2344 ... yes 685/2345 ... yes 685/2346 ... yes 654/2347 ... yes 654/2348 ... yes 655/2356 ... yes 645/2377 ... yes 685/2383 ... yes 685/2384 ... yes 689/2385 ... yes 689/2386 ... yes 689/2387 ... yes 689/2388 ... yes 689/2389 ... yes 689/2390 ... yes 689/2391 ... yes 689/2392 ... yes 689/2393 ... yes 689/2394 ... yes 689/2395 ... yes 689/2396 ... yes 689/2397 ... yes 689/2398 ... yes 689/2399 ... yes 689/2400 ... yes 689/2401 ... yes 689/2402 ... yes 689/2403 ... yes 689/2404 ... yes 689/2405 ... yes 689/2406 ... yes 689/2407 ... yes 689/2408 ... yes 689/2409 ... yes 689/2410 ... yes 689/2411 ... yes 689/2412 ... yes 689/2413 ... yes 689/2414 ... yes 689/2415 ... yes 689/2416 ... yes 689/2417 ... yes 689/2418 ... yes 689/2419 ... yes 689/2420 ... yes 689/2421 ... yes 689/2422 ... yes 689/2423 ... yes 689/2424 ... yes 689/2425 ... yes 689/2426 ... yes 689/2427 ... yes 689/2428 ... yes 689/2429 ... yes 689/2430 ... yes 689/2431 ... yes 689/2432 ... yes 689/2433 ... yes 689/2434 ... yes 689/2435 ... yes 689/2436 ... yes 689/2437 ... yes 689/2438 ... yes 689/2439 ... yes 689/2440 ... yes 689/2441 ... yes 689/2442 ... yes 689/2443 ... yes 689/2444 ... yes 689/2445 ... yes 689/2446 ... yes 689/2447 ... yes 689/2448 ... yes 689/2449 ... yes 689/2450 ... yes 689/2451 ... yes 689/2452 ... yes 689/2453 ... yes 689/2454 ... yes 689/2455 ... yes 689/2456 ... yes 689/2457 ... yes 689/2458 ... yes 689/2459 ... yes 689/2460 ... yes 689/2461 ... yes 689/2462 ... yes 689/2463 ... yes 689/2464 ... yes 689/2465 ... yes 689/2466 ... yes 689/2467 ... yes 689/2468 ... yes 689/2469 ... yes 689/2470 ... yes 689/2471 ... yes 689/2472 ... yes 689/2473 ... yes 689/2474 ... yes 689/2475 ... yes 689/2476 ... yes 689/2477 ... yes 689/2478 ... yes 689/2479 ... yes 689/2480 ... yes 689/2481 ... yes 689/2482 ... yes 689/2483 ... yes 689/2484 ... yes 689/2489 ... yes 689/2490 ... yes 689/2491 ... yes 689/2492 ... yes 689/2493 ... yes 689/2494 ... yes 689/2495 ... yes 689/2496 ... yes 689/2497 ... yes 689/2498 ... yes 689/2499 ... yes 689/2500 ... yes 689/2501 ... yes 689/2502 ... yes 689/2503 ... yes 689/2504 ... yes 689/2505 ... yes 689/2506 ... yes 689/2507 ... yes 689/2508 ... yes 689/2509 ... yes 689/2510 ... yes 689/2511 ... yes 689/2512 ... yes 689/2513 ... yes 689/2514 ... yes 689/2515 ... yes 689/2516 ... yes 689/2517 ... yes 689/2518 ... yes 689/2519 ... yes 689/2520 ... yes 689/2521 ... yes 689/2522 ... yes 689/2523 ... yes 689/2524 ... yes 689/2525 ... yes 689/2526 ... yes 689/2527 ... yes 689/2528 ... yes 689/2529 ... yes 689/2530 ... yes 689/2531 ... yes 689/2532 ... yes 689/2533 ... yes 689/2534 ... yes 689/2535 ... yes 689/2536 ... yes 689/2537 ... yes 689/2538 ... yes 689/2539 ... yes 689/2540 ... yes 689/2541 ... yes 689/2542 ... yes 689/2543 ... yes 689/2544 ... yes 689/2545 ... yes 689/2546 ... yes 689/2547 ... yes 689/2548 ... yes 689/2549 ... yes 689/2551 ... yes 689/2552 ... yes 689/2553 ... yes 689/2554 ... yes 689/2555 ... yes 689/2556 ... yes 689/2557 ... yes 689/2558 ... yes 689/2559 ... yes 689/2560 ... yes 689/2561 ... yes 689/2562 ... yes 689/2563 ... yes 689/2564 ... yes 689/2565 ... yes 689/2566 ... yes 689/2567 ... yes 689/2568 ... yes 689/2569 ... yes 689/2570 ... yes 689/2571 ... yes 689/2572 ... yes 689/2573 ... yes 689/2574 ... yes 689/2575 ... yes 689/2576 ... yes 689/2577 ... yes 689/2578 ... yes 689/2579 ... yes 689/2580 ... yes 689/2581 ... yes 689/2582 ... yes 689/2583 ... yes 689/2584 ... yes 689/2585 ... yes 689/2586 ... yes 689/2587 ... yes 689/2588 ... yes 689/2589 ... yes 689/2590 ... yes 689/2591 ... yes 689/2592 ... yes 689/2593 ... yes 689/2594 ... yes 689/2595 ... yes 689/2596 ... yes 689/2597 ... yes 689/2598 ... yes 689/2599 ... yes 689/2600 ... yes 689/2601 ... yes 689/2602 ... yes 689/2603 ... yes 689/2604 ... yes 689/2605 ... yes 689/2606 ... yes 689/2607 ... yes 689/2608 ... yes 689/2609 ... yes 689/2610 ... yes 689/2611 ... yes 689/2612 ... yes 689/2613 ... yes 689/2614 ... yes 689/2615 ... yes 689/2616 ... yes 689/2617 ... yes 689/2618 ... yes 689/2619 ... yes 689/2620 ... yes 689/2621 ... yes 689/2622 ... yes 689/2623 ... yes 689/2624 ... yes 689/2625 ... yes 689/2626 ... yes 689/2627 ... yes 689/2628 ... yes 689/2629 ... yes 689/2630 ... yes 689/2631 ... yes 689/2632 ... yes 689/2633 ... yes 689/2634 ... yes 689/2635 ... yes 689/2636 ... yes 689/2637 ... yes 689/2638 ... yes 689/2639 ... yes 689/2640 ... yes 689/2641 ... yes 689/2642 ... yes 689/2643 ... yes 689/2644 ... yes 689/2645 ... yes 689/2646 ... yes 689/2647 ... yes 689/2648 ... yes 689/2649 ... yes 689/2650 ... yes 689/2651 ... yes 689/2652 ... yes 689/2653 ... yes 689/2654 ... yes 689/2655 ... yes 689/2656 ... yes 689/2657 ... yes 689/2658 ... yes 689/2659 ... yes 645/2660 ... yes 689/2661 ... yes 689/2662 ... yes 689/2663 ... yes 689/2664 ... yes 689/2665 ... yes 689/2666 ... yes 689/2667 ... yes 689/2668 ... yes 689/2669 ... yes 689/2670 ... yes 689/2671 ... yes 689/2672 ... yes 689/2673 ... yes 689/2674 ... yes 689/2675 ... yes 689/2676 ... yes 689/2677 ... yes 689/2678 ... yes 689/2679 ... yes 689/2680 ... yes 689/2681 ... yes 689/2682 ... yes 689/2683 ... yes 689/2684 ... yes 689/2685 ... yes 689/2686 ... yes 689/2687 ... yes 689/2688 ... yes 689/2689 ... yes 689/2690 ... yes 689/2691 ... yes 689/2692 ... yes 689/2693 ... yes 689/2694 ... yes 689/2695 ... yes 689/2696 ... yes 689/2697 ... yes 689/2698 ... yes 689/2699 ... yes 689/2700 ... yes 689/2701 ... yes 689/2702 ... yes 689/2703 ... yes 689/2704 ... yes 689/2705 ... yes 689/2706 ... yes 689/2707 ... yes 689/2708 ... yes 689/2709 ... yes 689/2710 ... yes 689/2711 ... yes 689/2712 ... yes 689/2713 ... yes 689/2714 ... yes 689/2715 ... yes 689/2716 ... yes 689/2717 ... yes 689/2718 ... yes 689/2719 ... yes 689/2720 ... yes 689/2721 ... yes 689/2722 ... yes 689/2723 ... yes 689/2724 ... yes 689/2725 ... yes 689/2726 ... yes 689/2727 ... yes 689/2728 ... yes 689/2729 ... yes 689/2730 ... yes 689/2731 ... yes 689/2732 ... yes 689/2733 ... yes 689/2734 ... yes 689/2735 ... yes 689/2736 ... yes 689/2737 ... yes 689/2738 ... yes 689/2739 ... yes 689/2740 ... yes 689/2741 ... yes 689/2742 ... yes 689/2743 ... yes 689/2744 ... yes 689/2745 ... yes 689/2746 ... yes 689/2747 ... yes 689/2748 ... yes 689/2749 ... yes 689/2750 ... yes 689/2751 ... yes 689/2752 ... yes 689/2753 ... yes 689/2754 ... yes 689/2755 ... yes 689/2756 ... yes 689/2757 ... yes 689/2758 ... yes 689/2759 ... yes 689/2760 ... yes 689/2761 ... yes 689/2762 ... yes 689/2763 ... yes 689/2764 ... yes 689/2765 ... yes 689/2766 ... yes 689/2767 ... yes 689/2768 ... yes 689/2769 ... yes 689/2770 ... yes 689/2771 ... yes 689/2772 ... yes 689/2773 ... yes 689/2774 ... yes 689/2775 ... yes 689/2776 ... yes 689/2777 ... yes 689/2778 ... yes 689/2779 ... yes 689/2780 ... yes 689/2781 ... yes 689/2782 ... yes 689/2783 ... yes 689/2784 ... yes 689/2785 ... yes 689/2786 ... yes 689/2787 ... yes 689/2788 ... yes 689/2789 ... yes 689/2790 ... yes 689/2791 ... yes 689/2792 ... yes 689/2793 ... yes 689/2794 ... yes 689/2795 ... yes 689/2796 ... yes 689/2797 ... yes 689/2798 ... yes 689/2799 ... yes 689/2800 ... yes 689/2801 ... yes 689/2802 ... yes 689/2803 ... yes 689/2804 ... yes 689/2805 ... yes 689/2806 ... yes 689/2807 ... yes 689/2808 ... yes 689/2809 ... yes 689/2810 ... yes 689/2811 ... yes 689/2812 ... yes 689/2813 ... yes 689/2814 ... yes 689/2815 ... yes 689/2816 ... yes 689/2817 ... yes 689/2818 ... yes 689/2819 ... yes 689/2820 ... yes 689/2821 ... yes 689/2822 ... yes 689/2823 ... yes 689/2824 ... yes 689/2825 ... yes 689/2826 ... yes 689/2827 ... yes 689/2828 ... yes 689/2829 ... yes 689/2830 ... yes 689/2831 ... yes 689/2832 ... yes 689/2833 ... yes 689/2834 ... yes 689/2835 ... yes 689/2836 ... yes 689/2837 ... yes 689/2838 ... yes 689/2839 ... yes 689/2840 ... yes 689/2841 ... yes 689/2842 ... yes 689/2843 ... yes 689/2844 ... yes 689/2845 ... yes 689/2846 ... yes 689/2847 ... yes 689/2848 ... yes 689/2849 ... yes 689/2850 ... yes 689/2851 ... yes 689/2852 ... yes 689/2853 ... yes 689/2854 ... yes 689/2855 ... yes 689/2856 ... yes 689/2857 ... yes 689/2858 ... yes 689/2859 ... yes 689/2860 ... yes 689/2861 ... yes 689/2862 ... yes 689/2863 ... yes 689/2864 ... yes 689/2865 ... yes 689/2866 ... yes 689/2867 ... yes 689/2868 ... yes 689/2869 ... yes 689/2870 ... yes 689/2871 ... yes 689/2872 ... yes 689/2873 ... yes 689/2874 ... yes 689/2875 ... yes 689/2876 ... yes 689/2877 ... yes 689/2878 ... yes 689/2879 ... yes 689/2880 ... yes 689/2881 ... yes 689/2882 ... yes 689/2883 ... yes 689/2884 ... yes 689/2885 ... yes 689/2886 ... yes 689/2887 ... yes 689/2888 ... yes 689/2889 ... yes 689/2890 ... yes 689/2891 ... yes 689/2892 ... yes 689/2893 ... yes 689/2894 ... yes 689/2895 ... yes 689/2896 ... yes 689/2897 ... yes 689/2898 ... yes 689/2899 ... yes 689/2900 ... yes 689/2901 ... yes 689/2902 ... yes 689/2903 ... yes 689/2904 ... yes 689/2905 ... yes 689/2906 ... yes 689/2907 ... yes 689/2908 ... yes 689/2909 ... yes 689/2910 ... yes 689/2911 ... yes 689/2912 ... yes 689/2913 ... yes 689/2914 ... yes 689/2915 ... yes 689/2916 ... yes 689/2917 ... yes 689/2918 ... yes 689/2919 ... yes 689/2920 ... yes 689/2921 ... yes 689/2922 ... yes 689/2923 ... yes 689/2924 ... yes 689/2925 ... yes 689/2926 ... yes 689/2927 ... yes 689/2928 ... yes 689/2929 ... yes 689/2930 ... yes 689/2931 ... yes 689/2932 ... yes 689/2933 ... yes 689/2934 ... yes 689/2935 ... yes 689/2936 ... yes 689/2937 ... yes 689/2938 ... yes 689/2939 ... yes 689/2940 ... yes 689/2941 ... yes 689/2942 ... yes 689/2943 ... yes 689/2944 ... yes 689/2945 ... yes 689/2946 ... yes 689/2947 ... yes 689/2948 ... yes 689/2949 ... yes 689/2950 ... yes 689/2951 ... yes 689/2952 ... yes 689/2953 ... yes 689/2954 ... yes 689/2955 ... yes 689/2956 ... yes 689/2957 ... yes 689/2958 ... yes 689/2959 ... yes 689/2960 ... yes 689/2961 ... yes 689/2962 ... yes 689/2963 ... yes 689/2964 ... yes 689/2965 ... yes 689/2966 ... yes 689/2967 ... yes 689/2968 ... yes 689/2969 ... yes 689/2970 ... yes 689/2971 ... yes 689/2972 ... yes 689/2973 ... yes 689/2974 ... yes 689/2975 ... yes 689/2976 ... yes 689/2977 ... yes 689/2978 ... yes 689/2979 ... yes 689/2980 ... yes 689/2981 ... yes 689/2982 ... yes 689/2983 ... yes 689/2984 ... yes 689/2985 ... yes 689/2986 ... yes 689/2987 ... yes 689/2988 ... yes 689/2989 ... yes 689/2990 ... yes 689/2991 ... yes 689/2992 ... yes 689/2993 ... yes 689/2994 ... yes 689/2995 ... yes 689/2996 ... yes 689/2997 ... yes 689/2998 ... yes 689/2999 ... yes 689/3000 ... yes 689/3001 ... yes 689/3002 ... yes 689/3003 ... yes 689/3004 ... yes 689/3005 ... yes 689/3006 ... yes 689/3007 ... yes 689/3008 ... yes 689/3009 ... yes 689/3010 ... yes 689/3011 ... yes 689/3012 ... yes 689/3013 ... yes 689/3014 ... yes 689/3015 ... yes 689/3016 ... yes 689/3017 ... yes 689/3018 ... yes 689/3019 ... yes 689/3020 ... yes 689/3021 ... yes 689/3022 ... yes 689/3023 ... yes 689/3024 ... yes 689/3025 ... yes 689/3026 ... yes 689/3027 ... yes 689/3028 ... yes 689/3029 ... yes 689/3030 ... yes 689/3031 ... yes 689/3032 ... yes 689/3033 ... yes 689/3034 ... yes 689/3035 ... yes 689/3036 ... yes 689/3037 ... yes 689/3038 ... yes 689/3039 ... yes 689/3040 ... yes 689/3041 ... yes 689/3042 ... yes 689/3043 ... yes 689/3044 ... yes 689/3045 ... yes 689/3046 ... yes 689/3047 ... yes 689/3048 ... yes 689/3049 ... yes 689/3050 ... yes 689/3051 ... yes 689/3052 ... yes 689/3053 ... yes 689/3054 ... yes 689/3055 ... yes 689/3056 ... yes 689/3057 ... yes 689/3058 ... yes 689/3059 ... yes 689/3060 ... yes 689/3061 ... yes 689/3062 ... yes 689/3063 ... yes 689/3064 ... yes 689/3065 ... yes 689/3066 ... yes 689/3067 ... yes 689/3068 ... yes 689/3069 ... yes 689/3070 ... yes 689/3071 ... yes 689/3072 ... yes 689/3073 ... yes 689/3074 ... yes 689/3075 ... yes 689/3076 ... yes 689/3077 ... yes 689/3078 ... yes 689/3079 ... yes 689/3080 ... yes 689/3081 ... yes 689/3082 ... yes 689/3083 ... yes 689/3084 ... yes 689/3085 ... yes 689/3086 ... yes 689/3087 ... yes 689/3088 ... yes 689/3089 ... yes 689/3090 ... yes 689/3091 ... yes 689/3092 ... yes 689/3093 ... yes 689/3094 ... yes 689/3095 ... yes 689/3096 ... yes 689/3097 ... yes 689/3098 ... yes 689/3099 ... yes 689/3100 ... yes 689/3101 ... yes 689/3102 ... yes 689/3103 ... yes 689/3104 ... yes 689/3105 ... yes 689/3106 ... yes 689/3107 ... yes 689/3108 ... yes 689/3109 ... yes 689/3110 ... yes 689/3111 ... yes 689/3112 ... yes 689/3113 ... yes 689/3114 ... yes 689/3115 ... yes 689/3116 ... yes 689/3117 ... yes 689/3118 ... yes 689/3119 ... yes 689/3120 ... yes 689/3121 ... yes 689/3122 ... yes 689/3123 ... yes 689/3124 ... yes 689/3125 ... yes 689/3126 ... yes 689/3127 ... yes 689/3128 ... yes 689/3129 ... yes 689/3130 ... yes 689/3131 ... yes 689/3132 ... yes 689/3133 ... yes 689/3134 ... yes 689/3135 ... yes 689/3136 ... yes 689/3137 ... yes 689/3138 ... yes 689/3139 ... yes 689/3140 ... yes 689/3141 ... yes 689/3142 ... yes 689/3143 ... yes 689/3144 ... yes 689/3145 ... yes 689/3146 ... yes 689/3147 ... yes 689/3148 ... yes 689/3149 ... yes 689/3150 ... yes 689/3151 ... yes 689/3152 ... yes 689/3153 ... yes 689/3154 ... yes 689/3155 ... yes 689/3156 ... yes 689/3157 ... yes 689/3158 ... yes 689/3159 ... yes 689/3160 ... yes 689/3161 ... yes 689/3162 ... yes 689/3163 ... yes 689/3164 ... yes 689/3165 ... yes 689/3166 ... yes 689/3167 ... yes 689/3168 ... yes 689/3169 ... yes 689/3170 ... yes 689/3171 ... yes 689/3172 ... yes 689/3173 ... yes 689/3174 ... yes 689/3175 ... yes 689/3176 ... yes 689/3177 ... yes 689/3178 ... yes 689/3179 ... yes 689/3180 ... yes 689/3181 ... yes 689/3182 ... yes 689/3183 ... yes 689/3184 ... yes 689/3185 ... yes 689/3186 ... yes 689/3187 ... yes 689/3188 ... yes 689/3189 ... yes 689/3190 ... yes 689/3191 ... yes 689/3192 ... yes 689/3193 ... yes 689/3194 ... yes 689/3195 ... yes 689/3196 ... yes 689/3197 ... yes 689/3198 ... yes 689/3199 ... yes 689/3200 ... yes 689/3201 ... yes 689/3202 ... yes 689/3203 ... yes 689/3204 ... yes 689/3205 ... yes 689/3206 ... yes 689/3207 ... yes 689/3208 ... yes 689/3209 ... yes 689/3210 ... yes 689/3211 ... yes 689/3212 ... yes 689/3213 ... yes 689/3214 ... yes 689/3215 ... yes 689/3216 ... yes 689/3217 ... yes 689/3218 ... yes 689/3219 ... yes 689/3220 ... yes 689/3221 ... yes 689/3222 ... yes 689/3223 ... yes 689/3224 ... yes 689/3225 ... yes 689/3226 ... yes 689/3227 ... yes 689/3228 ... yes 689/3229 ... yes 689/3230 ... yes 689/3231 ... yes 689/3232 ... yes 689/3233 ... yes 689/3234 ... yes 689/3235 ... yes 689/3236 ... yes 689/3238 ... yes 689/3239 ... yes 689/3240 ... yes 689/3241 ... yes 689/3242 ... yes 689/3243 ... yes 689/3244 ... yes 689/3245 ... yes 689/3246 ... yes 689/3247 ... yes 689/3248 ... yes 689/3249 ... yes 689/3250 ... yes 689/3251 ... yes 689/3252 ... yes 689/3253 ... yes 689/3254 ... yes 689/3255 ... yes 689/3256 ... yes 689/3257 ... yes 689/3258 ... yes 689/3259 ... yes 689/3260 ... yes 689/3261 ... yes 689/3262 ... yes 689/3263 ... yes 689/3264 ... yes 689/3265 ... yes 689/3266 ... yes 689/3267 ... yes 689/3268 ... yes 689/3269 ... yes 689/3270 ... yes 689/3271 ... yes 689/3272 ... yes 689/3273 ... yes 689/3274 ... yes 689/3275 ... yes 689/3276 ... yes 689/3277 ... yes 689/3278 ... yes 689/3279 ... yes 689/3280 ... yes 689/3281 ... yes 689/3282 ... yes 689/3283 ... yes 689/3284 ... yes 689/3285 ... yes 689/3286 ... yes 689/3287 ... yes 689/3288 ... yes 689/3289 ... yes 689/3290 ... yes 689/3291 ... yes 689/3292 ... yes 689/3293 ... yes 689/3294 ... yes 689/3295 ... yes 689/3296 ... yes 689/3297 ... yes 689/3298 ... yes 689/3299 ... yes 689/3300 ... yes 689/3301 ... yes 689/3302 ... yes 689/3303 ... yes 689/3304 ... yes 689/3305 ... yes 689/3306 ... yes 689/3307 ... yes 689/3308 ... yes 689/3309 ... yes 689/3310 ... yes 689/3311 ... yes 689/3312 ... yes 689/3313 ... yes 689/3314 ... yes 689/3315 ... yes 689/3316 ... yes 689/3317 ... yes 689/3318 ... yes 689/3319 ... yes 689/3320 ... yes 689/3321 ... yes 689/3322 ... yes 689/3323 ... yes 689/3324 ... yes 689/3325 ... yes 689/3326 ... yes 689/3327 ... yes 689/3328 ... yes 689/3329 ... yes 689/3330 ... yes 689/3331 ... yes 689/3332 ... yes 689/3333 ... yes 689/3334 ... yes 689/3335 ... yes 689/3336 ... yes 689/3337 ... yes 689/3338 ... yes 689/3339 ... yes 689/3340 ... yes 689/3341 ... yes 689/3342 ... yes 689/3343 ... yes 689/3344 ... yes 689/3345 ... yes 689/3346 ... yes 689/3347 ... yes 689/3348 ... yes 689/3349 ... yes 689/3350 ... yes 689/3351 ... yes 689/3352 ... yes 689/3353 ... yes 689/3354 ... yes 689/3355 ... yes 689/3356 ... yes 689/3357 ... yes 689/3358 ... yes 689/3359 ... yes 689/3360 ... yes 689/3361 ... yes 689/3362 ... yes 689/3363 ... yes 689/3364 ... yes 689/3365 ... yes 689/3366 ... yes 689/3367 ... yes 689/3368 ... yes 689/3369 ... yes 689/3370 ... yes 689/3371 ... yes 689/3372 ... yes 689/3373 ... yes 689/3374 ... yes 689/3375 ... yes 689/3376 ... yes 689/3377 ... yes 689/3378 ... yes 689/3379 ... yes 689/3380 ... yes 689/3381 ... yes 689/3382 ... yes 689/3383 ... yes 689/3384 ... yes 689/3385 ... yes 689/3386 ... yes 689/3387 ... yes 689/3388 ... yes 689/3389 ... yes 689/3390 ... yes 689/3391 ... yes 687/3393 ... yes 687/3395 ... yes 645/3397 ... yes 643/3398 ... yes 687/3399 ... yes 687/3410 ... yes 687/3411 ... yes 687/3412 ... yes 687/3413 ... yes 687/3414 ... yes 687/3415 ... yes 687/3416 ... yes 687/3417 ... yes 687/3418 ... yes 687/3419 ... yes 643/3442 ... yes 643/3443 ... yes 552/3446 ... yes 690/3447 ... yes 690/3448 ... yes 691/3449 ... yes 691/3450 ... yes 690/3451 ... yes 702/3452 ... yes 715/3453 ... yes 13/3454 ... yes 13/3455 ... yes 606/3456 ... yes 552/3457 ... yes 702/3505 ... yes 717/3506 ... yes 782/3619 ... yes 782/3620 ... yes 782/3621 ... yes 782/3622 ... yes 782/3623 ... yes 782/3624 ... yes 782/3625 ... yes 782/3626 ... yes 782/3627 ... yes 782/3628 ... yes 782/3629 ... yes 782/3630 ... yes 782/3631 ... yes 782/3632 ... yes 782/3633 ... yes 782/3634 ... yes 782/3635 ... yes 641/3637 ... yes 641/3642 ... yes 641/3643 ... yes 782/3661 ... yes 803/3662 ... yes 641/3663 ... yes 642/3710 ... yes 816/3712 ... yes 816/3713 ... yes 806/3714 ... yes 806/3715 ... yes 642/3721 ... yes 823/3722 ... yes 824/3723 ... yes 535/3724 ... yes 823/3725 ... yes 552/3726 ... yes 552/3727 ... yes 228/3728 ... yes 782/3731 ... yes 31/3734 ... yes 830/3746 ... yes 642/3750 ... yes 642/3751 ... yes 642/3752 ... yes 642/3753 ... yes 641/3755 ... yes 214/3756 ... yes 214/3758 ... yes 642/3767 ... yes 821/3768 ... yes 617/3783 ... yes 642/3784 ... yes 861/3807 ... yes 863/3808 ... yes 10/3868 ... yes 870/4421 ... yes 685/4832 ... yes 685/4834 ... yes 685/4835 ... yes 685/4836 ... yes 685/4837 ... yes 871/4841 ... yes 872/4842 ... yes 642/4851 ... yes 875/4855 ... yes 875/4856 ... yes 875/4857 ... yes 875/4858 ... yes 875/4859 ... yes 875/4860 ... yes 875/4861 ... yes 875/4862 ... yes 875/4863 ... yes 875/4864 ... yes 875/4865 ... yes 875/4866 ... yes 875/4867 ... yes 875/4868 ... yes 875/4869 ... yes 875/4870 ... yes 875/4871 ... yes 875/4872 ... yes 875/4873 ... yes 875/4874 ... yes 875/4875 ... yes 875/4876 ... yes 875/4877 ... yes 875/4878 ... yes 875/4879 ... yes 875/4880 ... yes 875/4881 ... yes 875/4882 ... yes 875/4883 ... yes 875/4884 ... yes 875/4885 ... yes 875/4886 ... yes 875/4887 ... yes 875/4888 ... yes 875/4889 ... yes 875/4890 ... yes 875/4891 ... yes 875/4892 ... yes 875/4893 ... yes 875/4894 ... yes 875/4895 ... yes 875/4896 ... yes 875/4897 ... yes 875/4898 ... yes 875/4899 ... yes 875/4900 ... yes 875/4901 ... yes 875/4902 ... yes 875/4903 ... yes 875/4904 ... yes 875/4905 ... yes 875/4906 ... yes 875/4907 ... yes 875/4908 ... yes 875/4909 ... yes 875/4910 ... yes 875/4911 ... yes 875/4912 ... yes 875/4913 ... yes 875/4914 ... yes 875/4915 ... yes 875/4916 ... yes 875/4917 ... yes 875/4918 ... yes 875/4919 ... yes 875/4920 ... yes 875/4921 ... yes 875/4922 ... yes 875/4923 ... yes 875/4924 ... yes 875/4925 ... yes 875/4926 ... yes 875/4927 ... yes 875/4928 ... yes 875/4929 ... yes 875/4930 ... yes 875/4931 ... yes 875/4932 ... yes 875/4933 ... yes 875/4934 ... yes 875/4935 ... yes 875/4936 ... yes 875/4937 ... yes 875/4938 ... yes 875/4939 ... yes 875/4940 ... yes 875/4941 ... yes 875/4942 ... yes 875/4943 ... yes 875/4944 ... yes 875/4945 ... yes 875/4946 ... yes 875/4947 ... yes 875/4948 ... yes 875/4949 ... yes 875/4950 ... yes 875/4951 ... yes 875/4952 ... yes 875/4953 ... yes 875/4954 ... yes 875/4955 ... yes 875/4956 ... yes 875/4957 ... yes 875/4958 ... yes 875/4959 ... yes 875/4960 ... yes 875/4961 ... yes 875/4962 ... yes 875/4963 ... yes 875/4964 ... yes 875/4965 ... yes 875/4966 ... yes 875/4967 ... yes 875/4968 ... yes 875/4969 ... yes 875/4970 ... yes 875/4971 ... yes 875/4972 ... yes 875/4973 ... yes 875/4974 ... yes 875/4975 ... yes 875/4976 ... yes 875/4977 ... yes 875/4978 ... yes 875/4979 ... yes 875/4980 ... yes 875/4981 ... yes 875/4982 ... yes 875/4983 ... yes 875/4984 ... yes 875/4985 ... yes 875/4986 ... yes 875/4987 ... yes 875/4988 ... yes 875/4989 ... yes 875/4990 ... yes 875/4991 ... yes 875/4992 ... yes 875/4993 ... yes 875/4994 ... yes 875/4995 ... yes 875/4996 ... yes 875/4997 ... yes 875/4998 ... yes 875/4999 ... yes 875/5000 ... yes 875/5001 ... yes 875/5002 ... yes 875/5003 ... yes 875/5004 ... yes 875/5005 ... yes 875/5006 ... yes 875/5007 ... yes 875/5008 ... yes 875/5009 ... yes 875/5010 ... yes 875/5011 ... yes 875/5012 ... yes 875/5013 ... yes 875/5014 ... yes 875/5015 ... yes 875/5016 ... yes 875/5017 ... yes 875/5018 ... yes 875/5019 ... yes 875/5020 ... yes 875/5021 ... yes 875/5022 ... yes 875/5023 ... yes 875/5024 ... yes 875/5025 ... yes 875/5026 ... yes 875/5027 ... yes 875/5028 ... yes 875/5029 ... yes 875/5030 ... yes 875/5031 ... yes 875/5032 ... yes 875/5033 ... yes 875/5034 ... yes 875/5035 ... yes 875/5036 ... yes 875/5037 ... yes 875/5038 ... yes 875/5039 ... yes 875/5040 ... yes 875/5041 ... yes 875/5042 ... yes 875/5043 ... yes 875/5044 ... yes 875/5045 ... yes 875/5046 ... yes 875/5047 ... yes 875/5048 ... yes 875/5049 ... yes 875/5050 ... yes 875/5051 ... yes 875/5052 ... yes 875/5053 ... yes 875/5054 ... yes 875/5055 ... yes 875/5056 ... yes 875/5057 ... yes 875/5058 ... yes 875/5059 ... yes 875/5060 ... yes 875/5061 ... yes 875/5062 ... yes 875/5063 ... yes 875/5064 ... yes 875/5065 ... yes 875/5066 ... yes 875/5067 ... yes 875/5068 ... yes 875/5069 ... yes 875/5070 ... yes 875/5071 ... yes 875/5072 ... yes 875/5073 ... yes 875/5074 ... yes 875/5075 ... yes 875/5076 ... yes 875/5077 ... yes 875/5078 ... yes 875/5079 ... yes 875/5080 ... yes 875/5081 ... yes 875/5082 ... yes 875/5083 ... yes 875/5084 ... yes 875/5085 ... yes 875/5086 ... yes 875/5087 ... yes 875/5088 ... yes 875/5089 ... yes 875/5090 ... yes 875/5091 ... yes 875/5092 ... yes 875/5093 ... yes 875/5094 ... yes 875/5095 ... yes 875/5096 ... yes 875/5097 ... yes 875/5098 ... yes 875/5099 ... yes 875/5100 ... yes 875/5101 ... yes 875/5102 ... yes 875/5103 ... yes 875/5104 ... yes 875/5105 ... yes 875/5106 ... yes 875/5107 ... yes 875/5108 ... yes 875/5109 ... yes 875/5110 ... yes 875/5111 ... yes 875/5112 ... yes 875/5113 ... yes 875/5114 ... yes 875/5115 ... yes 875/5116 ... yes 875/5117 ... yes 875/5118 ... yes 875/5119 ... yes 875/5120 ... yes 875/5121 ... yes 875/5122 ... yes 875/5123 ... yes 875/5124 ... yes 875/5125 ... yes 875/5126 ... yes 875/5127 ... yes 875/5128 ... yes 875/5129 ... yes 875/5130 ... yes 875/5131 ... yes 875/5132 ... yes 875/5133 ... yes 875/5134 ... yes 875/5135 ... yes 875/5136 ... yes 875/5137 ... yes 875/5138 ... yes 875/5139 ... yes 875/5140 ... yes 875/5141 ... yes 875/5142 ... yes 875/5143 ... yes 875/5144 ... yes 875/5145 ... yes 875/5146 ... yes 875/5147 ... yes 875/5148 ... yes 875/5149 ... yes 875/5150 ... yes 875/5151 ... yes 875/5152 ... yes 875/5153 ... yes 875/5154 ... yes 875/5155 ... yes 875/5156 ... yes 875/5157 ... yes 875/5158 ... yes 875/5159 ... yes 875/5160 ... yes 875/5161 ... yes 875/5162 ... yes 875/5163 ... yes 875/5164 ... yes 875/5165 ... yes 875/5166 ... yes 875/5167 ... yes 875/5168 ... yes 875/5169 ... yes 875/5170 ... yes 875/5171 ... yes 875/5172 ... yes 875/5173 ... yes 875/5174 ... yes 875/5175 ... yes 875/5176 ... yes 875/5177 ... yes 875/5178 ... yes 875/5179 ... yes 875/5180 ... yes 875/5181 ... yes 875/5182 ... yes 875/5183 ... yes 875/5184 ... yes 875/5185 ... yes 875/5186 ... yes 875/5187 ... yes 875/5188 ... yes 875/5189 ... yes 875/5190 ... yes 875/5191 ... yes 875/5192 ... yes 875/5193 ... yes 875/5194 ... yes 875/5195 ... yes 875/5196 ... yes 875/5197 ... yes 875/5198 ... yes 875/5199 ... yes 875/5200 ... yes 875/5201 ... yes 875/5202 ... yes 875/5203 ... yes 875/5204 ... yes 875/5205 ... yes 875/5206 ... yes 875/5207 ... yes 875/5208 ... yes 875/5209 ... yes 875/5210 ... yes 875/5211 ... yes 875/5212 ... yes 875/5213 ... yes 875/5214 ... yes 875/5215 ... yes 875/5216 ... yes 875/5217 ... yes 875/5218 ... yes 875/5219 ... yes 875/5220 ... yes 875/5221 ... yes 875/5222 ... yes 875/5223 ... yes 875/5224 ... yes 875/5225 ... yes 875/5226 ... yes 875/5227 ... yes 875/5228 ... yes 875/5229 ... yes 875/5230 ... yes 875/5231 ... yes 875/5232 ... yes 875/5233 ... yes 875/5234 ... yes 875/5235 ... yes 875/5236 ... yes 875/5237 ... yes 875/5238 ... yes 875/5239 ... yes 875/5240 ... yes 875/5241 ... yes 875/5242 ... yes 875/5243 ... yes 875/5244 ... yes 875/5245 ... yes 875/5246 ... yes 875/5247 ... yes 875/5248 ... yes 875/5249 ... yes 875/5250 ... yes 875/5251 ... yes 875/5252 ... yes 875/5253 ... yes 875/5254 ... yes 875/5255 ... yes 875/5256 ... yes 875/5257 ... yes 875/5258 ... yes 875/5259 ... yes 875/5260 ... yes 875/5261 ... yes 875/5262 ... yes 875/5263 ... yes 875/5264 ... yes 875/5265 ... yes 875/5266 ... yes 875/5267 ... yes 875/5268 ... yes 875/5269 ... yes 875/5270 ... yes 875/5271 ... yes 875/5272 ... yes 875/5273 ... yes 875/5274 ... yes 875/5275 ... yes 875/5276 ... yes 875/5277 ... yes 875/5278 ... yes 875/5279 ... yes 875/5280 ... yes 875/5281 ... yes 875/5282 ... yes 875/5283 ... yes 875/5284 ... yes 875/5285 ... yes 875/5286 ... yes 875/5287 ... yes 875/5288 ... yes 875/5289 ... yes 875/5290 ... yes 875/5291 ... yes 875/5292 ... yes 875/5293 ... yes 875/5294 ... yes 875/5295 ... yes 875/5296 ... yes 875/5297 ... yes 875/5298 ... yes 875/5299 ... yes 875/5300 ... yes 875/5301 ... yes 875/5302 ... yes 875/5303 ... yes 875/5304 ... yes 875/5305 ... yes 875/5306 ... yes 875/5307 ... yes 875/5308 ... yes 875/5309 ... yes 875/5310 ... yes 875/5311 ... yes 875/5312 ... yes 875/5313 ... yes 875/5314 ... yes 875/5315 ... yes 875/5316 ... yes 875/5317 ... yes 875/5318 ... yes 875/5319 ... yes 875/5320 ... yes 875/5321 ... yes 875/5322 ... yes 875/5323 ... yes 875/5324 ... yes 875/5325 ... yes 875/5326 ... yes 875/5327 ... yes 875/5328 ... yes 875/5329 ... yes 875/5330 ... yes 875/5331 ... yes 875/5332 ... yes 875/5333 ... yes 875/5334 ... yes 875/5335 ... yes 875/5336 ... yes 875/5337 ... yes 875/5338 ... yes 875/5339 ... yes 875/5340 ... yes 875/5341 ... yes 875/5342 ... yes 875/5343 ... yes 875/5344 ... yes 875/5345 ... yes 875/5346 ... yes 875/5347 ... yes 875/5348 ... yes 875/5349 ... yes 875/5350 ... yes 875/5351 ... yes 875/5352 ... yes 875/5353 ... yes 875/5354 ... yes 875/5355 ... yes 875/5356 ... yes 875/5357 ... yes 875/5358 ... yes 875/5359 ... yes 875/5360 ... yes 875/5361 ... yes 875/5362 ... yes 875/5363 ... yes 875/5364 ... yes 875/5365 ... yes 875/5366 ... yes 875/5367 ... yes 875/5368 ... yes 875/5369 ... yes 875/5370 ... yes 875/5371 ... yes 875/5372 ... yes 875/5373 ... yes 875/5374 ... yes 875/5375 ... yes 875/5376 ... yes 875/5377 ... yes 875/5378 ... yes 875/5379 ... yes 875/5380 ... yes 875/5381 ... yes 875/5382 ... yes 875/5383 ... yes 875/5384 ... yes 875/5385 ... yes 875/5386 ... yes 875/5387 ... yes 875/5388 ... yes 875/5389 ... yes 875/5390 ... yes 875/5391 ... yes 875/5392 ... yes 875/5393 ... yes 875/5394 ... yes 875/5395 ... yes 875/5396 ... yes 875/5397 ... yes 875/5398 ... yes 875/5399 ... yes 875/5400 ... yes 875/5401 ... yes 875/5402 ... yes 875/5403 ... yes 875/5404 ... yes 875/5405 ... yes 875/5406 ... yes 875/5407 ... yes 875/5408 ... yes 875/5409 ... yes 875/5410 ... yes 875/5411 ... yes 875/5412 ... yes 875/5413 ... yes 875/5414 ... yes 875/5415 ... yes 875/5416 ... yes 875/5417 ... yes 875/5418 ... yes 875/5419 ... yes 875/5420 ... yes 875/5421 ... yes 875/5422 ... yes 875/5423 ... yes 875/5424 ... yes 875/5425 ... yes 875/5426 ... yes 875/5427 ... yes 875/5428 ... yes 875/5429 ... yes 875/5430 ... yes 875/5431 ... yes 875/5432 ... yes 875/5433 ... yes 875/5434 ... yes 875/5435 ... yes 875/5436 ... yes 875/5437 ... yes 875/5438 ... yes 875/5439 ... yes 875/5440 ... yes 875/5441 ... yes 875/5442 ... yes 875/5443 ... yes 875/5444 ... yes 875/5445 ... yes 875/5446 ... yes 875/5447 ... yes 875/5448 ... yes 875/5449 ... yes 875/5450 ... yes 875/5451 ... yes 875/5452 ... yes 875/5453 ... yes 875/5454 ... yes 875/5455 ... yes 875/5456 ... yes 875/5457 ... yes 875/5458 ... yes 875/5459 ... yes 875/5460 ... yes 875/5461 ... yes 875/5462 ... yes 875/5463 ... yes 875/5464 ... yes 875/5465 ... yes 875/5466 ... yes 875/5467 ... yes 875/5468 ... yes 875/5469 ... yes 875/5470 ... yes 875/5471 ... yes 875/5472 ... yes 875/5473 ... yes 875/5474 ... yes 875/5475 ... yes 875/5476 ... yes 875/5477 ... yes 875/5478 ... yes 875/5479 ... yes 875/5480 ... yes 875/5481 ... yes 875/5482 ... yes 875/5483 ... yes 875/5484 ... yes 875/5485 ... yes 875/5486 ... yes 875/5487 ... yes 875/5488 ... yes 875/5489 ... yes 875/5490 ... yes 875/5491 ... yes 875/5492 ... yes 875/5493 ... yes 875/5494 ... yes 875/5495 ... yes 875/5496 ... yes 875/5497 ... yes 875/5498 ... yes 875/5499 ... yes 875/5500 ... yes 875/5501 ... yes 875/5502 ... yes 875/5503 ... yes 875/5504 ... yes 875/5505 ... yes 875/5506 ... yes 875/5507 ... yes 875/5508 ... yes 875/5509 ... yes 875/5510 ... yes 875/5511 ... yes 875/5512 ... yes 875/5513 ... yes 875/5514 ... yes 875/5515 ... yes 875/5516 ... yes 875/5517 ... yes 875/5518 ... yes 875/5519 ... yes 875/5520 ... yes 875/5521 ... yes 875/5522 ... yes 875/5523 ... yes 875/5524 ... yes 875/5525 ... yes 875/5526 ... yes 875/5527 ... yes 875/5528 ... yes 875/5529 ... yes 875/5530 ... yes 875/5531 ... yes 875/5532 ... yes 875/5533 ... yes 875/5534 ... yes 875/5535 ... yes 875/5536 ... yes 875/5537 ... yes 875/5538 ... yes 875/5539 ... yes 875/5540 ... yes 875/5541 ... yes 875/5542 ... yes 875/5543 ... yes 875/5544 ... yes 875/5545 ... yes 875/5546 ... yes 875/5547 ... yes 875/5548 ... yes 875/5549 ... yes 875/5550 ... yes 875/5551 ... yes 875/5552 ... yes 875/5553 ... yes 875/5554 ... yes 875/5555 ... yes 875/5556 ... yes 875/5557 ... yes 875/5558 ... yes 875/5559 ... yes 875/5560 ... yes 875/5561 ... yes 875/5562 ... yes 875/5563 ... yes 875/5564 ... yes 875/5565 ... yes 875/5566 ... yes 875/5567 ... yes 875/5568 ... yes 875/5569 ... yes 875/5570 ... yes 875/5571 ... yes 875/5572 ... yes 875/5573 ... yes 875/5574 ... yes 875/5575 ... yes 875/5576 ... yes 875/5577 ... yes 875/5578 ... yes 875/5579 ... yes 875/5580 ... yes 875/5581 ... yes 875/5582 ... yes 875/5583 ... yes 875/5584 ... yes 875/5585 ... yes 875/5586 ... yes 875/5587 ... yes 875/5588 ... yes 875/5589 ... yes 875/5590 ... yes 875/5591 ... yes 875/5592 ... yes 875/5593 ... yes 875/5594 ... yes 875/5595 ... yes 875/5596 ... yes 875/5597 ... yes 875/5598 ... yes 875/5599 ... yes 875/5600 ... yes 875/5601 ... yes 875/5602 ... yes 875/5603 ... yes 875/5604 ... yes 875/5605 ... yes 875/5606 ... yes 875/5607 ... yes 875/5608 ... yes 875/5609 ... yes 875/5610 ... yes 875/5611 ... yes 875/5612 ... yes 875/5613 ... yes 875/5614 ... yes 875/5615 ... yes 875/5616 ... yes 875/5617 ... yes 875/5618 ... yes 875/5619 ... yes 875/5620 ... yes 875/5621 ... yes 875/5622 ... yes 875/5623 ... yes 875/5624 ... yes 875/5625 ... yes 875/5626 ... yes 875/5627 ... yes 875/5628 ... yes 875/5629 ... yes 875/5630 ... yes 875/5631 ... yes 875/5632 ... yes 875/5633 ... yes 875/5634 ... yes 875/5635 ... yes 875/5636 ... yes 875/5637 ... yes 875/5638 ... yes 875/5639 ... yes 875/5640 ... yes 875/5641 ... yes 875/5642 ... yes 875/5643 ... yes 875/5644 ... yes 875/5645 ... yes 875/5646 ... yes 875/5647 ... yes 875/5648 ... yes 875/5649 ... yes 875/5650 ... yes 875/5651 ... yes 875/5652 ... yes 875/5653 ... yes 875/5654 ... yes 875/5655 ... yes 875/5656 ... yes 875/5657 ... yes 875/5658 ... yes 875/5659 ... yes 875/5660 ... yes 875/5661 ... yes 875/5662 ... yes 875/5663 ... yes 875/5664 ... yes 875/5665 ... yes 875/5666 ... yes 875/5667 ... yes 875/5668 ... yes 875/5669 ... yes 875/5670 ... yes 875/5671 ... yes 875/5672 ... yes 875/5673 ... yes 875/5674 ... yes 875/5675 ... yes 875/5676 ... yes 875/5677 ... yes 875/5678 ... yes 875/5679 ... yes 875/5680 ... yes 875/5681 ... yes 875/5682 ... yes 875/5683 ... yes 875/5684 ... yes 875/5685 ... yes 875/5686 ... yes 875/5687 ... yes 875/5688 ... yes 875/5689 ... yes 875/5690 ... yes 875/5691 ... yes 875/5692 ... yes 875/5693 ... yes 875/5694 ... yes 875/5695 ... yes 875/5696 ... yes 875/5697 ... yes 875/5698 ... yes 875/5699 ... yes 875/5700 ... yes 875/5701 ... yes 875/5702 ... yes 875/5703 ... yes 875/5704 ... yes 875/5705 ... yes 875/5706 ... yes 875/5707 ... yes 875/5708 ... yes 875/5709 ... yes 875/5710 ... yes 875/5711 ... yes 875/5712 ... yes 875/5713 ... yes 875/5714 ... yes 875/5715 ... yes 875/5716 ... yes 875/5717 ... yes 875/5718 ... yes 875/5719 ... yes 875/5720 ... yes 875/5721 ... yes 875/5722 ... yes 875/5723 ... yes 875/5724 ... yes 875/5725 ... yes 875/5726 ... yes 875/5727 ... yes 875/5728 ... yes 875/5729 ... yes 875/5730 ... yes 875/5731 ... yes 875/5732 ... yes 875/5733 ... yes 875/5734 ... yes 875/5735 ... yes 875/5736 ... yes 875/5737 ... yes 875/5738 ... yes 875/5739 ... yes 875/5740 ... yes 875/5741 ... yes 875/5742 ... yes 875/5743 ... yes 875/5744 ... yes 875/5745 ... yes 875/5746 ... yes 875/5747 ... yes 875/5748 ... yes 875/5749 ... yes 875/5750 ... yes 875/5751 ... yes 875/5752 ... yes 875/5753 ... yes 875/5754 ... yes 875/5755 ... yes 875/5756 ... yes 875/5757 ... yes 875/5758 ... yes 875/5759 ... yes 875/5760 ... yes 875/5761 ... yes 875/5762 ... yes 875/5763 ... yes 875/5764 ... yes 875/5765 ... yes 875/5766 ... yes 875/5767 ... yes 875/5768 ... yes 875/5769 ... yes 875/5770 ... yes 875/5771 ... yes 875/5772 ... yes 875/5773 ... yes 875/5774 ... yes 875/5775 ... yes 875/5776 ... yes 875/5777 ... yes 875/5778 ... yes 875/5779 ... yes 875/5780 ... yes 875/5781 ... yes 875/5782 ... yes 875/5783 ... yes 875/5784 ... yes 875/5785 ... yes 875/5786 ... yes 875/5787 ... yes 875/5788 ... yes 875/5789 ... yes 875/5790 ... yes 875/5791 ... yes 875/5792 ... yes 875/5793 ... yes 875/5794 ... yes 875/5795 ... yes 875/5796 ... yes 875/5797 ... yes 875/5798 ... yes 875/5799 ... yes 875/5800 ... yes 875/5801 ... yes 875/5802 ... yes 875/5803 ... yes 875/5804 ... yes 875/5805 ... yes 875/5806 ... yes 875/5807 ... yes 875/5808 ... yes 875/5809 ... yes 875/5810 ... yes 875/5811 ... yes 875/5812 ... yes 875/5813 ... yes 875/5814 ... yes 875/5815 ... yes 875/5816 ... yes 875/5817 ... yes 875/5818 ... yes 875/5819 ... yes 875/5820 ... yes 875/5821 ... yes 875/5822 ... yes 875/5823 ... yes 875/5824 ... yes 875/5825 ... yes 875/5826 ... yes 875/5827 ... yes 875/5828 ... yes 875/5829 ... yes 875/5830 ... yes 875/5831 ... yes 875/5832 ... yes 875/5833 ... yes 875/5834 ... yes 875/5835 ... yes 875/5836 ... yes 875/5837 ... yes 875/5838 ... yes 875/5839 ... yes 875/5840 ... yes 875/5841 ... yes 875/5842 ... yes 875/5843 ... yes 875/5844 ... yes 875/5845 ... yes 875/5846 ... yes 875/5847 ... yes 875/5848 ... yes 875/5849 ... yes 875/5850 ... yes 875/5851 ... yes 875/5852 ... yes 875/5853 ... yes 875/5854 ... yes 876/5855 ... yes 876/5856 ... yes 876/5857 ... yes 876/5858 ... yes 876/5859 ... yes 877/5861 ... yes 642/5862 ... yes 642/5863 ... yes 642/5864 ... yes 642/5865 ... yes 642/5866 ... yes 642/5867 ... yes 642/5868 ... yes 642/5888 ... yes 642/5889 ... yes 642/5890 ... yes 642/5891 ... yes 642/5894 ... yes 642/5895 ... yes 642/5896 ... yes 642/5897 ... yes 642/5898 ... yes 642/5899 ... yes 642/5900 ... yes 642/5913 ... yes 880/5916 ... yes 880/5920 ... yes 882/5936 ... yes 870/5946 ... yes 31/5968 ... yes 2/5973 ... yes 642/5974 ... yes 642/5975 ... yes 31/5976 ... yes 642/5977 ... yes 716/5978 ... yes 716/5979 ... yes 716/5980 ... yes 716/5981 ... yes 31/6000 ... yes 888/6048 ... yes 539/6049 ... yes 13/6054 ... yes 13/6055 ... yes 874/6060 ... yes 716/6076 ... yes 874/6086 ... yes 874/6087 ... yes 1176/6097 ... yes 642/6131 ... yes 891/6150 ... yes 87/6152 ... yes 895/6153 ... yes 642/6197 ... yes 642/6198 ... yes 642/6199 ... yes 642/6200 ... yes 642/6201 ... yes 642/6202 ... yes 642/6203 ... yes 642/6204 ... yes 642/6205 ... yes 642/6206 ... yes 642/6231 ... yes 642/6232 ... yes 642/6233 ... yes 642/6234 ... yes 642/6235 ... yes 642/6236 ... yes 642/6237 ... yes 642/6238 ... yes 642/6239 ... yes 642/6240 ... yes 642/6241 ... yes 642/6242 ... yes 642/6243 ... yes 642/6244 ... yes 642/6245 ... yes 642/6246 ... yes 642/6247 ... yes 642/6248 ... yes 642/6249 ... yes 642/6250 ... yes 642/6251 ... yes 642/6252 ... yes 642/6253 ... yes 642/6254 ... yes 642/6255 ... yes 897/6256 ... yes 897/6257 ... yes 897/6258 ... yes 897/6259 ... yes 897/6260 ... yes 642/6316 ... yes 642/6317 ... yes 898/6318 ... yes 899/6319 ... yes 642/6320 ... yes 642/6321 ... yes 642/6322 ... yes 642/6325 ... yes 642/6326 ... yes 642/6327 ... yes 900/6328 ... yes 900/6329 ... yes 642/6336 ... yes 642/6337 ... yes 642/6338 ... yes 642/6339 ... yes 642/6340 ... yes 901/6341 ... yes 901/6342 ... yes 901/6343 ... yes 902/6344 ... yes 902/6345 ... yes 910/6349 ... yes 642/6363 ... yes 642/6364 ... yes 642/6365 ... yes 642/6366 ... yes 913/6377 ... yes 642/6382 ... yes 642/6383 ... yes 642/6384 ... yes 642/6385 ... yes 914/6386 ... yes 914/6387 ... yes 830/6516 ... yes 830/6518 ... yes 916/6548 ... yes 642/6556 ... yes 646/6557 ... yes 642/6558 ... yes 646/6559 ... yes 970/6572 ... yes 891/6574 ... yes 891/6575 ... yes 874/6577 ... yes 972/6578 ... yes 973/6579 ... yes 973/6580 ... yes 642/6622 ... yes 891/6762 ... yes 891/6763 ... yes 973/6892 ... yes 984/6893 ... yes 994/6898 ... yes 1167/6938 ... yes 1164/6939 ... yes 1169/6993 ... yes 1170/6994 ... yes 1170/6995 ... yes 1172/7005 ... yes 1176/7018 ... yes 870/7120 ... yes 870/7213 ... yes 969/7315 ... yes 1179/7395 ... yes 1179/7408 ... yes 1184/7847 ... yes 1184/7848 ... yes 1184/7849 ... yes 1184/7850 ... yes 1184/7851 ... yes 1184/7876 ... yes 1311/7899 ... yes 1343/7900 ... yes 1365/7992 ... yes 1378/7993 ... yes 1386/7994 ... yes 642/8046 ... yes 642/8095 ... yes 642/8096 ... yes 642/8097 ... yes 891/8098 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git user has default SSH configuration? ... yes Active users: ... 1165 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
My understanding is that the default config of this cron job should not be in the Gitlab.ee do
block, but above, such that it gets executed unconditionally.
I see in !69165 (diffs) that it has been like that from the beginning, and I don't understand why, since this LFK mechanism does not seem specific to commercial editions. I've seen no mention of that matter in the !69165 (merged) comments/reviews.
I've not yet tried to workaround this issue; I know that running LooseForeignKeys::CleanupWorker
from the Rails console works as expected (one run was enough to process all deleted records on a small GitLab installation), but I've not yet tried to define the missing cron myself. We're using gitlab-omnibus, and while it would let me override the cron schedule (['loose_foreign_keys_cleanup_worker']['cron']
), I don't think I can set ['loose_foreign_keys_cleanup_worker']['job_class']
, which I assume would still be missing.
Of course, if not defining this cron for self-managed/Free installations was indeed on purpose, then sorry for the wrong report... But I would then much appreciate knowing the rationales. (@ahegyi maybe?)