Commit 97386dd2 authored by Benjamin Winger's avatar Benjamin Winger

Initial work on v2.0 with support for the OpenMW Mod Repository

parent d572b15c
__pycache__
*.pyc
# openMMM
openMorrowind Mod Manager. A cli tool for linux to manage mods for openMW
# OpenMMM
OpenMW Mod Manager. A cli tool for linux to manage mods for OpenMW
Forked from https://github.com/Korons/openMMM
## Optional Dependencies
## Dependencies
### patool
http://wummel.github.io/patool/
Highly recommended. OpenMMM can be used without patool, but will only be able to install from uncompressed directories, not archives.
### colorama
### appdirs
## Optional Dependencies
### mlox
https://github.com/mlox/mlox
......@@ -29,52 +35,21 @@ To install into `/usr/bin` (requires root), run
To install into `~/.local/bin`, run
```./setup.py install --user```
If you already have mods installed and would like use openmmm to reinstall or uninstall existing mods, or would like openmmm to install mods in a certain directory, set the `mod_dir` variable in `~/.openmmm.cfg` to be the path of that directory. An example config file is included in the file `openmmm.cfg.default`. Otherwise OpenMMM will install mods in the default location of `~/MWmods`
Alternatively, or if you have mods installed in multiple directories already, use the `--collect-mods` flag to move all mods referenced in the config to the mod directory.
You will want to create a config file in ~/.config/openmmm/openmmm.cfg. A sample is included in the root of the repository. Primarily you will want to set the USE variable and the ARCH variable to reflect your openmw setup.
## Usage
Mods can be installed with the `-a` flag and the path to the mod you want to install. OpenMMM supports both uncompressed directories and archives (with the use of patool).
When adding a mod OpenMMM does the following:
1. Extracts the archive into `/tmp` (Please ensure that `/tmp` is large enough to hold the mod being installed)
2. Attempts to detect the location of the `Data Files` directory within the mod, as well as any optional assets, then prompts you for which of these you want to use if multiple are found.
3. Attempts to fix incorrectly named normal and specular texture files (renames `_nm` and `_normal(s)` to `_n`; `_reflection` and `_ref` to `_spec`. If you know of any other suffixes used in mods, create an issue or PR and they can be added)
4. Installs mod into `~/MWmods/MOD_NAME` (directory is configurable via `~/.openmmm.cfg`). Version numbers, such as from the end of archive names, are removed from the installed directory name.
5. Detects bsa, esp, esm and omwaddon files. bsa files are added automatically to `openmw.cfg`, while a prompt is provided so that you can choose which esp, esm and omwaddon files to enable (currently always prompts even if there is only one of these)
6. Updates `openmw.cfg`, adding bsa and esp etc. from previous step, as well as the `Data Files` locations from step 2. If the mod's data directories previously existed in the config file a prompt will be given to determine if the mod is reinstalled at the same location or at the end of the file.
```openmmm.py -a yourMod```
Mods can be reinstalled with the `-r` flag. This is the same as the `-a` flag but preserves the install order if the mod has been previously installed. OpenMMM will detect if the mod has been previously installed and prompt you for what to do if you use the `-a` flag instead, so all this does is skip that check.
```openmmm.py -r yourMod```
Mods can be removed with the `-R` flag. Note that the path given should be the install location, not an archive. OpenMMM does not currently attempt to uninstall mods from the default location if you run this on an archive.
```openmmm.py -R ~/MWmods/yourInstalledMod```
There are also two other flags that can be used to invoke other utilities. The `-m` flag invokes mlox to update the load order of the scripts and print out mlox's notes on the scripts that you have installed. The `-L` flag invokes omwllf to fix the levelled lists. These can be run on their own, or in combination with other flags when doing updates. If you are installing or removing many mods at once, it is recommended to run them only when you are done as they take considerably more time than installing mods
```openmmm.py -mL```
The `--validate` option checks to see if mods that are in the OpenMW config file exist, and that mods in the mod directory are in the config file and prints a warning if it encounters anything unexpected.
```openmmm.py --validate```
The `--rename-all` option renames all installed mod directories, stripping trailing version numbers from the directory names. It will also replace spaces with underscores if `spaces = False` is set in openmmm.cfg, and replace underscores with spaces if `spaces = True` is set.
Mods can be installed by passing the relevant atoms as command line arguments. E.g.:
`openmmm.py abandoned-flat`
```openmmm.py --rename-all```
They will automatically be downloaded, configured and installed.
As rename-all, but just prints what will be moved.
The `-c` flag will remove the specified mods and all mods that depend on, or are dependencies of, the specified mods.
```openmmm.py --fake-rename-all```
The `-C` flag will remove the specified mods, ignoring dependencies
## Notes
- Make sure the openmw-launcher is not running. If you add mods when the openmw-launcher is running the mods will be removed from the config file when the openmw-launcher is closed.
- While OpenMMM attempts to clean up mod names by removing version numbers and standardizing spaces/underscores, it cannot do anything for mods from archives that don't have a meaningful archive name. If you want to ensure your mods are installed with nice names, rename the archives to use those names before installing.
This project is not affiliated with openMW
[general]
mod_dir = ~/MWmods
MOD_DIR=~/.local/share/openmmm/mods
[naming]
# Enables renaming of mod dirs, which tries to standardize directory names by stripping off trailing version numbers etc.
rename = True
# If True: replaces underscores with spaces; if False: replaces spaces with underscores; if unset: leaves spaces and underscores alone
spaces = False
# Valid global use flags are "morrowind", "bloodmoon", "tribunal"
USE=morrowind
# Valid Architectures are "openmw" "tes3mp"
ARCH=openmw
This diff is collapsed.
from colorama import init, Fore, Back, Style
init()
def colour(colour, text):
return '{}{}{}'.format(colour, text, Style.RESET_ALL)
from openmmm.globals import OPENMW_CONFIG, OPENMW_CONFIG_DIR
import fnmatch
import shutil
import re
# Returns config file as a list of strings (one string per line)
def read_config():
with open(OPENMW_CONFIG, mode='r') as config:
return config.read().splitlines()
# Replaces config file with the given
def write_config(new_config):
# We make a back up of the config file
shutil.copy(OPENMW_CONFIG, OPENMW_CONFIG_DIR + 'openmw.cfg.bak')
print("Saving changes made to MW config")
with open(OPENMW_CONFIG, mode='w') as config:
for line in new_config:
print(line, file=config)
def add_config(config, prefix, name, index = -1):
if check_config(config, prefix, name) != -1:
return
line = "{0}={1}".format(prefix, name)
if index == -1:
config.append(line)
else:
config.insert(index, line)
print('Added "{}" to config'.format(line))
# Checks if config contains a line matching the parameters. Supports globbing in the prefix and name
def check_config(config, prefix, name):
for line in config:
if fnmatch.fnmatch(line, '{}={}'.format(prefix, name)) or fnmatch.fnmatch(line, '{}="{}"'.format(prefix, name)):
return config.index(line)
return -1
def check_config_subdirs(config, prefix, name):
index = check_config(config, prefix, name)
if index == -1:
return check_config(config, prefix, '{}/*'.format(name))
return index
# Returns index-value pairs for lines in config that match the given prefix and name
def find_config(config, prefix, name):
lines = []
for (index, line) in enumerate(config):
if fnmatch.fnmatch(line, '{}={}'.format(prefix, name)) or fnmatch.fnmatch(line, '{}="{}"'.format(prefix, name)):
match = re.match('^{}=("(.*)"|(.*))'.format(prefix), line)
newline = match.group(2) if match.group(2) else match.group(1)
lines.append((index, newline))
return lines
def find_config_subdirs(config, prefix, name):
return find_config(config, prefix, name) + find_config(config, prefix, '{}/*'.format(name))
def Diff(li1, li2):
li_dif = [i for i in li1 + li2 if i not in li1 or i not in li2]
return li_dif
# Removes lines from the config matching the parameters. Supports globbing in the prefix and name
def remove_config(config, prefix, name):
to_remove = find_config(config, prefix, name)
for (index, line) in to_remove:
del config[index]
print('Removed "{}" from config'.format(line))
return to_remove
def remove_config_subdirs(config, prefix, name):
return remove_config(config, prefix, name) + remove_config(config, prefix, '{}/*'.format(name))
import os
import sys
import tempfile
import configparser
from appdirs import user_data_dir, user_cache_dir, user_config_dir
APP_NAME = 'openmmm'
APP_AUTHOR = 'openmmm'
HOME = os.getenv("HOME")
OPENMW_CONFIG_DIR = user_config_dir('openmw', 'openmw')
OPENMW_CONFIG = os.path.join(OPENMW_CONFIG_DIR, 'openmw.cfg')
TMP_DIR = os.path.join(tempfile.gettempdir(), "openmmm")
OPENMMM_CONFIG_DIR = user_config_dir(APP_NAME, APP_AUTHOR)
SET_DIR = os.path.join(OPENMMM_CONFIG_DIR, 'sets')
OPENMMM_CONFIG = os.path.join(OPENMMM_CONFIG_DIR, 'openmmm.cfg')
OPENMMM_LOCAL_DIR = user_data_dir(APP_NAME, APP_AUTHOR)
_default_mod_dir = os.path.join(OPENMMM_LOCAL_DIR, 'mods')
CACHE_DIR = user_cache_dir(APP_NAME, APP_AUTHOR)
REPO = 'https://gitlab.com/bmwinger/openmmr.git'
LOCAL_REPO = os.path.join(OPENMMM_LOCAL_DIR, 'repo')
if os.path.exists(OPENMMM_CONFIG):
mmm_config = configparser.ConfigParser()
mmm_config.read(OPENMMM_CONFIG)
if mmm_config['general']:
mod_dir = mmm_config['general'].get('MOD_DIR', _default_mod_dir)
GLOBAL_USE = mmm_config['general'].get('USE', '').split()
ARCH = mmm_config['general'].get('ARCH', None)
else:
mod_dir = _default_mod_dir
GLOBAL_USE = []
else:
mod_dir = _default_mod_dir
MOD_DIR = os.path.normpath(os.path.expanduser(mod_dir))
import os
import sys
import shutil
import patoolib
from colorama import Fore
from distutils.dir_util import copy_tree
from openmmm.globals import MOD_DIR, TMP_DIR
from openmmm.repo.download import download_mod
from openmmm.colour import colour
def remove_mod(mod):
print("Removing " + colour(Fore.GREEN, mod.M))
mod.uninstall()
path = os.path.join(MOD_DIR, mod.C, mod.MN)
shutil.rmtree(path)
print("Finished Removing " + colour(Fore.GREEN, mod.M))
def install_mod(mod):
print("Starting installation of " + colour(Fore.GREEN, mod.M))
archives = download_mod(mod)
if not archives:
print("Unable to download mod. Aborting.")
return False
mod.SOURCE_DIR = os.path.join(TMP_DIR, mod.M, 'src')
mod.INSTALL_DIR = os.path.join(TMP_DIR, mod.M, 'pkg')
os.makedirs(mod.SOURCE_DIR, exist_ok = True)
os.makedirs(mod.INSTALL_DIR, exist_ok = True)
os.chdir(TMP_DIR)
print("Unpacking Mod...")
for archive in archives:
patoolib.extract_archive(archive, outdir=os.path.join(mod.SOURCE_DIR, os.path.basename(archive)))
mod.prepare()
mod.install()
final_install_dir = os.path.join(MOD_DIR, mod.C)
os.makedirs(final_install_dir, exist_ok=True)
final_install = os.path.join(final_install_dir, mod.MN)
print("Installing into {}".format(final_install))
if os.path.exists(final_install): shutil.rmtree(final_install)
shutil.copytree(mod.INSTALL_DIR, final_install)
print("Installed " + colour(Fore.GREEN, mod.M))
return True
import re
def parse_atom(atom):
expr = re.compile(r"(?P<O>([<>]=?|[<>=]))?((?P<C>[a-z\-]+)/)?(?P<M>(?P<MN>[a-z\-]+)(-(?P<MV>[0-9\.]+))?)(-(?P<MR>r[0-9]+))?(::(?P<R>.*))?$")
match = expr.match(atom)
return {
'M' : match.group('M'),
'MN' : match.group('MN'),
'MV' : match.group('MV'),
'MR' : match.group('MR'),
'C' : match.group('C'),
'R' : match.group('R'),
'O' : match.group('O'),
}
def is_valid_atom(atom):
parsed = parse_atom(atom)
if parsed['O'] != '' and parsed['MV'] == '':
return False
# Determines if a fully qualified atom (can only refer to a single package) satisfies a generic atom
def atom_sat(fq_atom, atom):
parfq = parse_atom(fq_atom)
para = parse_atom(atom)
if parfq['MN'] != para['MN']:
# Mods must have the same name
return False
if not para['C'] and para['C'] != parfq['C']:
# If para defines category, it must match
return False
if not para['R'] and para['R'] != parfq['R']:
# If para defines repo, it must match
return False
if not parfq['O']:
# Simple atom, either one version or all versions will satisfy
# Check if version is correct
if not para['MV'] and parfq['MV'] != para['MV']:
return False
# Check if revision is correct
if not para['MR'] and parfq['MR'] != para['MR']:
return False
else:
# TODO: ordering of version numbers and revisions
pass
return True
from enum import Enum
from colorama import Fore
from openmmm.repo.loader import load_mod
from openmmm.repo.atom import atom_sat
from openmmm.repo.sets import get_set
from openmmm.colour import colour
from openmmm.globals import GLOBAL_USE
class Trans(Enum):
DELETE = "d"
NEW = "N"
UPGRADE = "U" # TODO: Detect upgrades
DOWNGRADE = "D"
REINSTALL = "R"
def print_transactions(mods):
for (trans, mod) in mods:
if trans == Trans.DELETE:
trans_colour = Fore.RED
elif trans == Trans.NEW:
trans_colour = Fore.GREEN
elif trans == Trans.REINSTALL:
trans_colour = Fore.YELLOW
elif trans == Trans.DOWNGRADE or trans == Trans.UPGRADE:
trans_colour = Fore.BLUE
print("[{}] {}".format(colour(trans_colour, trans.value), colour(Fore.GREEN, mod.ATOM)))
def satisfies_use(uselist, mod):
if len(uselist) == 0:
return True
# TODO: load opts for mod and check if optlist is satisfied by them
return False
def get_most_recent(modlist):
newest = None
for mod in modlist:
if not newest:
newest = mod
newest_ver = mod.MV.split('.')
else:
ver = mod.MV.split('.')
for index,val in enumerate(ver):
if val > newest_ver[index]:
newest = mod
newest_ver = ver
break
elif val < newest_ver[index]:
break
return newest
def find_dependent(transactions):
# TODO: Implement
return []
def find_dependencies(transactions):
installed = set([mod.ATOM for (trans, mod) in transactions]).union(get_set('installed')).union(set(GLOBAL_USE))
worklist = transactions.copy()
new_mods = transactions.copy()
while len(worklist) > 0:
(trans, mod) = worklist.pop()
for atom in mod.DEPENDS:
# Check if atom is satisfied by a mod in the installed set
satisfied = False
for i in installed:
if atom_sat(i, atom):
satisfied = True
break
if not satisfied:
# find mod that satisfies atom and add to mods
pending_mods = load_mod(atom)
if len(pending_mods) > 0:
new_mod = get_most_recent(pending_mods)
new_mods.append((Trans.NEW, new_mod))
installed.add(new_mod.ATOM)
worklist.append((Trans.NEW, new_mod))
else:
print("ERROR: unable to satisfy atom {}".format(atom))
return []
return new_mods
import hashlib
import urllib.request
import os
from openmmm.globals import CACHE_DIR
BUF_SIZE = 65536
def get_filename(basename):
return os.path.join(CACHE_DIR, 'downloads', basename)
def download(url, destName):
os.makedirs(os.path.join(CACHE_DIR, 'downloads'), exist_ok=True)
urllib.request.urlretrieve(url, get_filename(destName))
def check_hash(filename, checksum):
sha = hashlib.sha512()
with open(filename, mode='rb') as archive:
while True:
data = archive.read(BUF_SIZE)
if not data:
break
sha.update(data)
return sha.hexdigest() == checksum
def get_download(name, checksum):
if os.path.exists(get_filename(name)) and check_hash(get_filename(name), checksum):
return get_filename(name)
return False
def download_mod(mod):
download_list = []
for source in mod.SOURCES:
cached = get_download(source.NAME, source.SHASUM)
if cached:
# Download is in cache. Nothing to do.
print("Using " + cached)
download_list.append(cached)
else:
# Download archive
filename = get_filename(source.NAME)
print("Downloading...")
download(source.URL, source.NAME)
print("Downloaded " + filename)
if not check_hash(filename, source.SHASUM):
print("Error: Source file {} has invalid checksum!".format(source.NAME))
return False
# Finally, add to list
download_list.append(filename)
return download_list
import importlib.machinery
import glob
import configparser
import os
from openmmm.repo.atom import parse_atom, atom_sat
from openmmm.globals import LOCAL_REPO, OPENMMM_CONFIG
repos = [LOCAL_REPO]
# We store a cache of mods so that they are only loaded once when doing dependency resolution.
# Stores key-value pairs of the form (filename, Mod Object)
__mods={}
def load_mod(atom):
parse = parse_atom(atom)
mods = [] # We will return every single mod matching this name. There may be multiple versions in different repos, as well versions with different version or release numbers
for repo in repos:
if parse['C']:
path = os.path.join(repo, parse['C'], parse['MN'])
else:
for dirname in glob.glob(os.path.join(repo, '*')):
path = os.path.join(repo, dirname, parse['MN'])
if os.path.exists(path):
break
if os.path.exists(path):
for file in glob.glob(os.path.join(path, '*.pybuild')):
if __mods.get(file, False):
mod = __mods[file]
else:
loader = importlib.machinery.SourceFileLoader(parse['MN'], file)
mod = loader.load_module().Mod()
__mods[file] = mod
if atom_sat(atom, mod.ATOM):
mods.append(mod)
return mods
import os
from openmmm.globals import SET_DIR
def get_set(mod_set):
set_file = os.path.join(SET_DIR, mod_set)
if os.path.exists(set_file):
with open(set_file, "r") as file:
return set(file.read().splitlines())
else:
return set()
def add_set(mod_set, atom):
set_file = os.path.join(SET_DIR, mod_set)
os.makedirs(SET_DIR, exist_ok=True)
if os.path.exists(set_file):
with open(set_file, "r+") as file:
for line in file:
if atom in line:
break
else:
print(atom, file=file)
else:
with open(set_file, "a+") as file:
print(atom, file=file)
def remove_set(mod_set, atom):
set_file = os.path.join(SET_DIR, mod_set)
with open(set_file,"r+") as f:
new_f = f.readlines()
f.seek(0)
for line in new_f:
if atom not in line:
f.write(line)
f.truncate()
from pybuild.pybuild import Pybuild1, InstallDir, Esp, Source, ModInfo
import os
import sys
from pathlib import Path
from operator import itemgetter
from distutils.dir_util import copy_tree
from openmmm.globals import MOD_DIR
from openmmm.repo.atom import parse_atom
from openmmm.config import read_config, write_config, add_config, remove_config
from openmmm.repo.deps import satisfies_use
import inspect
# This is a terrible hack function, but it does the job. This should only be used within pybuild files
# macropy may provide a more robust way of implementing this
def ModInfo():
frame = inspect.currentframe()
try:
g = frame.f_back.f_globals
filename = g['__file__']
category = Path(filename).resolve().parent.parent.name
repo_path = Path(filename).resolve().parent.parent.parent / 'repo_name'
with open (repo_path, "r") as repo_file:
repo=repo_file.readlines()[0].replace('\n', '')
ATOM = '{}/{}::{}'.format(category, os.path.basename(filename)[:-len('.pybuild')], repo)
(g['M'], g['MN'], g['MV'], g['MR'], C, R) = itemgetter('M', 'MN', 'MV', 'MR', 'C', 'R')(parse_atom(ATOM))
finally:
del frame
# If called within a class in this module, returns a dict with the variables for the file that instantiated the class (i.e. a pybuild).
def _ModInfo():
frame = inspect.currentframe()
try:
g = frame.f_back.f_back.f_globals
filename = g['__file__']
category = Path(filename).resolve().parent.parent.name
repo_path = Path(filename).resolve().parent.parent.parent / 'repo_name'
with open (repo_path, "r") as repo_file:
repo=repo_file.readlines()[0].replace('\n', '')
ATOM = '{}/{}::{}'.format(category, os.path.basename(filename)[:-len('.pybuild')], repo)
(M, MN, MV, MR, C, R) = itemgetter('M', 'MN', 'MV', 'MR', 'C', 'R')(parse_atom(ATOM))
finally:
del frame
return { 'M': M, 'MN': MN, 'MV': MV, 'MR': MR, 'C': C, 'R': R }
class InstallDir():
def __init__(self, path, **kwargs):
self.PATH = path
self.USE = kwargs.get('USE', '').split()
self.DESTPATH = kwargs.get('DESTPATH', '.')
self.ESPS = kwargs.get('ESPS', [])
self.BSAS = kwargs.get('BSAS', [])
self.SOURCE = kwargs.get('SOURCE', None)
self.RESOLVES = kwargs.get('RESOLVES', '').split()
class Esp():
def __init__(self, name, **kwargs):
self.NAME = name
self.USE = kwargs.get('USE', '').split()
self.RESOLVES = kwargs.get('RESOLVES', '').split()
self.CONFLICTS = kwargs.get('CONFLICTS', '').split()
class Source():
def __init__(self, url, shasum, name=None):
self.URL = url
self.SHASUM = shasum
if not name:
self.NAME = _ModInfo()['M']
else:
self.NAME = name
class Pybuild1():
CONFLICTS=""
DEPENDS=""
DATA_OVERRIDES=""
def __init__(self):
filename = sys.modules[self.__class__.__module__].__file__
category = Path(filename).resolve().parent.parent.name
repo_path = Path(filename).resolve().parent.parent.parent / 'repo_name'
with open (repo_path, "r") as repo_file:
repo=repo_file.readlines()[0].replace('\n', '')
self.ATOM = '{}/{}::{}'.format(category, os.path.basename(filename)[:-len('.pybuild')], repo)
(self.M, self.MN, self.MV, self.MR, self.C, self.R) = itemgetter('M', 'MN', 'MV', 'MR', 'C', 'R')(parse_atom(self.ATOM))
# Turn strings of space-separated atoms into lists
self.DEPENDS = self.DEPENDS.split()
self.CONFLICTS = self.CONFLICTS.split()
self.DATA_OVERRIDES = self.DATA_OVERRIDES.split()
def prepare(self):
pass
def update_config(self, config, install_dir):
path = os.path.normpath(os.path.join(MOD_DIR, self.C, self.MN, install_dir.DESTPATH))
# Add data directory
add_config(config, "data", "\"" + path + "\"")
# Process BSAs
for bsa in install_dir.BSAS:
add_config(config, "fallback-archive", bsa)
# Process ESPs
for esp in install_dir.ESPS:
if satisfies_use(esp.USE, self):
add_config(config, "content", esp.NAME)
def clean_config(self, config, install_dir):
path = os.path.normpath(os.path.join(MOD_DIR, self.C, self.MN, install_dir.DESTPATH))
# Add data directory
remove_config(config, "data", "\"" + path + "\"")
# Process BSAs
for bsa in install_dir.BSAS:
remove_config(config, "fallback-archive", bsa)
# Process ESPs
for esp in install_dir.ESPS:
remove_config(config, "content", esp.NAME)
def install(self):
config = read_config()
for install_dir in self.INSTALL_DIRS:
if satisfies_use(install_dir.USE, self):
source = os.path.join(self.SOURCE_DIR, self.M, install_dir.PATH)
dest = os.path.join(self.INSTALL_DIR, install_dir.DESTPATH)
copy_tree(source, dest)
self.update_config(config, install_dir)
write_config(config)
def uninstall(self):
config = read_config()
for install_dir in self.INSTALL_DIRS:
if not install_dir.USE:
self.clean_config(config, install_dir)
write_config(config)
......@@ -6,4 +6,5 @@ setup(name='OpenMMM',
version='1.0',
url='https://gitlab.com/bmwinger/openMMM',
scripts=['openmmm.py'],
packages=['openmmm', 'openmmm.repo', 'pybuild']
)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment