Commit 4680f62b authored by Ricardo Avila's avatar Ricardo Avila
Browse files

merge posts from authoring

parents e8725c6e aba430fe
Pipeline #140415401 passed with stage
in 1 minute and 9 seconds
......@@ -22,14 +22,14 @@ I have a large collection of electronic books, which I manage using [Calibre](ht
Thus, I have access to a very convenient and ever-growing virtual library of books, which I like to use on the go, and for exploratory research. Nevertheless, whenever I find a particularly good book, the thing that I want most, is to own a physical copy.
![](/assets/images/abebooks/calibre.png)
![](/assets/images/abebooks/calibre.png "My calibre e-book library")
Enter here AbeBooks.com. Next to Amazon, and occasionally Ebay, it is my go-to site for buying cheap used textbooks.
Given that I have stored the ISBN data for most of my electronic books, I would like to be able to automatically fetch pricing information for any book in my virtual library, perhaps even keeping track of changes in price over time.
However, until now, the main problem stopping me from writing a script to do this was that AbeBooks does not have a publicly available API... or at the very least, none that is explicitly documented.
![](/assets/images/abebooks/abebooks.png)
![](/assets/images/abebooks/abebooks.png "AbeBooks search results page")
## REST APIs
......@@ -53,17 +53,17 @@ Several common REST methods exist: GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT,
I found that inspecting the network packets for an AbeBooks search results page is simple, and yields promising results. If we open Firefox's developer tools, under the Network tab, we can see a list of all the packets that are loaded. In particular we are interested in those that have a JSON response, highlighted in red below:
![](/assets/images/abebooks/packets.png)
![](/assets/images/abebooks/packets.png "Network packet inspection in Firefox ")
We can see that there are four POST requests, to a service called "pricingservice", and one GET request to a "RecomendationsApi".
If we look more closely at one of the POST requests, we can see which parameters it takes in:
![](/assets/images/abebooks/params.png)
![](/assets/images/abebooks/params.png "POST request parameters")
ISBN! Just what we needed! Furthermore, looking at the response tab, we can see that this request returns the prices for new and used books, among other things:
![](/assets/images/abebooks/response.png)
![](/assets/images/abebooks/response.png "Example POST response")
## Wrapping the API in Python
......
---
title: "Switching Your GTK Theme Based on Time of Day"
excerpt: "Using systemd timers to automatically switch between light/dark GTK themes in a GNOME desktop. This may not be a data science post, but knowing systemd timers can certainly be handy for many applications where we want to run scheduled jobs on a Linux workstation."
tags:
- Linux
- GNOME
---
A new trend in software UI/UX is the introduction of dark and light options for user interfaces. Windows and MacOS have both added support for dark window themes in their latest versions, and many iOS and Android apps offer this feature through their application settings. Even many websites are enabling support for this feature using the [prefers-color-scheme](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme) CSS tag, which alerts a website about the user's theme preference at OS level. Linux naturally, has had the ability to customize user interface themes for decades, but what is new, is the focus on continuously adapting user interfaces.
For example, If you are using GNOME desktop, you may be aware that the default GNOME wallpaper transitions in color and tone throughout the day, providing brighter colors in the morning, and gradually transitioning to darker tones towards the evening. Furthermore, most operating systems now support "Night light" - a feature that allows a device's screen to gradually shift from blue tones to red, in order to reduce eye strain at night.
A neat feature missing from most desktop environments is the ability to automatically switch GTK themes, in order to provide a darker environment at night, and improve brightness and contrast during the day. The good news is that in Linux it is not difficult to implement this functionality with systemd timers.
![](/assets/images/gtk/gtk.png "Dark and light GTK themes in Nautilus")
## Cron Jobs vs Systemd Timers
So, what are systemd timers? Old time UNIX users will be familiar with cron jobs. Cron is a utility that runs commands, typically from a file stored under: `/etc/cronbtab`
### An example crontab job
```
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
```
### Relevant xkcd
![](https://imgs.xkcd.com/comics/cron_mail.png "Cron Mail")
> "Take THAT, piece of 1980s-era infrastructure I've inexplicably maintained on my systems for 15 years despite never really learning how it works."
Systemd on the other hand, is a manager for system processes and services. It has replaced much of the functionality that was previously handled by the UNIX `init` daemon, and has several nice features that the cron utility lacks:
The big benefit for our purpose, is that with cron, if the computer is powered off, a scheduled cron job does not run. Systemd, on the other hand, can run the tasks that it missed the next time that it powers on.
Other advantages of systemd timers:
- CPU and memory limits
- Randomized scheduling
- Jobs can be easily started independently of their timers
- Jobs are logged in systemd journal, which makes easier debugging
## Creating a Systemd Service
Systemd services have the extension `.service`. All user-created systemd scripts will be stored in `~/.config/systemd/user/`. Here are the contents of "dark-theme.sevice", out systemd service for switching to the Adwaita-dark GTK theme:
```
[Unit]
Description=Change the GTK theme to dark mode.
After=graphical.target
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'gsettings set org.gnome.desktop.interface gtk-theme Adwaita-dark'
[Install]
WantedBy=default.target
```
We will need to create a separate "light-theme.service" file to switch to a light GTK theme.
## Creating a Systemd Timer
Now, to create a timer, we make another file in the same directory with the same rootname plus the extension `.timer`. In this case, we named our file "dark-theme.timer". Here are the contents of this file:
```
[Unit]
Description=Change the GTK theme daily at a given time.
[Timer]
OnCalendar=*-*-* 16:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
The OnCalendar setting specifies that this particular timer should run every day at 16:00 hrs. You can also create timers that run every other day, or on specific days of the week. For more on this, I have found that the best resource for documentation is the ArchLinux Wiki: [https://wiki.archlinux.org/index.php/Systemd/Timers](https://wiki.archlinux.org/index.php/Systemd/Timers).
We will additionally need to create a "light-theme.timer" to change theme in the morning.
## Running Services as a User
When we are done creating the four configuration files (two for each of dark and light), we need to enable the services. It is recommended that we enable them at the user level (since requiring sudo access would be a hazzle and potential security risk).
Enabling the services:
```
systemctl --user enable dark-theme.service
systemctl --user enable light-theme.service
systemctl --user enable dark-theme.timer
systemctl --user enable light-theme.timer
```
Now the operating system will automatically change the GTK theme at the specified times.
Furthermore, if we wish to manually change the theme, or test that the service works, we may do so using:
```
systemctl --user start light-theme.service
```
or:
```
systemctl --user start dark-theme.service
```
## What can we use this for?
A couple other applications come to mind:
- Daily data backups
- Daily data ingestion
Let me know if you come up with other interesting applications!
---
layout: article
title: About
excerpt: Ricardo Avila is a bioinformatics nerd, artist, and lover of open source software.
---
Hi! 👋 My name is Ricardo Avila. I enjoy researching computational problems in biology, ranging from protein structures, to genomic sequences and chemical structures. I find that the most difficult and rewarding problems often require connecting seemingly unrelated fields. Hence, I enjoy being in the overlap of different disciplines, figuring out how things come together.
......
---
layout: article
title: Blogs
show_date: false
---
A Curated List of My Favorite Blogs.
## Cheminformatics:
[Is life worth living?](https://iwatobipen.wordpress.com/)
[Practical Cheminformatics](http://practicalcheminformatics.blogspot.com/)
## Linux
[GNOME Shell & Mutter](https://blogs.gnome.org/shell-dev/)
# Script for converting Jupyter notebooks to markdown Jekyll posts.
#
# Usage:
# Make sure your notebook filename is in the following format:
# YYYY MM DD Title of Post.ipynb
#
# jupyter nbconvert --to markdown <notebook>.ipynb --config ../jekyll.py
try:
from urllib.parse import quote # Py 3
except ImportError:
from urllib2 import quote # Py 2
import os
import sys
f = None
for arg in sys.argv:
if arg.endswith('.ipynb'):
f = arg.split('.ipynb')[0]
break
c = get_config()
c.NbConvertApp.export_format = 'markdown'
# Point this to your jekyll template file
c.MarkdownExporter.template_path = ['../templates']
c.MarkdownExporter.template_file = 'jekyll'
# c.Application.verbose_crash=True
# Modify this function to point your images to a custom path
# by default this saves all images to a directory 'images' in the root
# of the blog directory
def path2support(path):
"""Turn a file path into a URL"""
return '{{ BASE_PATH }}/assets/images/' + os.path.basename(path)
#return '../assets/images/' + os.path.basename(path)
c.MarkdownExporter.filters = {'path2support': path2support}
if f:
c.NbConvertApp.output_base = f.lower().replace(' ', '-')
# Point this to your build directory
c.FilesWriter.build_directory = '../_posts'
......@@ -9,4 +9,6 @@ show_date: false
Notes on various topics.
## [Blogs](/blogs)
A curated list of my favorite blogs.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reverse Engineering a REST API"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pricing Service"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first REST method that we will implement is the POST method that fetches prices for a given book. From inspecting the page elements, we know that the URL for this service is:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"url = \"https://www.abebooks.com/servlet/DWRestService/pricingservice\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Furthermore, we can see from these requests that there are three main parameter groups, and from them we can infer their purpose. (Variable parameters are show in bold below)\n",
"\n",
"Search prices by ISBN:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataByISBN|\n",
"|isbn|**isbn**|\n",
"|container|pricingService-9781250297662|\n",
"\n",
"Search prices by title and author:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataForAuthorTitleStandardAddToBasket|\n",
"|an|**author**|\n",
"|tn|**title**|\n",
"|container|oe-search-all|\n",
"\n",
"Search prices by title, author, and hardcover/softcover edition:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataForAuthorTitleBindingRefinements|\n",
"|isbn|9781250297662|\n",
"|an|**author**|\n",
"|tn|**title**|\n",
"|container|**priced-from-soft** OR **priced-from-hard**|\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending a POST request with Python\n",
"\n",
"We can store a reqest's parameters as a dictionary, and send them to the request's post method."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"payload1 = {'action': 'getPricingDataByISBN',\n",
" 'isbn': 9781250297662,\n",
" 'container': 'pricingService-9781250297662'}\n",
" \n",
"payload2 = {'action': 'getPricingDataForAuthorTitleStandardAddToBasket',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars',\n",
" 'container': 'oe-search-all'}\n",
" \n",
"payload3 = {'action':'getPricingDataForAuthorTitleBindingRefinements',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars contemporary chinese',\n",
" 'container': 'priced-from-soft'}\n",
" \n",
"payload4 = {'action':'getPricingDataForAuthorTitleBindingRefinements',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars contemporary chinese',\n",
" 'container': 'priced-from-hard'}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\n"
]
},
{
"data": {
"text/plain": [
"{'errorTexts': [None],\n",
" 'errorCodes': [None],\n",
" 'success': True,\n",
" 'newExists': True,\n",
" 'usedExists': True,\n",
" 'pricingInfoForBestNew': {'bestListingid': 30410510568,\n",
" 'totalResults': 16,\n",
" 'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 7.26',\n",
" 'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 7.26',\n",
" 'domesticShippingPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 4.50',\n",
" 'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 6.00',\n",
" 'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 6.00',\n",
" 'shippingDestinationNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorCountryNameInSurferLanguage': 'Canada',\n",
" 'vendorId': 71361,\n",
" 'bestPriceInPurchaseCurrencyValueOnly': '7.26',\n",
" 'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '6.0',\n",
" 'listingCurrencySymbol': 'US$',\n",
" 'purchaseCurrencySymbol': 'US$',\n",
" 'nonPaddedPriceInListingCurrencyValueOnly': '7.26',\n",
" 'refinementList': None,\n",
" 'internationalEdition': False,\n",
" 'bookCondition': 'New',\n",
" 'bookDescription': 'Hardcover. Publisher overstock,...',\n",
" 'freeShipping': False},\n",
" 'pricingInfoForBestUsed': {'bestListingid': 30529767259,\n",
" 'totalResults': 8,\n",
" 'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 6.55',\n",
" 'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 6.55',\n",
" 'domesticShippingPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingDestinationNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorCountryNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorId': 71597499,\n",
" 'bestPriceInPurchaseCurrencyValueOnly': '6.55',\n",
" 'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '3.99',\n",
" 'listingCurrencySymbol': 'US$',\n",
" 'purchaseCurrencySymbol': 'US$',\n",
" 'nonPaddedPriceInListingCurrencyValueOnly': '6.55',\n",
" 'refinementList': None,\n",
" 'internationalEdition': False,\n",
" 'bookCondition': 'As New',\n",
" 'bookDescription': 'Like brand new book.',\n",
" 'freeShipping': False},\n",
" 'pricingInfoForBestAllConditions': None,\n",
" 'isbn': '9781250297662',\n",
" 'totalResults': 24,\n",
" 'containerId': 'pricingService-9781250297662',\n",
" 'refinementList': [{'name': 'collectibleJacket',\n",
" 'label': 'Dust Jacket',\n",
" 'count': 2,\n",
" 'url': 'dj=on&isbn=9781250297662&sortby=17'},\n",
" {'name': 'freeShipping',\n",
" 'label': 'Free US Shipping',\n",
" 'count': 9,\n",
" 'url': 'isbn=9781250297662&n=100046078&sortby=17'},\n",
" {'name': 'bindingHard',\n",
" 'label': 'Hardcover',\n",
" 'count': 23,\n",
" 'url': 'bi=h&isbn=9781250297662&sortby=17'},\n",
" {'name': 'collectibleFirstEdition',\n",
" 'label': 'First Edition',\n",
" 'count': 3,\n",
" 'url': 'fe=on&isbn=9781250297662&sortby=17'}],\n",
" 'bibliographicDetail': {'author': '', 'title': ''}}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"resp = requests.post(url, data=payload1)\n",
"print(resp.status_code, resp.reason)\n",
"resp.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Implementing a GET request\n",
"\n",
"The API also has a GET method for obtaining book recomendation, given an ISBN. The url is different, but the parameters are similar:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|pageId|plp|\n",
"|itemIsbn13|**isbn**|"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"url = \"https://www.abebooks.com/servlet/RecommendationsApi\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example: Searching price by ISBN"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\n"
]
},
{
"data": {
"text/plain": [
"{'widgetResponses': [{'slotName': 'detail-1',\n",
" 'title': 'Customers who bought this item also bought',\n",
" 'algoName': 'abeBooksBlendedPurchaseSims',\n",
" 'ref': 'pd_b_p_1',\n",
" 'recommendationItems': [{'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384201-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765384201?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765384201',\n",
" 'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',\n",
" 'author': 'Liu, Ken'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781250306029-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781250306029?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9781250306029',\n",
" 'title': 'The Redemption of Time: A Three-Body Problem Novel...',\n",
" 'author': 'Baoshu'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765389312-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765389312?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765389312',\n",
" 'title': 'Waste Tide',\n",
" 'author': 'Qiufan, Chen'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384195-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765384195?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765384195',\n",
" 'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',\n",
" 'author': 'Liu, Ken'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781784978518-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781784978518?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9781784978518',\n",
" 'title': 'The Wandering Earth',\n",
" 'author': 'Liu, Cixin'}]},\n",
" {'slotName': 'ext-search-detail-1',\n",
" 'title': None,\n",
" 'algoName': 'heroWidgetIsbnSims',\n",
" 'ref': 'pd_hw_i_1',\n",
" 'recommendationItems': [{'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780804172448-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780804172448?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',\n",
" 'subTitle': 'Best Selling',\n",
" 'isbn13': '9780804172448',\n",
" 'title': 'Station Eleven',\n",
" 'author': 'Mandel, Emily St. John'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781786073495-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781786073495?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',\n",
" 'subTitle': 'Top Rated',\n",
" 'isbn13': '9781786073495',\n",
" 'title': 'Zuleikha',\n",
" 'author': 'Yakhina, Guzel'}]}]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"payload = {'pageId': 'plp',\n",
" 'itemIsbn13': 9781250297662}\n",
"\n",
"resp = requests.get(url, params=payload)\n",
"print(resp.status_code, resp.reason)\n",
"resp.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using the new Python Module"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"from abebooks import AbeBooks\n",
"\n",
"ab = AbeBooks()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"results = ab.getPriceByISBN(9780062941503)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"if results['success']:\n",
" best_new = results['pricingInfoForBestNew']\n",
" best_used = results['pricingInfoForBestUsed']"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},