Commit d6dd4b28 authored by Ricardo Avila's avatar Ricardo Avila
Browse files

Merge branch 'master' of https://gitlab.com/ravilabio/ravilabio.gitlab.io into projects

parents 2475295d f0aff07a
......@@ -6,7 +6,7 @@ show_date: false
A Curated List of My Favorite Blogs.
## Cheminformatics:
## Cheminformatics
[Is life worth living?](https://iwatobipen.wordpress.com/)
......
# Script for converting Jupyter notebooks to markdown Jekyll posts.
#
# Usage:
# Make sure your notebook filename is in the following format:
# YYYY MM DD Title of Post.ipynb
#
# jupyter nbconvert --to markdown <notebook>.ipynb --config ../jekyll.py
try:
from urllib.parse import quote # Py 3
except ImportError:
from urllib2 import quote # Py 2
import os
import sys
f = None
for arg in sys.argv:
if arg.endswith('.ipynb'):
f = arg.split('.ipynb')[0]
break
c = get_config()
c.NbConvertApp.export_format = 'markdown'
# Point this to your jekyll template file
c.MarkdownExporter.template_path = ['../templates']
c.MarkdownExporter.template_file = 'jekyll'
# c.Application.verbose_crash=True
# Modify this function to point your images to a custom path
# by default this saves all images to a directory 'images' in the root
# of the blog directory
def path2support(path):
"""Turn a file path into a URL"""
return '{{ BASE_PATH }}/assets/images/' + os.path.basename(path)
#return '../assets/images/' + os.path.basename(path)
c.MarkdownExporter.filters = {'path2support': path2support}
if f:
c.NbConvertApp.output_base = f.lower().replace(' ', '-')
# Point this to your build directory
c.FilesWriter.build_directory = '../_posts'
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reverse Engineering a REST API"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pricing Service"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first REST method that we will implement is the POST method that fetches prices for a given book. From inspecting the page elements, we know that the URL for this service is:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"url = \"https://www.abebooks.com/servlet/DWRestService/pricingservice\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Furthermore, we can see from these requests that there are three main parameter groups, and from them we can infer their purpose. (Variable parameters are show in bold below)\n",
"\n",
"Search prices by ISBN:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataByISBN|\n",
"|isbn|**isbn**|\n",
"|container|pricingService-9781250297662|\n",
"\n",
"Search prices by title and author:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataForAuthorTitleStandardAddToBasket|\n",
"|an|**author**|\n",
"|tn|**title**|\n",
"|container|oe-search-all|\n",
"\n",
"Search prices by title, author, and hardcover/softcover edition:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|action|getPricingDataForAuthorTitleBindingRefinements|\n",
"|isbn|9781250297662|\n",
"|an|**author**|\n",
"|tn|**title**|\n",
"|container|**priced-from-soft** OR **priced-from-hard**|\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending a POST request with Python\n",
"\n",
"We can store a reqest's parameters as a dictionary, and send them to the request's post method."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"payload1 = {'action': 'getPricingDataByISBN',\n",
" 'isbn': 9781250297662,\n",
" 'container': 'pricingService-9781250297662'}\n",
" \n",
"payload2 = {'action': 'getPricingDataForAuthorTitleStandardAddToBasket',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars',\n",
" 'container': 'oe-search-all'}\n",
" \n",
"payload3 = {'action':'getPricingDataForAuthorTitleBindingRefinements',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars contemporary chinese',\n",
" 'container': 'priced-from-soft'}\n",
" \n",
"payload4 = {'action':'getPricingDataForAuthorTitleBindingRefinements',\n",
" 'an': 'liu ken',\n",
" 'tn': 'broken stars contemporary chinese',\n",
" 'container': 'priced-from-hard'}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\n"
]
},
{
"data": {
"text/plain": [
"{'errorTexts': [None],\n",
" 'errorCodes': [None],\n",
" 'success': True,\n",
" 'newExists': True,\n",
" 'usedExists': True,\n",
" 'pricingInfoForBestNew': {'bestListingid': 30410510568,\n",
" 'totalResults': 16,\n",
" 'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 7.26',\n",
" 'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 7.26',\n",
" 'domesticShippingPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 4.50',\n",
" 'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 6.00',\n",
" 'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 6.00',\n",
" 'shippingDestinationNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorCountryNameInSurferLanguage': 'Canada',\n",
" 'vendorId': 71361,\n",
" 'bestPriceInPurchaseCurrencyValueOnly': '7.26',\n",
" 'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '6.0',\n",
" 'listingCurrencySymbol': 'US$',\n",
" 'purchaseCurrencySymbol': 'US$',\n",
" 'nonPaddedPriceInListingCurrencyValueOnly': '7.26',\n",
" 'refinementList': None,\n",
" 'internationalEdition': False,\n",
" 'bookCondition': 'New',\n",
" 'bookDescription': 'Hardcover. Publisher overstock,...',\n",
" 'freeShipping': False},\n",
" 'pricingInfoForBestUsed': {'bestListingid': 30529767259,\n",
" 'totalResults': 8,\n",
" 'bestPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 6.55',\n",
" 'bestPriceInSurferCurrencyWithCurrencySymbol': 'US$ 6.55',\n",
" 'domesticShippingPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingToDestinationPriceInPurchaseCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingToDestinationPriceInSurferCurrencyWithCurrencySymbol': 'US$ 3.99',\n",
" 'shippingDestinationNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorCountryNameInSurferLanguage': 'U.S.A.',\n",
" 'vendorId': 71597499,\n",
" 'bestPriceInPurchaseCurrencyValueOnly': '6.55',\n",
" 'bestShippingToDestinationPriceInPurchaseCurrencyValueOnly': '3.99',\n",
" 'listingCurrencySymbol': 'US$',\n",
" 'purchaseCurrencySymbol': 'US$',\n",
" 'nonPaddedPriceInListingCurrencyValueOnly': '6.55',\n",
" 'refinementList': None,\n",
" 'internationalEdition': False,\n",
" 'bookCondition': 'As New',\n",
" 'bookDescription': 'Like brand new book.',\n",
" 'freeShipping': False},\n",
" 'pricingInfoForBestAllConditions': None,\n",
" 'isbn': '9781250297662',\n",
" 'totalResults': 24,\n",
" 'containerId': 'pricingService-9781250297662',\n",
" 'refinementList': [{'name': 'collectibleJacket',\n",
" 'label': 'Dust Jacket',\n",
" 'count': 2,\n",
" 'url': 'dj=on&isbn=9781250297662&sortby=17'},\n",
" {'name': 'freeShipping',\n",
" 'label': 'Free US Shipping',\n",
" 'count': 9,\n",
" 'url': 'isbn=9781250297662&n=100046078&sortby=17'},\n",
" {'name': 'bindingHard',\n",
" 'label': 'Hardcover',\n",
" 'count': 23,\n",
" 'url': 'bi=h&isbn=9781250297662&sortby=17'},\n",
" {'name': 'collectibleFirstEdition',\n",
" 'label': 'First Edition',\n",
" 'count': 3,\n",
" 'url': 'fe=on&isbn=9781250297662&sortby=17'}],\n",
" 'bibliographicDetail': {'author': '', 'title': ''}}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"resp = requests.post(url, data=payload1)\n",
"print(resp.status_code, resp.reason)\n",
"resp.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Implementing a GET request\n",
"\n",
"The API also has a GET method for obtaining book recomendation, given an ISBN. The url is different, but the parameters are similar:\n",
"\n",
"|Parameter|Value|\n",
"|:--------|:----|\n",
"|pageId|plp|\n",
"|itemIsbn13|**isbn**|"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"url = \"https://www.abebooks.com/servlet/RecommendationsApi\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example: Searching price by ISBN"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\n"
]
},
{
"data": {
"text/plain": [
"{'widgetResponses': [{'slotName': 'detail-1',\n",
" 'title': 'Customers who bought this item also bought',\n",
" 'algoName': 'abeBooksBlendedPurchaseSims',\n",
" 'ref': 'pd_b_p_1',\n",
" 'recommendationItems': [{'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384201-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765384201?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765384201',\n",
" 'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',\n",
" 'author': 'Liu, Ken'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781250306029-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781250306029?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9781250306029',\n",
" 'title': 'The Redemption of Time: A Three-Body Problem Novel...',\n",
" 'author': 'Baoshu'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765389312-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765389312?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765389312',\n",
" 'title': 'Waste Tide',\n",
" 'author': 'Qiufan, Chen'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780765384195-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780765384195?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9780765384195',\n",
" 'title': 'Invisible Planets: Contemporary Chinese Science Fiction...',\n",
" 'author': 'Liu, Ken'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781784978518-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781784978518?cm_sp=rec-_-pd_b_p_1-_-plp&reftag=pd_b_p_1',\n",
" 'subTitle': None,\n",
" 'isbn13': '9781784978518',\n",
" 'title': 'The Wandering Earth',\n",
" 'author': 'Liu, Cixin'}]},\n",
" {'slotName': 'ext-search-detail-1',\n",
" 'title': None,\n",
" 'algoName': 'heroWidgetIsbnSims',\n",
" 'ref': 'pd_hw_i_1',\n",
" 'recommendationItems': [{'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9780804172448-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9780804172448?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',\n",
" 'subTitle': 'Best Selling',\n",
" 'isbn13': '9780804172448',\n",
" 'title': 'Station Eleven',\n",
" 'author': 'Mandel, Emily St. John'},\n",
" {'attributes': [],\n",
" 'thumbNailImgUrl': 'https://pictures.abebooks.com/isbn/9781786073495-us-300.jpg',\n",
" 'itemLink': '/products/isbn/9781786073495?cm_sp=rec-_-pd_hw_i_1-_-plp&reftag=pd_hw_i_1',\n",
" 'subTitle': 'Top Rated',\n",
" 'isbn13': '9781786073495',\n",
" 'title': 'Zuleikha',\n",
" 'author': 'Yakhina, Guzel'}]}]}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"payload = {'pageId': 'plp',\n",
" 'itemIsbn13': 9781250297662}\n",
"\n",
"resp = requests.get(url, params=payload)\n",
"print(resp.status_code, resp.reason)\n",
"resp.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using the new Python Module"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"from abebooks import AbeBooks\n",
"\n",
"ab = AbeBooks()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"results = ab.getPriceByISBN(9780062941503)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"if results['success']:\n",
" best_new = results['pricingInfoForBestNew']\n",
" best_used = results['pricingInfoForBestUsed']"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"US$ 21.49\n"
]
}
],
"source": [
"print(best_new['bestPriceInPurchaseCurrencyWithCurrencySymbol'])"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"US$ 24.42\n"
]
}
],
"source": [
"print(best_used['bestPriceInPurchaseCurrencyWithCurrencySymbol'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
{% extends 'markdown.tpl' %}
{%- block header -%}
---
title: "{{resources['metadata']['name']}}"
tags:
- Python
- notebook
---
{%- endblock header -%}
{% block in_prompt %}
{% endblock in_prompt %}
{% block input %}
{{ '{% highlight python %}' }}
{{ cell.source }}
{{ '{% endhighlight %}' }}
{% endblock input %}
{% block data_svg %}
![svg]({{ output.metadata.filenames['image/svg+xml'] | path2support }})
{% endblock data_svg %}
{% block data_png %}
![png]({{ output.metadata.filenames['image/png'] | path2support }})
{% endblock data_png %}
{% block data_jpg %}
![jpeg]({{ output.metadata.filenames['image/jpeg'] | path2support }})
{% endblock data_jpg %}
{% block markdowncell scoped %}
{{ cell.source | wrap_text(80) }}
{% endblock markdowncell %}
{% block headingcell scoped %}
{{ '#' * cell.level }} {{ cell.source | replace('\n', ' ') }}
{% endblock headingcell %}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment