Commit ebbc4e03 authored by Ricardo Avila's avatar Ricardo Avila
Browse files

Update post

parent 23d6e243
Pipeline #137409527 passed with stage
in 1 minute and 8 seconds
......@@ -6,6 +6,15 @@ tags:
- books
---
- [Motivation](#motivation)
- [REST APIs](#rest-apis)
- [Exploring Network Packets](#exploring-network-packets)
- [Wrapping the API in Python](#wrapping-the-api-in-python)
- [POST requests](#post-requests)
- [A GET request](#a-get-request)
- [An Object-Oriented Module](#an-object-oriented-module)
- [Using the new Module](#using-the-new-module)
## Motivation
I have a large collection of electronic books, which I manage using [Calibre](https://calibre-ebook.com). Using Calibre's "Extract ISBN" plugin, I am able to parse the ISBN identifier from most of my files, which then makes fetching the rest of the metadata very easy. (Below is an example of my library's metadata.)
......@@ -23,23 +32,35 @@ However, until now, the main problem stopping me from writing a script to do thi
## REST APIs
Everything we do in the internet involves packets.
REST, or Representational State Transfer, is an architecture that
REST, or Representational State Transfer, is a system used by the HTTP protocol to provide interoperability between servers. It is based on a request/response system, where a request is a "payload", normally formatted as HTML, XML, or JSON., and the response can be a link to a resource, a data payload in any of the aforementioned formats, or a confirmation that some data was modified in the server.
Several common REST methods exist: GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE. Among these, the two most common are GET and POST:
**GET**
- Used to request data from a server.
- Parameter data is stored in the URL of the query as string parameters
- Number of parameters is limited to the length that can fit in the URL
- Not secure for sensitive information. (Passwords can be easily seen)
**POST**
- Used to submit data to a server, and can modify server contents.
- Parameters are passed in the message body, rather than the URL.
- It has no restrictions on the number of parameters.
- Is more secure for sending sensitive information.
## Exploring Network Packets
In this case, I found that inspecting the network packets for an AbeBooks search results page is simple, and yields promising results. If we open Firefox's developer tools, under the Network tab, we can see a list of all the packets that are loaded. In particular we are interested in those that have a JSON response, highlighted in red below:
I found that inspecting the network packets for an AbeBooks search results page is simple, and yields promising results. If we open Firefox's developer tools, under the Network tab, we can see a list of all the packets that are loaded. In particular we are interested in those that have a JSON response, highlighted in red below:
![](/assets/images/abebooks/packets.png)
We can see that there are four POST requests, to a service called "pricingservice", and one GET request to a "RecomendationsApi". All of these requests are of the type xhr.
We can see that there are four POST requests, to a service called "pricingservice", and one GET request to a "RecomendationsApi".
If we look more closely at one of the POST pricingservice requests, we can see which parameters it takes in:
If we look more closely at one of the POST requests, we can see which parameters it takes in:
![](/assets/images/abebooks/params.png)
ISBN! Just what we needed! Furthermore, looking at the response tab, we can see that it returns the prices for new and used books, among other things:
ISBN! Just what we needed! Furthermore, looking at the response tab, we can see that this request returns the prices for new and used books, among other things:
![](/assets/images/abebooks/response.png)
......@@ -65,52 +86,54 @@ There seem to be three main parameter groups, and we can infer their purpose. (P
Searching prices by ISBN:
|Parameter|Value|
|:--------|:----|
|action|getPricingDataByISBN|
|isbn|**isbn**|
|container|pricingService-**isbn**|
| Parameter | Value |
| :-------- | :---------------------- |
| action | getPricingDataByISBN |
| isbn | **isbn** |
| container | pricingService-**isbn** |
Searching prices by title and author:
|Parameter|Value|
|:--------|:----|
|action|getPricingDataForAuthorTitleStandardAddToBasket|
|an|**author**|
|tn|**title**|
|container|oe-search-all|
| Parameter | Value |
| :-------- | :---------------------------------------------- |
| action | getPricingDataForAuthorTitleStandardAddToBasket |
| an | **author** |
| tn | **title** |
| container | oe-search-all |
Searching prices by title, author, and hardcover/softcover binding:
|Parameter|Value|
|:--------|:----|
|action|getPricingDataForAuthorTitleBindingRefinements|
|isbn|9781250297662|
|an|**author**|
|tn|**title**|
|container|**priced-from-soft** OR **priced-from-hard**|
| Parameter | Value |
| :-------- | :--------------------------------------------- |
| action | getPricingDataForAuthorTitleBindingRefinements |
| isbn | 9781250297662 |
| an | **author** |
| tn | **title** |
| container | **priced-from-soft** OR **priced-from-hard** |
The parameters are stored as a dictionary, and sent to the
The parameters can be stored as a dictionary, and sent to the
request's post method. For example:
{% highlight python %}
# Search prices by ISBN
#- Search prices by ISBN
payload1 = {'action': 'getPricingDataByISBN',
'isbn': 9781250297662,
'container': 'pricingService-9781250297662'}
# Search prices by author and title
#- Search prices by author and title
payload2 = {'action': 'getPricingDataForAuthorTitleStandardAddToBasket',
'an': 'liu ken',
'tn': 'broken stars',
'container': 'oe-search-all'}
# Sending a request
#- Sending a request
resp = requests.post(url, data=payload1)
print(resp.status_code, resp.reason)
resp.json()
{% endhighlight %}
The response is:
200 OK
......@@ -192,14 +215,14 @@ The url and parameter names are different, but the way we send the request is ve
url = "https://www.abebooks.com/servlet/RecommendationsApi"
{% endhighlight %}
|Parameter|Value|
|:--------|:----|
|pageId|plp|
|itemIsbn13|**isbn**|
| Parameter | Value |
| :--------- | :------- |
| pageId | plp |
| itemIsbn13 | **isbn** |
{% highlight python %}
# Get book recommendations by ISBN
#- Get book recommendations by ISBN
payload = {'pageId': 'plp',
'itemIsbn13': 9781250297662}
......@@ -208,6 +231,8 @@ print(resp.status_code, resp.reason)
resp.json()
{% endhighlight %}
Response:
200 OK
......@@ -271,7 +296,7 @@ resp.json()
## An Object-Oriented Module
I created a small Python module for ease of use. The full code is below:
I created a small Python module [abebooks.py]() to encapsulate the requests. The full code is below:
{% highlight python %}
import requests
......@@ -349,7 +374,7 @@ class AbeBooks:
{% endhighlight %}
## Using the new Module
## Using the AbeBooks Module
{% highlight python %}
......@@ -363,7 +388,7 @@ if results['success']:
{% endhighlight %}
{% highlight python %}
# Best New Price
#- Best New Price
print(best_new['bestPriceInPurchaseCurrencyWithCurrencySymbol'])
{% endhighlight %}
......@@ -372,7 +397,7 @@ print(best_new['bestPriceInPurchaseCurrencyWithCurrencySymbol'])
{% highlight python %}
# Best Used Price
#- Best Used Price
print(best_used['bestPriceInPurchaseCurrencyWithCurrencySymbol'])
{% endhighlight %}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment