Commit c999d134 authored by Ricardo Avila's avatar Ricardo Avila
Browse files

Update post

parent 30047f55
Pipeline #137388032 passed with stage
in 1 minute and 7 seconds
---
title: "Web-scraping Abebooks.com (Reverse Engineering a REST API)"
title: "Web-scraping AbeBooks.com (Reverse Engineering a REST API)"
tags:
- Python
- REST API
- books
---
## Motivation
I have a large collection of electronic books, which I manage using [Calibre](https://calibre-ebook.com). Using Calibre's "Extract ISBN" plugin, I am able to parse the ISBN identifier from most of my files, which then makes fetching the rest of the metadata very easy. (Below is an example of my library's metadata.)
Thus, I have access to a very convenient and ever-growing virtual library of books, which I like to use on the go, and for exploratory research. Nevertheless, whenever I find a particularly good book, the thing that I want most, is to own a physical copy.
![](/assets/images/abebooks/calibre.png)
Enter here AbeBooks.com. Next to Amazon, and occasionally Ebay, it is my go-to site for buying cheap used textbooks.
Given that I have stored the ISBN data for most of my electronic books, I would like to be able to automatically fetch pricing information for any book in my virtual library, perhaps even keeping track of changes in price over time.
However, until now, the main problem stopping me from writing a script to do this, was that AbeBooks does not have a publicly available API... or at the very least, none that is explicitly documented.
![](/assets/images/abebooks/abebooks.png)
## Rest APIs
Nevertheless, with some persistence, I learned that inspecting the page elements can yield fruitful results. In Firefox's developer tools, under the Network tab,
![](/assets/images/abebooks/packets.png)
![](/assets/images/abebooks/params.png)
![](/assets/images/abebooks/response.png)
{% highlight python %}
import requests
import json
......@@ -53,7 +82,7 @@ Search prices by title, author, and hardcover/softcover edition:
|container|**priced-from-soft** OR **priced-from-hard**|
## Sending a POST request with Python
## A POST request with Python
We can store a reqest's parameters as a dictionary, and send them to the
request's post method.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment