Trending hashtags

I'm doing this mainly as an exercise in dealing with data, getting this project bootstrapped with a few things, and as practice for dealing with feeds. After much deliberation about how to store the data, how to process it, and whether to use SQLite or LMDB, this is where I'm at (it's SQLite):

-- Tag usages table
CREATE TABLE tag_usages (
  tag TEXT NOT NULL,
  pubkey8 TEXT NOT NULL,
  inserted_at DATETIME NOT NULL,
);
CREATE INDEX idx_time_tag ON tag_usages(inserted_at, tag);

-- Query top 10 trending tags within a time range
SELECT tag, COUNT(DISTINCT pubkey8)
  FROM tag_usages
  WHERE inserted_at >= $1 AND inserted_at < $2
  GROUP BY tag
  ORDER BY COUNT(DISTINCT pubkey8)
  DESC LIMIT 10;

-- Inserting a row
INSERT INTO tag_usages (tag, pubkey8, inserted_at)
  VALUES ($1, $2, $3);

-- Cleaning up old data
DELETE FROM tag_usages WHERE inserted_at < $1;

pubkey8 is the pubkey truncated to the first 8 characters, to save space.

I investigated this in LMDB too, but decided Sqlite would be cheaper for a number of reasons. Also, the query would warm a cache in the background, and the API would always serve a cache.

Assignee Loading
Time tracking Loading