Feeds migration
We use the following table to maintain indexes of feeds:
```sql
CREATE TABLE minds.entities_by_time (
key text,
column1 text,
value text,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
```
A typical newsfeed will look like this:
| key | column1 | value |
| ------ | ------ | ------ |
| mark | 952852823127826445 | timestamp |
| mark | 932852823127826445 | timestamp |
We use 64 bit numerical GUID's which allow use to easily sort linearly.
Due to the fact the we use `TEXT` and the column1 scala type, the ordering system will break in a matter of months. `1000000000000000001` is treated as less than `992852823127826445` because `Cassandra` is ordering by looking at the first letter(number).
| key | column1 | value |
| ------ | ------ | ------ |
| mark | 952852823127826445 | timestamp |
| mark | 932852823127826445 | timestamp |
| mark | 1000000000000000001 | YIKES |
## Proposed solution
#### Simple cassandra tables
```sql
CREATE TABLE minds.subscribed_feeds (
user_guid varint,
type text,
guid varint,
owner_guid varint,
PRIMARY KEY (user_guid, type, timestamp, guid)
);
CREATE TABLE minds.owner_feeds (
owner_guid varint,
type text,
guid varint,
PRIMARY KEY (owner_guid, type, guid)
);
```
#### Elasticsearch solution
Is something achievable with elasticsearch?
- Benefits
- Users could order their own channel feed based on scores
- Tradeoffs
- Replicated data. Accurate?
Interesting read: http://www.quentinsuire.com/presentations/a-news-feed-with-elasticsearch/index.html#/23
POST: /subscriptions/users/100000000000000063
```json
{
"guids": [
100000000000001111,
100000000000000134,
100000000000000341
]
}
````
query:
```json
{
"query": {
"terms": {
"owner_guid": {
"index": "subscriptions",
"type": "users",
"id": "100000000000000063",
"path": "guids"
}
}
},
"sort": [{ "@timestamp": { "order" : "desc" } }]
}
```
result:
```json
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2499,
"max_score": 8.710042,
"hits": [
{
"_index": "minds_badger",
"_type": "activity",
"_id": "787815555636662272",
"_score": 8.710042,
"_source": {
"interactions": 10,
"guid": "787815555636662272",
"type": "activity",
"time_created": 1513205865,
"access_id": "2",
"owner_guid": "100000000000001111",
"container_guid": "100000000000001111",
"mature": false,
"message": "",
"name": "",
"title": "",
"blurb": "",
"description": "",
"paywall": false,
"@timestamp": 1513205865000,
"taxonomy": "activity",
"public": true,
"tags": []
}
},
...
]
}
}
```
## Migration
Migration is is most probably going to be required in most scenarios.
epic