Skip to content

Feat/metascraper stage 2 front#5815

Ben requested to merge feat/metascraper-stage-2-5815 into master

Ticket(s) / Related Merge Requests

Summary of Changes

Use Metascraper for parsing Twitter and Youtube importers.

Testing Considerations

This is all going to be quite tricky to test on sandboxes, however:

Youtube
source /vault/secrets/env && php cli.php YouTubeImporter import --video_id='4drttTSsWjM' --channel_id='UCGksFXDnH3swdUfRV6Dhs3w'`

Check /nemofin to see the video. To repeat, you will need to delete Redis to it reboots due to rate limiting.

Alternatively do it yourself, connect a Youtube channel and then swap the params in the above command to post it to your channel.

Note the thumbnail isn't appearing - I believe this to be a CDN issue because it also doesn't work for newsfeed preview which isn't touched by these changes when on sandbox BUT the thumbnail works fine on newsfeed post preview on production and locally.

Twitter

To get TwitterSync to work I've had to do quite a bit of hacking around bits of code in the pods engine branch to make the request work, manually inserting the user GUID into the URL etc, and added various var_dumps.

For some reason my Twitter account is linked to https://feat-metascraper-stage-2-5815.minds.io/tanyatest4/

I've set it so the timestamp is ALWAYS the start of this month, so you can run the below command repeatedly and it should sync the same tweets to that channel.

source /vault/secrets/env &&  php cli.php TwitterSync sync

From above:

Note the thumbnail isn't appearing - I believe this to be a CDN issue because it also doesn't work for newsfeed preview which isn't touched by these changes when on sandbox BUT the thumbnail works fine on newsfeed post preview on production and locally.

Similar here, it works on prods newsfeed and locally but not on sandboxes. HOWEVER there are KNOWN issues with Metascraper and Twitter so it would not surprise me if some do not work. See: https://github.com/microlinkhq/metascraper/issues/260

Deployment Considerations

Needs deploying to runners. Gated by feature flag front-5815-metascraper-stage-2 which will need flipping.

Be aware stage 1 (direct feed posting / preview endpoint) is controlled by front-5392-metascraper-previews

Regression Scope

Youtube Importer and Twitter Sync

Platform Affected (web, mobile, etc)

Runners

Developer Testing Completed

Manual, unit.

Screenshots / Screen Recording

N/A

Does this impact

  • Localization
  • Dark/light mode
  • Guest mode

Definition of Done Checklist

  • The Acceptance Criteria has been met
  • Code is tested: Testing includes unit/spec, E2E/automated and manual testing
  • Merge requests description has been filled out
Edited by Ben

Merge request reports