Pleroma bot is failing to pull tweets from a non-protected account

[AyotoCorp]

I've been using the bot to mirror my Twitter (thanks for the work put into it btw, greatly appreciated) and for some reason between roughly 30 to 12 hours ago the bot started requiring OAuth keys, despite not pulling from a protected account.

ℹ 2023-01-12 01:38:47,691 - pleroma_bot - INFO - config path: /root/config.yml 
ℹ 2023-01-12 01:38:47,691 - pleroma_bot - INFO - tweets temp folder: /root/tweets 
ℹ 2023-01-12 01:38:47,697 - pleroma_bot - INFO - ====================================== 
ℹ 2023-01-12 01:38:47,697 - pleroma_bot - INFO - Processing user:       Ayoto 
Gathering tweets... 0
✖ 2023-01-12 01:38:49,106 - pleroma_bot - ERROR - Unable to retrieve tweets. Is the account protected? If so, you need to provide the following OAuth 1.0a fields in the user config:
 - consumer_key 
 - consumer_secret 
 - access_token_key 
 - access_token_secret (cli.py:642) 
ℹ 2023-01-12 01:38:49,106 - pleroma_bot - INFO - Current pinned:        1531787041461215233 
ℹ 2023-01-12 01:38:49,107 - pleroma_bot - INFO - Previous pinned:       1531787041461215233 
ℹ 2023-01-12 01:38:50,380 - pleroma_bot - INFO - Updating profile:       <Response [200]> 

And the config file

pleroma_base_url: https://pfeed.ayotocorp.com/
max_tweets: 40
users:
- twitter_username: AyotoCorp
  pleroma_username: Ayoto
  pleroma_token: XXXXXXXXXXXXXXXXXXXXXX
  signature: true
  bio_text: "\U0001F916 BEEP BOOP \U0001F916 \n I'm a bot that mirrors the {{ twitter_username }} Twitter account until it dies a true death.\n \n"
  fields:
      - name: "\U0001F426 Birdsite"
        value: "{{ twitter_url }}"
      - name: "Status"
        value: "I am completely operational, and all my circuits are functioning perfectly."
      - name: "All The Socials"
        value: "{{ website }}"

Apologies if this is something stupid I'm missing, but I have no idea what would have changed between when I got it running several days ago and now

[AyotoCorp]

So, uh, weird update but it works again. I have no idea what caused that behavior though, and finding out the cause and solution would probably be really handy if/when it happens again.

[robertoszek]

Hi!
Thank you for the kind words, I really appreciate you found the bot useful!

I can see reading your config that you're using Guest Tokens.
Bad news, it looks like Twitter has been changing and restricting the search endpoint we use to scrape tweets as a guest since yesterday:
https://twitter.com/i/api/2/search/adaptive.json

The change may be related to the removal of the "Latest" tab on the search for non-logged in users:

They also seem to have blacklisted the TLS fingerprint that the Python package requests uses and returns HTTP 404 if you try to use the endpoint in any way.

I'll investigate if we can get around this by setting custom ciphers or some other way.

I have an in-progress draft version with custom ciphers you can try to install to confirm if that's the issue you're facing:
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc3

But even after doing that, the pagination also seems to be broken (so it only retrieves 47 tweets at most) and I need to probably fix the pinned tweet endpoint too.

Unfortunately, the only fully-working alternative at the moment is using Twitter Tokens or an RSS feed with the bot, which are unaffected and are still working as expected after Twitter's changes to the search.

I'll keep trying to see if we can circumvent the restrictions somehow.

[AyotoCorp]

I'll keep an eye on it, the next time it goes down I'll see if that change will fix things. "Unfortunately" right now it seems to be functioning as intended, and I'm not sure why. The only thing I'm noticing is that the tweets that were made during the time it wasn't working aren't being picked up by the bot when I run it, even when using --forcedate 2023-01-01 to make sure it's not just grabbing things from afterwards.

[AyotoCorp]

Alright, the issue is back. Running that install didn't fix it, so I'll just have to wait and see if it'll go away on its own or if this is the end of the auto-tweet pulls