[TwitterBridge] Migration to API V1.1 #2433

arnd-s · 2022-01-21T12:28:33Z

[TwitterBridge] Migration to API V1.1

Migrate all API calls to V1.1 This is an intermediate PR, so the bridge will be working again.

Final goal should be a move to V2.0

…t step

arnd-s · 2022-01-21T12:29:44Z

Currently only search by username without filters is migrated.

Also the PR currently contains garbage and debug elements.

…me for text attribute of tweet

csisoap · 2022-01-22T14:16:21Z

I hope V2 will have a way to filter out Pinned Tweet, Promo Tweet and any Reply to another user. I'm only want self-Reply from that user and old PR has done a great job from doing this.

arnd-s · 2022-01-22T14:22:05Z

I hope V2 will have a way to filter out Pinned Tweet, Promo Tweet and any Reply to another user. I'm only want self-Reply from that user and old PR has done a great job from doing this.

I will later add this filter again. It's currently commented out until i matched everything back to tweet structure from V1.1
V2 will be a separate PR later when everything is ported to at least 1.1. Primary goal is, to have it running ;)

Looks like for V2 authentication has to be changed again, based on some quick tests i made

toelke · 2022-01-23T19:07:01Z

FWIW, I am running this branch for a few hours and it is way more "stable" than main. I now got error 401 for one feed. I will keep monitoring.

Added API call to fetch user object.

…added generic makeApiCall funtion

Also commented out some not anymore used functions

arnd-s · 2022-01-24T12:34:37Z

@anyone using this branch: It would be nice go get some feedback about whats broken or changed on the output side. For some cases I'm missing data to compare it with past behavior and I'm also missing old API documentation. So I'm currently trying my best to match everything from reading the old code against official API documentation.
Also general change requests or additions are welcomed and i will try my best to integrate them.

linoth · 2022-01-24T14:45:45Z

Disclaimer: Not trying to advocate for one behavior over another, just trying to list differences that I've noticed.

Currently, retweets link to what I guess is the action of retweeting?, rather than the original tweet. For example, the feed for @\Twitter now links to: https://twitter.com/TwitterBlue/status/1484314336772321282

which Twitter then resolves to: https://twitter.com/Twitter/status/1484314336772321282 ("Twitter Retweeted foo")

Previous behavior would have been to just link straight to: https://twitter.com/TwitterBlue/status/1484226494708662273

The current behavior also won't show replies when you open the URL unless you click on the tweet's time to go to the tweet itself. This behavior also impacts any feed reader that used URL to detect duplicate entries, for better or worse. Username behavior has also changed, and would have been "TwitterBlue (@\TwitterBlue) RT: @\Twitter"

Possibly an artifact of the above, the timestamp seems like it may be when the user took the action of retweeting, rather than the timestamp of the original tweet. Using the example, 2022-01-20T23:58:09+00:00 is used instead of 2022-01-20T18:09:??+00:00.
Replies are now present, but you've also already pointed out that is temporary.
I feel like image resizing just defaults to "orig" which I have no problem with, but that previously required noimgscaling to be set, and I'm sure others' use case may make that not ideal.

I'm sleep deprived and that's everything that comes to mind at the moment. Thank you for picking up this torch and running hard with it from the get-go.

Edit: corrected username behavior and attempted to remove account pinging.

… retweeted

arnd-s · 2022-01-24T15:12:01Z

@linoth Thanks for the input

replies should now be filtered if norep parameter was given. I think, based on old code, filtering out replies was always the case in the past but i thought it would be better to use the already available parameter for this.
timestamp should now be from original tweet instead of the retweet.

mnalis · 2022-01-25T13:14:26Z

hm, doesn't seem to work for me at all. I get latest one just now from https://raw.githubusercontent.com/arnd-s/rss-bridge/migrate_to_api_1.1/bridges/TwitterBridge.php and replace original one from 2022-01-20+dfsg1-1 with it, and search only for username without adding any checkboxes. I get this error:

Bridge returned error 403! (19017)
2022-01-25T13:09:21+00:00

Twitter @elonmusk was unable to receive or process the remote website's content!
Error message: `Unexpected response from upstream.
cUrl error: (0)
PHP error: `
Query string: `action=display&bridge=Twitter&context=By+username&u=elonmusk&format=Html`
Version: `dev.2022-01-20`

    Press Return to check your input parameters
    Press F5 to retry
    Check if this issue was already reported on GitHub (give it a thumbs-up)
    Open a GitHub Issue if this error persists

UPDATE: Minor downtimes of few minutes and restarts didn't help. But after disabling rss-bridge twitter bridge for half a day, and enabling it again, now it seems to work ok!

em92 · 2022-01-26T04:57:42Z

Yesterday evening deployed on https://feed.eugenemolotov.ru/pr2433/
In logs I got 2 times this:

[26-Jan-2022 01:41:21 UTC] Exception: Unexpected response from upstream.
cUrl error:  (0)
PHP error:  in /var/www/rss-bridge/pr2433/lib/error.php:24
Stack trace:
#0 /var/www/rss-bridge/pr2433/lib/contents.php(203): returnError()
#1 /var/www/rss-bridge/pr2433/bridges/TwitterBridge.php(735): getContents()
#2 /var/www/rss-bridge/pr2433/bridges/TwitterBridge.php(252): TwitterBridge->makeApiCall()
#3 /var/www/rss-bridge/pr2433/actions/DisplayAction.php(137): TwitterBridge->collectData()
#4 /var/www/rss-bridge/pr2433/index.php(38): DisplayAction->execute()
#5 {main}

My feed reader uses one feed on that deployment. 40 minutes between queries.

arnd-s · 2022-01-26T07:29:58Z

@mnalis This looks like expired/invalid authentication information. I will rewrite some code today and try to catch this kind of error or at least return a more informative error message to solve this.

@em92 Do you got also 403 errors ?

@linoth the noimgscaling option has no use. This was also the case for old code before i started the rewrites. The parts for embedding images wasn't touched by me so far.
Links for retweets will be fixed today

getapikey will normaly now only called once and store all required data inside class object. When an API call fails with status code 403 the API keys will be renewed and the query will be tried a second time.

yamanq · 2022-04-01T14:46:37Z

The only issue I've encountered are the "error 500"s. Whether they pollute the feed depend on a setting in config.ini

Can you tell us what should be put in /etc/rss-bridge/config.ini.php in order so those do not pollute the feed, @berenddeschouwer ? That would be very helpful!

You can also edit report_limit under [error] which you can raise to a higher number. Then errors will need to repeat multiple times before they are pushed to the feed.

bridges/TwitterBridge.php

em92 · 2022-04-02T11:53:15Z

I have following account: https://twitter.com/___em92 It gives following error
Requested username can't be found.
Twitter @___em92
https://feed.eugenemolotov.ru/?action=display&bridge=Twitter&context=By+username&u=___em92&format=Html

False alert here.

em92 · 2022-04-02T11:57:14Z

I am almost ready to merge this. I have left comments that need to be resolved before merge: https://github.com/RSS-Bridge/rss-bridge/pull/2433/files
Not sure if @arnd-s is active yet. If he does not answer next 7 days, feel free anyone to continue his work in separate PR.

Bockiii · 2022-04-03T17:42:21Z

@em92 if the thing holding us up are your reviews on the "instead of comment out, remove", we can just do that in github ui, merge the recommendations and be done with it.

Bockiii

Remove all the commented out lines.

bridges/TwitterBridge.php

github-actions · 2022-04-03T17:48:39Z

Pull request artifacts

file	last change
`Twitter-current-context1`	2022-04-04, 14:11:38
`Twitter-current-context2`	2022-04-04, 14:11:38
`Twitter-current-context3`	2022-04-04, 14:11:38
`Twitter-current-context4`	2022-04-04, 14:11:38
`Twitter-pr-context1`	2022-04-04, 14:11:38
`Twitter-pr-context2`	2022-04-04, 14:11:38
`Twitter-pr-context3`	2022-04-04, 14:11:38
`Twitter-pr-context4`	2022-04-04, 14:11:38

Bockiii · 2022-04-03T18:00:22Z

There we go. I removed all commented lines that were not needed and updated the examplevalues. The "by hashtag" had a wrong syntax in it's examplevalue and the "by list" had no list name at all. "by list" now retrieves the same list id as the "by list id" one.

@em92 you can merge this if it pleases you now :)

Bockiii

linting and fixing

bridges/TwitterBridge.php

Bockiii

Empty line fixing

bridges/TwitterBridge.php

Bockiii · 2022-04-03T18:13:41Z

Okay... NOW its fine :)

arnd-s · 2022-04-04T10:02:00Z

Thanks @Bockiii

Sorry for not responding the last weeks. I was rather busy and occupied. I will adept the getContents call to the new exception based handling from lib/contents.php until tomorrow.

If their are no further problems or blocking points, a merge would be good. New PRs can then be made for further fixes/tweaks.

…m global lib instead own internal privat method

arnd-s · 2022-04-04T14:12:53Z

I removed the private method getContents and make use of the new exception class.

Bockiii · 2022-04-04T17:50:45Z

Alright, I think we are fine. Thanks to everyone involved in writing and testing!

[TwitterBridge] Migration to API V1.1 - Get Tweets by username / firs…

f25d0a2

…t step

arnd-s mentioned this pull request Jan 21, 2022

Twitter @user_name failed with error 429 #2432

Closed

arnd-s added 5 commits January 21, 2022 13:59

Fixed Retweets

d0fa0b6

Commented out some old code without function and fixed json object na…

61eb33e

…me for text attribute of tweet

Fix phpcs errors

b2c5478

Switch to extended Tweet format to get full text again

a285cfb

Get the full text from retweeted_status element

3416e3e

arnd-s added 5 commits January 24, 2022 10:46

In usersearch mode, load userdata to hide pinned tweets.

006e2ec

Added API call to fetch user object.

start to restructure collectData function for cleaning up code, also …

2750c17

…added generic makeApiCall funtion

Migrated "by keyword or hashtag' context

aa5a551

Reactivated filter for replies in "by username" context.

7abb03a

Also commented out some not anymore used functions

Fix phpcs error

5b3eef4

arnd-s added 4 commits January 24, 2022 16:00

Store apikey and guesttoken inside class instance to save some cycles

ea1c82b

Updated tweet filtering to hide replies independend from calling context

b23d6af

On retweets take timestamp from original tweet instead of when it was…

e77fcc9

… retweeted

fix phpcs errors

94b02ec

arnd-s added 4 commits January 26, 2022 09:14

Fix link to tweet in case of a retweet

1efd2ae

Filterout retweets only for search by hashtag or key

662e48a

fix bug with continue not working in case of filtering out a tweet

e88def6

Rewrote Authcode

ecd8d8e

getapikey will normaly now only called once and store all required data inside class object. When an API call fails with status code 403 the API keys will be renewed and the query will be tried a second time.

em92 reviewed Apr 2, 2022

View reviewed changes

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

em92 reviewed Apr 2, 2022

View reviewed changes

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

Bockiii reviewed Apr 3, 2022

View reviewed changes

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

Bockiii added 4 commits April 3, 2022 19:47

comment removal 1

f04e9fa

comment removal 2

c708d32

comment removal 3

cea98b9

comment removal 4

ae652e6

Update examplevalues

f9ddd04

Merge branch 'master' into migrate_to_api_1.1

ac5f134

Bockiii approved these changes Apr 3, 2022

View reviewed changes

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

Bockiii added 2 commits April 3, 2022 20:05

more removals

12f3d85

lint fixing

ddc30b4

Bockiii reviewed Apr 3, 2022

View reviewed changes

bridges/TwitterBridge.php Show resolved Hide resolved

bridges/TwitterBridge.php Show resolved Hide resolved

Bockiii added 2 commits April 3, 2022 20:08

empty line 1

d1e7393

empty line 2

8665079

Bockiii reviewed Apr 3, 2022

View reviewed changes

bridges/TwitterBridge.php Outdated Show resolved Hide resolved

mistake on my end. typo

8436226

Change to use new version of getContents with improved exceptions fro…

eb87ef2

…m global lib instead own internal privat method

Bockiii merged commit 0d305f1 into RSS-Bridge:master Apr 4, 2022

Kwbmm pushed a commit to Kwbmm/rss-bridge that referenced this pull request Jun 17, 2022

[TwitterBridge] Migration to API V1.1 (RSS-Bridge#2433)

3572f00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TwitterBridge] Migration to API V1.1 #2433

[TwitterBridge] Migration to API V1.1 #2433

arnd-s commented Jan 21, 2022

arnd-s commented Jan 21, 2022

csisoap commented Jan 22, 2022

arnd-s commented Jan 22, 2022

toelke commented Jan 23, 2022

arnd-s commented Jan 24, 2022

linoth commented Jan 24, 2022 •

edited

Loading

arnd-s commented Jan 24, 2022

mnalis commented Jan 25, 2022 •

edited

Loading

em92 commented Jan 26, 2022

arnd-s commented Jan 26, 2022

yamanq commented Apr 1, 2022

em92 commented Apr 2, 2022

em92 commented Apr 2, 2022

Bockiii commented Apr 3, 2022

Bockiii left a comment

github-actions bot commented Apr 3, 2022 •

edited

Loading

Bockiii commented Apr 3, 2022

Bockiii left a comment

Bockiii left a comment

Bockiii commented Apr 3, 2022

arnd-s commented Apr 4, 2022

arnd-s commented Apr 4, 2022

Bockiii commented Apr 4, 2022

[TwitterBridge] Migration to API V1.1 #2433

[TwitterBridge] Migration to API V1.1 #2433

Conversation

arnd-s commented Jan 21, 2022

arnd-s commented Jan 21, 2022

csisoap commented Jan 22, 2022

arnd-s commented Jan 22, 2022

toelke commented Jan 23, 2022

arnd-s commented Jan 24, 2022

linoth commented Jan 24, 2022 • edited Loading

arnd-s commented Jan 24, 2022

mnalis commented Jan 25, 2022 • edited Loading

em92 commented Jan 26, 2022

arnd-s commented Jan 26, 2022

yamanq commented Apr 1, 2022

em92 commented Apr 2, 2022

em92 commented Apr 2, 2022

Bockiii commented Apr 3, 2022

Bockiii left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 3, 2022 • edited Loading

Pull request artifacts

Bockiii commented Apr 3, 2022

Bockiii left a comment

Choose a reason for hiding this comment

Bockiii left a comment

Choose a reason for hiding this comment

Bockiii commented Apr 3, 2022

arnd-s commented Apr 4, 2022

arnd-s commented Apr 4, 2022

Bockiii commented Apr 4, 2022

linoth commented Jan 24, 2022 •

edited

Loading

mnalis commented Jan 25, 2022 •

edited

Loading

github-actions bot commented Apr 3, 2022 •

edited

Loading