Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PokerStars compatible Hand Histories #576

Open
4 of 12 tasks
ctm opened this issue Apr 15, 2021 · 26 comments
Open
4 of 12 tasks

PokerStars compatible Hand Histories #576

ctm opened this issue Apr 15, 2021 · 26 comments
Assignees
@ctm
Labels
easy Trivial to do (even when tired!) and semi-worthwhile enhancement New feature or request

Comments

@ctm
Copy link
Owner

ctm commented Apr 15, 2021 •
edited
Loading

Make the hand histories use Poker Stars format.

If mb2 spits out Poker Stars compatible hand histories that will make it much easier for sites like PokeIt to provide nice graphical depictions of what happened. In fact, they could have that working before the new UI is deployed, which could be a big win for a number of reasons.

Now that the scaffolding / glue code is in place, "all" (heh) we need to do get PS-compatible hand histories is to implemment new for each of the data structures that make up a poker_stars::History. Each of them have many parts, but I'll start out with just the top-level data structures and only add sub-check-boxes as I dig into the top-level ones (in order, probably).

  • Heading
    • HandStart
    • TableInfo
      • table
      • n_seats
      • play_money (from events rather than from the table name)
      • button (from a combination of the first Status as well as Structure::is_excusively_buttonless)
  • ForcedBet
  • PlayerInfo
  • HandLines
  • Resolution
  • Summary
@ctm ctm added enhancement New feature or request easy Trivial to do (even when tired!) and semi-worthwhile agdgat labels Apr 15, 2021
@ctm ctm self-assigned this Apr 15, 2021
@amanjyaku
Copy link

I'm not sure that getting Pokeit to display Binglaha and Scrotum hands will be all that easy.

@ctm
Copy link
Owner Author

ctm commented Apr 17, 2021

Understood, and I can't allow this to eat a bunch of my time, but I think I can make a lot of progress with relatively little effort.

Thanks for poking around, so to speak.

@ctm ctm removed the agdgat label Oct 20, 2021
@ctm ctm mentioned this issue Feb 14, 2022
Closed
@ctm
Copy link
Owner Author

ctm commented Jan 5, 2025

I'm going to do a little work on this today, because today is likely to be a day where I don't interact with people much due to (perceived) tiredness. I also haven't added any new features recently (although I've added and greatly improved debugging facilities).

This will be very similar to how structure sheets are implemented in that there will be an HTTP API that provides the information (rather than messaging through a Web Socket). However, since public table messages have evolved over time, we'll need to create a fallible conversion from JSON and a timestamp to a given message (identified by the variant), where we use various heuristics to build the current representation of any given message and if the heuristics fail, we use the timestamp to figure out how to proceed and if that fails, we log and report an error.

I do not expect to get this done today, although it'll be neat if I have enough of the parts built to show something. I'll use hand 435561 as a goal and will not initially even bother with PokerStars compatibility.

This was referenced Jan 5, 2025
Closed
Closed
@ctm
Copy link
Owner Author

ctm commented Jan 6, 2025

Yesterday was like swimming in cement. I made progress, but it was so slow and annoying. Today I finished up the first part of the API and deployed it so I could check on hand 435561. Success. I should probably make it so clicking that link fails (#1534), but I'd like to add a hand-history text page before doing that.

@amanjyaku
Copy link

GitHub just put me through a verification ringer, but if you need any help I may have some time at the end of the week
;)

@ctm
Copy link
Owner Author

ctm commented Jan 6, 2025

That's a very kind offer. I will probably stumble my way through it in a proof-of-concept way, at least for now.

This was referenced Jan 7, 2025
Closed
Closed
@ctm
Copy link
Owner Author

ctm commented Jan 7, 2025 •
edited
Loading

I've added some of the scaffolding to make /hand/NNN be a component, albeit with todo!() invocations. Off the top of my head, the next steps are:

  • Move HandMessage and friends to mb2_commands
    • Bite the bullet and duplicate RawHandMessage since one has to be in models and one has to be in mb2_commands
  • Make a HandInfo (or similar name) struct that has a HashMap<PlayerId, Nick> and a Vec<Result<HandMessage, ...>> in it
  • Add an impl From<HashMap<PlayerId, Nick> for ShareablePlayers (or make a new fn that takes unknown)
  • Send the HandInfo back from the API
    • Hoist the guts out of ToNickMapper::all_referenced_player_ids
  • Do a super hacky copy-and-paste proof-of-concept Component implementation
  • View
    • transform \n in text into <br/> elements
    • omit non-history messages (reminds are particularly annoying because the current text isn't even correct) (look at end of table_message.rs for some useful predicates)
    • add the code that currently says TODO 1
    • add the code that currently says TODO 2
    • use close to PokerStars formatting
  • Figure out a clean refactor (which itself will probably have a bunch of steps)

@ctm
Copy link
Owner Author

ctm commented Jan 8, 2025

@amanjyaku

GitHub just put me through a verification ringer, but if you need any help I may have some time at the end of the week ;)

If you have (or can generate) PS hand history files and either attach them to this issue or email them (I don't have a preference), that alone will be a fairly big help, because it'll allow me to proceed w/o getting distracted by trying to get back on PS (and then spending time there when I should be doing other things).

BTW, if you've identified yourself (perhaps via email or some other way), I no longer remember who you are. If you haven't and want to, that's fine. If you don't want to identify yourself, that's fine, too. I assume you're a BARGEr due to your mention of Binglaha and Scrotum, but it turns out I like non-BARGErs, too.

@amanjyaku
Copy link

amanjyaku commented Jan 8, 2025 via email

@amanjyaku
Copy link

I am away from home and was unable to access any hand histories. Sorry.

@ctm
Copy link
Owner Author

ctm commented Jan 9, 2025

Turns out, there's enough on a Wikipedia page for me to at least structure the code that will do the conversion from the messages that we get from the database into what is needed to produce that output. I don't know how much I'll actually do of it any time soon. My financial situation is a bit higher of a priority.

@amanjyaku
Copy link

amanjyaku commented Jan 9, 2025 via email

@ctm
Copy link
Owner Author

ctm commented Jan 9, 2025

Turns out the histories on Wikipedia are too old for me to use, but I have a PokerStars account and am trying to fire it up now (it's downloading an update).

This is fairly low priority, but it's something I can work on when I'm very tired which happens all too much these days. In 2025 I need to expand mb2's use beyond BARGErs and although one key component to that is an acceptable user-interface, it may be a little while before I can pay for more work on that. OTOH, I can do the hand history stuff myself and then once we're generating PS-compatible hand histories, we'll be able to demonstrate compatibility with some of the other online tools.

As it is, mb2 has a database of 437, 990 hands but without the fix-up code I've mentioned above, only the most recent ones can be accessed, even in the non-PS ugly format currently available. The fix-up code goes in between the JSON that we get out of the database and the final emitter of text and in between we need a representation that is sufficient for any hand history format that we want, but starting with one that is a super-set of PS (and can be limited to PS-only) makes sense in light of the compatibility that gives us.

For today, my plan is simply to write a bunch of boilerplate that I can then peck away at a little at a time. Implementing PS-compatibility decomposes into a ton of trivial things to do, where none of them takes much mental effort.

@ctm
Copy link
Owner Author

ctm commented Jan 9, 2025

D'oh! Futzing around on pokerstars.net makes me think you can't get hand histories from there anymore. I'll have to ask more people about them, but I will probably not do it today (and may not for a while).

@amanjyaku
Copy link

amanjyaku commented Jan 9, 2025 via email

@ctm
Copy link
Owner Author

ctm commented Jan 9, 2025 via email

@ctm
Copy link
Owner Author

ctm commented Jan 26, 2025 •
edited
Loading

FWIW, I was sent a zip of 117 files collectively containing 8,929 hand histories from a variety of games, both play money and tournaments. I'm writing a parser (using nom) that parses all of them into this struct:

pub struct History {
    heading: Heading,
    player_info: PlayerInfo,
    forced_bets: Vec<ForcedBet>,
    hand_lines: HandLines,
    resolution: Resolution,
    summary: Summary,
}

I'll then write a method to generate a string from that struct and add a check to make sure that the string that is constructed is character-per-character the same (including trailing spaces!) as the input. That will show that my encoding hasto the ability to capture every nuance (including trailing spaces!) detectable from these 8,929 hands.

Once I have all of that working, I can then take the stream of messages that are currently returned by or existing hand history code and turn them into this PokerStars compatible representation. I'll still have to pay attention to subtle details (e.g., kicker representation).

Currently I have code for everything except the summary, but it doesn't yet get through all the hands. I discover edge cases one at a time and each time I find a new edge case and fix it, I get to see another one. I am hoping that today I'll get through all the edge cases for what I've already written and start working on the Summary (which probably doesn't have many edge cases).

A snapshot of where I am right now is:

[history-576]% tokei lib/poker-stars/src/history/parser.rs 
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1         2001         1636          120          245
===============================================================================
 Total                   1         2001         1636          120          245
===============================================================================

113 lines of parser.rs are in the test module.

The next edge case for me to support is splitting bounties (#1549).

@ctm
Copy link
Owner Author

ctm commented Jan 26, 2025 •
edited
Loading

For reference, I wrote a parser for WRGPT years ago (just before I wrote mb2) and it's a similar size:

[history-576]% tokei mb2/src/wrgpt.rs
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1         1711         1460           72          179
===============================================================================
 Total                   1         1711         1460           72          179
===============================================================================

@ctm
Copy link
Owner Author

ctm commented Jan 26, 2025

Now that I think about it, parser.rs isn't quite comparable to wrgpt.rs, because a lot of the structs and enums that parser.rs uses are in history.rs, where wrgpt.rs has both the data structures and the parsing code. OTOH, history.rs contains the beginning of some code for creating the strings out of the data structures and also has some redundant code that I'll be refactoring due to the difference between representing a player by a PlayerId and by a Nick. So, with that caveat:

[history-576]% tokei lib/poker-stars/src/history.rs 
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1          628          445           93           90
===============================================================================
 Total                   1          628          445           93           90
===============================================================================

@amanjyaku
Copy link

amanjyaku commented Jan 26, 2025 via email

@ctm
Copy link
Owner Author

ctm commented Jan 26, 2025

My guess is it's stabilized. There are a few reasons why I'm doing PokerStars compatible hand history work now, some of them are even good reasons.

One is that I believe there are several third party tools that use PS hand histories and I'd like them to work with mb2. CPC and mb2 are too darned small for anyone (except an elite few) to care about, but I think a few more people will be interested if they can look at some mb2 stuff via a decent interface like some of the third party tools. I suspect either PS doesn't want to break those tools or the toolmakers themselves want to stay on top of things, in which case, once I gain a foothold, perhaps I can work with the third party toolmakers, even if PS does change.

Another reason to work on PS compatible hand histories is that mb2 has recorded all of the messages associated with all the hands back to hand number one, but those message formats themselves have changed and in order to provide people with hand histories that go back in time, I need to think about how best to do that and knowing what info PS displays (and to a lesser extent how and when it displays it) helps me think about that organization. Turns out, even if PS were to completely change their representation, that wouldn't really affect this aspect of doing PS hand history work.

Another reason to work on PS compatible hand histories is that it's something I can do when I'm tired. In theory, that's a great idea, but in practice, what happens is when I'm tired I do a ton of grunt work and then get to something that requires a bit of thought and then when I'm rested I work on the thing that requires thought rather than on more important matters. Right now, for example, I need to find a CPA in Albuquerque and although I've asked two different entities, neither has replied yet and I need to poke them, which I'll do on Monday. OTOH, I haven't written a status report to potential investors in a long time and I could do that today with my current level of alertness…

@ctm
Copy link
Owner Author

ctm commented Jan 26, 2025

For the playing at home crowd, everything but the summary section is now parsed for all 8,929 hands, although I am not yet reproducing the text output from my parsed data structures so there are probably a few places where I've made tiny mistakes in parsing what I've got. I'm probably not overlooking anything major, because the parser would "jump the rails" and refuse to parse.

The last edge case I had to deal with was the possibility that the deck would be reshuffled. That only happened once in the 8,929 hands that I'm working with. If it hadn't happened at all, I wouldn't have added that functionality. To see what it took to add reshuffling in, here's the diff:

diff --git a/lib/poker-stars/src/history/parser.rs b/lib/poker-stars/src/history/parser.rs
index 0e83057e..663f3d58 100644
--- a/lib/poker-stars/src/history/parser.rs
+++ b/lib/poker-stars/src/history/parser.rs
@@ -702,6 +702,12 @@ struct FourthStreetNick {
     pair_on_board_disclaimer: bool,
 }
 
+#[derive(Clone, Debug)]
+struct MaybeReshuffledDrawsNick {
+    reshuffled: bool,
+    draws: Vec<PlayerDrawOrAnytimeNick>,
+}
+
 #[derive(Clone, Debug)]
 enum CardActionNick {
     Flop(Flop),
@@ -714,8 +720,8 @@ enum CardActionNick {
     SeventhStreet(Vec<DealtToNick>),
     DealingHands(Vec<DealtToNick>),
     FirstDraw(Vec<PlayerDrawOrAnytimeNick>),
-    SecondDraw(Vec<PlayerDrawOrAnytimeNick>),
-    ThirdDraw(Vec<PlayerDrawOrAnytimeNick>),
+    SecondDraw(MaybeReshuffledDrawsNick),
+    ThirdDraw(MaybeReshuffledDrawsNick),
     ImplicitSingleDraw(Vec<PlayerDrawOrAnytimeNick>),
 }
 
@@ -895,16 +901,22 @@ fn nick_draw_or_anytime(input: &str) -> IResult<&str, PlayerDrawOrAnytimeNick> {
 
 fn nth_draw<'a>(
     to_match: &'static str,
-    constructor: fn(Vec<PlayerDrawOrAnytimeNick>) -> CardActionNick,
+    constructor: fn(MaybeReshuffledDrawsNick) -> CardActionNick,
 ) -> impl FnMut(&'a str) -> IResult<&'a str, CardActionNick> {
     map(
-        preceded(tag(to_match), many1(nick_draw_or_anytime)),
-        constructor,
+        tuple((
+            preceded(tag(to_match), opt_tag("The deck is reshuffled\n")),
+            many1(nick_draw_or_anytime),
+        )),
+        move |(reshuffled, draws)| constructor(MaybeReshuffledDrawsNick { reshuffled, draws }),
     )
 }
 
 fn first_draw(input: &str) -> IResult<&str, CardActionNick> {
-    nth_draw("*** FIRST DRAW ***\n", CardActionNick::FirstDraw)(input)
+    map(
+        preceded(tag("*** FIRST DRAW ***\n"), many1(nick_draw_or_anytime)),
+        CardActionNick::FirstDraw,
+    )(input)
 }
 
 fn second_draw(input: &str) -> IResult<&str, CardActionNick> {

If you look carefully, you may see that I only allow reshuffling at the beginning of the second and third draws. I don't think it's ever needed at the beginning of the first draw. I didn't bother to make my nth_draw function sufficiently generic for it to be used for first_draw, so there's a little bit of duplicated code between first_draw (which can't detect, much less record a reshuffle) and nth_draw which is used for the second and third draws, where a reshuffle can occur.

@ctm
Copy link
Owner Author

ctm commented Jan 27, 2025 •
edited
Loading

I'm now parsing all the hands, including the summaries. I got pretty close before bedtime yesterday, but my regimented sleep schedule kicked in and I was able to sleep (hooray!) and then get back on it this morning.

IIRC, I just need to add time zones to one of the data structures and then they'll all be self-contained, meaning I can implement Display on them and then I'll be generating PS-compatible hand histories, albeit initially solely from PS-parsed hand histories. Once they're character-for-character perfect (including trailing spaces!!!), I'll probably put this issue aside briefly, because the next push will require some (but not much) thought, a lot of concentration and (a few) big blocks of time.

@ctm
Copy link
Owner Author

ctm commented Jan 29, 2025

My code now parses all of the hand histories that I have available and is then able to produce a string from the parsed data. That string is character-for-character the exact same data that was parsed (including trailing spaces!!!!!!!!!!—I'm not bitter).

I sent the source to a few people who might be interested. I'll probably do a talk in February using my code as a launch point.

@ctm ctm mentioned this issue Feb 9, 2025
Closed
11 tasks
@ctm
Copy link
Owner Author

ctm commented Feb 24, 2025

FWIW, I started back on this yesterday, because it will play a role in revealing cards (#1566) in games where there's more than just hole cards (e.g., draw games). I added enough scaffolding that we're now constructing a (sub-minimal) History, but with most of the Heading properly filled in.

I'm going to edit in some check-boxes to the description, since the remaining steps are well defined.

@ctm ctm mentioned this issue Feb 24, 2025
Open
@ctm
Copy link
Owner Author

ctm commented Feb 26, 2025

I made a small amount of progress on this yesterday, but not as much as I'd have liked. I changed some of the helper functions so that we wouldn't have to look for the relevant Status message more than once, and I made a helper to find a given message (or the JSON for the message if it couldn't be parsed due to a format change). I also wrote the code to back-patch starting stacks by adding back in the forced bets.

So, although I didn't get to click any of the check-boxes, I did the preliminary work to finish TableInfo as well as get ForcedBet and PlayerInfo done. Finishing those three is less than a day's work, but my time is going to be very fragmented today :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ctm ctm

Labels
easy Trivial to do (even when tired!) and semi-worthwhile enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ctm @amanjyaku