HybridCache : implement the tag expiration feature

[mgravell]

(note: this is a re-do of #5748 / #5781 which went "a bit wrong" when rebasing)

Implement tag expiration in HybridCache

The HybridCache system has a "tags" feature; rather than building a bespoke per-backend secondary index (like how the redis OutputCache backend works), here we move everything into the core library:

• in the DefaultHybridCache, we maintain a map of tags (string, note * means everything) to expirations, and we track the creation time against each CacheItem
• when fetching items from L1 (in-proc) cache, we check the item's creation data against the cache's wildcard and per-tag expirations; if the creation date is earlier than any of these: it is treated as disallowed
• to allow persistence of invalidation between sessions when a L2 (backend cache) is in play, we additionally store the timeout data via additional simple values in the cache
• we also pre-fetch any missing tag data from L2 as-needed, and enforce the previous expirations
• to facilitate this, and allow reliability that the data we are parsing is the data requested: the data stored in L2 is now embedded inside a payload that includes the creation timestamp, duration, key, tags, etc as well as the payload
• in the rare event that the L2 data contains additional tags to those advertised by the caller: these additional tags are also fetched and enforced

Because the tag fetches are async, the data in the lookup is not (timestamp), it is Task<(timestamp)>

Outstanding:

• additional logging
  • add log when L2 data is rejected for (reason)
  • add metric of the number of tag expirations being tracked in L1
• resolve netfx test failure (actually: niche timing problem in tag invalidation)
• address option member inheritance (or not - per @jodydonetti comments here)

Microsoft Reviewers: Open in CodeFlow

[jodydonetti]

Note

Reposting my comment here from there for ease of discussion.

Hi Marc, not strictly related to tagging, but I'm looking at some code change in the PR and I spotted this xml comment regarding HybridCacheEntryOptions:

If options are specified at the individual call level, the non-null values are merged
(with the per-call options being used in preference to the global options). If no value is
specified for a given option (globally or per-call), the implementation can choose a reasonable default.

When we touched on this subject some time ago I remember we agreed on the fact that inheritance (from per-call up to the DefaultEntryOptions) would have to be done on an all-or-nothing approach, and not on a per-single-option approach.
What I mean is that "if null is passed as an HybridCacheEntryOptions param" than it would fall back to the DefaultEntryOptions, but single options inside that object (like Expiration, LocalCacheExpiration) should not have inheritance of values.

All of this was because of the impossibility to express "undefined" or, to better say it, the impossibility to differentiate between null meaning "I'm saying null because I want a local fallback" (for example from LocalCacheExpiration to Expiration) and null meaning "I'm saying null because I want a global fallback" (for example to the same option in the DefaultEntryOptions").

A practical example of a problem this may create: in the DefaultEntryOptions I set Expiration to 5min and LocalCacheExpiration to 10s to refresh L1 more frequently. Then in a certain call I pass a specific HybridCacheEntryOptions instance where I set Expiration to 1h and leave LocalCacheExpiration to null, thinking it will be the same as Expiration, when in reality it will become 10s. Because of this, there will be for me no way to keep the 2 options in-sync at every call, unless I keep specifying the same value for both Expiration and LocalCacheExpiration at every call.

Can you clarify what is the current rationale here?

I'm asking for 2 reasons: first I'm interested in general, and second because as you know I'm creating a 3rd party implementation, and I'd like the observable external behaviour to be the same for both implementations.

Thanks!

ps: if you like I can open a different issue for this, just let me know.
pps: will post more about the tagging implementation as soon as I'll finish reading the massive code changes.

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.14 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84.49 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=912260&view=codecoverage-tab

[jodydonetti]

An extra point to think about, again related the entry options inheritance thing, is how this method (marked internal, and maybe it will not be used in the end) would be affected.

[BrennanConroy]

to allow persistence of invalidation between sessions when a L2 (backend cache) is in play, we additionally store the timeout data via additional simple values in the cache

Shouldn't we be concerned about machine clocks not being synchronized? i.e. Machine 1 says it's 10:05:03 and adds a timeout for 10:10:03, but Machine 2 says it's 10:12:05 and when it reads the cache item it will determine it to be invalid.

[jodydonetti]

Shouldn't we be concerned about machine clocks not being synchronized? i.e. Machine 1 says it's 10:05:03 and adds a timeout for 10:10:03, but Machine 2 says it's 10:12:05 and when it reads the cache item it will determine it to be invalid.

Eh, welcome to the world of clock shifting/drifting/skewing & friends: it's not something that's solvable via software, at least not at this level (to the best of my knowledge).

Just to make an example: typically Microsoft/Meta/Google/AWS have their own private atomic clocks around the world to avoid this problem (... as much as possible), usually dedicated to specific services (like Google Spanner with TrueTime).

Some interesting reads:

• https://en.wikipedia.org/wiki/Clock_synchronization
• https://en.wikipedia.org/wiki/Clock_drift
• https://www.theverge.com/2012/11/26/3692392/google-spanner-atomic-clocks-GPS
• https://engineering.fb.com/2021/08/11/open-source/time-appliance/
• https://cloud.google.com/spanner/docs/true-time-external-consistency
• https://www.bigdatawire.com/2024/12/03/aws-takes-on-google-spanner-with-atomic-clock-powered-distributed-dbs/
• https://people.mpi-inf.mpg.de/~clenzen/pubs/LLSW10clock.pdf

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	78.59 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.65 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=913450&view=codecoverage-tab

[mgravell]

Clock skew: ultimately as stated, not easily solvable. But in the context of cache servers: we don't need super complexity or accuracy: IMO if any layer is unhappy re expiration, that's good enough to consider it invalid. As a side-effect, it also makes the logic time-testable.

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	78.59 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.65 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=931777&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.35 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=931922&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	77.96 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	78.19 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	82.83 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=933691&view=codecoverage-tab

[jodydonetti]

Hi Marc, I'm looking at this and wow, those are a lot of tests 💪

Just to doublecheck so we are on the same page: the idea is still this?

ps: the amount of work you are doing on HC is spectacular, kudos.

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	79.42 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	83.84 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=934881&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	84.72 🔻
Microsoft.Extensions.Caching.Hybrid	Branch	86	84.34 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=934939&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻
Microsoft.Extensions.Caching.Hybrid	Line	86	85.48 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=935018&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.OpenAI	77	78

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=935225&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Caching.Hybrid	Line	86	82 🔻
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89
Microsoft.Extensions.AI.Abstractions	83	84

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=938365&view=codecoverage-tab

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.AI.Ollama	Line	80	78.25 🔻
Microsoft.Gen.MetadataExtractor	Line	98	57.35 🔻
Microsoft.Gen.MetadataExtractor	Branch	98	62.5 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.AI.Abstractions	83	84
Microsoft.Extensions.Caching.Hybrid	86	87
Microsoft.Extensions.AI.OpenAI	77	78
Microsoft.Extensions.AI	88	89

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=938418&view=codecoverage-tab

[DeagleGross]

Approved; but would be happy if someone else takes a look (PR is long living). Below asked a couple of specific code questions; overall purpose of PR is kind of understandable, but I dont have any background with HybridCache

[jodydonetti]

Congrats for the big milestone 🎉

[mgravell]

@jodydonetti if I made the "snap" last night, this should go out ~11th as a final month of [Experimental], then it'll be GA the following month; I have some docs improvements from internal review, which apply to both HC and FC (things like guidance on appropriate key construction), but: eesh, that was indeed a slog.

[jodydonetti]

@jodydonetti if I made the "snap" last night, this should go out ~11th as a final month of [Experimental],

every time I see an [Experimental] usage I get a dumb smile thinking about the dense fella who proposed it 🤣

then it'll be GA the following month; I have some docs improvements from internal review,

Eh, enjoy every second of stress test in the public preview, it's really worth it. I got random minor edge case stuff came up with FC (and something is still coming up) so that's really precious, although frustrating sometimes 🥲

which apply to both HC and FC (things like guidance on appropriate key construction)

Eagerly awaiting this part to see all coming together on both HC/FC, can't wait.

but: eesh, that was indeed a slog.

Amazing job 💪