Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement cache in schema loading #385

Merged
merged 1 commit into from
Mar 19, 2023
Merged

Implement cache in schema loading #385

merged 1 commit into from
Mar 19, 2023

Conversation

tdeo
Copy link
Contributor

@tdeo tdeo commented Feb 24, 2023

In the application I'm working on, we check all of our endpoints against OpenApi schemas, and we take a big performance hit.

For instance one of our file performs 102 examples against a very simple controller, and right now here's the output of ruby prof for this test file:

Measure Mode: wall_time
Thread ID: 15080
Fiber ID: 869160
Total: 127.745651
Sort by: self_time

 %self      total      self      wait     child     calls  name                           location
 10.24     96.512    13.082     0.000    83.430 27013392  *OpenAPIParser::SchemaLoader#load_data_by_schema_loader /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader.rb:39
  8.41     25.498    10.748     0.000    14.750 20716320   OpenAPIParser::SchemaLoader::ValuesLoader#load_data /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/values_loader.rb:6
  5.96     12.147     7.619     0.000     4.528 27013392   OpenAPIParser::SchemaLoader::Base#variable_set /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/base.rb:25
  4.54     96.510     5.800     0.000    90.710  3899328  *OpenAPIParser::SchemaLoader::Creator#build_openapi_object_from_option /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/creator.rb:37
  3.94      9.559     5.036     0.000     4.523  3899328   OpenAPIParser::SchemaLoader::Creator#build_object_reference_from_base /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/creator.rb:17
  3.90      4.982     4.982     0.000     0.000  5170643   Array#+                        
  3.60     96.513     4.595     0.000    91.918  1277376  *OpenAPIParser::SchemaLoader#load_data /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader.rb:29
  3.55      4.537     4.537     0.000     0.000 27048841   Kernel#instance_variable_set   
  3.16      4.032     4.032     0.000     0.000 30747851   Symbol#to_s                    
  2.66      3.404     3.404     0.000     0.000 54026784   OpenAPIParser::SchemaLoader#target_object 
  2.45      3.784     3.129     0.000     0.656  4837296   OpenAPIParser::SchemaLoader::Creator#escape_reference /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/creator.rb:33
  2.44      3.123     3.123     0.000     0.000 27013392   OpenAPIParser::Schemas::Base#raw_schema 
  1.89      2.419     2.419     0.000     0.000     1014   PG::Connection#exec_params     
  1.82     96.511     2.331     0.000    94.181  2961360  *OpenAPIParser::SchemaLoader::ObjectsLoader#create_attr_object /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/objects_loader.rb:14
  1.71     14.044     2.187     0.000    11.856  1156416  *OpenAPIParser::Expandable#expand_reference /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/expandable.rb:5
  1.67      2.136     2.136     0.000     0.000 27013392   OpenAPIParser::SchemaLoader::Base#variable_name 
  1.66      2.116     2.116     0.000     0.000 26639040   OpenAPIParser::SchemaLoader::Base#schema_key 
  1.60     10.184     2.044     0.000     8.140  1277376   OpenAPIParser::SchemaLoader#all_loader /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader.rb:51
  1.57      2.001     2.001     0.000     0.000  7810647   Hash#values                    
  1.43     96.512     1.828     0.000    94.684  2961360  *OpenAPIParser::SchemaLoader::ObjectsLoader#load_data /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/objects_loader.rb:6
  1.34      2.251     1.713     0.000     0.539  3899328   OpenAPIParser::SchemaLoader#register_child /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader.rb:45
  1.32      1.681     1.681     0.000     0.000     1057   PG::Connection#async_exec      
  1.22     14.043     1.562     0.000    12.481  1156416  *OpenAPIParser::Expandable#expand_objects /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/expandable.rb:36

* recursively called methods

With the changes in this PR:

Measure Mode: wall_time
Thread ID: 15060
Fiber ID: 508180
Total: 19.379235
Sort by: self_time

 %self      total      self      wait     child     calls  name                           location
 12.15      2.354     2.354     0.000     0.000     1014   PG::Connection#exec_params     
  8.21      1.591     1.591     0.000     0.000     1057   PG::Connection#async_exec      
  4.96      0.961     0.961     0.000     0.000      104   <Class::Dir>#[]                <internal:dir>:127
  4.21      0.816     0.816     0.000     0.000   108586   String#split                   
  2.03      0.393     0.393     0.000     0.000   174462   <Class::File>#mtime            
  1.96      0.381     0.381     0.000     0.000     8931   Time#getlocal                  
  1.73      2.425     0.336     0.000     2.089   562779  *OpenAPIParser::SchemaLoader#load_data_by_schema_loader /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader.rb:39
  1.51      0.293     0.292     0.000     0.001   120061   Array#join                     
  1.42      0.651     0.274     0.000     0.376   431590   OpenAPIParser::SchemaLoader::ValuesLoader#load_data /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/values_loader.rb:6
  1.32      0.255     0.255     0.000     0.000   118012   Array#+                        
  1.21      0.234     0.234     0.000     0.000    69997   String#=~                      
  1.12      0.217     0.217     0.000     0.000       48   Digest::Base#update            
  1.09      0.212     0.212     0.000     0.000      311   PG::Connection#query           
  1.00      0.299     0.195     0.000     0.104   562779   OpenAPIParser::SchemaLoader::Base#variable_set /Users/thierry/.asdf/installs/ruby/3.2.1/lib/ruby/gems/3.2.0/gems/openapi_parser-1.0.0/lib/openapi_parser/concerns/schema_loader/base.rb:25

I tried to be safe and cache the the schema based on both the file contents and the parser options, let me know if there's anything else I could have missed.

By the way I'm realizing that most of the class methods in Committee::Drivers should actually be private in the first place, but I decided not to include extra changes in this PR

@ota42y
Copy link
Member

ota42y commented Mar 19, 2023

Thanks for the suggestion!
I originally didn't expect to have to load it multiple times, so that would cause problems.
There is no workaround for this, especially since committee support the file loading only, and I think this change is necessary!

@ota42y ota42y merged commit 6bdc5cf into interagent:master Mar 19, 2023
@tdeo tdeo deleted the td/schema_caching branch March 19, 2023 18:44
@quentindemetz quentindemetz restored the td/schema_caching branch March 20, 2023 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants