scientist

Scientist! A Ruby library for carefully refactoring critical paths. How do I science? Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load. require "scientist" class MyWidget def allows?(user) experiment = Scientist::Default.new "widget-permissions" experiment.use { model.check_user(user).valid? } # old way experiment.try { user.can?(:read, model) } # new way experiment.run end end Wrap a use block around the code's original behavior, and wrap try around the new behavior. experiment.run will always return whatever the use block returns, but it does a bunch of stuff behind the scenes: It decides whether or not to run the try block, Randomizes the order in which use and try blocks are run, Measures the wall time and cpu time of all behaviors in seconds, Compares the result of try to the result of use, Swallow and record exceptions raised in the try block when overriding raised, and Publishes all this information. The use block is called the control. The try block is called the candidate. Creating an experiment is wordy, but when you include the Scientist module, the science helper will instantiate an experiment and call run for you: require "scientist" class MyWidget include Scientist def allows?(user) science "widget-permissions" do |experiment| experiment.use { model.check_user(user).valid? } # old way experiment.try { user.can?(:read, model) } # new way end # returns the control value end end If you don't declare any try blocks, none of the Scientist machinery is invoked and the control value is always returned. Making science useful The examples above will run, but they're not really doing anything. The try blocks don't run yet and none of the results get published. Replace the default experiment implementation to control execution and reporting: require "scientist/experiment" class MyExperiment include Scientist::Experiment attr_accessor :name def initialize(name) @name = name end def enabled? # see "Ramping up experiments" below true end def raised(operation, error) # see "In a Scientist callback" below p "Operation '#{operation}' failed with error '#{error.inspect}'" super # will re-raise end def publish(result) # see "Publishing results" below p result end end When Scientist::Experiment is included in a class, it automatically sets it as the default implementation via Scientist::Experiment.set_default. This set_default call is skipped if you include Scientist::Experiment in a module. Now calls to the science helper will load instances of MyExperiment. Controlling comparison Scientist compares control and candidate values using ==. To override this behavior, use compare to define how to compare observed values instead: class MyWidget include Scientist def users science "users" do |e| e.use { User.all } # returns User instances e.try { UserService.list } # returns UserService::User instances e.compare do |control, candidate| control.map(&:login) == candidate.map(&:login) end end end end If either the control block or candidate block raises an error, Scientist compares the two observations' classes and messages using ==. To override this behavior, use compare_errors to define how to compare observed errors instead: class MyWidget include Scientist def slug_from_login(login) science "slug_from_login" do |e| e.use { User.slug_from_login login } # returns String instance or ArgumentError e.try { UserService.slug_from_login login } # returns String instance or ArgumentError compare_error_message_and_class = -> (control, candidate) do control.class == candidate.class && control.message == candidate.message end compare_argument_errors = -> (control, candidate) do control.class == ArgumentError && candidate.class == ArgumentError && control.message.start_with?("Input has invalid characters") && candidate.message.start_with?("Invalid characters in input") end e.compare_errors do |control, candidate| compare_error_message_and_class.call(control, candidate) || compare_argument_errors.call(control, candidate) end end end end Adding context Results aren't very useful without some way to identify them. Use the context method to add to or retrieve the context for an experiment: science "widget-permissions" do |e| e.context :user => user e.use { model.check_user(user).valid? } e.try { user.can?(:read, model) } end context takes a Symbol-keyed Hash of extra data. The data is available in Experiment#publish via the context method. If you're using the science helper a lot in a class, you can provide a default context: class MyWidget include Scientist def allows?(user) science "widget-permissions" do |e| e.context :user => user e.use { model.check_user(user).valid? } e.try { user.can?(:read, model) } end end def destroy science "widget-destruction" do |e| e.use { old_scary_destroy } e.try { new_safe_destroy } end end def default_scientist_context { :widget => self } end end The widget-permissions and widget-destruction experiments will both have a :widget key in their contexts. Expensive setup If an experiment requires expensive setup that should only occur when the experiment is going to be run, define it with the before_run method: # Code under test modifies this in-place. We want to copy it for the # candidate code, but only when needed: value_for_original_code = big_object value_for_new_code = nil science "expensive-but-worthwhile" do |e| e.before_run do value_for_new_code = big_object.deep_copy end e.use { original_code(value_for_original_code) } e.try { new_code(