GUILD-690: Initial draft of new Active Record Consumer Generator

CHANGELOG.md

@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## UNRELEASED
 
+# 1.18.3 - 2022-11-16
+
+### Features :star:
+
+-  Added ActiveRecordConsumerGenerator, which streamlines the process into a single flow
+-  Creates a database migration, Rails model, consumer, consumer config, and Deimos schema classes
+
 # 1.18.2 - 2022-11-14
 
 -  Fixes bug related to wait between runs when no records are fetched by updating poll_info

README.md

@@ -22,6 +22,7 @@ Built on Phobos and hence Ruby-Kafka.
         * [Instrumentation](#instrumentation)
         * [Kafka Message Keys](#kafka-message-keys)
    * [Consumers](#consumers)
+        * [Active Record Consumer Generator](#active-record-consumer-generator)   
    * [Rails Integration](#rails-integration)
         * [Controller Mixin](#controller-mixin)
    * [Database Backend](#database-backend)
@@ -309,6 +310,49 @@ class MyConsumer < Deimos::Consumer
 end
 ```
 
+## Active Record Consumer Generator
+
+Deimos provides a generator that streamlines Deimos consumer creation. It
+creates all the necessary components needed in a Ruby on Rails application when 
+creating a new Active Record Consumer. 
+
+It will take an existing schema and generate the following files based on its fields:
+
+    Database Migration          
+    Rails Model                 
+    Consumer Class              
+    Deimos Consumer Config 
+    Generated Schema Classes     
+
+
+By default, any complex sub-types (such as records or arrays) are turned into JSON 
+(if supported) or string columns.
+
+Before running this generator, you must first copy the schema into your repo in the 
+correct path.
+
+To generate a migration, model, consumer class, consumer config and schema classes, run the following:
+
+Usage:
+
+    rails g deimos:active_record_consumer FULL_SCHEMA_NAME [CONFIG_FILE_PATH]
+
+    Options are...
+            FULL_SCHEMA_NAME            (required) Fully qualified schema name, located in config.schema.path that defines the Avro schema for the Consumer.
+            CONFIG_FILE_PATH            (optional) Path to existing config file. If not specified, defaults to config/initializers/deimos.rb.
+
+Example:
+
+    rails g deimos:active_record_consumer com.my-namespace.Widget
+
+...would generate:
+
+    db/migrate/20221111134112_create_widgets.rb
+    app/models/widget.rb
+    app/lib/kafka/models/widget_consumer.rb
+    config/initializers/deimos.rb
+    app/lib/schema_classes/**/*.rb
+
 ### Fatal Errors
 
 The recommended configuration is for consumers *not* to raise errors

lib/generators/deimos/active_record_consumer/USAGE

@@ -0,0 +1,23 @@
+Description:
+    Streamlines Deimos consumer creation. Creates all the necessary components needed in a Ruby on Rails application when creating a new Active Record Consumer.
+
+Usage:
+    bin/rails generate deimos:active_record_consumer FULL_SCHEMA_NAME [CONFIG_FILE_PATH]
+
+    Options are...
+        FULL_SCHEMA_NAME            (required) Fully qualified schema name, located in config.schema.path that defines the Avro schema for the Consumer.
+        CONFIG_FILE_PATH            (optional) Path to existing config file. If not specified, defaults to config/initializers/deimos.rb.
+
+Example:
+    bin/rails generate deimos:active_record_consumer com.test.Widget config/initializers/myconfig.config
+
+    This will create:
+        Database Migration          db/migrate/20221111134112_create_widgets.rb
+        Rails Model                 app/models/widget.rb
+        Consumer Class              app/lib/kafka/models/widget_consumer.rb
+        Deimos Consumer Config      config/initializers/deimos.rb or adds config to CONFIG_FILE_PATH
+        Generated Schema Classes    app/lib/schema_classes or config.schema.generated_class_path
+
+
+
+

lib/generators/deimos/active_record_consumer/templates/config.rb.tt

@@ -0,0 +1,29 @@
+
+Deimos.configure do
+  logger Logger.new(STDOUT)
+  # Nested config field
+  kafka.seed_brokers ['my.kafka.broker:9092']
+
+  schema.generate_namespace_folders true
+  schema.generated_class_path 'app/lib/schemas'
+
+  # Multiple nested config fields via block
+  consumers do
+    session_timeout 30
+    offset_commit_interval 10
+  end
+
+  # Define a consumer
+  consumer do
+    class_name '<%= consumer_name.classify %>'
+    topic 'TopicToConsume'
+    schema '<%= schema %>'
+    namespace '<%= namespace %>'
+    key_config field: :<%= key_field %>
+    # include Phobos / RubyKafka configs
+    start_from_beginning true
+    heartbeat_interval 10
+    use_schema_classes true
+  end
+
+end

lib/generators/deimos/active_record_consumer/templates/consumer.rb.tt

@@ -0,0 +1,45 @@
+class <%= consumer_name.classify %> < Deimos::ActiveRecordConsumer
+    record_class <%= model_name.classify %>
+
+  # Optional override of the way to fetch records based on payload and
+  # key. Default is to use the key to search the primary key of the table.
+  # Only used in non-batch mode.
+  def fetch_record(klass, payload, key)
+    super
+  end
+
+  # Optional override on how to set primary key for new records.
+  # Default is to set the class's primary key to the message's decoded key.
+  # Only used in non-batch mode.
+  def assign_key(record, payload, key)
+    super
+  end
+
+  # Optional override of the default behavior, which is to call `destroy`
+  # on the record - e.g. you can replace this with "archiving" the record
+  # in some way.
+  # Only used in non-batch mode.
+  def destroy_record(record)
+    super
+  end
+
+  # Optional override to change the attributes of the record before they
+  # are saved.
+  def record_attributes(payload, key)
+    super.merge(:some_field => 'some_value')
+  end
+
+  # Optional override to change the attributes used for identifying records
+  def record_key(payload)
+    super
+  end
+
+  # Optional override, returns true by default.
+  # When this method returns true, a record corresponding to the message
+  # is created/updated.
+  # When this method returns false, message processing is skipped and a
+  # corresponding record will NOT be created/updated.
+  def process_message?(payload)
+    super
+  end
+end

lib/generators/deimos/active_record_consumer/templates/migration.rb.tt

@@ -0,0 +1,30 @@
+class <%= migration_class_name %> < ActiveRecord::Migration<%= migration_version %>
+  def up
+    if table_exists?(:<%= table_name %>)
+      warn "<%= table_name %> already exists, exiting"
+      return
+    end
+    create_table :<%= table_name %> do |t|
+      <%- fields.each do |key| -%>
+      <%- next if %w(id message_id timestamp updated_at created_at).include?(key.name) -%>
+      <%- sql_type = schema_base.sql_type(key)
+         if %w(record array map).include?(sql_type)
+           conn = ActiveRecord::Base.connection
+           sql_type = conn.respond_to?(:supports_json?) && conn.supports_json? ? :json : :string
+         end
+      -%>
+      t.<%= sql_type %> :<%= key.name %>
+      <%- end -%>
+
+      t.timestamps
+
+      # TODO add indexes as necessary
+    end
+  end
+
+  def down
+    return unless table_exists?(:<%= table_name %>)
+    drop_table :<%= table_name %>
+  end
+
+end

lib/generators/deimos/active_record_consumer/templates/model.rb.tt

@@ -0,0 +1,5 @@
+class <%= table_name.classify %> < ApplicationRecord
+<%- fields.select { |f| f.enum_values.any? }.each do |field| -%>
+  enum <%= field.name %>: {<%= field.enum_values.map { |v| "#{v}: '#{v}'"}.join(', ') %>}
+<% end -%>
+end

lib/generators/deimos/active_record_consumer_generator.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+require 'rails/generators'
+require 'rails/generators/active_record/migration'
+require 'generators/deimos/schema_class_generator'
+require 'rails/version'
+require 'erb'
+
+# Generates a new Active Record Consumer, as well
+# as all the necessary files and configuration.
+# Streamlines the creation process into a single flow.
+module Deimos
+  module Generators
+    # Generator for ActiveRecordConsumer db migration, Rails model, Consumer, Consumer config
+    # and Deimos Schema class
+    class ActiveRecordConsumerGenerator < Rails::Generators::Base
+      include Rails::Generators::Migration
+      if Rails.version < '4'
+        extend(ActiveRecord::Generators::Migration)
+      else
+        include ActiveRecord::Generators::Migration
+      end
+      source_root File.expand_path('active_record_consumer/templates', __dir__)
+
+      argument :full_schema, desc: 'The fully qualified schema name.', required: true
+      argument :config_path, desc: 'The path to the deimos configuration file, relative to the root directory.', required: false
+
+      no_commands do
+
+        # Creates database migration for creating new table
+        def create_db_migration
+          migration_template('migration.rb', "#{db_migrate_path}/create_#{table_name.underscore}.rb")
+        end
+
+        # Creates Rails Model
+        def create_rails_model
+          template('model.rb', "app/models/#{model_name}.rb")
+        end
+
+        # Creates Kafka Consumer file
+        def create_consumer
+          template('consumer.rb', "app/lib/kafka/models/#{consumer_name.underscore}.rb")
+        end
+
+        # Adds consumer config to config file.
+        # Defaults to deimos.rb if config_path arg is not specified.
+        def create_consumer_config
+          config_file = 'deimos.rb'
+          config_file_path = "#{initializer_path}/#{config_file}"
+          config_file_path = config_path if config_path.present?
+
+          if File.exist?(config_file_path)
+            config_template = File.expand_path(find_in_source_paths('config.rb'))
+            insert_into_file(config_file_path.to_s, CapturableERB.new(::File.binread(config_template)).result(binding))
+          else
+            template('config.rb', config_file_path.to_s)
+          end
+        end
+
+        # Generates schema classes
+        def create_deimos_schema_class
+          Deimos::Generators::SchemaClassGenerator.start
+        end
+
+        # @return [String]
+        def db_migrate_path
+          if defined?(Rails.application) && Rails.application
+            paths = Rails.application.config.paths['db/migrate']
+            paths.respond_to?(:to_ary) ? paths.to_ary.first : paths.to_a.first
+          else
+            'db/migrate'
+          end
+        end
+
+        # @return [String]
+        def migration_version
+          "[#{ActiveRecord::Migration.current_version}]"
+        rescue StandardError
+          ''
+        end
+
+        # @return [String]
+        def table_class
+          table_name.classify
+        end
+
+        # @return [String]
+        def schema
+          last_dot = self.full_schema.rindex('.')
+          self.full_schema[last_dot + 1..-1]
+        end
+
+        # @return [String]
+        def namespace
+          last_dot = self.full_schema.rindex('.')
+          self.full_schema[0...last_dot]
+        end
+
+        # @return [Deimos::SchemaBackends::Base]
+        def schema_base
+          @schema_base ||= Deimos.schema_backend_class.new(schema: schema, namespace: namespace)
+        end
+
+        # @return [Array<SchemaField>]
+        def fields
+          schema_base.schema_fields
+        end
+
+        # @return [String]
+        def table_name
+          schema.tableize
+        end
+
+        # @return [String]
+        def model_name
+          table_name.underscore.singularize
+        end
+
+        # @return [String]
+        def consumer_name
+          "#{schema.classify}Consumer"
+        end
+
+        # @return [String]
+        def initializer_path
+          if defined?(Rails.application) && Rails.application
+            paths = Rails.application.config.paths['config/initializers']
+            paths.respond_to?(:to_ary) ? paths.to_ary.first : paths.to_a.first
+          else
+            'config/initializers'
+          end
+        end
+
+        # @return [String] Returns the name of the first field in the schema, as the key
+        def key_field
+          fields.first.name
+        end
+      end
+
+      # desc 'Generate necessary files and configuration for a new Active Record Consumer.'
+      # @return [void]
+      def generate
+        create_db_migration
+        create_rails_model
+        create_consumer
+        create_consumer_config
+        create_deimos_schema_class
+      end
+    end
+  end
+end