So say you have a chat room and only the owner of a message is allowed to delet its message.

-# app/views/messages/_room.html.haml

= turbo_stream_from dom_id(current_user)
= turbo_stream_from dom_id(room)
- room.messages.each do |message|
  = render partial: 'messages/message', locals: {message: message, owner: message.user == current_user}

Here we have two streams one for the current user, which is used to add the delete link, and the one for the chat room itself to add the newly create message.

The chat message partial looks like.

-# app/views/messages/_message.html.haml

= message.initials
= message.username
= I18n.l message.created_at, format: :short
= raw message.message

= turbo_frame_tag [dom_id(message), 'delete'] do
  - if owner
    = render partial: 'messages/delete', locals: {message: message}

Now then somebody sends a message we using in the model callback the message user to send also the delete link to the owner.

# app/models/message.rb

class Message < ApplicationRecord
  belongs_to :room, counter_cache: true
  belongs_to :user

  after_create_commit :create_broadcast

  def create_broadcast
    broadcast_append_to room,
                        partial: 'messages/message',
                        locals: {message: self, owner: false}

    broadcast_replace_to "user_#{user.id}",
                         target: "message_#{id}_delete",
                         partial: 'messages/delete',
                         locals: {message: self}
    end
  end
end

The trick is to send the delete broadcast to the user stream, with the delete target defined by the turbo frame above.

It took a while for me to find it out, how to make us of it in a Rails 8 application, together with propshaft and importmaps.

At the end it was quite easy, simply add following:

# config/initializers/assets.rb
Rails.application.config.assets.paths << Rails.root.join("app/components")
Rails.application.config.importmap.cache_sweepers << Rails.root.join("app/components")

# config/importmap.rb
pin_all_from "app/components", under: "controllers", to: ""

Here some lessons that I learned.

Brew

⚠️ Do this only, if you don’t need “legacy” apps.

What do I mean: In my case, I need use rbenv to manage my rubys, in one project an oracle access is required, will be done with ruby-oci8. This gems needs the Oracle Instantclient, but this one is only for Mac Intel available.

on your old machine do:

• save your delevopment data, like PG databases
• list all installed formulaes and save the Brewfile with

  brew bundle dump
  

• remove Brew completely with

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"
  

on your new machine:

• ensure you have the xcode command line tools installed
• install brew, see above, but now install.sh
• reinstall your formulas with

  brew bundle
  

I did it … 🤬

to fix it had to do following steps

1. go in Finder to the Termninal and open “Information” (⌘+i)
2. check “open with Rosetta” (or similar)
3. install homebrew but now with given architecture

   arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
   

4. and reinstall your formulas as above

How to use Mx Apps?

You have to create the folder structure under /opt, see here Untar anywhere

And specify the architecture then, e.g.

arch -arm64 brew install tiles

or even better, create your own alias … 😉

… to be continued

In this process we found some unexpected solutions, for existend behaviour. Unexpected because they are not documented. Here we want to share these finding.

Findings

1 Mixing

According to the Searchkick documentation, you have several possibilities to influence the search behavior, by specifying the analyzer to use, so for our example with

class Product < ApplicationRecord
  searchkick text_middle: [:name],
             word_start: [:content]
end

and use then the same setting in the match field … 🤔

But we wanted to search for both fields at once. This was not documented, so we had to dive into the code (→ idea), where the field option will be evaluated, so it seems an Array of Hashes will be accepted.

The solution is, to provide for each field the appropiated analyzer.

Product.search query, fields: [
  { 'name^3': :text_middle }, # use togehter with boost
  { content: :word_start },
]

By using this method, avoid using the match key, cause it overwrites above analyzers.

2 Settings

Above we saw how to use the options. But what are these options? That are shortcuts for predefined Anlayzers, to use for processing the String/Text on indexing.

One can see, that each of them is built up the three keys type, tokenizer and filter. It specifies the Analysis, which should be applied to this field.

Another finding are the sizes of the min_gram and max_gram. They are good for most cases, as are all predefined analyzers. But we wanted to change it to our needs, cause our content can be a very long description, so using a max_ngram of 50, will result in a very big index, without improving the search quality itself.

So we have to adapt it to our needs, and yes you guessed it … 😉 But the settings are used, so it must be possible to specify them as well. With a little trial and error, we found it

Searchkick.model_options = {
  settings: {
    analysis: {
      filter: {
        searchkick_edge_ngram: { type: 'edge_ngram', min_gram: 1, max_gram: 17 },
        searchkick_ngram: { type: 'ngram', min_gram: 1, max_gram: 17 }
      }
    },
    index: {
      max_ngram_diff: 23
    }
  }
}

But what, if we want to define our own analyzer?

It must be said that this is not recommended, unless you know exactly what you are doing, because as already said, the available ones are very good and sufficient for most cases.

Ok, one can not add own ones to the predefined analyzer list. But one can overwrite an existend one … for example changing the above text_middle analyzer

Searchkick.model_options = {
  settings: {
    analysis: {
      analyzer: {
        searchkick_text_middle_index: {
          type: 'custom',
          tokenizer: 'whitespace',
          filter: %w[lowercase snowball_german_umlaut unique name_ngram]
        },
      },
      filter: {
        name_ngram: { type: 'ngram', min_gram: 1, max_gram: 3, preserve_original: true },
        snowball_german_umlaut: { type: 'snowball', name: 'German2' },
      }
    }
  }
}

it doesn’t make so much sense, but you get the idea. For possible Analyzer options refer to the Analysis documentation.

By the way, it also works with OpenSearch, but actual the ES documentation, especially the Lucene related one is much better.

Don’t forget to reindex each time, after one of the obove settings are changed!

This requires mostly the knowledge of the count of items, so the OFFSET can be calculated correctly.

In following some log output for the count and select statements, results table contains ~4,5M rows …

(129.8ms)  SELECT COUNT(*) FROM "results"
…
Result Load (0.4ms)  SELECT "results".* FROM "results" LIMIT $1 OFFSET $2  [["LIMIT", 100], ["OFFSET", 0]]

one can see that COUNT statement needs multiple times more then the SELECT statement.
But do we need every time we are fetching the next span of data a new COUNT?

Think, NO.

Separate endpoints for count and data fetching

Add another endpoint for getting the count only when its needed, for example on page load.¹

get 'count' do
  results = ::Result.filter(params)

  { count: results.count }
end

get 'filter' do
  limit = params[:limit] || 500
  offset = params[:offset] || 0
  results = ::Result.filter(params).limit(limit).offset(offset)

  { items: results }
end

Ok this results in an decreased count of executing of the count statement, cause of hitting the count endpoint only, when parameters are changed, but doesn’t speed it up.

Estimate

By thinking and researching about, what can be improved, I found this inspiring post count-performance.

Ok, why not … the exact count isn’t really needed, so I decided to give it a try.

The Function

First we have to add a SQL function, which does the estimation. For that, the function from the post above will be packed into a migration.

class CreateEstimateFunction < ActiveRecord::Migration[6.0]
  def up
    function = "CREATE FUNCTION count_estimate(query text) RETURNS integer AS $$
      DECLARE
        rec   record;
        rows  integer;
      BEGIN
        FOR rec IN EXECUTE 'EXPLAIN ' || query LOOP
          rows := substring(rec.\"QUERY PLAN\" FROM ' rows=([[:digit:]]+)');
          EXIT WHEN rows IS NOT NULL;
        END LOOP;


        RETURN rows;
      END;
      $$ LANGUAGE plpgsql VOLATILE STRICT;"

    ActiveRecord::Base.connection.execute(function)
  end
end

Remember the name of the function (here: count_estimate) for later usage.

Now we have the function place, we should make usage of it. So we are changing the endpoint into …

get 'count' do
  count = ::Result.counting(params)

  { count: count }
end

to call the new model method. Note, that it also take the same parameters as for filter endpoint.

def self.filter(params)
  # filter implementation
  # returns an ActiveRecord::Relation object
end

def self.counting(params)
  query = if <search params given?>
            insert = ActiveRecord::Base.connection.quote(filter(params).to_sql)
            search_total_query(insert)
          else
            total_query
          end

  result = ActiveRecord::Base.connection.exec_query(query)
  result.rows.flatten.first.to_i
end

def self.search_total_query(query)
  "SELECT count_estimate(" + query + ");"
end

# helper methods
def self.total_query
  "SELECT
    (reltuples/relpages) * (pg_relation_size('results') /
    (current_setting('block_size')::integer)) AS count
  FROM pg_class where relname = 'results';"
end

To be consequent, we will use both estimations and decide via search params given?, which one to choose. This is useful if want to make search queries which are decreasing the result space, so the estimation want not differ too much.

For that the value of the filter(params) method have to be an ActiveRecord::Relation object, so that we can reuse it here to generate the SQL string.

First some comparisions:

The difference is evident.

All was fine, cause the data once imported will never change, only one thing to have in mind is, refreshing the view after data import. Naming the model in this example Result ends up in this worker:

class ResultRefreshJob
  include Sidekiq::Worker
  sidekiq_options queue: :views, retry: false

  def perform
    Result.refresh
  end
end

it calls the refresh method of our materialized view as it is documented here.

All fine, but some months later, a feature request comes in, to add another column. Sounds after a low-hanging-fruit, but these data can occur 0..n times and it ends up in a flatten pivot table — presenting it as pivot table was not wanted —, means, the data grows exponential with count of entries. And a second problem was, the data can be changed, very quickly.

Its naïve to think the above solution can handle the boost of jobs, which blocking itself.

Goal

So the count of running jobs must be minimised
and it have to be ensured the view is refreshed.

I came up with a second job, which checks if refresh jobs are running, if not it enqueues a new one, and if one exists, an error will be raised and it will be enqueued again.

It looks like following:

class ResultEnqueueJob
  include Sidekiq::Worker
  sidekiq_options queue: :views, retry: 3

  # TODO: this should be observed and maybe adapted
  def perform
    running_worker = Sidekiq::Workers.new.map(&:last).select { |x| x['queue'] == 'views' }.last
    can_run = if running_worker
                run_at = DateTime.strptime("#{running_worker['run_at']}", '%s')
                run_at < DateTime.current - 1.minutes
              else
                true
              end

    if can_run
      ResultRefreshJob.perform_async
    else
      raise StandardError, "---> another ResultRefreshJob is running"
    end
  end
end

ok, looks quite simple, but what’s the trick here?

We have here 3 parameters, number one the retry count (rc), number two the timespan we go back (ts) and number three, the unknown one, we can only estimate and observe, the time to rebuild the view (tr).

The actual values are tc = 3, ts = 1min and we estimate tr to round 2min, now we have to optimize these values, to met the above goal and to adapt it on changes.

Have in mind, that the time to retry will exponential growing, see: automatic job retry.

Example:

t_x : job₁ checks if can start → no job running, starts
t_x+ts : job₂ checks if can start → job₁ is running, can not
t_x+ts+30s : job₂, checks again → job₁ is running, can not
t_x+ts+30s+46s : job₂, checks again → job₁ finished, starts

For the example here an API skeleton will be build with grape-starter, so in the result you have a running API app. There are two namespaces created, one for simple arrays and on for arrays of objects. For both POST and PUT endpoints will be implemented.

One thing to emphasize is the param_type, it is set to body, this is needed for JSON requests. If it will not be specified, a FormData parameter will be expected and documented.

Array of simple Datatypes

The implementation self:

namespace :arrays do
  params do
    optional :names, type: Array[String],
             documentation: { param_type: 'body' }
  end
  post do
    present :names, params[:names]
  end

  params do
    requires :id
  end
  route_param :id do
    params do
      optional :names, type: Array[String],
               documentation: { param_type: 'body' }
    end
    put do
      present :names, params[:names]
    end
  end
end

usage

• POST request to: /arrays
• PUT request to: /arrays/1

with following JSON body:

{
  "names": [
    "a", "b"
  ]
}

Array of JSON objects

The implementation self:

namespace :complex do
  params do
    requires :addresses, type: Array[JSON],
             documentation: { param_type: 'body' } do
               requires :street
               requires :city
               requires :code
             end
  end
  post do
    present :addresses, params[:addresses]
  end

  params do
    requires :id
  end
  route_param :id do
    params do
      requires :addresses, type: Array[JSON],
               documentation: { param_type: 'body' } do
                 requires :street
                 requires :city
                 requires :code
               end
    end
    put do
      present :addresses, params[:addresses]
    end
  end
end

usage

• POST request to: /complex
• PUT request to: /complex/1

with following JSON body:

{
  "addresses": [
    {
      "street": "some street a",
      "city": "city a",
      "code": "12345"
    }, {
      "street": "another street",
      "city": "city b",
      "code": "67890"
    }
  ]
}

The repo can be found here.

For Rails developers, there is a nice gem – pg_search – out there, it does all what is needed for it. But this is deeply bundled to Rails itself.

So I had to do it by hand, means:

1. Understanding what it is and how is done in PG.
2. How can this be integrated into my App?

For 1.

… this post full-text-search-ruby-rails-postgres is very informative for the beginning and understanding of the basics of the topic.

For 2.

For my app, it means a simple grape API, for the building of it grape-starter will be used. There I use ActiveRecord for accessing the database.

So that was the setup …

At the end of it, a model file exists, where the search will be implemented¹.

def self.search(query)
  find_by_sql([sql, query, query])
end

def self.sql
  <<-SQL.strip_heredoc
    SELECT
      id,
      data,
      created_at,
      updated_at,
      ts_rank(
        to_tsvector(',language', 'data'),
        to_tsquery(',language', '<query>')
      ) AS rank
    FROM zettel
    WHERE
      to_tsvector(',language', 'data') @@
      to_tsquery(',language', '<query>')
    ORDER BY rank DESC
    LIMIT 3
  SQL
end

1. Ensure you have a folder _drafts.
2. Run bin/new_post.rb <Title, with whitespaces> this generates a Markdown file with name <2020-10-10-Title,_with_whitespaces>.md in _drafts folder, if exists, else in project folder.
3. To run it local bundle exec jekyll serve --draft or bin/run
4. If you are ready/finishing writing, move it into to _posts folder to publish.
5. push the changes