Skip to Content

Backpack indexing for KiokuDB and Search::GIN Syndicate content

Dave Sherohman's picture

Call it loose coupling, application of encapsulation, avoiding god objects, plugin-based architecture, or whatever else, but I don't like having objects which carry out the functions of another class's objects. I believe that each class should contain the full measure of its functionality, so that adding it to a project is as simple as use MyClass; without requiring modification of any existing code for the new class to do its thing. (Existing code will often need to be modified to tell it to use the new class, of course, but that's not what I'm talking about here.)

Unfortunately, KiokuDB::Tutorial's example of using Search::GIN hardcodes knowledge of how the Person class should be indexed into the database ->connect call. I'm sure this was just done for the sake of showing the simplest possible example, but here's how I set things up so that my ->connects don't have to change when a new class or index is added.

My core concern here is that the class and the knowledge of how to index it should be self-contained, rather than spreading that information out across multiple locations. Traditionally, this might be accomplished via inheritance, but it's not really something that should be tied to any specific lineage of classes - any class could potentially be indexable. But there are also likely to be many classes which don't need to be indexed, so it would be overkill to put this into a root class from which all others derive. This makes it a perfect candidate for a Moose role:

role Lillon::Role::GINIndexing {
  # Returns a hashref of index_name => index_value pairs for GIN
  requires 'extract_index';

Nice and simple.

Now that we have our role, we can connect to the database with instructions to use that role to extract indexes from objects as they're saved, without having to tell the database anything about the objects being indexed:

sub init_db {
  my $dsn = shift;

  confess 'DSN is missing!' unless $dsn;

  require Search::GIN::Extract::Callback;
  require Search::GIN::Extract::Class;
  require Search::GIN::Extract::Multiplex;

  my $dir = KiokuDB->connect( $dsn, create => 1,
    extract => Search::GIN::Extract::Multiplex->new(
      extractors => [
          extract => sub {
            my ( $obj, $extractor, @args ) = @_;
            return $obj->extract_index($extractor, @args)
              if $obj->does('Lillon::Role::GINIndexing');

  return $dir;

Note that, in addition to each class's custom indexing provided by the role I defined, I'm also using Search::GIN::Extract::Class so that I'll be able to look up objects by class in addition to whatever information they index about themselves. And that means I also have to pull in Search::GIN::Extract::Multiplex, as ->connect only appears to accept a single extract argument directly.

Finally, the last piece in order to make this work is to define a class which consumes our indexing role:

class Lillon::User with Lillon::Role::GINIndexing {
method extract_index {
  return {
    last_active => $self->last_active,


The User class will now be indexed on its last activity timestamp, while also remaining completely self-contained.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Very cool. I've been exited

Anonymous's picture

Very cool. I've been exited to give KiokuDB + Search::GIN a try. Thanks for the incentive. :-)

Happy to help!

Dave Sherohman's picture

Happy to help!

Side note: If you're going to try my extraction code, you may need to grab the latest Search::GIN from github. Someone else was having trouble using Extract::Multiplex with Extract::Class on #kiokudb last night; it was a problem I'd previously encountered and I think updating Search::GIN was what fixed it.

Also be sure you've got the latest (0.08 / Nov 27) KiokuDB::Backend::DBI. Version 0.07 had a bug where it didn't delete old GIN indexes, so objects were being indexed with every value they'd ever had rather than only the current value.