DFK Indexer Engine

Grant Proposal for DFK Indexer

Description

I am nearing completion of a DFK specific blockchain indexer as it’s become a necessity for many of the projects and ventures I participate in.

After Brian’s proposal for a DFK Community Extended API I got inspired and thought I’d make the effort for a grant proposal in order to open source my indexer.

Indexer Goals and Requirements

  • Transactionally walk through the blockchain, marking each block that has been successfully indexed.
  • Focus on events emitted by DFK contracts as a more elegant and targeted indexing operation.
  • Index multiple chains concurrently and independently.
  • Index in bulk ranges (from X block till last block mined).
  • Index in real-time (as blocks are mined).
  • Assymetric indexing between indexers (contracts get deployed on different blocks - allow for indexing them appropriately).
  • Easy addition of new indexers as new functionality lands from DFK.
  • Robust operation with very high degree of error and issue handling.
  • Elaborate logging and reporting infrastructure, as well as performance measuring.
  • Indexers should be able to handle being run out of sequence (i.e. index block 100, then index block 80).
  • Indexers should be able to handle being run twice or more on the same block.
  • Test coverage that exceeds 93%.
  • CI/CD of builds and all contributions.

Implementation Details

  • Node.js platform, vanilla JS
  • Extreme degree of static analysis tooling for quality control.
  • Postgres database.
  • Command line execution & arguments.
  • Server mode operation (real-time indexing).

Indexers Implemented - First Major Release

The first implementation will produce the following indexers:

  • Heroes Up to date representation of the heroes model across all chains.
  • Auction Sales Index auction sales events across all chains.
  • Auction Rent Up to date rental market, all chains.
  • Summoning Index both the summoning and the opening of crystals, all chains.

Indexers Implemented - Second Major Release

At a later stage, the indexer will perform more complicated indexing operations as well as add all the additional parts of the game:

Complicated Indexing Operations

  • Heroes historical state Being able to answer what were a hero’s stats at a specific point in time.
  • Historic Auction Sales Plex heroes historical states with auction sales to produce a dataset that includes the hero sale event as well as the state of the hero at the time.
  • Historical Auction Rents As per sales.
  • Historical Quest Rewards Quest rewards accross all quests (including TQs) with the state of the hero at the time of the quest completion.
  • Historical Summons Plex the state of the summoner and assistant heroes at the time of the summon.

Rest of Indexers

The 3 or 4 more popular and requested indexers on game functions will be chosen based on community feedback (i.e. should we implement stamina potion consumption or level up events or pet hatching, etc?).

At about that time, the indexer being open source, well documented and tested and with enough indexers implemented, we can then expect the contributions to start comming in with more detailed and purpose built indexers.

Grant Rationale

A “go-to” indexer is the building block for any and all DFK applications, whether they are a tavern equivalent, a tax report website or a summoning calculator.

It will greatly expedite and proliferate the development of applications for the DFK ecosystem and lower the barrier to entry.

I am confident this indexing engine also has the potential to be used in-house from DFK to further the functionality of their operations and game.

Grant Amount

50K USD for the development and maintenance of the DFK Indexing Engine.

Payment Schedule

Funds will be used for the compensation of development.

Team Information

Hi, I am Thanos, my short bio is bellow, but before that, I’d like to mention why I am particularly well suited to perform this task:

  • I have been doing Open Source software for nearly 15 years to date.
  • I have honed my documentation and testing skills to a very high degree, producing packages that delight Developers and are focused on their experience.
  • In late 2021, I developed a multi-blockchain indexing service when building what is now the Core of [vfat v2.0][vfat] (to be released soon-ish).

Thanos’ Short Bio

Thanos Polychronakis (thanpolas): I am a family man in my 40s and I am a seasoned engineer. I have “retired” to web3 after a fulfilling and rewarding career in Web2. During my career I helped navigate startups through brutal hardships and held the keys to kingdom come of many businesses and dreams.

My Open Source work exceeds 200 repositories on Github and more than 50 published packages on NPM. I am also the author of an open source library for DFK, that is the scaffolding of an SDK.

I am a moderator on the (third-party) DFK Developers discord and I have received grants for my Open Source and private contributions from Uniswap, Synthetix, Balancer and AAVE.

Today, I work full-time producing tools, services and content for DFK.

I am fully doxed. If you google me, which I recommend you do, you will find dozens and dozens of pages of my community contributions on every imaginable setting (open source, meetups, conferences, etc). It’s just what I love doing, helping people. Start here: https://thanpol.as/.

Requests from DFK Team

Nothing much, this is a “read-only” operation and I have everything I need.

Deliverables, Timeline and Milestones

Deliverables as listed on:

Long Term Plan

  • Upkeep and maintain the indexing engine.
  • Process and accept incoming contributions (PRs).
  • Contribute myself with new indexers as new features come out.

External Links

N/A, Indexing Engine is currently being developed as proprietary software.

10 Likes

Nice work Thanos. This would be such a boon to all builders and could unlock many capabilities/features for the community. It also has much potential beyond DFK.

1 Like

I really like the capturing of sale price data. that has been challenging for us to work with over at ADFK. This would be an enormous help! love the project idea.

Love the concept and I agree that this would benefit many builders! My question though would be where does the indexed data go? Into a database? If so, what schema? This info could be useful to determine the usability of your indexer. Thanks!

Hey 9dorf, thank you for the question, give me the opportunity to clarify that this proposal is for open sourcing the engine.

It is not about providing a service or an infrastructure. This is addressed by @bgeihsgt’s proposal: DFK Community Extended API with whom I look forward to working together on making the service available, powered by the indexer of this proposal.

To be more specific around how this will work, it will be a standalone node.js application. You download it, install the NPM dependencies and configure it according to instructions in the README.

Once you have configured credentials to your database and run the application, it will automatically create the tables and schemas and start operating according to the issued command.

I am also considering providing pre-indexed data, in the form of data dumps so as to get a head-start on indexing and not having to re-index the whole chain from the start.

Hope this answers your question.

2 Likes