We started our "idea labs" to:
Help mentor / collaborate in the open
Serve some good
Kick the tyres on cool/interesting tech
With those in mind, our first lab was to create a "real-time analytics data-mesh" -- but what the chuff even is that?!?
Why did we choose this for our first lab?
As this was our first lab, we wanted to choose something with broad appeal for tech learners to have for their practical experience, but we weren't sure who would show up.
What if people are wanting to do cloud / platform stuff? Or front-end / graphics? What about back-end, architecture, or software engineering??? Whatever their interests, we wanted a broach church, as well as to show how all these pieces come together.
With this lab, there's something for everyone, as it uses:
a micro front-end (architecture)
using svelte (front-end, cool new tech ... about to release version 5)
on top of kubernetes (platform - great in-demand, portable stack)
using Apache Pinot (real-time streaming database, reached version 1.0 while we did this lab)
and Apache Kafka (excellent streaming tech, great for aspiring data-engineers)
What good does this do / what problem does it solve?
This lab is really two concepts in one:
1) a decoupled micro front-end platform
2) an example "real-time streaming" component widget for that platform
The micro front-end platform is just a way products can take advantage of simple, pluggable component "widgets". We separate out a general skeleton dashboard from the content displayed in that dashboard. The effect is small, separate "green-field" mini-products which can each have their own teams (or even single contributors) and release cycles dedicated to specific types of users.
This concept is familiar to financial traders who enjoy having dedicated "quants" who deliver exactly the information and signals the traders need to do their best work.
It addresses a number of problems...
Problem #1: Success leading to products creaking under their own weight
Successful products can often become victims of their own success. What once starts as a simple product which serves a distinct purpose can soon accrete features, becoming more complex both for its users and the engineers who have to maintain it.
Problem #2: Remaining agile while serving different "user personas"
Product teams often struggle prioritising who to please and who to disappoint, as different users often use technology for different purposes. It's not just "power users" vs "casual users", content creators vs consumers vs moderators or admins, or finance vs marketing vs creatives vs engineers vs anybody else.
People are diverse, and two people who may even have the same "persona" or be in the same role may want to view information or use software differently.
Having a single monolithic product forces people to make compromises and trade-offs. It also can significantly ("orders of magnitude" significantly) slow down the number of experiments we're able to try.
Problem #3: Maintaining the monolith
Even if there is a clear product steer, the technical people (designers, architects, engineers) behind the product will have to support a growing product. Features which were commissioned by someone who has since left the company are still there, confusing new users and complicating the code-base.
We get further away from "single responsibility principle", we have more "dependency hell" issues, builds get longer, and there is too much context/knowledge people need to make what should be trivial change, so innovation suffers.
Putting these together...
By separating out our product into widgets, we have small, single-responsibility components produced by teams optimised for flow and a supporting dashboard platform team (see team topologies).
We exploit concepts from biology and market economies by allowing popular widget components to thrive and unpopular ones to simply die of atrophy.
What is the "cool / interesting" tech?
Our labs take these ideas beyond an idea or blog post. We have some prototype code on our github repo at github.com/kindservices/idea-labs.
It's built on:
Kubernetes to exploit side-cars architecture and keep it portable
ArgoCD for excellent continuous deployment
Svelte for lovely, fast UI development
Scala for back-end services
Node and nginx in docker for serving the web components
web components - allowing people to create components in any web-component compatible technology or stack (react, vue, nextjs, etc)
Apache Kafka for streaming events in the example component
Apache Pinot for the OLAP database behind the simple graph components
Apache Pinot reached it's 1.0 milestone during this lab, and Svelte is releasing 5.0 with it's new "runes" concept shortly. These technologies may be too far to the early adopter extreme, so it was fun to be able to kick their tyres!
So ... What do YOU think?
We believe strongly in open-source and scrutiny, so not only welcome everyone's critiques, thoughts and criticism, but encourage them - and at any level. What's wrong with our premise? What have we neglected? How would you improve it?
Your comments, suggestions or involvement are most certainly welcome!
Equally, do you have an idea for a lab? Do you need guidance or help to ensure your organisation is making effective use of technology in a way which serves your users and teams? Or are you using or wanting to use these technologies effectively? There's a lot more beyond the prototype code to put projects like this into production, and we'd love to help if we can!
Yorumlar