Personal Knowledge Graphs
A new generation of note-taking tools helps us quickly organize thoughts as knowledge graphs
Last year, my longtime friend Arun Batchu introduced me to the new generation of software for creating Personal Knowledge Graphs (PKG). Although the concept of a PKG goes back to 2019, the term has recently become popular because of a new generation of note-taking software that uses knowledge graph representations.
This blog will introduce you to PKGs and then discuss how PKGs might impact your overall corporate knowledge management strategy and how they might eventually work with Enterprise Knowledge Graphs.
Spoiler alert: I am hooked!
What is a Personal Knowledge Graph?
A PKG is a new class of software that allows you to efficiently take notes in the form of flexible non-linear knowledge graphs. PKGs have evolved from older linear note-taking and outlining software. PKGs are quickly gaining popularity among students, researchers, software developers, bloggers, and creative content authors.
A Brief History of Note-taking using Linked Notecards
Back in the 1500s, researchers in Germany developed a knowledge capture system called Zettelkasten. It was essentially a way of putting ideas on index cards that showed how ideas or concepts were related to other concepts. The ideas around One-Concept-Per-Card and Concept-Linking developed in Zettlekasten were then transferred to electronic form in the 1960s through the concept of HyperMedia, where any part of a document could be linked to any other part of that document or any other document. HyperMedia evolved into hypertext which became the basis of the Hypertext Markup Language (HTML). In 2001, the authors of the worldwide web (Tim Berners-Lee) also attempted adding relationship types to these links, which became the basis for the Semantic Web. The semantic web was an attempt to make it easier to store distributed but still connected information on web pages.
In parallel to the evolution of concept linking, we saw the growth of the personal note-taking software industry. These electronic tools, such as outlining tools, mind-mapping tools, and cross-platform tools such as Evernote, allowed individuals to create and retain their notes on many different devices such as their cell phones and tablets. These disparate devices were all tied to a cloud-connected storage system so all your notes on all your devices could stay in sync.
The Rise of Markdown for Knowledge Capture and Knowledge Interchange
Although these note-taking tools were handy, they lacked a simple way to capture and interchange linked knowledge between systems. Markup syntax such as HTML is portable, yet most users didn’t want to capture their notes in hypertext because it required too much work to format things such as links to other concepts. What was needed was a “lightweight” version of HTML that was easier to type in.
In 2004 the Markdown format was created to be just that. A lightweight way for users to quickly enter knowledge without requiring them to create highly structured markup. Any text editor such as Notepad can be used to create markup, and a few hundred lines of Python code are all that is needed to convert Markdown into HTML. Since 2004 Markdown has steadily grown to be the number one way software engineers and data scientists, create documentation about their code. It is now the default way to capture documentation on GitHub, in Data Science with Jupyter Notebooks, and create project documentation with Microsites. Even traditional products such as Google Docs are adding Markdown features.
Adding “smart autocomplete” into IDEs such as VisualStudio makes it easier to create and edit markup. Want to point to an image in your repo? Just type a “ space. It started with innovative projects such as Roam Research and the open-source Obsidian product.
Why Are Personal Knowledge Graphs So Popular? Second brain theory.
The amount of new information about almost any field of study grows exponentially. Doing a simple Google search on a topic is like drinking out of a firehose — it is hard to figure out what is essential in this continual stream of incoming information. Here are a few common reasons people tell me they love their PKGs.
Advantage 1: Quickly Organize Thoughts
PKGs allow you to quickly organize the information coming out of these streams to figure out what is important, make new connections between existing topics and create your insights. You can import any text, and just by highlighting a word or phrase and typing “[[“it magically turns into a vertex in your concept graph! As you type, the “autocomplete” can tell you if you already have it in your PKG, and you can thus quickly link items together.
Advantage 2: Lowers Your Cognitive Load
Before using PKGs, I would frequently get stressed during and after my constant back-to-back meeting schedules. I rarely had time between meetings to write down my thoughts and try to re-prioritize tasks as I learned new content. Now I can both take notes during meetings and link the notes to my existing knowledge base. My stress levels have come way down!
The problem is that our brain’s short-term memory can only hold a fixed number of things. If we don’t get them committed to documents, they disappear. I recall I neglected to write down these new ideas only days or weeks later.
Many of these are related to the “offloading” of the task of remembering concepts and their relationships to another system. These are often referred to as the “Second Brain” arguments on why PKGs are useful. Your PKG becomes a “persistent shadow” of your real brain — it stores the key concepts and their relationships in a non-linear structure — a graph — similar to how our brain works.
Advantage 3: Promotes Non-linear Thinking
The third thing I see often is that PKGs help us break away from the “Tyranny of Linear Thinking.” The order of traversal is usually fixed. This contrasts with traditional note-taking using documents that contain a fixed order of items lists. Document order requires us to have a series of sections and subsections.
But the inherent order you encounter information while taking notes might not be the real way that you have insights into the world. Outlining tools can allow you to rearrange the order of topics and pop down and pop up the order of sections. Still, it is nowhere near as flexible and powerful as connecting each topic to an existing knowledge graph!
Advantage 4: Long-term Persistence
It seems like companies change the way they store knowledge every few months. It might be MS Word stored in file folders one year, a wiki the following year, then perhaps Microsoft Sharepoint, and now we are seeing a shift to microsites on GitHub. Every time an organization changes its tools, the knowledge is lost. Because they are built around standards like easy to edit Markdown, we now have hope that this data can persist for a long time. Although things like concept links, tags, and aliases are not yet standardized across platforms, our hope is that lightweight PKI inoperability standards without the need to implement a complete DocBook compatibility checklist.
Advantage 5: Breaks Writers’ Block
One of my goals is to try to write a blog every other week. Sometimes I have an idea, write the first paragraph, and I get blocked. I can’t figure out how to organize my thoughts in any way that will be somewhat coherent to my readers. If I keep lots of little graphs around for ideas, I can see them grow over time. The metaphor is to plant your “Digital Garden of Ideas” and let them grow naturally. When I have about a dozen good ideas in a subgraph it might be time to make it an actual blog.
These advantages are my own personal view on how I use PKGs. If you search on Google, you will find that students, researchers, creative writers, and bloggers are all using PKGs in different ways.
Challenges with PKG Tools
Although the PKG tools such as Roam Research and Obsidian are relatively new, they are adding new features (both free and for fees) quickly. My biggest challenge has been trying to share sub-sections of my PKG with others and publishing subgraph layouts on microsites. Right now we need to do crude import/export operations to combine our knowledge graphs to a place where we can collaborate. However, I suspect these new features are coming quickly.
My biggest challenge with the current generation of tools is the limited ways I can generate visualizations of subgraphs. Like using the powerful GraphViz libraries, I want to be able to try multiple constrained layout algorithms and play with top-down and left-to-right layouts. I want to specify these layout rules in a dot-like language. I would like to “pin” the arrangement of various nodes in the layout so they don’t randomly change as I add new items. I am sure all these features will come with time. For now, we can write Python scripts that take the Markdown and convert them into Mermaid structures for automatic layouts in our published microsites.
These tools also don’t allow me to add types or attributes to my relationships. That might also come in a future release. Remember that a little bit of semantics goes a long way!
Impact on the Enterprise Knowledge Graph Market
Vertex-level Role-based Access Control (RBAC) is a hallmark of enterprise-grade knowledge sharing systems like TigerGraph or Smartlogic’s Ontology Editing tools.
Right now, these PKG systems are all cloud-backed desktop tools or websites. They don’t have fine-grain role-based access control rules so I can grant my teams read or write access to a subset of my PKG.
However, I can’t believe that the quickly growing PKG market will not find customers who want to combine note-taking’s free-flowing nature and formal curated and approved knowledge graph systems. As APIs evolve, I could easily see functions such as “Publish this subgraph to the corporate ontology server for review.”
Legal Question: Who Owns Your PKG?
I can’t help but think how the complexity of people building their super-high-quality PKGs will overlap with the concerns of the intellectual property of organizations. When I worked for AT&T Bell Labs in the 1980s all our written engineering notes were required to be done using the standardized AT&T Bell Laboratories blue-lined notebooks with individual page numbers. We were required to date each entry so that we could defend our intellectual property and patents carefully using these formal note-taking systems. We were told that adding a date, a few lines of text, and simple diagrams could save the company millions of dollars. Yet the word “Personal” implies that you own your own second brain. This will run against the concerns of corporate legal staff protecting their intellectual property.
Getting Started
My last comment is that it may take you a while to get used to these new tools. They are a bit rough around the edges and don’t yet have the same presentation and flexibility as more mature note-taking tools. My suggestion is to be patient and give yourself a few days/weeks to adjust the tool to meet your style of note-taking and publishing. Try out different ways of organizing concepts and see if you can find a pattern that works.
Enjoy!
