past on [https://forum.obsidian.md/t/your-post-in-beyond-plain-text-expanding-obsidian-s-connection-concept-to-build-a-closed-loop-architecture-for-knowledge-base-object-storage-lightweight-publishing/117215]
I. Introduction and Overview (Problem Statement and Proposed Solution)
As a localized knowledge base, Obsidian emphasizes local storage and the connections between knowledge points (or Markdown documents) to transform fragmented information into systematic knowledge. This aligns exceptionally well with the security requirements of personal information and systematic learning. In particular, the natural compatibility between Markdown files and AI models makes Obsidian an increasingly vital tool. However, with the explosive growth of information and the surging demand to collect, organize, and share knowledge using AI technology, Obsidian reveals several architectural pain points:
- Storage Expansion and Accumulation of Orphaned Objects: As images and videos accumulate inside local Markdown documents, file sizes balloon, leading to continuous vault expansion. Consequently, many users turn to Object Storage (S3/OSS/COS) to store media files, keeping only remote links within Obsidian. While this is a practical workaround, it introduces a major problem: when documents are deleted or modified inside Obsidian, the object storage retains numerous orphaned, “zombie” images and videos, which continue to accumulate over time.
- Severe Isolation Between Control Plane and Data Plane: Once media files are moved to object storage, Obsidian becomes a completely passive receiver of links. It cannot perceive the real-time status of the object storage, nor can it browse or manage remote assets from within the app. Conversely, when Markdown documents change and links are removed, the object storage remains completely unaware and cannot perform corresponding actions (such as marking files for deletion). Thus, despite appearing connected, Obsidian and object storage remain completely isolated and unaware of each other.
- Unreliable Management and Publishing: Object storage prevents local vault bloat and makes publishing notes much lighter and easier. However, because Obsidian notes and object storage remain isolated, managing, publishing, and sharing knowledge becomes unreliable and highly inconvenient.
To address these pain points, I believe Obsidian’s current architecture can undergo an evolutionary expansion:
- Expanding the Dimension of Connection (Two-Way Perception): We can expand Obsidian’s core concept—connection—beyond relationships between note points to include the relationship between referenced media links inside notes and the actual assets stored in object storage. By bringing image and video links into Obsidian’s management scope, we achieve true manageability and control. When Markdown documents change (e.g., deletion), the object storage layer can be notified to take corresponding actions (such as marking assets as broken links or eligible for deletion).
- In-App Object Management (Visual & Controllable): Obsidian can enable users to browse and manage remote assets directly inside the app. When a Markdown document referencing an image or video is deleted, orphaned objects can be automatically or manually cleaned up.
- Reliable & Lightweight Publishing (Link Health Monitoring): Systematic knowledge accumulation isn’t just about personal study—sharing is an equally or even more important aspect. Therefore, besides making knowledge collection easier, Obsidian should also be a tool that makes sharing seamless. By connecting Obsidian’s publishing workflow with object storage, notes published to web servers can remain extremely lightweight (containing only text and links). Meanwhile, link validity can be monitored to guarantee that published content is both lightweight and reliable.
By expanding Obsidian’s core concept of connection, we can achieve three main goals:
- Lightweight Storage: Obsidian offloads large media files to object storage while keeping only pure text and links locally, ensuring a lightweight local vault.
- Closed-Loop Control: Users can browse, display, and manage remote assets without leaving Obsidian, maintaining link validity while preventing the accumulation of orphaned zombie objects.
- Reliable Publishing: Users can share notes effortlessly while continuously monitoring the validity of embedded links in real time.
II. Mindmap of the Proposed Architecture Extension
# Expanding Obsidian's Connection Concept: Knowledge Base + Object Storage + Reliable Publishing
1. Current Status & Pain Points
├── Local vault balloons rapidly as media (images/videos) increases
├── Using Object Storage leaves "orphaned zombie files" in the cloud when notes are deleted/edited
├── Isolation: Obsidian cannot perceive cloud state or manage assets inside the app
└── Publishing/sharing lacks link health monitoring, making external links unreliable
2. Core Architectural Shift: Expanding the Concept of "Connection"
├── Expand from "managing connections between note points" to "two-way binding between notes & storage objects"
└── Upgrade positioning: From "local Markdown notepad" to "Digital Asset & Content Management System (DAM/CMS)"
3. Three Core Goals & Feature Designs
├── ① Lightweight Local Storage: Offload media to cloud; keep pure text locally for speed & AI friendliness
├── ② Closed-Loop Cloud Asset Control: In-app visual browser for OSS/S3; Garbage Collection (GC) based on reference count
└── ③ Reliable Lightweight Publishing: Deploy pure text; direct media to CDN; pre-publish 404 & permission checks
4. Value & Development Roadmap
├── Value: Eliminate storage waste, preserve AI-text compatibility, close the loop from "collect -> organize -> share"
└── Roadmap:
├── Phase 1: Develop "Two-Way Indexer & Orphan Scanner" plugin
├── Phase 2: Integrate "In-App Object Storage Manager" view
└── Phase 3: Implement "Pre-flight Diagnostics + One-Click Lightweight Publishing" workflow
III. Core Analysis and Deconstruction of the Mindmap
The mindmap above provides a systematic deconstruction of the proposal, grounded in an architectural shift aimed at "breaking the barrier between text and cloud storage":
5. Root Cause: Isolation Between Control Plane and Data Plane
Currently, integrating Object Storage (S3/OSS/COS) into Obsidian remains at a one-way passive reference stage—uploading an image and returning a text URL.
Orphaned Zombie Files: Occur because cloud storage lacks a "Garbage Collection" mechanism when Markdown text drops a link.
Isolation: Stems from Obsidian lacking a Control Plane for remote object storage, leaving it blind to the status of cloud assets.
2. Architectural Breakthrough: Two-Way Perception & Reference Counting
Achieving two-way binding between notes and cloud objects requires introducing two classic software engineering concepts at the plugin layer:
Two-Way Metadata Indexing: Maintain a lightweight local index (e.g., SQLite inside Obsidian) mapping Document ↔ Remote URL bidirectionally.
Active State Perception: When a document is deleted or a link removed, the reverse index updates. Once the reference count drops to zero, the plugin automatically tags the cloud object as Pending Delete / Orphan, enabling one-click cleanup inside Obsidian.
3. Practical Impact: From Local Notepad to "Lightweight Publishing Console"
By expanding the connection dimension, Obsidian undergoes a fundamental evolution:
Built-in Cloud Asset View: Browse, preview, grid-manage, and drag-and-drop cloud assets directly from Obsidian's sidebar without switching to web consoles or third-party tools.
Pre-Publish Automated Diagnostics (Pre-publish CI/CD): Before sharing notes publicly, the system automatically runs link validity checks (404 checks), uploads missing local images, and verifies permissions—ensuring published content is both ultra-lightweight and 100% reliable.
This closed-loop design eliminates local vault bloat while preserving Markdown's natural compatibility with AI workflows. It seamlessly bridges the gap between a private local knowledge repository and an on-demand, highly reliable public sharing pipeline.