Reading up on data-extraction

The Log Is the Agent

@ishaansehgal

So... "ai, explain what I just did": we ingested 9M+ directed edges from X into a weighted influence graph, then ran a personalized PageRank variant — tuned for human accounts — to surface eigenvector centrality at scale. once you know the high-centrality nodes, you watch their signal propagation before it hits the long tail. beta soon. @digg @basic_in_

@kevinrose

I've been in the process of building a custom home for 5 years. Bought the land in 2021. Got the building permit this year. Haven't started construction yet. During those 5 years, I accumulated thousands of emails with dozens of architects, engineers, surveyors, contractors, government agencies, title companies, and others. Hundreds of PDFs I opened once and never found again. My project management system was email search and my own memory. I could always find individual emails when I needed them. What I couldn't do was see the project. How much money have we actually spent, and on what? Who are all the contractors we talked to, and how did we find each one? What happened with the easement, not one email about it, but the full arc across three years? Why did we stop using the original surveyor? The answers were all in my inbox. But they were spread across hundreds of threads. No single email contained the story. The story only existed in the connections between them. So I tried something. I pointed OpenClaw at my full email inbox and said: read all my emails in chronological order and figure out what happened with this project over the last 5 years. Build me a timeline. Find all the documents. Track the money. Map the people. That's it. I didn't sort anything. I didn't classify anything. I didn't tell it which threads mattered. I just pointed at the inbox and let it work. And it worked way better than I expected. It found 1,850 emails across 450 threads involving 58 people at 35 organizations. From that, it produced 511 timeline events describing what actually happened over 5 years. Not "Daniel emailed the architect" but "Easement delay threatens grading permit" or "architect warns the entire permit depends on securing the neighbor's access agreement." Real project history in PM language. It identified 690 documents and classified each one: invoice, permit, survey map, legal agreement, environmental report, estimate, and so on, and it linked them to the timeline events that referenced them. It extracted 170 finance records from email bodies and PDF attachments. Invoices, payments, estimates, and receipts with amounts, dates, and payees pulled from messy documents. It mapped out 58 contacts with their roles, their organizations, and how they related to the project over time. All interlinked. Click a timeline event, see the emails that produced it and the documents attached. Click a payment, trace it back to the invoice and the email thread. Click a person, see every event they were involved in. It built a dashboard on top of it and for the first time in 5 years, I could actually see the whole project. The full arc. Every dollar. Every person. Every decision. Stitched together from raw correspondence into something I can sit down and browse. The key insight for me was realizing this is basically an ETL process: Extract, Transform, and Load. The emails are the source data. The agent does the extraction from emails and loading into a database. But the really powerful part is the Transform: the LLM reads the raw correspondence with enough context to do intelligent enrichment across hundreds of threads spanning months and years. And by enrichment I don't mean summarization. I mean it actually reconstructed the narrative of the project. It traced how we almost hired the wrong well driller. We originally hired one company, paid a deposit, and were ready to go. Then the architect heard from someone in his network that they weren't reliable. We pivoted to a different driller who came recommended through a chain of referrals the agent traced back to its origin. The new company came out, drilled 140 feet, hit an artesian well with water pressure above ground level, and finished in two weeks. The original deposit got refunded. The agent reconstructed that entire sequence from first contact to final invoice, across dozens of emails and multiple contractors, and presented it as one coherent story. It reconstructed the full permit saga. Four separate permits with the county, each with its own cycle of applications, reviews, correction letters, resubmissions, and approvals. Years of back and forth. The agent built the complete timeline for each permit and linked every document and payment to the right stage. It tracked the money flow end to end. Not just "we paid the architect X." It found every invoice, matched them to the work described in the email threads, categorized the spending, and produced a financial history of the entire project broken down by architect, engineer, surveyor, contractor, county fees, and everything else. It mapped out relationships between people that I had half-forgotten. Which engineer referred which surveyor. Which contractor's crew member later became a separate vendor. Which county reviewer handled which permit. All of it was in the email, I just never had the time to stitch it together myself. One of the most fun things it did was writing honest personality profiles for each contact based purely on their communication style. How responsive they are. How they handle pushback. Whether they tend to over-promise. Whether they're the kind of person who answers at 11pm or takes five days to reply. Reading an AI's unfiltered take on the people you've been doing business with for years, based on nothing but their emails, is surprisingly entertaining and uncomfortably accurate. The thing that surprised me most is how much structure was already hiding in the email. I didn't add information. The agent found what was already there. The timeline, the document graph, the money flows, the cast of characters. It was all latent in the correspondence. Five years of decisions and negotiations and payments, all recorded in email, just never connected. I think a lot of people are sitting on projects like this without realizing it. Your renovation emails are a project database waiting to be assembled. Your legal correspondence is a case file. Your immigration threads are an application history. The raw material has been accumulating for months or years. It's rich, timestamped, and complete. It's just in a format designed for messaging, not for understanding. Point an agent at it. Let it read everything. Let it do the transform. The whole story was in my inbox the entire time. I just needed something that could read all of it at once.

@dvassallo

https://t.co/koRUvkFila

@phoebeyao

https://t.co/rc5GHb38Tq time

@thekitze