HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
john-friedman
datamule-python
Python2024–2026
john-friedman / datamule-python

datamule-python

A package to work with SEC data. Incorporates datamule endpoints.

Explore on GitHubRead Hacker News discussions
2Distinct people
10Mentions
0Explicit recommenders
560GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A package to work with SEC data. Incorporates datamule endpoints.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 11:56:56 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month10 dated mentions

Observed mentions: 2024-09-10 21:27:07 UTC to 2026-03-20 02:39:57 UTC.

0122024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 commentsSep 20242024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsJun 20252025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-03-01 00:00:00 UTC to 2026-04-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsMar 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC220
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC110
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC110
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC220
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC110
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC000
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC110
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC000
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC110
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC000
Mar 20262026-03-01 00:00:00 UTC2026-04-01 00:00:00 UTC110

What Hacker News says

10 sources
jgfriedman1999·Mar 20, 2026Submission

Show HN: I wrote an open source SEC filing compliance package

The U.S. Securities and Exchange Commission requires companies and individuals to submit data in SEC specific formats. Usually this means taking a columnar dataset and converting it to a specific XML schema. In practice, this usually means paying a company for proprietary filing software that is annoying to use, and is not modifiable. Here it an open source solution: https://github.com/john-friedman/secfiler Note that this is a side project. My main project is manipulating SEC data. That project required me to figure out how to parse every SEC XML file into constituent tables. Since I had to do that, I might as well do the reverse -- which is how secfiler came about. The other project is https://github.com/john-friedman/datamule-python.
1 points · Open discussion
jgfriedman1999·Jul 31, 2025Submission

New budget financial API, based on EDGAR data

Hey everyone, I'm the developer of an open-source (MIT License) python package to convert SEC submissions into useful data. I've recently put a bunch of stuff in the cloud for a nominal convenience fee. Cloud: 1. SEC Websocket - notifies you of new submissions as they come out. (Free) 2. SEC Archive - download SEC submissions without rate limits. ($1/100,000 downloads) 3. MySQL RDS ($1/million rows returned) - XBRL - Fundamentals - Institutional Holdings - Insider Transactions - Proxy Voting Records Posting here, in case someone finds it useful. Links: Datamule (Package) GitHub: https://github.com/john-friedman/datamule-python Documentation: https://john-friedman.github.io/datamule-python/datamule-python/sheet/sheet/ Get an API Key: https://datamule.xyz/dashboard2.html
7 points · Open discussion
jgfriedman1999·May 23, 2025Submission

Show HN: Doc2dict a fast, open-source document to dict converter – No AI

doc2dict is a python package that converts html and pdf documents into dictionaries preserving hierarchy. It also supports table extraction for html files. https://github.com/john-friedman/doc2dict Speed: * html - 500 pages per second single threaded. * pdf - 200 pages per second, pdf must have an underlying text structure. Multithreading is not possible due to the limitations of PDFium. Here's an example output from Microsoft's Annual Report: > "title": "PART I", "standardized_title": "parti", "class": "part", "contents": { "38": { "title": "ITEM 1. BUSINESS", "standardized_title": "item1", "class": "item", "contents": { "39": { "title": "GENERAL",
Read the rest
"standardized_title": "", "class": "predicted header", "contents": { "40": { "title": "Embracing Our Future", "standardized_title": "", "class": "predicted header", "contents": { "41": { "text": "Microsoft is a technolo... Raw: https://html-preview.github.io/?url=https://raw.githubuserco... Parsed dictionary: https://github.com/john-friedman/doc2dict/blob/main/example_... Simple description of algorithm: * Take complicated document such as pdf or html, and created a simplified representation for it as a list of a list of dicts where each dict is a text block with key features such as "bold", "font-size", etc and each line represents a new html block or line on a pdf. * Convert the simplified representation to a dictionary using a set of predetermined rules, e.g. smaller font-size for a heading means it should be nested under the larger font-size heading. Note that I am working on making the last part more modular by creating predetermined instructions that users can tweak for their use-case without rewriting the parser. I call these "mapping dicts". doc2dict also includes visualization tools for the debugging process: * visualize simplified representation https://html-preview.github.io/?url=https://github.com/john-... * visualize output dictionary https://html-preview.github.io/?url=https://github.com/john-... Why I made this: I'm currently working on another open source python package to make it easy to exploit Securities & Exchanges Commission data. Writing a generalized document parser that can be tweaked is easier than writing 100 or so specialized parsers for each document type. Also, converting html and pdf files to dictionary representation reduces document size by a factor of 10 or so. Not sure what I can do with that, but planning on some fun NoSQL database experiments. Link to other package (datamule) https://github.com/john-friedman/datamule-python
4 points · Open discussion
jgfriedman1999·Jan 7, 2025Submission

Predicting good news using regulatory disclosure patterns, Python, SEC

I was curious if I could use insider trading patterns to predict whether the sentiment of 8-K filings were positive or negative. https://github.com/john-friedman/datamule-python/blob/experimental/examples/predic… Doesn't look like it - not enough data. However, there is an interesting result. Trading behavior after a positive 8-K is very different for insiders. They wait to sell, possibly to let good news trickle out. https://github.com/john-friedman/datamule-python/blob/experimental/examples/predic… I'm considering looking at companies using their entire corpus of filings (e.g. 10-K to IRAN NOTICE) for a year and applying finbert for another try, but it seems expensive in terms of compute. I've mostly solved the download problem by h
Read the rest
osting my own SEC archive - it takes about 45 minutes to download 50,000 8-Ks and 150,000 Form 4s (1 year), but compute seems harder. I used Loughran McDonald dictionaries to do the sentiment parsing - took about an hour, so I imagine applying finbert on a larger corpus would take weeks on my laptop. Not sure if there's a better option?
6 points · Open discussion
jgfriedman1999·Dec 23, 2024Submission

Show HN: A faster and compliant way to scrape the SEC

The SEC rate limits you to 5 requests per second. The SEC has more than 16 million submissions, with each submission having multiple documents. For example, Tesla's 2021 10-K SEC page has 14 documents. Scraping every SEC submission document by document would take about 200 days. There is a faster approach. Submissions to the SEC are stored in Standard Generalized Markup Language in one file. That means that if you have a SGML parser, you only have to download the submission .txt file. Scraping every SEC SGML submission takes about 40 days. There is a much faster approach. SEC data is stored in SGML in daily archives accessible here going back to 1994: https://www.sec.gov/Archives/edgar/Feed/1995/. Since the SEC rate limits you to 30mb/s of egress and there is 3tb of data, downloading every archive takes slightly more than a day. I've writ
Read the rest
ten a fast SGML parser here using Cython: https://github.com/john-friedman/datamule-python
1 points · Open discussion
jgfriedman1999·Oct 14, 2024Submission

Show HN: OS SEC Chatbot with Artifacts

A basic implementation of a tool-using SEC chatbot without reasoning ability. Features: * Browse and download Company XBRL facts * View specific sections in filings, e.g. "Tesla's Management Discussion & Analysis" Future: * Reasoning ability + toggle * Integration with simplified MD&A database allowing for comparing hundreds of companies at once within one context window * Integration with Table Parser API GitHub: https://github.com/john-friedman/datamule-python
2 points · Open discussion
jgfriedman1999·Sep 14, 2024Submission

Show HN: Package to parse SEC 10Ks into JSON and other fun things

Features: * parse textual filings into simplified html, interactive html, or structured json. * download sec filings quickly and easily Datasets: * 2024 Management Discussion and Analysis * 2024 10Ks in json format Datasets can be downloaded using the package, or here https://www.dropbox.com/scl/fo/byxiish8jmdtj4zitxfjn/AAaiwwu... All APIs are free and do not require auth, so you can use them directly :) e.g. I'm currently using the parser api with js to run my filing viewer demo: https://datamule.xyz/filings_viewer
1 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.