HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
simonw
ttok
Python2023–2024
simonw / ttok

ttok

Count and truncate text based on tokens

3Distinct people
3Mentions
0Explicit recommenders
409GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Count and truncate text based on tokens

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:03:16 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month3 dated mentions

Observed mentions: 2023-09-22 23:43:55 UTC to 2024-09-26 02:51:59 UTC.

012023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsSep 20232023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsMar 20242024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsSep 2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC101
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC000
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC000
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC000
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC101
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC101

What Hacker News says

3 sources
4-turbo should have a knowledge up to December 2023. But it doesn't know anything about the cli tools you created before then (except shot-scraper). Compare 4-turbo to 4o llm -m 4t "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Among the tools you've listed, "shot-scraper" is the only one that I could provide detailed information about based on widespread use and documentation available as of the last update. As for "llm," "ttok," and "symbex," there isn't readily available, widely recognized information on tools precisely matching those names, so it's possible they might be lesser-known tools, newly released, or possibly regional or specialized tools not widely documented in widely-used platforms. (Truncated) llm -m 4t "Tell me about these cli tools and link to them:
Read the rest
- llm: - ttok - symbex - shot-scraper " As of my last knowledge update in September 2023, here is the information on the CLI tools you are asking about: 1. *llm*: - I couldn't find a specific command line tool named "llm" that is widely recognized or used. It could be a lesser-known tool, a new tool, or potentially a typographical error. Please verify the tool's name and provide additional details if possible. 2. *ttok*: - As with "llm," there isn't any widely known CLI tool named "ttok." (Truncated) llm -m 4o "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Sure! Here are brief descriptions of the CLI tools you mentioned along with links to their official repositories or webpages: ### 1. llm `llm` is a command-line interface (CLI) tool to interact with large language models like OpenAI's GPT-3. It allows you to query these models directly from the terminal, making it easy to integrate AI capabilities into your workflow or automate tasks. - *Link*: [llm GitHub Repository](https://github.com/simonw/llm) ### 2. ttok `ttok` is a CLI tool for working with tokenizers. It's a tool under development aimed at helping users understand how tokenization processes text, which is essential for preparing data for machine learning models, especially in NLP applications. - *Link*: [ttok GitHub Repository](https://github.com/simonw/ttok) ### 3. symbex `Symbex` is a CLI tool for performing symbolic execution on Python bytecode. It can analyze Python code to find potential bugs, vulnerabilities, or logical errors by exploring all possible execution paths. - *Link*: [symbex GitHub Repository](https://github.com/simonw/symbex) (Truncated) llm -m 4o "Tell me about these cli tools and link to them: - llm: - ttok - symbex - shot-scraper " Sure, here is a brief overview of each CLI tool along with links to their respective repositories or documentation: 1. *llm (Language Model CLI)* - *Description*: The `llm` command-line tool leverages large language models, such as OpenAI's GPT-3, to make it easier to incorporate AI functionalities into your command-line tasks. You can use it to generate text, answer questions, and assist with coding or other language-based tasks directly from your terminal. - **Link**: [llm on GitHub](https://github.com/simonw/llm) 2. *ttok* - *Description*: `ttok` is a simple command-line utility that provides tokenization for text, particularly useful for breaking down input for language models. It helps to preprocess text, count tokens, or manage tokens more efficiently from the command line. - **Link**: [ttok on GitHub](https://github.com/simonw/ttok) 3. *symbex (Symbolic Execution)* - *Description*: `symbex` is a CLI tool designed for searching codebases. It employs symbolic execution to analyze code paths and find functions, variables, or other code elements. This can be particularly useful for developers looking for higher-precision searches in large codebases. - **Link**: [symbex on GitHub](https://github.com/paulgb/symbex) (Truncated) Clearly, openai did a lot more web scraping for this model.
In Show HN: Interactive Graph by LLM (GPT-4o)
Awesome work! I used it to summarize an hour long podcast I had been meaning to watch and it worked fabulously. What's amazing is that the transcript is auto-generated and of a conversation between two individuals without any indication of who's actually speaking. Yet GPT-3.5 is able to make sense of it. Out of curiosity I downloaded the transcript myself with `youtube_transcript_api --format text` and counted the tokens via ttok [0], it was a tad over 16k. So what does your site do in that case? Is the transcript truncated? [0]: https://github.com/simonw/ttok
In Show HN: Summary Cat, a YouTube Video Summary Generator

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.