docquery

There is a visual demo here: https://sites.google.com/view/llm4html/home. This work is very exciting to me for a few reasons: - HTML is an incredibly rich source of visually structured information, with a semi-structured representation. This is as opposed to PDFs, which are usually fed into models with a "flat" representation (words + bounding boxes). Intuitively, this offers the model a more direct way to learn about nested structure, over an almost unlimited source of unsupervised pre-training data. - Many projects (e.g. Pix2Struct https://arxiv.org/pdf/2210.03347.pdf, also from Google) operate on pixels, which are expensive (both to render and process in the transformer). Operating on HTML directly means smaller, faster, more efficient models. - (If open sourced) it will be the first (AFAIK) open pre-trained ready-to-go model for the R

DocQuery (https://github.com/impira/docquery), a project I work on, allows you to do something similar, but search over semantic information in the PDF files (using a large language model that is pre-trained to query business documents). For example: $ docquery scan "What is the due date?" /my/invoices/ /my/invoices/Order1.pdf What is the due date?: 4/27/2022 /my/invoices/Order2.pdf What is the due date?: 9/26/2022 ... It's obviously a lot slower than "grepping", but very powerful.