dlt

rom the question of composability: how distinct would a ggplot IR be from a metrics layer spec? could i use ggsql to create an IR that I then use R's ggplot to render (or vise versa maybe?) as for the deployment story today, I'll likely learn most by doing (with agents). My experiment will be to kick off an agent to do something like: extract this dataset to S3 using dlt [2], model it using dbt [3], then use ggsql to visualize. p.s. @thomasp85, I was a big fan of tidygraph back in the day [4]. love how small our data world is. [1]: https://github.com/posit-dev/ggsql/blob/main/Cargo.toml [2]: https://github.com/dlt-hub/dlt [3]: https://github.com/dbt-labs/dbt-fusion [4]: https://stackoverflow.com/questions/46466351/how-to-hide-unc...

tom Python code to create a dataset. OpenAPI specifications standardized API definitions, making it easier to interact with APIs. Now, dlt-init-openapi leverages these OpenAPI specs to automate the manual work around dataset creation you had to do before. OpenAPI is a growing standard with almost every API using it, and used by default in frameworks like FastAPI. What can dlt-init-openapi do for you? - It generates Python scripts with dlt pipelines that you can run to pull data from your API into a structured destination of your choice (Parquet files, SQL DBs, Databricks, Snowflake, etc.) - Infers and evolves schema for the endpoints from the actual data! - Discovers pagination style for each endpoint - Finds and unwraps data entities for each endpoint, also for deep, nested JSON responses - Discovers the primary key for each entity - Discovers authentication schema and generates code and config files to pass required credentials - You always have the last say. The generated code is declarative and ready to hack in case we pick the wrong paginator or response entity. The tool and dlt are open source, find the code here: https://github.com/dlt-hub/dlt-init-openapi and here: https://github.com/dlt-hub/dlt

Python. You can read more about us here: https://news.ycombinator.com/item?id=37999527 --- Your Task and Responsibilities: - Contribute to dlt, including code, tests, and documentation - Maintain the open source project with the team (e.g., review PRs, resolve issues, talk with community contributors, etc.) Who You Are: - You really like Python and are fluent in writing Python code (e.g., Python typing, unit testing, writing docstrings, etc.) - You are interested in the Python ecosystem (i.e., popular libraries, tools, Python internals, PEPs, how Python is used outside of software engineering, etc.) - You have some experience with databases and data warehouses (i.e., you understand the relational data model, transactions, concurrency, etc.) You are familiar with GitHub workflows (e.g., pull requests, code reviews, CI/CD services, etc.) Nice to Have: - You might have experience with data engineering (e.g., building data pipelines, dataset modeling, enabling others to use the data, etc.) -You might have experience with machine learning (e.g., the toolset, the workflows, practical applications, etc.) If the role sounds interesting, please apply here: https://apply.workable.com/dlthub/j/4C5D016FF5/apply/

ine-schema - in a schema file: https://dlthub.com/docs/walkthroughs/adjust-a-schema if you want to enforce schema and data contracts: - you can use pydantic models to validate data (if you use pydantic model as a table definition, this is the default) - we have soon-to-be-merged schema contract PR: https://github.com/dlt-hub/dlt/pull/594 My observations are that it is more than 1% of people that are fine with auto-generated schemas. But that could be selection bias (they use our library because they like it).

easy to use Python library called dlt that will automate most data engineering tasks. It hides the complexities of data loading and automatically generates a structured and clean datasets for immediate querying and sharing. — — — At its core, dlt removes the need to create the dataset schemas, react to changing data, generate append or merge statements, and to move the data in transactional and idempotent manner. Those things are automated and can be declared right in the Python code, just by decorating functions. Add @dlt.resource decorator, give it a few hints, and convert any data into a simple pipeline that creates and updates datasets. dlt gets the details out of your way: 1. You do not need to worry about the structure of a database or parquet files dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contracts and Pydantic models on top to keep your data clean. 2. You do not need to write any INSERT/UPDATE or data copy statements dlt will push the data to DuckDB, Weaviate, storage buckets and many popular SQL stores. It will align the data types, file formats, and identifier names automatically 3. You do not need to worry when you need to add new data or update the changes. dlt lets you declare how to load the data, how to increment it and will keep the loading state together so they are always in sync. 4. You keep how you develop and test your code Iterate and test quickly on your laptop or in a dev container. Run locally on DuckDB and just swap destination name to go to the cloud - your code, schema and data will stay the same. 5. You can work with data on your laptop. Combine dlt with other tools and libraries to process data locally. duckdb, Pandas, Arrow tables and Rust based loading libraries like ConnectorX work nicely with dlt and process data blazingly fast, compared to the cloud. 6. You do not need to worry if your pipeline will work when you deploy it. dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to work on constrained environments like AWS Lambda or run with Airflow, GitHub Actions or Dagster. dlt has an Apache 2.0 license. We plan to make money by offering organizations a paid control plane, where dlt users can track and policy what every pipeline does, manage schemas and contracts across organization, create data catalogues, and share them with the team members and customers.