dlt-init-openapi

tom Python code to create a dataset. OpenAPI specifications standardized API definitions, making it easier to interact with APIs. Now, dlt-init-openapi leverages these OpenAPI specs to automate the manual work around dataset creation you had to do before. OpenAPI is a growing standard with almost every API using it, and used by default in frameworks like FastAPI. What can dlt-init-openapi do for you? - It generates Python scripts with dlt pipelines that you can run to pull data from your API into a structured destination of your choice (Parquet files, SQL DBs, Databricks, Snowflake, etc.) - Infers and evolves schema for the endpoints from the actual data! - Discovers pagination style for each endpoint - Finds and unwraps data entities for each endpoint, also for deep, nested JSON responses - Discovers the primary key for each entity - Discovers authentication schema and generates code and config files to pass required credentials - You always have the last say. The generated code is declarative and ready to hack in case we pick the wrong paginator or response entity. The tool and dlt are open source, find the code here: https://github.com/dlt-hub/dlt-init-openapi and here: https://github.com/dlt-hub/dlt

ine-schema - in a schema file: https://dlthub.com/docs/walkthroughs/adjust-a-schema if you want to enforce schema and data contracts: - you can use pydantic models to validate data (if you use pydantic model as a table definition, this is the default) - we have soon-to-be-merged schema contract PR: https://github.com/dlt-hub/dlt/pull/594 My observations are that it is more than 1% of people that are fine with auto-generated schemas. But that could be selection bias (they use our library because they like it).