pandas

It does appear to potentially be possible: https://github.com/pandas-dev/pandas/pull/64585

I’d like to share my work with you: https://news.ycombinator.com/item?id=47137452 It shows that I posted just a little later than you. I agree that the chances of something going wrong with timestamps are low, but I still think it’s worth considering as a potential security risk or injection vector — although I’m not sure how realistic that threat actually is. Regarding UUIDs, you can get the best of both worlds: sequential IDs are convenient, and adding a small number of random characters can help avoid name collisions. Since the random component only needs to ensure uniqueness across local repositories, it doesn’t require many characters. I can see that you’ve put a great deal of effort into this, especially with the various bridging components. I built a simple bridge myself and ran some tests using the pandas project (https://github.c

Unrelated, but I personally am not satisfied with the performance of Panda's XLSX export. As you can see here [0], the code does really strange things. It takes cell.style and throws it into json.dumps() to generate a key for a dictionary so that they can cache the XlsxStyler.convert(cell.style) result. Except, the vast majority of cells do not have any styling whatsoever, so json.dumps is producing the string "null", which is then used to lookup None. The low hanging fruit are jaw dropping. You can easily speed up the code 10%+ by adding a simple check "if cell.style is not None or fmt is not None:" and switching from json.dumps(cell.style) to str(cell.style). If I wanted an easy weekend project that positively impacts many people this is what I'd work on. [0] https://github.com/pandas-dev/pandas/blob/main/pandas/io/exc...

Is it possible to apply these optimizations to Arrow? Arrow's struct is called StructArray. Fields of StructArray have a StructType. StructType has .bit_width and .byte_width attrs in Python and probably the other implementations too: https://arrow.apache.org/docs/python/generated/pyarrow.Struc... Arrow supports bitfields with BooleanArray, and enums with categoricals but "BUG: Categorical columns using the PyArrow backend requires 4x more memory" (open) https://github.com/pandas-dev/pandas/issues/58062 : > On disk Parquet appears to store the category data as logical type String which is compressed with snappy and encoded Arrow Flight RPC handles nested structs with enums over the wire somehow too FWIU

Can this be merged into pandas? Pandas does not currently install tqdm by default. pandas-dev/pandas//pyproject.toml [project.optional-dependencies] https://github.com/pandas-dev/pandas/blob/8943c97c597677ae98... Dask solves for various adjacent problems; IDK if pandas, dask, or dask-cudf would be faster with async? Dask docs > Scheduling > Dask Distributed (local) https://docs.dask.org/en/stable/scheduling.html#dask-distrib... : > Asynchronous Futures API Dask docs > Deploy Dask Clusters; local multiprocessing poll, k8s (docker desktop, podman-desktop,), public and private clouds, dask-jobqueue (SLURM,), dask-mpi: https://docs.dask.org/en/stable/deploying.html#deploy-dask-c... Dask docs > Dask DataFrame: https://docs.dask.org/en/stable/dataframe.html : > Dask DataFrames are a collection of many pandas DataFrames. > The API is the sam

They are when the only reason they are flagged as security updates is because some a single group deems a very rare, obscure edge case as a HIGH severity vuln when in practice it rarely is => this leads to having to upgrade a minor version of a library that ends up causing breaking changes. This is the recent thread I'm down. Pandas 2.2 broke SQLalchemy backwards compatibility: https://stackoverflow.com/questions/38332787/pandas-to-sql-t... + https://github.com/pandas-dev/pandas/issues/57049#issuecomme...

That is something I’d like to see but I don’t want to wade into the already very complicated discussion around arrow strings in pandas. If a Pandas developer wanted to take this on I think that would make things easier since there’s so much complexity around strings in Pandas. That said there is a branch that gets most of the way there: https://github.com/pandas-dev/pandas/pull/58578. The remaining challenges are mostly around getting consensus around how to introduce this change. If NumPy had StringDType in 2019 instead of 2024 I think Pandas might have had an easier time. Sadly the timing didn’t quite work out.

hm https://github.com/pandas-dev/pandas/blob/main/pandas/io/par... says "fastparquet" engine must be available if no pyarrow https://pypi.org/project/fastparquet/

there is pull request to integrate in both pydantic extra types and into pandas cose [1] [1]: https://github.com/pandas-dev/pandas/issues/53999

And for an example of the practical limits of @overload, take a look at the Pandas type hints: https://github.com/pandas-dev/pandas/blob/dc5586baa9e4731805... Meanwhile it's not even possible to express such things in other static type systems. So I'm not exactly an unhappy customer, but it does put certain things tantalizingly close, but still out of reach without a ton of clunky boilerplate and LoC explosion.

I asked GPT-4 this can you visit https://pandas.pydata.org/about/governance.html and tell me if I am allowed to use the term 'pandas' in the name of another unaffiliated project, for example 'pandas-ai' -- Based on the BSD 3-Clause License under which pandas is released, neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission[ 1] . This means that to use the term 'pandas' in the name of another unaffiliated project such as 'pandas-ai', you would likely need to get written permission from the pandas project's copyright holders. However, please note that this is not legal advice, and it would be a good idea to consult with a lawyer who specializes in open-source software or intellectual property law to ensure th