httpx

From the software license: >THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. https://github.com/encode/httpx/blob/master/LICENSE.md The contrib

Biggest problem with httpx I see is authors sanity https://github.com/encode/httpx/discussions/3784. I do not even understand what is his problem (too much or too little woke internet?).

Read for yourself: https://github.com/encode/httpx/discussions/3784 They also effectively killed off MkDocs: https://github.com/mkdocs/mkdocs/discussions/3677

Anthropic made the same change a few weeks after OpenAI did: https://github.com/anthropics/anthropic-sdk-python/releases/... The problem with httpx as a dependency is that it's currently working towards a 1.0 release which will be full of breaking changes. The httpx2 project is essentially a fork that promises not to break the existing API, which makes it a more stable dependency to build against. I wrote a pretty long comment about my concerns for the breaking 1.0 version last year - https://github.com/encode/httpx/discussions/3344#discussionc... - in that comment I recommended the HTTPX project release their 1.0 as a package called httpx2 instead, but a year later we now have an httpx2 (released by a different maintainer) that keeps the old API.

https://github.com/encode/httpx/discussions/3784 > I've closed off access to issues and discussions. > I don't want to continue allowing an online environment with such an absurdly skewed gender representation. I find it intensely unwelcoming, and it's not reflective of the type of working environments I value. https://www.reddit.com/r/Python/comments/1rl5kuq/anyone_know...

> I made a PR to fix it but the maintainers haven't applied it. :-( https://github.com/encode/httpx/pull/3733 Unfortunately, you may be the wrong gender to contribute to Encode repositories like httpx: > I've closed off access to issues and discussions. > I don't want to continue allowing an online environment with such an absurdly skewed gender representation. I find it intensely unwelcoming, and it's not reflective of the type of working environments I value. — https://github.com/encode/httpx/discussions/3784 Discussed on Hacker News here: https://news.ycombinator.com/item?id=47193563 A fork discussed here: https://news.ycombinator.com/item?id=47514603

If you are using "httpx", it's likely caused by a reference cycle. I made a PR to fix it but the maintainers haven't applied it. :-( https://github.com/encode/httpx/pull/3733 The reference cycle httpx creates is kind of a worst-case scenario for the incremental GC issue. Both the generational (3.13 and older) and incremental GC are triggered by the net new "container" objects (objects that have references to others, like lists and not like ints and floats). The short summary is that you need to create more container objects before the incremental GC triggers. In the case of the httpx reference cycle, you have a relatively small number of container objects hanging on to a lot of memory, due the SSL context data (which is a big memory hog). Reverting back to the generational GC was the wise thing to do, even though it's a bit scary to

A very surprising development indeed. The original author (who is also the leading maintainer of httpx, another huge package) started a crusade against male-dominated spaces such as... Github issues and discussions. >So… sharing the source isn’t the problem here. The issue is the GitHub working environment. I’m not interested in the smokey boys-only-club atmosphere, it feels starkly unprofessional. >We’re not having any more conversation in all-male online spaces. Not happening. It looks like 3 weeks ago a similar change happened in httpx - github discussions were disabled. https://github.com/encode/httpx/discussions/3784 >I don't want to continue allowing an online environment with such an absurdly skewed gender representation. I find it intensely unwelcoming, and it's not reflective of the type of working environments I value. As a mkdo

Httpx/httpcore are abandoned and their connection pooling etc implementations are completely broken https://github.com/encode/httpx/issues/3215 Many PRs trying to fix varying issues https://github.com/encode/httpcore/pulls But maintainer(s) are not intrested in fixing any of the issues.

Python has lacked a batteries-included HTTP library that would have both async and sync clients. Httpx (httpcore), which has offered this, is unfortunately pretty much unmaintained and suffering from huge perf issues (https://github.com/encode/httpx/issues/3215). I built pyreqwest HTTP client for Python that is fully Rust based on top of reqwest. It includes all features reqwest offers plus some more. Also including unit testing utilities (mocking, ASGI app support). Go check https://github.com/MarkusSintonen/pyreqwest :)

Exactly as sibling comment says, e.g. let's pretend the popular httpx cli was my project to deploy and run on the server. With only uv installed, i can: uv tool run "httpx[cli] @ git+https://github.com/encode/httpx" To be clear in this example i'm not pulling a package published on pypi, i'm running the HEAD of that git repo (i could do a branch or tag instead). I could use the "uvx" shortcut instead of "uv tool run". I could specify a specific python version (either one already installed on this OS or choose a dist which uv downloads for me). This caches the deps in an isolated virtual env for me. It'll only download the deps in the first run.

What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud. There are many tools, see links below Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience. To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms. https://github.com/lexiforest/curl_cffi https://github.com/encode/httpx https://github.com/scrapy/scrapy https://github.com/apify/crawlee