fn.py

xt of mailing list archives posts these days? From "[Python-ideas] Re: Introduce constant variables in Python" (2021) https://mail.python.org/archives/list/python-ideas@python.or... : - pyrsistent: PMap, PVector I'm out of time for this; (reformatting this for HN so that URLs will be auto-linkified but newlines won't be eliminated) here's the full email as .txt, the mailing list archive has a hyperlinkified version with newlines preserved. GH Markdown and CommonMark Markdown also preserve newlines and auto-linkify: From: [@westurner] Date: Thu, Jun 17, 2021, 10:43 AM Subject: Re: [Python-ideas] Re: Introduce constant variables in Python Cc: python-ideas <python-ideas@python.org> On Mon, May 24, 2021 at 5:43 PM Chris Angelico <rosuav@gmail.com> wrote: Requiring that a name not be rebound is well-defined and testable. Requiring that an object not change is either trivial (in the case of, say, an integer) or virtually impossible (in the case of most objects). What would be the advantage of such a declaration? ChrisA ## Existing threads re: consts and applications thereof So, `/? from:me pyrsistent` I found a few results: - "[Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)" 2016-04 https://mail.python.org/pipermail/python-dev/2016-April/143958.html - ~Sandboxing python within python is nontrivial to impossible; consts might help a bit - https://mail.python.org/pipermail/python-dev/2016-April/143958.html - "Proposal to add const-ness type hints to PEP-484" https://mail.python.org/archives/list/python-ideas@python.org/thread/OVPF5I6IOVF6G… - https://github.com/python/typing/issues/242 - "Final names and attributes" https://github.com/python/mypy/pull/5522 This is where `typing.Final` comes from. - "[Python-ideas] "Immutable Builder" Pattern and Operator" https://mail.python.org/pipermail/python-ideas/2017-January/044374.html - [pyrsistent] and "fn.py [do] immutables: https://github.com/kachayev/fn.py/blob/master/README.rst#persistent-data-structures " - "[Python-ideas] Add recordlcass to collections module" https://groups.google.com/g/python-ideas/c/9crHfcCBgYs/m/6_EEaWJAAgAJ - ORMs (e.g. Django, SQLAlchemy) require "dirty state" checking to know which object attributes have changed and need an SQL statement to be executed to synchronize the state; this is relevant because when we're asking for mutable namedtuple we're often trying to do exactly this pattern. - "[Python-ideas] Suggestions: dict.flow_update and dict.__add__" https://www.google.com/search?q=%22%5BPython-ideas%5D+Suggestions%3A+dict.flow_upd… > dicttoolz has functions for working with these objects; including dicttoolz.merge (which returns a reference to the merged dicts but does not mutate the arguments passed). > > https://toolz.readthedocs.io/en/latest/api.html#dicttoolz > https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.merge > > pyrsistent has a PRecord class with invariants and type checking that precedes dataclasses. pyrsistent also has 'freeze' and 'thaw' functions for immutability. PRecord extends PMap, which implements __add__ as self.update(arg) (which does not mutate self) https://github.com/tobgu/pyrsistent/blob/master/README.rst#precord > > https://github.com/tobgu/pyrsistent/blob/master/pyrsistent/_pmap.py - "[Python-ideas] How to prevent shared memory from being corrupted ?" https://www.google.com/search?q=%22How+to+prevent+shared+memory+from+being+corrupt… > PyArrow Plasma object ids, "sealing" makes an object immutable, pyristent > > https://arrow.apache.org/docs/python/plasma.html#object-ids > https://arrow.apache.org/docs/python/plasma.html#creating-an-object-buffer > > Objects are created in Plasma in two stages. First, they are created, which allocates a buffer for the object. At this point, the client can write to the buffer and construct the object within the allocated buffer. [...] - [Python-ideas] Experimenting with dict performance, and an immutable dict https://mail.python.org/archives/list/python-ideas@python.org/message/DNBGUJHDH4UT… > https://pyrsistent.readthedocs.io/en/latest/intro.html#pyrsistent : > >> Pyrsistent is a number of persistent collections (by some referred to as functional data structures). Persistent in the sense that they are immutable. >> >> All methods on a data structure that would normally mutate it instead return a new copy of the structure containing the requested updates. The original structure is left untouched. >> >> This will simplify the reasoning about what a program does since no hidden side effects ever can take place to these data structures. You can rest assured that the object you hold a reference to will remain the same throughout its lifetime and need not worry that somewhere five stack levels below you in the darkest corner of your application someone has decided to remove that element that you expected to be there. >> >> Pyrsistent is influenced by persistent data structures such as those found in the standard library of Clojure. The data structures are designed to share common elements through path copying. It aims at taking these concepts and make them as pythonic as possible so that they can be easily integrated into any python program without hassle. > What would be the advantage of such a declaration? Constants don't need to be locked or unlocked; which is advantageous for parallelism and reasoning about program correctness. True consts (wherein everything referred to in that object is 'frozen' and immutable or at least only modifiable with e.g. copy-on-write) wouldn't require locks, which would be post-GIL advantageous. You could do consts by never releasing a threading.Lock (or similar): - https://docs.python.org/3/library/asyncio-sync.html#locks - https://docs.python.org/3/library/threading.html#lock-objects - This from https://docs.python.org/2/library/sets.html?highlight=immutable#immutable-transforms re ImmutableSet/FrozenSet is not present in the python 3 docs: https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset Though - even if Python enforced normal consts in the language - all of the other code objects would still be mutable, so you still have the impossibility of sandboxing python. Functional and contracts coding styles rely upon invariance; which can be accomplished with various third-party packages that enforce const-ness throughout what may be an object tree behind that reference that would otherwise need to be copy.deepcopy()'d. ## pyrsistent Src: https://github.com/tobgu/pyrsistent > - PVector, similar to a python list > - PMap, similar to dict > - PSet, similar to set > - PRecord, a PMap on steroids with fixed fields, optional type and invariant checking and much more > - PClass, a Python class fixed fields, optional type and invariant checking and much more > - Checked collections, PVector, PMap and PSet with optional type and invariance checks and more > - PBag, similar to collections.Counter > - PList, a classic singly linked list > - PDeque, similar to collections.deque > - Immutable object type (immutable) built on the named tuple > - freeze and thaw functions to convert between python standard collections and pyrsistent collections. > - Flexible transformations of arbitrarily complex structures built from PMaps and PVectors. ## icontract Src: https://github.com/Parquery/icontract > icontract provides design-by-contract to Python3 with informative violation messages and inheritance. > > It also gives a base for a flourishing of a wider ecosystem: > > - A linter pyicontract-lint, > - A sphinx plug-in sphinx-icontract, > - A tool icontract-hypothesis for automated testing and ghostwriting test files which infers Hypothesis strategies based on the contracts, together with IDE integrations such as icontract-hypothesis-vim, icontract-hypothesis-pycharm, and icontract-hypothesis-vscode, > - Directly integrated into CrossHair, a tool for automatic verification of Python programs, together with IDE integrations such as crosshair-pycharm and crosshair-vscode, and > - An integration with FastAPI through fastapi-icontract to enforce contracts on your HTTP API and display them in OpenAPI 3 schema and Swagger UI. https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Invariant_(mathematics)#Invari... [ https://en.wikipedia.org/wiki/Class_invariant ] > What is the difference between "invariant" and "constant" and "final"?

nd output processors where, or when I should use one or the other. > The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain. I also have troubles understanding ItemLoader details. They are totally optional though, and they are no longer in Scrapy tutorial (http://doc.scrapy.org/en/latest/intro/tutorial.html). Item loaders provide features very similar to https://github.com/Suor/funcy or https://github.com/kachayev/fn.py. > The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like? I'm not sure what problems did you have. Scrapy selectors library (https://github.com/scrapy/parsel) is quite similar to PyQuery (esp. when CSS selectors are used), and nothing prevents you from using PyQuery with Scrapy. In future we may add PyQuery (and BeautifulSoup?) support to parsel and provide PyQuery selectors as response.pq (like response.css and response.xpath), +1 to do that. > In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly). Yeah, library interface used to be a problem. It was improved in 1.0 release (there is an official API for integrating Scrapy with Twisted apps and running spiders from user scripts), but there is still more to go. See http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrap.... It probably won't be as easy to integrate with regular Python scripts as mechanize because Scrapy is async. On the other hand, Scrapy is easier to integrate with async servers like Twisted or Tornado. > Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. You may want to check the new 'Scrapy at glance' page (http://doc.scrapy.org/en/latest/intro/overview.html). The main advantage of Scrapy over mechanize is that it handles parallel downloads and have a wide range of built-in extensions you won't have to implement yourselves.

ression. It doesn't escape the scope it's defined in. Just like in the 1.+(_ * 2) example, what comes before it doesn't matter. It's the beginning of an expression. If that was true, then "1 + _ * 2" would be invalid, and "f(_)(1)" would be equivalent to "f(1)". But the former is valid and the latter doesn't work like that. > I was going to attempt to write a Ruby analogue, but I can't actually figure out a clean way to do that. There is a limited Python analogue: https://github.com/kachayev/fn.py#scala-style-lambdas-defini... That only works insofar that the _ placeholder object can seize control over the expression, so "f(_, 2)" won't work unless f explicitly handles _. Implementing the full feature would require language or macro support, but my point was more that a new dynamic language could trivially support it.