Learning Python 5th edition (ebook) by David Ascher and Mark Lutz

I’ll start by saying this is a massive book, covering so many things that’s hard to go through it cover-to-cover without forgetting or missing out on stuff. Overall the book is excellent but we always need to keep in mind it’s “the book over Python” so no matter how big it is, it will always skimp on something.

Rating 4/5 - just of its sheer size, otherwise an excellent book

Chapter1-4

• basic intro, running scripts, exec, open, reload
• basic data types
• immutability (strings, numbers, tuples)
• help, dir functions
• raw strings r’C:\text', bytes b’’
• lists and list comprehension
• tuples immutable (see as read-only)

Chapter 5

• floor division //
• ’{} or {0} string’.format()
• random.random, random.randint(a, b), random.choice([‘a’, ‘b’, ‘c’]), random.shuffle)
• Decimal, Fraction (limit_denominator)
• int(‘0xbo0-f’, base_2_8_16) to convert string to int in a specific base
• Sets {1, 2, 3}
• must be explicitly created
• can contain only immutable/hashable elements
• frozensets can contain any type of object

Chapter 6

• types live with objects not variables
• everything is garbage collected using reference counting, at the point when there are no references
• as variables might point to the same object, changing the object in-place (only for mutable objects), might affect more variables always use X.copy()
  • use copy or deepcopy from the copy module to achieve this
• testing for same object A is B, also one can use sys.getrefcount(A) to see references to this object

Chapter 7 - Strings

• encode decode format() r’rawstrings’ b’bytestrings’ u’unicode\u022’
• triple quotes or block strings used for comments or multi-line strings
• we can test for string_a in string_b
• string formatting % always makes a new string when printing
  • print(“string is %(named)d” % {‘named’: ‘long’})
  • ‘{motto}, {0} and {food}’.format(42, motto=3.14, food=[1, 2])
  • ’{}, {} and {}’.format(‘a’, ‘b’, ‘c’)
  • ‘My %(kind)s runs %(platform)s’ % dict(kind=’laptop’, platform=sys.platform)
• int() chr() ord() for int/char/string conversion

Chapter 8 - Lists and dictionaries

• lists: mutable, sub-nesting, accessed by offset, heterogeneous
• list comprehension [c*2 for c in ‘MONKEY’]
  • map does a similar thing, applying a function to a list(map(abs, [-1, -2, 1, 2, 3]))
• slice assignment a_list[0:2] = [‘one’, ‘two’]
  • [1, 2, ‘spam’].sort() fails in python3 but works in 2.x
• extends adds many elements, append only one
• del(x[element]) deletes one or more elements
• dictionaries: accessed by key, mutable, heterogenous, variable length

• for key in Dict

  (key, value) in Dict.items()

• any immutable (object) can be a key in a dictionary
• dict(zip(keys, values)) to quickly create a dictionary
• has_key is deprecated in python3, use in instead
• dictionary comprehension {k: v for (k,v) in zip([1,2], [‘a’, ‘b’])} or Dict = {k: None for k in ‘spam’}
• items, keys and values() are now views and not elements
• can be sorted using sorted(Iterable), which works for more any iterable

Chapter 9 - tuples, files, …

• tuple: ordered collections of elements, immutable, accessed by offset, can be seen as arrays of object references
  • the objects contained in tuples can be changed
  • namedtuples in collections module:
    • Object = namedtuple(‘Name’, [‘structure’, ‘field’])
    • instance = Object(‘Bob’, 11.23)
• Files are buffered, we need to call flush or close to make sure everything is persisted
  • in python3 there’s a clearer distinction between binary and text (including unicode) files
  • pickle is sort of a serialization basic function
  • there is also a struct module for handling C like structs
  • in pyhon2 there is also file() as well as open(), in py3 only open remained
• elements inside objects can be referneced by other objects so we need to be careful to work with copies if we need not to alter the originals, use slicing copy()/deepcopy() or create new objects
• we should not check for type(object) as it limits functionality

Chapter 10-13

• intro to indentation and some pythonic rules :)
• try: except: clause
• print([object, …][, sep=’ ‘][, end=’\n’][, file=sys.stdout][, flush=False]) for python3
• stream redirection (stdout, stderr), easiest is through assignment sys.stdout = open(‘file’, ‘w’)
• ellipsis can be used instead of pass def to_be_implemented(): …
• valid only in python3
• 2 or 3, 3 or 2, 2 and 3, 2 and []
• ternary expression Z = A if Condition else B
• while conditions: (…) else: (…) # if no break was encountered
• for target in object: (…) else: (…) # if no break was encountered
• zip() or map(); keys = [], values = [] => D = dict(zip(keys, values))
• for (off, item) in enumerate(‘iterable’): …

Chapter 14-17 - Iterators, Scope, …

• iterator, any object which exposes __next__() and raises StopIteration at end of results
  • iterator object (__next__) or iterable object(__iter__)
  • files have their own iterators
• f = open(…), next(f) - will actually call f.__next__()
• list comprehensions might run faster because they run using compiled C code, this is especially true for large sets
• lines = [line.rstrip() for line in open(‘script.py’)]
  • len([line for line in open(fname) if line.strip() != ‘’]) show # of lines
• similar to nested for loops: [x + y for x in ‘abc’ for y in ‘lmn’]
• map itself is also an iterable, applying a function call to a list list(map(str.upper, open(‘script2.py’)))
• sorted() returns a list, not an iterable
• *arg can be used to unpack values
• keys, values, items, range, map, filter, zip return iterable
• triple quoted strings can be accessed w/ __doc__, also existing modules documentation, i.e. sys.__doc__
• help(function/object/module) also gives valuable information
• online help can be served using the built-in server python -m pydoc
• Scope: LEGB rule - local (import builtins), enclosing (comprehension), global, built-in
  • namespace declarations: global, nonlocal; ie X = 1; def …: global X, X=2
• factory functions / closures ; lambda’s
• nonlocal - a name that can be changed in an enclosing scope

Chapter 18 - Arguments

• positional arguments matched from left to right
• keywords matched using the parameters names
• use * or ** varargs to match any number of parameters as tuples (positional) or dictionaries (keyword)
• if we have the following def f(a, *pargs, **kargs): print(a, pargs, kargs)
• then calling f(1, 2, 3, x=1, y=2) will result in 1 (2, 3) {‘y’: 2, ‘x’: 1}
• similarly def kwonly(a, *, b, c='spam'): print(a, b, c) will work for kwonly(1, b=’eggs’) => 1 eggs spam but not for kwonly(1, c=’eggs’) => TypeError: kwonly() missing 1 required keyword-only argument: ‘b’
• sys.getrecursionlimit() to get/set higher limits for stack
• functions can have annotations in Python3
• anonymous functions (lambdas): no ifs are allowed, local scope for variables
  • remember the scoped LEGB rule, the same applies for lambdas
  • f = lambda x, y, z: x + y + z
• functional programming: map, filter, reduce
  • apply function to an iterable - map - map(my_function, list)
  • i.e. list(map(pow, [10,10,10], [2,3,4]))
  • selecting items in iterables - filtering - list(filter((lambda x: x>0), [3, 5, -1, 9, -9, -100]))
  • reduce - returning an element after scanning an iterable: reduce((lambda x, y: x * y), [1, 2, 3, 4]) => 24
• list comprehension collect the result of applying an expression to an iterable, and return a list i.e. res = [ord(x) for x in 'spam']
  • sometimes map or list comprehension are better suited for the task-at-hand, for simple scenarios, you can get either of them quite straightforward
  • as in Python3 file is iterable [line.rstrip() for line in open(‘file’)] == list(map(lambda line: line.rstrip(), open(‘file’)))
• the timeit module used for time length statistics of function calls, pystones

Chapter 22-25 - Modules/Packages/Advanced Modules

• modules are imported only once per session, so in interactive mode you need to use reload instead of just calling import again
• we can use .pth files to define other paths where Python can search for modules
• prefer import over from, when working with modules but no strong preference is given to import
• import as can be used in order to rename modules or to avoid name conflicts
• namespace nesting works down (mod1 imports mod2 which imports module3) but not upwards, so module3 can’t access mod2 or mod1
• reload is imp.reload in Python3; a module must be imported first, before we can reload it
  • using from is not affected by reloads, so we must use import if we want to use reload
• each module folder must contain an __init__.py file and the code within this file is executed automatically on import
  • __all__ lists inside the files define what modules are imported on from * statements
  • from A import b will make it easier in the future if refactoring is needed where as import is a bit more cumbersome to update
  • import A as B addresses this issue and also makes it easier for future adjustments
• we will need to use import and not from only if we want to use the same property across modules and then we will need to specify the full path to the object
  • relative imports, specific to Python 3.x only apply to from imports and only to the interpackage imports starting with . or ..
  • in Python 3.x from A import B are always absolute imports
• types of imports for Python 3.3+
  • basic module imports: import MOD, from MOD import ATTR
  • package imports: import dir1.dir2.MOD, from dir1.MOD import ATTR
  • package relative imports: from . import MOD (relative), import MOD (absolute)
  • namespace packages: import splitdir.MOD
• we can hide some of the names for from by using either __all__ or _Underscore for declarations
• future imports backported to previous versions can make use of the __future__ feature
• use __name__ to be able to import or use as a module a certain file
  • we could also use this feature for (unit) testing the module
• by using triple quoted text blocks we can add comments parsed by the pydoc utility
• sys.path can be changed and updated, but by accessing it directly, it’s also easy to invalidate it
• recursive module reload example
• The path feature which allows. Pth extension files, to define extra paths to search for modules
• You can get current path setup with sys. Path

Chapter 26: OOP

• multiple inheritance is possible
• referring to current object (this - equivalent) is done by using the self keyword
• __init__ method used as a constructor, class name(inherit_1, inherit_2) using LtR order of precedence
• method names surround by double underscores are special hooks
• new-style classes available in Python3 have some built-in hooks available, but not the usual ones
• __str__, __repr__, __add__
  • str is usually used for user-friendly appearance and when calling print explicitly (i.e. print(Manager))
  • repr is used for low-level display on object or when calling the object (i.e. Manager)
• it is advised to use the explicit name of the super class (not super) when we customize a subclass
• example with customizing Person class and Manager which in __init__ calls Person(job=’manager’)
• prefixing a method name with double underscores __method_name will make it pseudo-private, and it will also contain the class name making it unique when looking it up
• pickle: serialize arbitrary python objects
  • pickled classes must be importable
• shelve module provides extra layer that allows storing of objects by key
  • import shelve, open new shelve db = shelve.open(‘file’) then we can call db[key] = object_name

Chapter 30-31: operator overloading and oop techniques

• many operations can be overloaded, i.e. __len/add/bool/lt/gt/iter__
• they always start and end with underscores; example for getitem used for slicing; as an extra, getitem is also used by the for-loop, so iteration, indexing and slicing
• iterable objects: iter and next; end of iteration => raise StopIteration we have to take into account space and time considering the fact we can reuse an object or create new objects for iterable
• using yield (which automatically saves state, generator function) and iter
• __getattr__ to access attributes of the class or raise AttributeError(missing) otherwise
• string representation of the object using repr or str - called automatically when instances are converted to strings
  • in case one is defined, the other remains as an extra representation
  • both must return strings
• add/radd/iadd for addition, right-addition and in-place, usually not all need to be implemented
• calling using __call__ allows to code objects that conform to functions, useful for API
• py3 specific: lt/gt, in py2 there’s a generic cmp overload operator available
• bool/len - any object in Python can be True/False, python tries to infer the boolean value from len
• __del__ is called at gc time, not used in Python
• name mangling: all __attribute in a class will get converted to _Class__attribute as pseudo-private attributes, in order to differentiate between different classes with same attributes
• unbound methods can be without using the class’ instance (no self), only works in py3
• mix-in class example

  def __attrnames(self):
   result = ""
   for attr in sorted(self.__dict__):
    result += "\t%s=%s\n" % (attr,self.__dict__[attr])
   return result
  def __str__(self):
    return "Instance of %s, address %s: %s" %  (self.__class__.__name__,id(self),self.__attrnames())
  

Ch 32: advanced class topics

• new style classes
  • classes and types are merged
  • object serves as root for all classes
  • diamond/MRO search for inheritance instead of DFLR, applies to multi-inheritance
  • proxy classes much more difficult to build and manage
• MRO algo and some examples :))
• __slots__ = array of ivars that need to be assigned before they can be used, and no other ivars can be used
  • optimization mechanism mostly, “best reserved for special cases”
  • special case using __dict__ as a slot ivar
• Properties
  • intercept access and compute variables arbitrarily
  • require new style classes to be used

class P():
   def getage(self): return 22
   def setage(self, new_age): self._age = new_age
   age = property(getage, setage, None, None)

• static methods - py3 allows for methods which don’t require a class to be called
  • explicit defining of class or instance method def A():pass A = classmethod/staticmethod(A)
• function and class decorations
  • extra logic/functionality to functions or classes, they can also be manage later calls to function/classes
• custom decorators are possible for functions/classes and will act as a proxy caller
• super
  • difficult to really grasp, used in MRO, and not widely agreed as best approach, so better to stay away from it unless you really understand what’s it that you’re doing
  • as soon as multiple inheritance is involved, super usage might become problematic as it’s very complex and require all classes to adhere to same calling ways
  • the quote though is really spot on “If the implementation is hard to explain, it’s a bad idea”

Exceptions (ch 34 & ch 35 & ch 36)

• finally code blocks are always executed
• else from try/except/else is executed only if no exception were raised during the try/except code block
• except can be empty, and then we catch all exceptions or we can specify 1 (except NameError) or more (except (A, B)) exceptions encountered
• assert used for trapping user errors and not programming errors
• with EXPRESSION as VARIABLE
  • EXPRESSION might run startup and cleanup code, context manager objects which need to implement __enter__ and exit methods
  • multiple context managers can be nested in newer versions
• exceptions are class-derived only in current versions
• try to create a class hierarchy for exceptions like Overall(Exception) Specific1(Overall) Specific2(Overall)
  • this way you can easily add or update exceptions
  • easy to catch exceptions by people using the code
  • to add more custom text we can implement repr or str
  • not all exceptions are errors, we can use an exception to exit from multiple loops
• to continue to run after exceptions encountered

  try:
    # ...run program...
  except:                         # All uncaught exceptions come here
    import sys
    print('uncaught!', sys.exc_info()[0], sys.exc_info()[1])
  

• operations that fail should be wrapped in try statements
• the profile module can be used to determine bottleneck points in the program

Unicode and bytestrings - ch 37

• encondings / help(encodings) should give some initial details
  • bytes immutable, bytesarray mutable
    • bytes doesn’t support format or % to print text
  • while working with files using ‘b’ binary when opening the file, sets also the data type read
  • py3: str.encode() and bytes.decode() to switch between str and bytes
  • chr/ord to convert chars to int and back
  • String . encode/decode(‘character_encoding’), can be handled automatically when opening file
    • open(…, encoding=’latin-1’)

Managed attributes - ch 38

• property: something = property(getterFunc, setterFunc, deleteFunc, ‘doc style info’)
  • decorators

@property
def address(self):
    return self._address
@address.setter
@def address(self, value):
    self._address = value

• descriptors provide an alternative way to intercept attributes access
  • it allows to route a specific attribute operation to a methods from a different class

Decorators - ch 39

• rebinding of methods and classes to other callables
  • e.g. tracing function calls

class tracer:
    def __init__(self, func):
        self.calls = 0
        self.func = func
    def __call__(self, *args):  # On @ decoration: save original func
        self.calls += 1         # On later calls: run original func
        print('call %s to %s' % (self.calls, self.func.__name__))
        self.func(*args)
        #[...]
@tracer
def spam(a, b, c):
print(a + b + c)         # spam = tracer(spam) # Wraps spam in a decorator object

Metaclasses - ch 40

• :)