The Future of Python Packaging

Page Status:Incomplete Last Reviewed:2014-04-09

The distutils cross-platform build and distribution system was added to the Python standard library in late 2000. This means the current Python software distribution ecosystem (which builds on distutils) has a foundation that is almost 15 years old, which poses a variety of challenges to successful evolution.

The current effort to improve the situation started when the packaging library (also known as distutils2) failed to be accepted into the standard library for Python 3.3. That effort had spanned from 2009 to 2012.

The current effort is managed by the Python Packaging Authority (PyPA), in cooperation with members of the Python core development team.

Goals

• To provide a relatively easy to use software distribution infrastructure that is also fast, reliable and reasonably secure. “Reasonably secure”, due to backwards compatibility constraints preventing turning off some insecure legacy features.
• Although it’s still being defined, to work towards a “Meta-Packaging” [1] system that:
  • Clearly delineates the phases of distribution
  • Allows for multiple interacting tools vs one monolithic tool
  • Specifically allows for alternative build systems, i.e. a “MetaBuild” system.
• To improve the docs for users, including the Python Packaging User Guide, anything related to packaging on docs.python.org, and the project docs for pip, setuptools, virtualenv, and wheel.
• To be progressive, but also be very mindful to not break things that are currently working, due to haste.
• To specifically not focus at first on adding something to the standard library as our solution to our packaging problems. Adding something to the standard library is hard, and once it’s added, it’s a slow process to change it. Most of the current effort is largely focused on 3rd party projects.

How to help

• Get involved with one of mainstream packaging projects.
• Help us catalog and discuss the current problems in packaging and installation. See the The issue tracker for the problems in packaging maintained by PyPA.
• Discuss PEPs and long terms plans at distutils-sig.

Presentations & Articles

In addition to this document, there have been some talks and presentations regarding current and future efforts related to packaging.

• PyCon US, March 2013
  • Directions in Packaging Q & A Panel (aka ”./setup.py install must die”)
• PyCon AU, July 2013
  • Nobody Expects the Python Packaging Authority
• linux.conf.au 2014
  • Python Packaging 2.0: Playing Well With Others (video, article)
• PyCon US, April 2014
  • What is coming in Python packaging
  • Python packaging simplified, for end users, app developers, and open source contributors
• Personal essays
  • Nick Coghlan

Major Todos

Metadata 2.0

See the Metadata Open Issues

• PEP426: Metadata for Python Software Packages 2.0
• PEP440: Version Identification and Dependency Specification
• PEP459: Standard Metadata Extensions for Python Software Packages
• Wheel 1.1
• sdist 2.0
• PEP for common naming schemes
• Installation Database 2.0 (replace PEP376)

PyPI Infrastructure

• Migration from the legacy PyPI server to Warehouse (the preview is available at https://warehouse.python.org/ running off the live PyPI data)
• PEP458: An integration of PyPI with the “The Update Framework (TUF)”
• Improved PyPI upload API

pip

• An internal stable api for pip
• Removal of older pip commands and options that aren’t popular or well maintained (#906, #1046)
• pip needs a real dependency resolver

Docs and Community

• Refactor the virtualenv, setuptools, and wheel docs to be consistent with the “PyPA Standard Docs Template”
• Document pip’s (and more generally pypa’s) deprecation policy (Issue 1611)
• A general release email list for all Pypa projects?

More PEPs

• A “MetaBuild” PEP that would allow projects to specify alternative build systems (i.e. something other than setuptools).
• Wheel 2.0

---

[1]	See Nick Coghlan’s The Phases of Distribution and A Meta-Packaging System