Podcast Summary

Podcast: Talkin' Bout [Infosec] News

Episode: BHIS Podcast: Py2K20 - Transitioning from Python2 to Python3
Host: Black Hills Information Security (BHIS), with lead speaker Joff Thyer
Date: May 31, 2019

Episode Overview

This episode dives into the critical migration from Python 2 to Python 3, discussing technical distinctions, core language changes, migration strategies, and the implications for infosec practitioners and toolsets. Joff Thyer, a pen tester, developer, and SANS instructor, walks listeners through both the practical and theoretical aspects of transitioning code and workflows as Python 2 reaches end-of-life. The session focuses on core language updates rather than third-party modules and is packed with examples, tips, and a lively Q&A.

Key Discussion Points & Insights

1. Why This Transition Matters

• The end-of-life (EOL) for Python 2 means no more updates or support from the core Python team after 2020.

• PEP 394 details how operating systems map the python command to specific Python versions, with Linux and Mac likely defaulting to Python 2 until it's safe to change due to widespread dependencies. For Windows, it's user-controlled and version-dependent.

• The real impact is in maintenance headaches and script breakage, especially for sysadmins and teams relying on legacy scripts.

  "The main barrier for changing the Python symbolic link to Python 3 is going to be breakage of course of third party scripts and packages." — Joff Thyer [05:44]

2. PEP 394 and OS Differences [03:10]

• Linux/Mac use symbolic links to point to Python binaries.
• Windows is more flexible: the user chooses which version is default.
• Regular review of PEP 394; distributions have freedom to change at will.

3. The future Module: A Migration Strategy [06:09]

• The from __future__ import ... statement in Python 2 allows developers to import features from Python 3 for forward compatibility.

• Examples: division behavior, print function.

  "The future statement is a migration feature that allows us to program in...Python 3 syntax within a Python 2 script, a very, very useful feature." — Joff Thyer [07:19]

4. Syntax & Language Changes: Key Differences

• Keywords [07:48]

  • Python 3 adds True, False, and None as true language keywords.

• Division [08:36–11:11]

  • In Python 2, / is integer division (unless operands are floats); Python 3, / always produces a float. Use // for integer (floor) division.

  • Migration tip: from __future__ import division in Python 2 to mimic Python 3.

    "In Python 3, all division mathematically is going to be floating point unless you explicitly use the Floor operator, which is a double slash." — Joff Thyer [09:14]

• Print Function [11:12–13:18]

  • Python 2 uses print as a statement; Python 3 requires parentheses: print().

  • In Python 2, print() prints a tuple, not a string; leads to subtle bugs.

  • New line suppression syntax changed: use end= parameter in Python 3.

    "In Python 3, print has actually become a function. So...you actually have to use parentheses." — Joff Thyer [12:03]

• Format Strings [14:21–17:43]

  • Python 2 supports C-like % formatting; Python 3 adds .format() with {} braces.
  • New .format() method is more explicit and flexible (alignment, fill character, number formatting).
  • Both methods exist in Python 3, but the new style is encouraged.

5. Handling Input [20:23]

• In Python 2, raw_input() is safe for user input, while input() can result in code execution.
• In Python 3, raw_input() is gone; input() is safe and preferred.
• Porting tip: reassign input = raw_input in Python 2 for compatibility.

6. Strings and Encodings [22:08–30:15]

• Major shift: strings in Python 2 = bytes; in Python 3 = Unicode (UTF-8 by default).

• Conversion between str and bytes is explicit; use .encode() and .decode().

• Codec-related functions change; use the codecs module for things like base64 encoding.

• File modes now include text ('t') and binary ('b') modes for reading/writing to handle encoding differences.

• Line ending conversions are handled automatically in text mode.

  "Python 3 treats all strings by the default encoding, which is, you know, UTF8." — Joff Thyer [22:23]

7. Iterables, Memory Efficiency, and Dict Methods [33:20–38:45]

• Python 2: built-ins like range and map return lists.
• Python 3: they return iterable objects/generators, reducing memory usage.
• Dictionary methods: Python 2 has keys(), values(), and items() (return lists) plus iterkeys(), itervalues(), iteritems() (generators).
• Python 3: keys(), values(), and items() return "views" (iterators) by default; no more iterkeys(), etc.
• Dict view objects are "live" and update with dictionary changes.

8. Variable Scope in List Comprehensions [41:08–42:45]

• Python 2: variables from list comprehensions leak into the containing scope.

• Python 3: comprehension variables are local to the expression, preventing accidental variable reuse or overwrites.

  "In Python 3...the x variable that we use inside the list comprehension actually has its own new local scope." — Joff Thyer [41:44]

Essential Quotes & Memorable Moments

• On future adoption fears:

  "I think this is probably the number one issue that scares everybody and holds them back from adopting the new version because this brings some baggage with it." — Joff Thyer on string encoding [22:38]

• On the ease of migration:

  "If you take care of your strings and your print statements and your input side of things...you've pretty much got 80 to 90% of the job done." — Joff Thyer [44:04]

• On Python philosophy:

  "Python should be an intuitive and easy language to learn...the goal was to make it fun, and honestly I think they achieved that goal." — Joff Thyer [50:22]

Memorable Q&A Highlights

• Floating point "weirdness"
  [18:59]

  • Q: "Why is 10/3 3.3333335 and not..., etc.?"
  • A: Floating point hardware limitations, rounding—infinite can't be represented by finite registers.

• Input and security
  [20:23]

  • "The input function in Python 2 is bad. In fact, so much so that, well, it is a code injection issue."

• On Red Team Tools and Modules
  [46:43]

  • "Most Red Team modules have moved to Python 3, but be careful when dealing with things like byte objects and HTTP requests—prefer the 'requests' module."

• IronPython & .NET Integration
  [47:57]

  • "I love Iron Python...I just hope that they go to Python 3, because that's...important, in my opinion."

Timestamps for Important Segments

• 00:29 — Introduction and episode context
• 01:45 — Agenda overview
• 03:10 — PEP 394, OS differences, and EOL
• 06:09 — __future__ module and migration practicality
• 08:36 — Core language changes: Division
• 11:12 — Print function overhaul
• 14:21 — Format strings: Old vs. new style
• 20:23 — Input handling and code injection
• 22:08 — Python 3 string encoding (UTF-8) and codecs
• 30:55 — File handling/modes and implications
• 33:20 — Iterable/generator objects for memory savings
• 38:45 — Dicts: keys/views and live updates
• 41:08 — List comprehensions and variable scope
• 44:04 — Practical conversion advice
• 46:08 — Q&A: Slicing, tool/library compatibility, IronPython, portable code (six lib), variable scope
• 50:09 — Programming philosophy and learning Python

Additional Resources & Takeaways

• Migration Tools: pythonconverter.com — Automates translation from Python 2 to 3.
• Reading:
  • Python docs: docs.python.org
  • SANS Course: Automating Information Security with Python
  • "Python 3 for Scientists" Transition Guide (ReadTheDocs)
  • Wiki: Porting to Py3k — a bilingual quick ref.

Final Recommendations

• Focus your migration on updating print statements, input handling, and proper string encoding.
• Use the from __future__ imports liberally in Python 2 to prep code for the transition.
• For HTTP and web handling, prefer the requests library over urllib.
• When moving tools/scripts, watch out for core language differences impacting security, especially with input.

Closing Remarks

The transition from Python 2 to Python 3 is non-trivial but highly manageable—especially by tackling the common pitfalls outlined: print, input, encoding, and type distinctions. Start modernizing scripts now using the tips Joff shares, and leverage both the official docs and automated tools to ease the process.

"Programming is a journey—learn early, repeat often." — Listener Tim [50:09]