Added support for progress callbacks in NUS download functions

* Unfinished wiiload module and LZ77 compression code

* Updated WIP LZ77 compressor, still not working yet

* Updated WIP LZ77 compressor, still not working yet (again)

* LZ77 compression is now fully functional! (But still very slow)

* Added compress_lz77 docstrings, temporarily removed unfinished wiiload module

README.md

@@ -2,7 +2,7 @@
 # libWiiPy
 libWiiPy is a modern Python 3 library for handling the various files and formats found on the Wii. It aims to be simple to use, well maintained, and offer as many features as reasonably possible in one library, so that a newly-written Python program could do 100% of its Wii-related work with just one library. It also aims to be fully cross-platform, so that any tools written with it can also be cross-platform.
 
-libWiiPy is inspired by [libWiiSharp](https://github.com/TheShadowEevee/libWiiSharp), which was originally created by `Leathl` and is now maintained by [@TheShadowEevee](https://github.com/TheShadowEevee). If you're looking for a Wii library that isn't in Python, then go check it out!
+libWiiPy is inspired by [libWiiSharp](https://github.com/TheShadowEevee/libWiiSharp), which was originally created by `Leathl` and is now maintained by [@TheShadowEevee](https://github.com/TheShadowEevee).
 
 
 # Features
@@ -10,14 +10,18 @@ This list will expand as libWiiPy is developed, but these features are currently
 - TMD and Ticket parsing/editing (`.tmd`, `.tik`)
 - Title parsing/editing, including content encryption/decryption (both retail and development)
 - WAD file parsing/editing (`.wad`)
-- Downloading titles from the NUS
+- Downloading titles and their components from the NUS
+- Certificate, TMD, and Ticket signature verification
 - Packing and unpacking U8 archives (`.app`, `.arc`)
 - Decompressing ASH files (`.ash`, both the standard variants and the variants found in My Pokémon Ranch)
+- Compressing/Decompressing LZ77-compressed files
 - IOS patching
 - NAND-related functionality:
   - EmuNAND title management (currently requires an existing EmuNAND)
   - `content.map` parsing/editing
+  - `setting.txt` parsing/editing
   - `uid.sys` parsing/editing
+- Limited channel banner parsing/editing
 - Assorted miscellaneous features used to make the other core features possible
 
 For a more detailed look at what's available in libWiiPy, check out our [API docs](https://ninjacheetah.github.io/libWiiPy).
@@ -68,6 +72,3 @@ Thank you to all of the contributors to the documentation on the WiiBrew pages t
 
 ### One additional special thanks to [@DamiDoop](https://github.com/DamiDoop)!
 She made the very cool banner you can see at the top of this README, and has also helped greatly with my sanity throughout debugging this library.
-
-**Note:** While libWiiPy is directly inspired by libWiiSharp and aims to have feature parity with it, no code from either libWiiSharp or Wii.py was used in the making of this library. All code is original and is written by [@NinjaCheetah](https://github.com/NinjaCheetah), [@rvtr](https://github.com/rvtr), and any other GitHub contributors.
-

docs/source/archive/archive.md

@@ -1,15 +1,18 @@
 # libWiiPy.archive Package
 
-## Modules
+## Description
+
 The `libWiiPy.archive` package contains modules for packing and extracting archive formats used by the Wii. This currently includes packing and unpacking support for U8 archives and decompression support for ASH archives.
 
+## Modules
+
 | Module                                 | Description                                             |
 |----------------------------------------|---------------------------------------------------------|
 | [libWiiPy.archive.ash](/archive/ash)   | Provides support for decompressing ASH archives         |
 | [libWiiPy.archive.lz77](/archive/lz77) | Provides support for the LZ77 compression scheme        |
 | [libWiiPy.archive.u8](/archive/u8)     | Provides support for packing and extracting U8 archives |
 
-### libWiiPy.archive Package Contents
+## Full Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/archive/ash.md

@@ -1,5 +1,7 @@
 # libWiiPy.archive.ash Module
 
+## Description
+
 The `libWiiPy.archive.ash` module provides support for handling ASH files, which are a compressed format primarily used in the Wii Menu, but also in some other titles such as My Pokémon Ranch.
 
 At present, libWiiPy only has support for decompressing ASH files, with compression as a planned feature for the future.

docs/source/archive/lz77.md

@@ -1,5 +1,7 @@
 # libWiiPy.archive.lz77 Module
 
+## Description
+
 The `libWiiPy.archive.lz77` module provides support for handling LZ77 compression, which is a compression format used across the Wii and other Nintendo consoles.
 
 ## Module Contents

docs/source/archive/u8.md

@@ -1,5 +1,7 @@
 # libWiiPy.archive.u8 Module
 
+## Description
+
 The `libWiiPy.archive.u8` module provides support for handling U8 archives, which are a non-compressed archive format used extensively on the Wii to join multiple files into one.
 
 This module exposes functions for both packing and unpacking U8 archives, as well as code to parse IMET headers. IMET headers are a header format used specifically for U8 archives containing the banner of a channel, as they store the localized name of the channel along with other banner metadata.

docs/source/conf.py

@@ -17,7 +17,13 @@ release = 'main'
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
-extensions = ['myst_parser', 'sphinx.ext.napoleon', 'sphinx_copybutton', 'sphinx_tippy', 'sphinx_design']
+extensions = [
+    'myst_parser',
+    'sphinx.ext.napoleon',
+    'sphinx_copybutton',
+    'sphinx_tippy',
+    'sphinx_design'
+]
 
 templates_path = ['_templates']
 exclude_patterns = ["Thumbs.db", ".DS_Store"]
@@ -31,7 +37,8 @@ html_logo = "banner.png"
 html_title = "libWiiPy API Docs"
 html_theme_options = {
     "repository_url": "https://github.com/NinjaCheetah/libWiiPy",
-    "use_repository_button": True
+    "use_repository_button": True,
+    "show_toc_level": 3
 }
 
 # MyST Configuration

docs/source/media/banner.md

@@ -1,5 +1,7 @@
 # libWiiPy.media.banner Module
 
+## Description
+
 The `libWiiPy.media.banner` module is essentially a stub at this point in time. It only provides one dataclass that is likely to become a traditional class when fully implemented. It is not recommended to use this module for anything yet.
 
 ## Module Contents

docs/source/media/media.md

@@ -1,13 +1,16 @@
 # libWiiPy.media Package
 
-## Modules
+## Description
+
 The `libWiiPy.media` package contains modules used for parsing and editing media formats used by the Wii. This currently only includes limited support for parsing channel banners.
 
+## Modules
+
 | Module                                 | Description                                       |
 |----------------------------------------|---------------------------------------------------|
 | [libWiiPy.media.banner](/media/banner) | Provides support for basic channel banner parsing |
 
-### libWiiPy.media Package Contents
+## Full Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/nand/emunand.md

@@ -1,5 +1,7 @@
 # libWiiPy.nand.emunand Module
 
+## Description
+
 The `libWiiPy.nand.emunand` module provides support for creating and managing Wii EmuNANDs. At present, you cannot create an EmuNAND compatible with something like NEEK on a real Wii with the features provided by this library, but you can create an EmuNAND compatible with Dolphin.
 
 ## Module Contents

docs/source/nand/nand.md

@@ -1,15 +1,18 @@
 # libWiiPy.nand Package
 
-## Modules
+## Description
+
 The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNANDs as well as modules for parsing and editing core system files found on the Wii's NAND.
 
+## Modules
+
 | Module                                 | Description                                                                                                                      |
 |----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
 | [libWiiPy.nand.emunand](/nand/emunand) | Provides support for parsing, creating, and editing EmuNANDs                                                                     |
 | [libWiiPy.nand.setting](/nand/setting) | Provides support for parsing, creating, and editing `setting.txt`, which is used to store the console's region and serial number |
 | [libWiiPy.nand.sys](/nand/sys)         | Provides support for parsing, creating, and editing `uid.sys`, which is used to store a log of all titles run on a console       |
 
-### libWiiPy.nand Package Contents
+## Full Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/nand/setting.md

@@ -1,5 +1,7 @@
 # libWiiPy.nand.setting Module
 
+## Description
+
 The `libWiiPy.nand.setting` module provides support for handling the Wii's `setting.txt` file. This file is stored as part of the Wii Menu's save data (stored in `/title/00000001/00000002/data/`) and is an encrypted text file that's primarily used to store your console's serial number and region information.
 
 This module allows you to encrypt or decrypt this file, and exposes the keys stored in it for editing.

docs/source/nand/sys.md

@@ -1,5 +1,7 @@
 # libWiiPy.nand.sys Module
 
+## Description
+
 The `libWiiPy.nand.sys` module provides support for editing system files used on the Wii. Currently, it only offers support for `uid.sys`, which keeps a record of the Title IDs of every title launched on the console, assigning each one a unique ID.
 
 ## Module Contents

docs/source/title/cert.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.cert Module
 
+## Description
+
 The `libWiiPy.title.cert` module provides support for parsing the various signing certificates used by the Wii for content validation.
 
 This module allows you to write your own code for validating the authenticity of a TMD or Ticket by providing the certificates from the Wii's certificate chain. Both retail and development certificate chains are supported.

docs/source/title/commonkeys.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.commonkeys Module
 
+## Description
+
 The `libWiiPy.title.commonkeys` module simply provides easy access to the Wii's common encryption keys.
 
 ## Module Contents

docs/source/title/content.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.content Module
 
+## Description
+
 The `libWiiPy.title.content` module provides support for parsing, adding, removing, and editing content files from a digital Wii title.
 
 ## Module Contents

docs/source/title/crypto.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.crypto Module
 
+## Description
+
 The `libWiiPy.title.crypto` module provides low-level cryptography functions required for handling digital Wii titles. It does not expose many functions that are likely to be required during typical use, and instead acts more as a dependency for other modules.
 
 ## Module Contents

docs/source/title/iospatcher.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.iospatcher Module
 
+## Description
+
 The `libWiiPy.title.iospatcher` module provides support for applying various binary patches to IOS' ES module. These patches and what they do can be found attached to the methods used to apply them.
 
 ## Module Contents

docs/source/title/nus.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.nus Module
 
+## Description
+
 The `libWiiPy.title.nus` module provides support for downloading digital Wii titles from the Nintendo Update Servers. This module provides easy methods for downloading TMDs, common Tickets (when present), encrypted content, and the certificate chain.
 
 ## Module Contents
@@ -9,4 +11,5 @@ The `libWiiPy.title.nus` module provides support for downloading digital Wii tit
    :members:
    :undoc-members:
    :show-inheritance:
+   :special-members: __call__
 ```

docs/source/title/ticket.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.ticket Module
 
+## Description
+
 The `libWiiPy.title.ticket` module provides support for handling Tickets, which are the license files used to decrypt the content of digital titles during installation. This module allows for easy parsing and editing of Tickets.
 
 ## Module Contents

docs/source/title/title.md

@@ -1,7 +1,9 @@
 # libWiiPy.title Package
 
+## Description
+The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, as it offers the functionality one would be most likely to need. As a result, it gets the most attention during development and should be the most reliable.
+
 ## Modules
-The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, and therefore offers the most functionality.
 
 | Module                                         | Description                                                                                                                   |
 |------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
@@ -17,7 +19,7 @@ The `libWiiPy.title` package contains modules for interacting with Wii titles. T
 | [libWiiPy.title.util](/title/util)             | Provides some simple utility functions relating to titles                                                                     |
 | [libWiiPy.title.wad](/title/wad)               | Provides support for parsing and editing WAD files, allowing you to load each component into the other available classes      |
 
-### libWiiPy.title Package Contents
+## Full Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/title/title.title.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.title Module
 
+## Description
+
 The `libWiiPy.title.title` module provides a high-level interface for handling all the components of a digital Wii title through one class. It allows for directly importing a WAD, and will automatically extract the various components and load them into their appropriate classes. Additionally, it provides duplicates of some methods found in those classes that require fewer arguments, as it has the context of the other components and is able to retrieve additional data automatically.
 
 An example of that idea can be seen with the method `get_content_by_index()`. In its original definition, which can be seen at <project:#libWiiPy.title.content.ContentRegion.get_content_by_index>, you are required to supply the Title Key for the title that the content is sourced from. In contrast, when using <project:#libWiiPy.title.title.Title.get_content_by_index>, you do not need to supply a Title Key, as the Title object already has the context of the Ticket and can retrieve the Title Key from it automatically. In a similar vein, this module provides the easiest route for verifying that a title is legitimately signed by Nintendo. The method <project:#libWiiPy.title.title.Title.get_is_signed> is able to access the entire certificate chain, the TMD, and the Ticket, and is therefore able to verify all components of the title by itself.
@@ -12,5 +14,4 @@ Because using <project:#libWiiPy.title.title.Title> allows many operations to be
 .. automodule:: libWiiPy.title.title
    :members:
    :undoc-members:
-   :show-inheritance:
 ```

docs/source/title/tmd.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.tmd Module
 
+## Description
+
 The `libWiiPy.title.tmd` module provides support for handling TMD (Title Metadata) files, which contain the metadata of both digital and physical Wii titles. This module allows for easy parsing and editing of TMDs.
 
 ## Module Contents

docs/source/title/util.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.util Module
 
+## Description
+
 The `libWiiPy.title.util` module provides common utility functions internally. It is not designed to be used directly.
 
 ## Module Contents

docs/source/title/wad.md

@@ -1,5 +1,7 @@
 # libWiiPy.title.wad Module
 
+## Description
+
 The `libWiiPy.title.wad` module provides support for handling WAD (Wii Archive Data) files, which is the format used to deliver digital Wii titles. This module allows for extracting the various components for a WAD, as well as properly padding and writing out that data when it has been edited using other modules.
 
 ## Module Contents

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "libWiiPy"
-version = "0.6.0"
+version = "1.0.0"
 authors = [
     { name="NinjaCheetah", email="ninjacheetah@ncxprogramming.com" },
     { name="Lillian Skinner", email="lillian@randommeaninglesscharacters.com" }
@@ -13,7 +13,7 @@ classifiers = [
     #   3 - Alpha
     #   4 - Beta
     #   5 - Production/Stable
-    "Development Status :: 4 - Beta",
+    "Development Status :: 5 - Production/Stable",
     "Intended Audience :: Developers",
     "License :: OSI Approved :: MIT License",
     "Operating System :: OS Independent",

src/libWiiPy/archive/lz77.py

@@ -5,6 +5,7 @@
 
 import io
 from dataclasses import dataclass as _dataclass
+from typing import List as _List
 
 
 _LZ_MIN_DISTANCE = 0x01   # Minimum distance for each reference.
@@ -20,19 +21,20 @@ class _LZNode:
     weight: int = 0
 
 
-def _compress_compare_bytes(byte1: bytes, offset1: int, byte2: bytes, offset2: int, abs_len_max: int) -> int:
-    # Compare bytes up to the maximum length we can match.
+def _compress_compare_bytes(buffer: _List[int], offset1: int, offset2: int, abs_len_max: int) -> int:
+    # Compare bytes up to the maximum length we can match. Start by comparing the first 3 bytes, since that's the
+    # minimum match length and this allows for a more optimized early exit.
     num_matched = 0
     while num_matched < abs_len_max:
-        if byte1[offset1 + num_matched] != byte2[offset2 + num_matched]:
+        if buffer[offset1 + num_matched] != buffer[offset2 + num_matched]:
             break
         num_matched += 1
     return num_matched
 
 
-def _compress_search_matches(buffer: bytes, pos: int) -> (int, int):
+def _compress_search_matches_optimized(buffer: _List[int], pos: int) -> (int, int):
     bytes_left = len(buffer) - pos
-    global _LZ_MAX_DISTANCE, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
+    global _LZ_MAX_DISTANCE, _LZ_MIN_LENGTH, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
     # Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
     # only look as far back as we've gone.
     max_dist = min(_LZ_MAX_DISTANCE, pos)
@@ -43,7 +45,7 @@ def _compress_search_matches(buffer: bytes, pos: int) -> (int, int):
     biggest_match, biggest_match_pos = 0, 0
     # Search for matches.
     for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
-        num_matched = _compress_compare_bytes(buffer, pos - i, buffer, pos, max_len)
+        num_matched = _compress_compare_bytes(buffer, pos - i, pos, max_len)
         if num_matched > biggest_match:
             biggest_match = num_matched
             biggest_match_pos = i
@@ -52,6 +54,25 @@ def _compress_search_matches(buffer: bytes, pos: int) -> (int, int):
     return biggest_match, biggest_match_pos
 
 
+def _compress_search_matches_greedy(buffer: _List[int], pos: int) -> (int, int):
+    # Finds and returns the first valid match, rather that finding the best one.
+    bytes_left = len(buffer) - pos
+    global _LZ_MAX_DISTANCE, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
+    # Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
+    # only look as far back as we've gone.
+    max_dist = min(_LZ_MAX_DISTANCE, pos)
+    # Default to only matching up to 18 bytes, unless fewer than 18 bytes remain, in which case we can only match
+    # up to that many bytes.
+    max_len = min(_LZ_MAX_LENGTH, bytes_left)
+    match, match_pos = 0, 0
+    for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
+        match = _compress_compare_bytes(buffer, pos - i, pos, max_len)
+        match_pos = i
+        if match >= _LZ_MIN_LENGTH or match == max_len:
+            break
+    return match, match_pos
+
+
 def _compress_node_is_ref(node: _LZNode) -> bool:
     return node.len >= _LZ_MIN_LENGTH
 
@@ -64,40 +85,30 @@ def _compress_get_node_cost(length: int) -> int:
     return 1 + (num_bytes * 8)
 
 
-def compress_lz77(data: bytes) -> bytes:
-    """
-    Compresses data using the Wii's LZ77 compression algorithm and returns the compressed result.
-
-    Parameters
-    ----------
-    data: bytes
-        The data to compress.
-
-    Returns
-    -------
-    bytes
-        The LZ77-compressed data.
-    """
+def _compress_lz77_optimized(data: bytes) -> bytes:
+    # Optimized compressor based around a node graph that finds optimal string matches. Originally the default
+    # implementation, but unfortunately it's very slow.
     nodes = [_LZNode() for _ in range(len(data))]
     # Iterate over the uncompressed data, starting from the end.
     pos = len(data)
     global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
+    data_list = list(data)
     while pos:
         pos -= 1
         node = nodes[pos]
         # Limit the maximum search length when we're near the end of the file.
-        max_search_len = min(_LZ_MAX_LENGTH, len(data) - pos)
+        max_search_len = min(_LZ_MAX_LENGTH, len(data_list) - pos)
         if max_search_len < _LZ_MIN_DISTANCE:
             max_search_len = 1
         # Initialize as 1 for each, since that's all we could use if we weren't compressing.
         length, dist = 1, 1
         if max_search_len >= _LZ_MIN_LENGTH:
-            length, dist = _compress_search_matches(data, pos)
+            length, dist = _compress_search_matches_optimized(data_list, pos)
         # Treat as direct bytes if it's too short to copy.
         if length == 0 or length < _LZ_MIN_LENGTH:
             length = 1
         # If the node goes to the end of the file, the weight is the cost of the node.
-        if (pos + length) == len(data):
+        if (pos + length) == len(data_list):
             node.len = length
             node.dist = dist
             node.weight = _compress_get_node_cost(length)
@@ -117,7 +128,7 @@ def compress_lz77(data: bytes) -> bytes:
             node.len = len_best
             node.dist = dist
             node.weight = weight_best
-    # Write the header data.
+    # Write the compressed data.
     with io.BytesIO() as buffer:
         # Write the header data.
         buffer.write(b'LZ77\x10')  # The LZ type on the Wii is *always* 0x10.
@@ -155,6 +166,75 @@ def compress_lz77(data: bytes) -> bytes:
     return out_data
 
 
+def _compress_lz77_greedy(data: bytes) -> bytes:
+    # Greedy compressor that processes the file start to end and saves the first matches found. Faster than the
+    # optimized implementation, but creates larger files.
+    global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
+    with io.BytesIO() as buffer:
+        # Write the header data.
+        buffer.write(b'LZ77\x10')  # The LZ type on the Wii is *always* 0x10.
+        buffer.write(len(data).to_bytes(3, 'little'))
+
+        src_pos = 0
+        data_list = list(data)
+        while src_pos < len(data):
+            head = 0
+            head_pos = buffer.tell()
+            buffer.write(b'\x00')  # Reserve a byte for the chunk head.
+
+            i = 0
+            while i < 8 and src_pos < len(data):
+                length, dist = _compress_search_matches_greedy(data_list, src_pos)
+                # This is a reference node.
+                if length >= _LZ_MIN_LENGTH:
+                    encoded = (((length - _LZ_MIN_LENGTH) & 0xF) << 12) | ((dist - _LZ_MIN_DISTANCE) & 0xFFF)
+                    buffer.write(encoded.to_bytes(2))
+                    head = (head | (1 << (7 - i))) & 0xFF
+                    src_pos += length
+                # This is a direct copy node.
+                else:
+                    buffer.write(data[src_pos:src_pos + 1])
+                    src_pos += 1
+                i += 1
+
+            pos = buffer.tell()
+            buffer.seek(head_pos)
+            buffer.write(head.to_bytes(1))
+            buffer.seek(pos)
+
+        buffer.seek(0)
+        out_data = buffer.read()
+    return out_data
+
+
+def compress_lz77(data: bytes, compression_level: int = 1) -> bytes:
+    """
+    Compresses data using the Wii's LZ77 compression algorithm and returns the compressed result. Supports two
+    different levels of compression, one based around a "greedy" LZ compression algorithm and the other based around
+    an optimized LZ compression algorithm. The greedy compressor, level 1, will produce a larger compressed file but
+    will run noticeably faster than the optimized compressor, which is level 2, especially for larger data.
+
+    Parameters
+    ----------
+    data: bytes
+        The data to compress.
+    compression_level: int
+        The compression level to use, either 1 and 2. Default value is 1.
+
+    Returns
+    -------
+    bytes
+        The LZ77-compressed data.
+    """
+    if compression_level == 1:
+        out_data = _compress_lz77_greedy(data)
+    elif compression_level == 2:
+        out_data = _compress_lz77_optimized(data)
+    else:
+        raise ValueError(f"Invalid compression level \"{compression_level}\"!\"")
+    return out_data
+
+
 def decompress_lz77(lz77_data: bytes) -> bytes:
     """
     Decompresses LZ77-compressed data and returns the decompressed result. Supports data both with and without the

src/libWiiPy/media/banner.py

@@ -14,16 +14,10 @@ class IMD5Header:
 
     An IMD5 header is always 32 bytes long.
 
-    Attributes
-    ----------
-    magic : str
-        Magic number for the header, should be "IMD5".
-    file_size : int
-        The size of the file this header precedes.
-    zeros : int
-        8 bytes of zero padding.
-    md5_hash : bytes
-        The MD5 hash of the file this header precedes.
+    :ivar magic: Magic number for the header, should be "IMD5".
+    :ivar file_size: The size of the file this header precedes.
+    :ivar zeros: 8 bytes of zero padding.
+    :ivar md5_hash: The MD5 hash of the file this header precedes.
     """
     magic: str  # Should always be "IMD5"
     file_size: int

src/libWiiPy/nand/emunand.py

@@ -128,6 +128,10 @@ class EmuNAND:
         uid_sys = _UidSys()
         if not uid_sys_path.exists():
             uid_sys.create()
+        else:
+            uid_sys.load(uid_sys_path.read_bytes())
+        uid_sys.add(title.tmd.title_id)
+        uid_sys_path.write_bytes(uid_sys.dump())
 
     def uninstall_title(self, tid: str) -> None:
         """
@@ -170,12 +174,8 @@ class EmuNAND:
         An InstalledTitles object that is used to track a title type and any titles that belong to that type that are
         installed to an EmuNAND.
 
-        Attributes
-        ----------
-        type : str
-            The type (Title ID high) of the installed titles.
-        titles : List[str]
-            The Title ID low of each installed title.
+        :ivar type: The type (Title ID high) of the installed titles.
+        :ivar titles: The Title ID low of each installed title.
         """
         type: str
         titles: List[str]

src/libWiiPy/nand/sys.py

@@ -77,7 +77,8 @@ class UidSys:
 
     def add(self, title_id: str | bytes) -> int:
         """
-        Adds a new Title ID to the uid.sys file and returns the UID assigned to that title.
+        Adds a new Title ID to the uid.sys file and returns the UID assigned to that title. The new entry will only
+        be added if the provided Title ID doesn't already have an assigned UID.
 
         Parameters
         ----------
@@ -106,6 +107,11 @@ class UidSys:
             title_id_converted = title_id
         else:
             raise TypeError("Title ID type is not valid! It must be either type str or bytes.")
+        # Ensure this TID hasn't already been assigned a UID. If it has, just exit early and return the UID.
+        if self.uid_entries.count != 0:
+            for entry in self.uid_entries:
+                if entry.title_id == title_id_converted:
+                    return entry.uid
         # Generate the new UID by incrementing the current highest UID by 1.
         try:
             new_uid = self.uid_entries[-1].uid + 1

src/libWiiPy/title/nus.py

@@ -4,9 +4,9 @@
 # See https://wiibrew.org/wiki/NUS for details about the NUS
 
 import requests
-import hashlib
-from typing import List
-from urllib.parse import urlparse as _urlparse
+#import hashlib
+from typing import Any, List, Protocol
+#from urllib.parse import urlparse as _urlparse
 from .title import Title
 from .tmd import TMD
 from .ticket import Ticket
@@ -14,13 +14,36 @@ from .ticket import Ticket
 _nus_endpoint = ["http://nus.cdn.shop.wii.com/ccs/download/", "http://ccs.cdn.wup.shop.nintendo.net/ccs/download/"]
 
 
+class DownloadCallback(Protocol):
+    """
+    The format of a callable passed to a NUS download function.
+    """
+    def __call__(self, done: int, total: int) -> Any:
+        """
+        This function will be called with the current number of bytes downloaded and the total size of the file being
+        downloaded.
+
+        Parameters
+        ----------
+        done : int
+            The number of bytes already downloaded.
+        total : int
+            The total size of the file being downloaded.
+        """
+        ...
+
+
 def download_title(title_id: str, title_version: int = None, wiiu_endpoint: bool = False,
-                   endpoint_override: str = None) -> Title:
+                   endpoint_override: str = None, progress: DownloadCallback = lambda done, total: None) -> Title:
     """
     Download an entire title and all of its contents, then load the downloaded components into a Title object for
-    further use. This method is NOT recommended for general use, as it has absolutely no verbosity. It is instead
+    further use. This method is NOT recommended for general use, as it has extremely limited verbosity. It is instead
     recommended to call the individual download methods instead to provide more flexibility and output.
 
+    Be aware that you will receive fairly vague feedback from this function if you attach a progress callback. The
+    callback will be connected to each of the individual functions called by this function, but there will be no
+    indication of which function is currently running, just the progress of its download.
+
     Parameters
     ----------
     title_id : str
@@ -32,27 +55,34 @@ def download_title(title_id: str, title_version: int = None, wiiu_endpoint: bool
     endpoint_override: str, optional
         A custom endpoint URL to use instead of the standard Wii or Wii U endpoints. Defaults to no override, and if
         set entirely overrides the "wiiu_endpoint" parameter.
+    progress: DownloadCallback, optional
+        A callback function used to return the progress of the downloads. The provided callable must match the signature
+        defined in DownloadCallback.
 
     Returns
     -------
     Title
         A Title object containing all the data from the downloaded title.
+
+    See Also
+    --------
+    libWiiPy.title.nus.DownloadCallback
     """
     # First, create the new title.
     title = Title()
     # Download and load the certificate chain, TMD, and Ticket.
     title.load_cert_chain(download_cert_chain(wiiu_endpoint, endpoint_override))
-    title.load_tmd(download_tmd(title_id, title_version, wiiu_endpoint, endpoint_override))
-    title.load_ticket(download_ticket(title_id, wiiu_endpoint, endpoint_override))
+    title.load_tmd(download_tmd(title_id, title_version, wiiu_endpoint, endpoint_override, progress))
+    title.load_ticket(download_ticket(title_id, wiiu_endpoint, endpoint_override, progress))
     # Download all contents
     title.load_content_records()
-    title.content.content_list = download_contents(title_id, title.tmd, wiiu_endpoint, endpoint_override)
+    title.content.content_list = download_contents(title_id, title.tmd, wiiu_endpoint, endpoint_override, progress)
     # Return the completed title.
     return title
 
 
 def download_tmd(title_id: str, title_version: int = None, wiiu_endpoint: bool = False,
-                 endpoint_override: str = None) -> bytes:
+                 endpoint_override: str = None, progress: DownloadCallback = lambda done, total: None) -> bytes:
     """
     Downloads the TMD of the Title specified in the object. Will download the latest version by default, or another
     version if it was manually specified in the object.
@@ -68,11 +98,18 @@ def download_tmd(title_id: str, title_version: int = None, wiiu_endpoint: bool =
     endpoint_override: str, optional
         A custom endpoint URL to use instead of the standard Wii or Wii U endpoints. Defaults to no override, and if
         set entirely overrides the "wiiu_endpoint" parameter.
+    progress: DownloadCallback, optional
+        A callback function used to return the progress of the download. The provided callable must match the signature
+        defined in DownloadCallback.
 
     Returns
     -------
     bytes
         The TMD file from the NUS.
+
+    See Also
+    --------
+    libWiiPy.title.nus.DownloadCallback
     """
     # Build the download URL. The structure is download/<TID>/tmd for latest and download/<TID>/tmd.<version> for
     # when a specific version is requested.
@@ -89,7 +126,7 @@ def download_tmd(title_id: str, title_version: int = None, wiiu_endpoint: bool =
         tmd_url += "." + str(title_version)
     # Make the request.
     try:
-        tmd_request = requests.get(url=tmd_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
+        response = requests.get(url=tmd_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
     except requests.exceptions.ConnectionError:
         if endpoint_override:
             raise ValueError("A connection could not be made to the NUS endpoint. Please make sure that your endpoint "
@@ -97,11 +134,16 @@ def download_tmd(title_id: str, title_version: int = None, wiiu_endpoint: bool =
         else:
             raise Exception("A connection could not be made to the NUS endpoint. The NUS may be unavailable.")
     # Handle a 404 if the TID/version doesn't exist.
-    if tmd_request.status_code != 200:
+    if response.status_code != 200:
         raise ValueError("The requested Title ID or TMD version does not exist. Please check the Title ID and Title"
                          " version and then try again.")
-    # Save the raw TMD.
-    raw_tmd = tmd_request.content
+    total_size = int(response.headers["Content-Length"])
+    progress(0, total_size)
+    # Stream the TMD's data in chunks so that we can post updates to the callback function (assuming one was supplied).
+    raw_tmd = b""
+    for chunk in response.iter_content(512):
+        raw_tmd += chunk
+        progress(len(raw_tmd), total_size)
     # Use a TMD object to load the data and then return only the actual TMD.
     tmd_temp = TMD()
     tmd_temp.load(raw_tmd)
@@ -109,7 +151,8 @@ def download_tmd(title_id: str, title_version: int = None, wiiu_endpoint: bool =
     return tmd
 
 
-def download_ticket(title_id: str, wiiu_endpoint: bool = False, endpoint_override: str = None) -> bytes:
+def download_ticket(title_id: str, wiiu_endpoint: bool = False, endpoint_override: str = None,
+                    progress: DownloadCallback = lambda done, total: None) -> bytes:
     """
     Downloads the Ticket of the Title specified in the object. This will only work if the Title ID specified is for
     a free title.
@@ -123,11 +166,18 @@ def download_ticket(title_id: str, wiiu_endpoint: bool = False, endpoint_overrid
     endpoint_override: str, optional
         A custom endpoint URL to use instead of the standard Wii or Wii U endpoints. Defaults to no override, and if
         set entirely overrides the "wiiu_endpoint" parameter.
+    progress: DownloadCallback, optional
+        A callback function used to return the progress of the download. The provided callable must match the signature
+        defined in DownloadCallback.
 
     Returns
     -------
     bytes
         The Ticket file from the NUS.
+
+    See Also
+    --------
+    libWiiPy.title.nus.DownloadCallback
     """
     # Build the download URL. The structure is download/<TID>/cetk, and cetk will only exist if this is a free
     # title.
@@ -141,18 +191,23 @@ def download_ticket(title_id: str, wiiu_endpoint: bool = False, endpoint_overrid
     ticket_url = endpoint_url + title_id + "/cetk"
     # Make the request.
     try:
-        ticket_request = requests.get(url=ticket_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
+        response = requests.get(url=ticket_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
     except requests.exceptions.ConnectionError:
         if endpoint_override:
             raise ValueError("A connection could not be made to the NUS endpoint. Please make sure that your endpoint "
                              "override is valid.")
         else:
             raise Exception("A connection could not be made to the NUS endpoint. The NUS may be unavailable.")
-    if ticket_request.status_code != 200:
+    if response.status_code != 200:
         raise ValueError("The requested Title ID does not exist, or refers to a non-free title. Tickets can only"
                          " be downloaded for titles that are free on the NUS.")
-    # Save the raw cetk file.
-    cetk = ticket_request.content
+    total_size = int(response.headers["Content-Length"])
+    progress(0, total_size)
+    # Stream the Ticket's data just like with the TMD.
+    cetk = b""
+    for chunk in response.iter_content(chunk_size=1024):
+        cetk += chunk
+        progress(len(cetk), total_size)
     # Use a Ticket object to load only the Ticket data from cetk and return it.
     ticket_temp = Ticket()
     ticket_temp.load(cetk)
@@ -212,7 +267,7 @@ def download_cert_chain(wiiu_endpoint: bool = False, endpoint_override: str = No
 
 
 def download_content(title_id: str, content_id: int, wiiu_endpoint: bool = False,
-                     endpoint_override: str = None) -> bytes:
+                     endpoint_override: str = None, progress: DownloadCallback = lambda done, total: None) -> bytes:
     """
     Downloads a specified content for the title specified in the object.
 
@@ -227,11 +282,18 @@ def download_content(title_id: str, content_id: int, wiiu_endpoint: bool = False
     endpoint_override: str, optional
         A custom endpoint URL to use instead of the standard Wii or Wii U endpoints. Defaults to no override, and if
         set entirely overrides the "wiiu_endpoint" parameter.
+    progress: DownloadCallback, optional
+        A callback function used to return the progress of the download. The provided callable must match the signature
+        defined in DownloadCallback.
 
     Returns
     -------
     bytes
         The downloaded content.
+
+    See Also
+    --------
+    libWiiPy.title.nus.DownloadCallback
     """
     # Build the download URL. The structure is download/<TID>/<Content ID>.
     content_id_hex = hex(content_id)[2:]
@@ -247,23 +309,29 @@ def download_content(title_id: str, content_id: int, wiiu_endpoint: bool = False
     content_url = endpoint_url + title_id + "/000000" + content_id_hex
     # Make the request.
     try:
-        content_request = requests.get(url=content_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
+        response = requests.get(url=content_url, headers={'User-Agent': 'wii libnup/1.0'}, stream=True)
     except requests.exceptions.ConnectionError:
         if endpoint_override:
             raise ValueError("A connection could not be made to the NUS endpoint. Please make sure that your endpoint "
                              "override is valid.")
         else:
             raise Exception("A connection could not be made to the NUS endpoint. The NUS may be unavailable.")
-    if content_request.status_code != 200:
+    if response.status_code != 200:
         raise ValueError("The requested Title ID does not exist, or an invalid Content ID is present in the"
                          " content records provided.\n Failed while downloading Content ID: 000000" +
                          content_id_hex)
-    content_data = content_request.content
-    return content_data
+    total_size = int(response.headers["Content-Length"])
+    progress(0, total_size)
+    # Stream the content just like the TMD/Ticket.
+    content = b""
+    for chunk in response.iter_content(chunk_size=1024):
+        content += chunk
+        progress(len(content), total_size)
+    return content
 
 
-def download_contents(title_id: str, tmd: TMD, wiiu_endpoint: bool = False,
-                      endpoint_override: str = None) -> List[bytes]:
+def download_contents(title_id: str, tmd: TMD, wiiu_endpoint: bool = False, endpoint_override: str = None,
+                      progress: DownloadCallback = lambda done, total: None) -> List[bytes]:
     """
     Downloads all the contents for the title specified in the object. This requires a TMD to already be available
     so that the content records can be accessed.
@@ -279,11 +347,18 @@ def download_contents(title_id: str, tmd: TMD, wiiu_endpoint: bool = False,
     endpoint_override: str, optional
         A custom endpoint URL to use instead of the standard Wii or Wii U endpoints. Defaults to no override, and if
         set entirely overrides the "wiiu_endpoint" parameter.
+    progress: DownloadCallback, optional
+        A callback function used to return the progress of the downloads. The provided callable must match the signature
+        defined in DownloadCallback.
 
     Returns
     -------
     List[bytes]
         A list of all the downloaded contents.
+
+    See Also
+    --------
+    libWiiPy.title.nus.DownloadCallback
     """
     # Retrieve the content records from the TMD.
     content_records = tmd.content_records
@@ -295,7 +370,7 @@ def download_contents(title_id: str, tmd: TMD, wiiu_endpoint: bool = False,
     content_list = []
     for content_id in content_ids:
         # Call self.download_content() for each Content ID.
-        content = download_content(title_id, content_id, wiiu_endpoint, endpoint_override)
+        content = download_content(title_id, content_id, wiiu_endpoint, endpoint_override, progress)
         content_list.append(content)
     return content_list
 
@@ -315,9 +390,10 @@ def _validate_endpoint(endpoint: str) -> str:
         The validated NUS endpoint with the proper path.
     """
     # Find the root of the URL and then assemble the correct URL based on that.
-    new_url = _urlparse(endpoint)
-    if new_url.netloc == "":
-        endpoint_url = "http://" + new_url.path + "/ccs/download/"
-    else:
-        endpoint_url = "http://" + new_url.netloc + "/ccs/download/"
-    return endpoint_url
+    # TODO: Rewrite in a way that makes more sense and un-stub
+    #new_url = _urlparse(endpoint)
+    #if new_url.netloc == "":
+    #    endpoint_url = "http://" + new_url.path + "/ccs/download/"
+    #else:
+    #    endpoint_url = "http://" + new_url.netloc + "/ccs/download/"
+    return endpoint

src/libWiiPy/title/ticket.py

@@ -75,7 +75,7 @@ class Ticket:
         self.title_version: int = 0  # Version of the ticket's associated title.
         self.permitted_titles: bytes = b''  # Permitted titles mask
         # "Permit mask. The current disc title is ANDed with the inverse of this mask to see if the result matches the
-        # Permitted Titles Mask."
+        # Permitted Titles Mask." -WiiBrew
         self.permit_mask: bytes = b''
         self.title_export_allowed: int = 0  # Whether title export is allowed with a PRNG key or not.
         self.common_key_index: int = 0  # Which common key should be used. 0 = Common Key, 1 = Korean Key, 2 = vWii Key
@@ -128,7 +128,7 @@ class Ticket:
             self.console_id = int.from_bytes(ticket_data.read(4))
             # Title ID.
             ticket_data.seek(0x1DC)
-            self.title_id = binascii.hexlify(ticket_data.read(8))
+            self.title_id = ticket_data.read(8)
             # Unknown data 1.
             ticket_data.seek(0x1E4)
             self.unknown1 = ticket_data.read(2)
@@ -202,7 +202,7 @@ class Ticket:
         # Console ID.
         ticket_data += int.to_bytes(self.console_id, 4)
         # Title ID.
-        ticket_data += binascii.unhexlify(self.title_id)
+        ticket_data += self.title_id
         # Unknown data 1.
         ticket_data += self.unknown1
         # Title version.
@@ -318,6 +318,8 @@ class Ticket:
                 return "Korean"
             case 2:
                 return "vWii"
+            case _:
+                return "Unknown"
 
     def get_title_key(self) -> bytes:
         """
@@ -343,7 +345,7 @@ class Ticket:
         """
         if len(title_id) != 16:
             raise ValueError("Invalid Title ID! Title IDs must be 8 bytes long.")
-        self.title_id = title_id.encode()
+        self.title_id = binascii.unhexlify(title_id.encode())
 
     def set_title_version(self, new_version: str | int) -> None:
         """

src/libWiiPy/title/title.py

@@ -194,6 +194,8 @@ class Title:
         bytes
             The decrypted content listed in the content record.
         """
+        if self.ticket.title_id == "":
+            raise ValueError("A Ticket must be loaded to get decrypted content.")
         dec_content = self.content.get_content_by_index(index, self.ticket.get_title_key(), skip_hash)
         return dec_content
 
@@ -213,6 +215,8 @@ class Title:
         bytes
             The decrypted content listed in the content record.
         """
+        if self.ticket.title_id == "":
+            raise ValueError("A Ticket must be loaded to get decrypted content.")
         dec_content = self.content.get_content_by_cid(cid, self.ticket.get_title_key(), skip_hash)
         return dec_content
 

src/libWiiPy/title/wiiload.py

@@ -1,62 +0,0 @@
-# "title/wiiload.py" from libWiiPy by NinjaCheetah & Contributors
-# https://github.com/NinjaCheetah/libWiiPy
-#
-# This code is adapted from "wiiload.py", which can be found on the WiiBrew page for Wiiload.
-# https://pastebin.com/4nWAkBpw
-#
-# See https://wiibrew.org/wiki/Wiiload for details about how Wiiload works
-
-import sys
-import zlib
-import socket
-import struct
-
-
-def send_bin_wiiload(target_ip: str, bin_data: bytes, name: str) -> None:
-    """
-    Sends an ELF or DOL binary to The Homebrew Channel via Wiiload. This requires the IP address of the console you
-    want to send the binary to.
-
-    Parameters
-    ----------
-    target_ip: str
-        The IP address of the console to send the binary to.
-    bin_data: bytes
-        The data of the ELF or DOL to send.
-    name: str
-        The name of the application being sent.
-    """
-    wii_ip = (target_ip, 4299)
-
-    WIILOAD_VERSION_MAJOR=0
-    WIILOAD_VERSION_MINOR=5
-
-    len_uncompressed = len(bin_data)
-    c_data = zlib.compress(bin_data, 6)
-
-    chunk_size = 1024*128
-    chunks = [c_data[i:i+chunk_size] for i  in range(0, len(c_data), chunk_size)]
-
-    args = [name]
-    args = "\x00".join(args) + "\x00"
-
-    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    s.connect(wii_ip)
-
-    s.send("HAXX")
-    s.send(struct.pack("B", WIILOAD_VERSION_MAJOR)) # one byte, unsigned
-    s.send(struct.pack("B", WIILOAD_VERSION_MINOR)) # one byte, unsigned
-    s.send(struct.pack(">H",len(args))) # bigendian, 2 bytes, unsigned
-    s.send(struct.pack(">L",len(c_data))) # bigendian, 4 bytes, unsigned
-    s.send(struct.pack(">L",len_uncompressed)) # bigendian, 4 bytes, unsigned
-
-    print(len(chunks),"chunks to send")
-    for piece in chunks:
-        s.send(piece)
-        sys.stdout.write("."); sys.stdout.flush()
-    sys.stdout.write("\n")
-
-    s.send(args)
-
-    s.close()
-    print("done")