Re-add wiiload module post-merge

docs/source/archive/archive.md

@@ -1,10 +1,7 @@
 # libWiiPy.archive Package
 
-## Description
-
-The `libWiiPy.archive` package contains modules for packing and extracting archive formats used by the Wii. This currently includes packing and unpacking support for U8 archives and decompression support for ASH archives.
-
 ## Modules
+The `libWiiPy.archive` package contains modules for packing and extracting archive formats used by the Wii. This currently includes packing and unpacking support for U8 archives and decompression support for ASH archives.
 
 | Module                                 | Description                                             |
 |----------------------------------------|---------------------------------------------------------|
@@ -12,7 +9,7 @@ The `libWiiPy.archive` package contains modules for packing and extracting archi
 | [libWiiPy.archive.lz77](/archive/lz77) | Provides support for the LZ77 compression scheme        |
 | [libWiiPy.archive.u8](/archive/u8)     | Provides support for packing and extracting U8 archives |
 
-## Full Package Contents
+### libWiiPy.archive Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/archive/ash.md

@@ -1,7 +1,5 @@
 # libWiiPy.archive.ash Module
 
-## Description
-
 The `libWiiPy.archive.ash` module provides support for handling ASH files, which are a compressed format primarily used in the Wii Menu, but also in some other titles such as My Pokémon Ranch.
 
 At present, libWiiPy only has support for decompressing ASH files, with compression as a planned feature for the future.

docs/source/archive/lz77.md

@@ -1,7 +1,5 @@
 # libWiiPy.archive.lz77 Module
 
-## Description
-
 The `libWiiPy.archive.lz77` module provides support for handling LZ77 compression, which is a compression format used across the Wii and other Nintendo consoles.
 
 ## Module Contents

docs/source/archive/u8.md

@@ -1,7 +1,5 @@
 # libWiiPy.archive.u8 Module
 
-## Description
-
 The `libWiiPy.archive.u8` module provides support for handling U8 archives, which are a non-compressed archive format used extensively on the Wii to join multiple files into one.
 
 This module exposes functions for both packing and unpacking U8 archives, as well as code to parse IMET headers. IMET headers are a header format used specifically for U8 archives containing the banner of a channel, as they store the localized name of the channel along with other banner metadata.

docs/source/conf.py

@@ -31,8 +31,7 @@ html_logo = "banner.png"
 html_title = "libWiiPy API Docs"
 html_theme_options = {
     "repository_url": "https://github.com/NinjaCheetah/libWiiPy",
-    "use_repository_button": True,
+    "use_repository_button": True
-    "show_toc_level": 3
 }
 
 # MyST Configuration

docs/source/media/banner.md

@@ -1,7 +1,5 @@
 # libWiiPy.media.banner Module
 
-## Description
-
 The `libWiiPy.media.banner` module is essentially a stub at this point in time. It only provides one dataclass that is likely to become a traditional class when fully implemented. It is not recommended to use this module for anything yet.
 
 ## Module Contents

docs/source/media/media.md

@@ -1,16 +1,13 @@
 # libWiiPy.media Package
 
-## Description
-
-The `libWiiPy.media` package contains modules used for parsing and editing media formats used by the Wii. This currently only includes limited support for parsing channel banners.
-
 ## Modules
+The `libWiiPy.media` package contains modules used for parsing and editing media formats used by the Wii. This currently only includes limited support for parsing channel banners.
 
 | Module                                 | Description                                       |
 |----------------------------------------|---------------------------------------------------|
 | [libWiiPy.media.banner](/media/banner) | Provides support for basic channel banner parsing |
 
-## Full Package Contents
+### libWiiPy.media Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/nand/emunand.md

@@ -1,7 +1,5 @@
 # libWiiPy.nand.emunand Module
 
-## Description
-
 The `libWiiPy.nand.emunand` module provides support for creating and managing Wii EmuNANDs. At present, you cannot create an EmuNAND compatible with something like NEEK on a real Wii with the features provided by this library, but you can create an EmuNAND compatible with Dolphin.
 
 ## Module Contents

docs/source/nand/nand.md

@@ -1,10 +1,7 @@
 # libWiiPy.nand Package
 
-## Description
-
-The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNANDs as well as modules for parsing and editing core system files found on the Wii's NAND.
-
 ## Modules
+The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNANDs as well as modules for parsing and editing core system files found on the Wii's NAND.
 
 | Module                                 | Description                                                                                                                      |
 |----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
@@ -12,7 +9,7 @@ The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNAN
 | [libWiiPy.nand.setting](/nand/setting) | Provides support for parsing, creating, and editing `setting.txt`, which is used to store the console's region and serial number |
 | [libWiiPy.nand.sys](/nand/sys)         | Provides support for parsing, creating, and editing `uid.sys`, which is used to store a log of all titles run on a console       |
 
-## Full Package Contents
+### libWiiPy.nand Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/nand/setting.md

@@ -1,7 +1,5 @@
 # libWiiPy.nand.setting Module
 
-## Description
-
 The `libWiiPy.nand.setting` module provides support for handling the Wii's `setting.txt` file. This file is stored as part of the Wii Menu's save data (stored in `/title/00000001/00000002/data/`) and is an encrypted text file that's primarily used to store your console's serial number and region information.
 
 This module allows you to encrypt or decrypt this file, and exposes the keys stored in it for editing.

docs/source/nand/sys.md

@@ -1,7 +1,5 @@
 # libWiiPy.nand.sys Module
 
-## Description
-
 The `libWiiPy.nand.sys` module provides support for editing system files used on the Wii. Currently, it only offers support for `uid.sys`, which keeps a record of the Title IDs of every title launched on the console, assigning each one a unique ID.
 
 ## Module Contents

docs/source/title/cert.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.cert Module
 
-## Description
-
 The `libWiiPy.title.cert` module provides support for parsing the various signing certificates used by the Wii for content validation.
 
 This module allows you to write your own code for validating the authenticity of a TMD or Ticket by providing the certificates from the Wii's certificate chain. Both retail and development certificate chains are supported.

docs/source/title/commonkeys.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.commonkeys Module
 
-## Description
-
 The `libWiiPy.title.commonkeys` module simply provides easy access to the Wii's common encryption keys.
 
 ## Module Contents

docs/source/title/content.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.content Module
 
-## Description
-
 The `libWiiPy.title.content` module provides support for parsing, adding, removing, and editing content files from a digital Wii title.
 
 ## Module Contents

docs/source/title/crypto.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.crypto Module
 
-## Description
-
 The `libWiiPy.title.crypto` module provides low-level cryptography functions required for handling digital Wii titles. It does not expose many functions that are likely to be required during typical use, and instead acts more as a dependency for other modules.
 
 ## Module Contents

docs/source/title/iospatcher.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.iospatcher Module
 
-## Description
-
 The `libWiiPy.title.iospatcher` module provides support for applying various binary patches to IOS' ES module. These patches and what they do can be found attached to the methods used to apply them.
 
 ## Module Contents

docs/source/title/nus.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.nus Module
 
-## Description
-
 The `libWiiPy.title.nus` module provides support for downloading digital Wii titles from the Nintendo Update Servers. This module provides easy methods for downloading TMDs, common Tickets (when present), encrypted content, and the certificate chain.
 
 ## Module Contents

docs/source/title/ticket.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.ticket Module
 
-## Description
-
 The `libWiiPy.title.ticket` module provides support for handling Tickets, which are the license files used to decrypt the content of digital titles during installation. This module allows for easy parsing and editing of Tickets.
 
 ## Module Contents

docs/source/title/title.md

@@ -1,9 +1,7 @@
 # libWiiPy.title Package
 
-## Description
-The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, as it offers the functionality one would be most likely to need. As a result, it gets the most attention during development and should be the most reliable.
-
 ## Modules
+The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, and therefore offers the most functionality.
 
 | Module                                         | Description                                                                                                                   |
 |------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
@@ -19,7 +17,7 @@ The `libWiiPy.title` package contains modules for interacting with Wii titles. T
 | [libWiiPy.title.util](/title/util)             | Provides some simple utility functions relating to titles                                                                     |
 | [libWiiPy.title.wad](/title/wad)               | Provides support for parsing and editing WAD files, allowing you to load each component into the other available classes      |
 
-## Full Package Contents
+### libWiiPy.title Package Contents
 
 ```{toctree}
 :maxdepth: 4

docs/source/title/title.title.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.title Module
 
-## Description
-
 The `libWiiPy.title.title` module provides a high-level interface for handling all the components of a digital Wii title through one class. It allows for directly importing a WAD, and will automatically extract the various components and load them into their appropriate classes. Additionally, it provides duplicates of some methods found in those classes that require fewer arguments, as it has the context of the other components and is able to retrieve additional data automatically.
 
 An example of that idea can be seen with the method `get_content_by_index()`. In its original definition, which can be seen at <project:#libWiiPy.title.content.ContentRegion.get_content_by_index>, you are required to supply the Title Key for the title that the content is sourced from. In contrast, when using <project:#libWiiPy.title.title.Title.get_content_by_index>, you do not need to supply a Title Key, as the Title object already has the context of the Ticket and can retrieve the Title Key from it automatically. In a similar vein, this module provides the easiest route for verifying that a title is legitimately signed by Nintendo. The method <project:#libWiiPy.title.title.Title.get_is_signed> is able to access the entire certificate chain, the TMD, and the Ticket, and is therefore able to verify all components of the title by itself.
@@ -14,4 +12,5 @@ Because using <project:#libWiiPy.title.title.Title> allows many operations to be
 .. automodule:: libWiiPy.title.title
    :members:
    :undoc-members:
+   :show-inheritance:
 ```

docs/source/title/tmd.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.tmd Module
 
-## Description
-
 The `libWiiPy.title.tmd` module provides support for handling TMD (Title Metadata) files, which contain the metadata of both digital and physical Wii titles. This module allows for easy parsing and editing of TMDs.
 
 ## Module Contents

docs/source/title/util.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.util Module
 
-## Description
-
 The `libWiiPy.title.util` module provides common utility functions internally. It is not designed to be used directly.
 
 ## Module Contents

docs/source/title/wad.md

@@ -1,7 +1,5 @@
 # libWiiPy.title.wad Module
 
-## Description
-
 The `libWiiPy.title.wad` module provides support for handling WAD (Wii Archive Data) files, which is the format used to deliver digital Wii titles. This module allows for extracting the various components for a WAD, as well as properly padding and writing out that data when it has been edited using other modules.
 
 ## Module Contents

src/libWiiPy/archive/lz77.py

@@ -5,7 +5,6 @@
 
 import io
 from dataclasses import dataclass as _dataclass
-from typing import List as _List
 
 
 _LZ_MIN_DISTANCE = 0x01   # Minimum distance for each reference.
@@ -21,20 +20,19 @@ class _LZNode:
     weight: int = 0
 
 
-def _compress_compare_bytes(buffer: _List[int], offset1: int, offset2: int, abs_len_max: int) -> int:
+def _compress_compare_bytes(byte1: bytes, offset1: int, byte2: bytes, offset2: int, abs_len_max: int) -> int:
-    # Compare bytes up to the maximum length we can match. Start by comparing the first 3 bytes, since that's the
+    # Compare bytes up to the maximum length we can match.
-    # minimum match length and this allows for a more optimized early exit.
     num_matched = 0
     while num_matched < abs_len_max:
-        if buffer[offset1 + num_matched] != buffer[offset2 + num_matched]:
+        if byte1[offset1 + num_matched] != byte2[offset2 + num_matched]:
             break
         num_matched += 1
     return num_matched
 
 
-def _compress_search_matches_optimized(buffer: _List[int], pos: int) -> (int, int):
+def _compress_search_matches(buffer: bytes, pos: int) -> (int, int):
     bytes_left = len(buffer) - pos
-    global _LZ_MAX_DISTANCE, _LZ_MIN_LENGTH, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
+    global _LZ_MAX_DISTANCE, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
     # Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
     # only look as far back as we've gone.
     max_dist = min(_LZ_MAX_DISTANCE, pos)
@@ -45,7 +43,7 @@ def _compress_search_matches_optimized(buffer: _List[int], pos: int) -> (int, in
     biggest_match, biggest_match_pos = 0, 0
     # Search for matches.
     for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
-        num_matched = _compress_compare_bytes(buffer, pos - i, pos, max_len)
+        num_matched = _compress_compare_bytes(buffer, pos - i, buffer, pos, max_len)
         if num_matched > biggest_match:
             biggest_match = num_matched
             biggest_match_pos = i
@@ -54,25 +52,6 @@ def _compress_search_matches_optimized(buffer: _List[int], pos: int) -> (int, in
     return biggest_match, biggest_match_pos
 
 
-def _compress_search_matches_greedy(buffer: _List[int], pos: int) -> (int, int):
-    # Finds and returns the first valid match, rather that finding the best one.
-    bytes_left = len(buffer) - pos
-    global _LZ_MAX_DISTANCE, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
-    # Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
-    # only look as far back as we've gone.
-    max_dist = min(_LZ_MAX_DISTANCE, pos)
-    # Default to only matching up to 18 bytes, unless fewer than 18 bytes remain, in which case we can only match
-    # up to that many bytes.
-    max_len = min(_LZ_MAX_LENGTH, bytes_left)
-    match, match_pos = 0, 0
-    for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
-        match = _compress_compare_bytes(buffer, pos - i, pos, max_len)
-        match_pos = i
-        if match >= _LZ_MIN_LENGTH or match == max_len:
-            break
-    return match, match_pos
-
-
 def _compress_node_is_ref(node: _LZNode) -> bool:
     return node.len >= _LZ_MIN_LENGTH
 
@@ -85,30 +64,40 @@ def _compress_get_node_cost(length: int) -> int:
     return 1 + (num_bytes * 8)
 
 
-def _compress_lz77_optimized(data: bytes) -> bytes:
+def compress_lz77(data: bytes) -> bytes:
-    # Optimized compressor based around a node graph that finds optimal string matches. Originally the default
+    """
-    # implementation, but unfortunately it's very slow.
+    Compresses data using the Wii's LZ77 compression algorithm and returns the compressed result.
+
+    Parameters
+    ----------
+    data: bytes
+        The data to compress.
+
+    Returns
+    -------
+    bytes
+        The LZ77-compressed data.
+    """
     nodes = [_LZNode() for _ in range(len(data))]
     # Iterate over the uncompressed data, starting from the end.
     pos = len(data)
     global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
-    data_list = list(data)
     while pos:
         pos -= 1
         node = nodes[pos]
         # Limit the maximum search length when we're near the end of the file.
-        max_search_len = min(_LZ_MAX_LENGTH, len(data_list) - pos)
+        max_search_len = min(_LZ_MAX_LENGTH, len(data) - pos)
         if max_search_len < _LZ_MIN_DISTANCE:
             max_search_len = 1
         # Initialize as 1 for each, since that's all we could use if we weren't compressing.
         length, dist = 1, 1
         if max_search_len >= _LZ_MIN_LENGTH:
-            length, dist = _compress_search_matches_optimized(data_list, pos)
+            length, dist = _compress_search_matches(data, pos)
         # Treat as direct bytes if it's too short to copy.
         if length == 0 or length < _LZ_MIN_LENGTH:
             length = 1
         # If the node goes to the end of the file, the weight is the cost of the node.
-        if (pos + length) == len(data_list):
+        if (pos + length) == len(data):
             node.len = length
             node.dist = dist
             node.weight = _compress_get_node_cost(length)
@@ -128,7 +117,7 @@ def _compress_lz77_optimized(data: bytes) -> bytes:
             node.len = len_best
             node.dist = dist
             node.weight = weight_best
-    # Write the compressed data.
+    # Write the header data.
     with io.BytesIO() as buffer:
         # Write the header data.
         buffer.write(b'LZ77\x10')  # The LZ type on the Wii is *always* 0x10.
@@ -166,75 +155,6 @@ def _compress_lz77_optimized(data: bytes) -> bytes:
     return out_data
 
 
-def _compress_lz77_greedy(data: bytes) -> bytes:
-    # Greedy compressor that processes the file start to end and saves the first matches found. Faster than the
-    # optimized implementation, but creates larger files.
-    global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
-    with io.BytesIO() as buffer:
-        # Write the header data.
-        buffer.write(b'LZ77\x10')  # The LZ type on the Wii is *always* 0x10.
-        buffer.write(len(data).to_bytes(3, 'little'))
-
-        src_pos = 0
-        data_list = list(data)
-        while src_pos < len(data):
-            head = 0
-            head_pos = buffer.tell()
-            buffer.write(b'\x00')  # Reserve a byte for the chunk head.
-
-            i = 0
-            while i < 8 and src_pos < len(data):
-                length, dist = _compress_search_matches_greedy(data_list, src_pos)
-                # This is a reference node.
-                if length >= _LZ_MIN_LENGTH:
-                    encoded = (((length - _LZ_MIN_LENGTH) & 0xF) << 12) | ((dist - _LZ_MIN_DISTANCE) & 0xFFF)
-                    buffer.write(encoded.to_bytes(2))
-                    head = (head | (1 << (7 - i))) & 0xFF
-                    src_pos += length
-                # This is a direct copy node.
-                else:
-                    buffer.write(data[src_pos:src_pos + 1])
-                    src_pos += 1
-                i += 1
-
-            pos = buffer.tell()
-            buffer.seek(head_pos)
-            buffer.write(head.to_bytes(1))
-            buffer.seek(pos)
-
-        buffer.seek(0)
-        out_data = buffer.read()
-    return out_data
-
-
-def compress_lz77(data: bytes, compression_level: int = 1) -> bytes:
-    """
-    Compresses data using the Wii's LZ77 compression algorithm and returns the compressed result. Supports two
-    different levels of compression, one based around a "greedy" LZ compression algorithm and the other based around
-    an optimized LZ compression algorithm. The greedy compressor, level 1, will produce a larger compressed file but
-    will run noticeably faster than the optimized compressor, which is level 2, especially for larger data.
-
-    Parameters
-    ----------
-    data: bytes
-        The data to compress.
-    compression_level: int
-        The compression level to use, either 1 and 2. Default value is 1.
-
-    Returns
-    -------
-    bytes
-        The LZ77-compressed data.
-    """
-    if compression_level == 1:
-        out_data = _compress_lz77_greedy(data)
-    elif compression_level == 2:
-        out_data = _compress_lz77_optimized(data)
-    else:
-        raise ValueError(f"Invalid compression level \"{compression_level}\"!\"")
-    return out_data
-
-
 def decompress_lz77(lz77_data: bytes) -> bytes:
     """
     Decompresses LZ77-compressed data and returns the decompressed result. Supports data both with and without the

src/libWiiPy/title/nus.py

@@ -4,7 +4,7 @@
 # See https://wiibrew.org/wiki/NUS for details about the NUS
 
 import requests
-#import hashlib
+import hashlib
 from typing import List
 from urllib.parse import urlparse as _urlparse
 from .title import Title

src/libWiiPy/title/wiiload.py

@@ -0,0 +1,62 @@
+# "title/wiiload.py" from libWiiPy by NinjaCheetah & Contributors
+# https://github.com/NinjaCheetah/libWiiPy
+#
+# This code is adapted from "wiiload.py", which can be found on the WiiBrew page for Wiiload.
+# https://pastebin.com/4nWAkBpw
+#
+# See https://wiibrew.org/wiki/Wiiload for details about how Wiiload works
+
+import sys
+import zlib
+import socket
+import struct
+
+
+def send_bin_wiiload(target_ip: str, bin_data: bytes, name: str) -> None:
+    """
+    Sends an ELF or DOL binary to The Homebrew Channel via Wiiload. This requires the IP address of the console you
+    want to send the binary to.
+
+    Parameters
+    ----------
+    target_ip: str
+        The IP address of the console to send the binary to.
+    bin_data: bytes
+        The data of the ELF or DOL to send.
+    name: str
+        The name of the application being sent.
+    """
+    wii_ip = (target_ip, 4299)
+
+    WIILOAD_VERSION_MAJOR=0
+    WIILOAD_VERSION_MINOR=5
+
+    len_uncompressed = len(bin_data)
+    c_data = zlib.compress(bin_data, 6)
+
+    chunk_size = 1024*128
+    chunks = [c_data[i:i+chunk_size] for i  in range(0, len(c_data), chunk_size)]
+
+    args = [name]
+    args = "\x00".join(args) + "\x00"
+
+    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    s.connect(wii_ip)
+
+    s.send("HAXX")
+    s.send(struct.pack("B", WIILOAD_VERSION_MAJOR)) # one byte, unsigned
+    s.send(struct.pack("B", WIILOAD_VERSION_MINOR)) # one byte, unsigned
+    s.send(struct.pack(">H",len(args))) # bigendian, 2 bytes, unsigned
+    s.send(struct.pack(">L",len(c_data))) # bigendian, 4 bytes, unsigned
+    s.send(struct.pack(">L",len_uncompressed)) # bigendian, 4 bytes, unsigned
+
+    print(len(chunks),"chunks to send")
+    for piece in chunks:
+        s.send(piece)
+        sys.stdout.write("."); sys.stdout.flush()
+    sys.stdout.write("\n")
+
+    s.send(args)
+
+    s.close()
+    print("done")