6 Commits

Author SHA1 Message Date
8adbef26b1 Updated README to reflect features since v0.5.0 2025-02-10 16:26:45 -05:00
5dde9f7835 Changed how the Title ID is handled in Tickets, stubbed out NUS endpoint validation 2025-02-10 13:36:39 -05:00
93abad1f31 LZ77 Compression: Now 12.5% faster! 2025-01-29 22:48:58 -05:00
9eabf2caee Added a lower LZ77 compression level that runs faster 2025-01-26 12:56:41 -05:00
5ae867197b (doc) Workaround for ScrollSpy issue breaking TOC 2025-01-25 20:40:44 -05:00
6552dc5fa8 Add LZ77 Compression Function (#22)
* Unfinished wiiload module and LZ77 compression code

* Updated WIP LZ77 compressor, still not working yet

* Updated WIP LZ77 compressor, still not working yet (again)

* LZ77 compression is now fully functional! (But still very slow)

* Added compress_lz77 docstrings, temporarily removed unfinished wiiload module
2025-01-23 22:26:34 -05:00
27 changed files with 306 additions and 26 deletions

View File

@@ -2,7 +2,7 @@
# libWiiPy # libWiiPy
libWiiPy is a modern Python 3 library for handling the various files and formats found on the Wii. It aims to be simple to use, well maintained, and offer as many features as reasonably possible in one library, so that a newly-written Python program could do 100% of its Wii-related work with just one library. It also aims to be fully cross-platform, so that any tools written with it can also be cross-platform. libWiiPy is a modern Python 3 library for handling the various files and formats found on the Wii. It aims to be simple to use, well maintained, and offer as many features as reasonably possible in one library, so that a newly-written Python program could do 100% of its Wii-related work with just one library. It also aims to be fully cross-platform, so that any tools written with it can also be cross-platform.
libWiiPy is inspired by [libWiiSharp](https://github.com/TheShadowEevee/libWiiSharp), which was originally created by `Leathl` and is now maintained by [@TheShadowEevee](https://github.com/TheShadowEevee). If you're looking for a Wii library that isn't in Python, then go check it out! libWiiPy is inspired by [libWiiSharp](https://github.com/TheShadowEevee/libWiiSharp), which was originally created by `Leathl` and is now maintained by [@TheShadowEevee](https://github.com/TheShadowEevee).
# Features # Features
@@ -10,14 +10,18 @@ This list will expand as libWiiPy is developed, but these features are currently
- TMD and Ticket parsing/editing (`.tmd`, `.tik`) - TMD and Ticket parsing/editing (`.tmd`, `.tik`)
- Title parsing/editing, including content encryption/decryption (both retail and development) - Title parsing/editing, including content encryption/decryption (both retail and development)
- WAD file parsing/editing (`.wad`) - WAD file parsing/editing (`.wad`)
- Downloading titles from the NUS - Downloading titles and their components from the NUS
- Certificate, TMD, and Ticket signature verification
- Packing and unpacking U8 archives (`.app`, `.arc`) - Packing and unpacking U8 archives (`.app`, `.arc`)
- Decompressing ASH files (`.ash`, both the standard variants and the variants found in My Pokémon Ranch) - Decompressing ASH files (`.ash`, both the standard variants and the variants found in My Pokémon Ranch)
- Compressing/Decompressing LZ77-compressed files
- IOS patching - IOS patching
- NAND-related functionality: - NAND-related functionality:
- EmuNAND title management (currently requires an existing EmuNAND) - EmuNAND title management (currently requires an existing EmuNAND)
- `content.map` parsing/editing - `content.map` parsing/editing
- `setting.txt` parsing/editing
- `uid.sys` parsing/editing - `uid.sys` parsing/editing
- Limited channel banner parsing/editing
- Assorted miscellaneous features used to make the other core features possible - Assorted miscellaneous features used to make the other core features possible
For a more detailed look at what's available in libWiiPy, check out our [API docs](https://ninjacheetah.github.io/libWiiPy). For a more detailed look at what's available in libWiiPy, check out our [API docs](https://ninjacheetah.github.io/libWiiPy).
@@ -68,6 +72,3 @@ Thank you to all of the contributors to the documentation on the WiiBrew pages t
### One additional special thanks to [@DamiDoop](https://github.com/DamiDoop)! ### One additional special thanks to [@DamiDoop](https://github.com/DamiDoop)!
She made the very cool banner you can see at the top of this README, and has also helped greatly with my sanity throughout debugging this library. She made the very cool banner you can see at the top of this README, and has also helped greatly with my sanity throughout debugging this library.
**Note:** While libWiiPy is directly inspired by libWiiSharp and aims to have feature parity with it, no code from either libWiiSharp or Wii.py was used in the making of this library. All code is original and is written by [@NinjaCheetah](https://github.com/NinjaCheetah), [@rvtr](https://github.com/rvtr), and any other GitHub contributors.

View File

@@ -1,15 +1,18 @@
# libWiiPy.archive Package # libWiiPy.archive Package
## Modules ## Description
The `libWiiPy.archive` package contains modules for packing and extracting archive formats used by the Wii. This currently includes packing and unpacking support for U8 archives and decompression support for ASH archives. The `libWiiPy.archive` package contains modules for packing and extracting archive formats used by the Wii. This currently includes packing and unpacking support for U8 archives and decompression support for ASH archives.
## Modules
| Module | Description | | Module | Description |
|----------------------------------------|---------------------------------------------------------| |----------------------------------------|---------------------------------------------------------|
| [libWiiPy.archive.ash](/archive/ash) | Provides support for decompressing ASH archives | | [libWiiPy.archive.ash](/archive/ash) | Provides support for decompressing ASH archives |
| [libWiiPy.archive.lz77](/archive/lz77) | Provides support for the LZ77 compression scheme | | [libWiiPy.archive.lz77](/archive/lz77) | Provides support for the LZ77 compression scheme |
| [libWiiPy.archive.u8](/archive/u8) | Provides support for packing and extracting U8 archives | | [libWiiPy.archive.u8](/archive/u8) | Provides support for packing and extracting U8 archives |
### libWiiPy.archive Package Contents ## Full Package Contents
```{toctree} ```{toctree}
:maxdepth: 4 :maxdepth: 4

View File

@@ -1,5 +1,7 @@
# libWiiPy.archive.ash Module # libWiiPy.archive.ash Module
## Description
The `libWiiPy.archive.ash` module provides support for handling ASH files, which are a compressed format primarily used in the Wii Menu, but also in some other titles such as My Pokémon Ranch. The `libWiiPy.archive.ash` module provides support for handling ASH files, which are a compressed format primarily used in the Wii Menu, but also in some other titles such as My Pokémon Ranch.
At present, libWiiPy only has support for decompressing ASH files, with compression as a planned feature for the future. At present, libWiiPy only has support for decompressing ASH files, with compression as a planned feature for the future.

View File

@@ -1,5 +1,7 @@
# libWiiPy.archive.lz77 Module # libWiiPy.archive.lz77 Module
## Description
The `libWiiPy.archive.lz77` module provides support for handling LZ77 compression, which is a compression format used across the Wii and other Nintendo consoles. The `libWiiPy.archive.lz77` module provides support for handling LZ77 compression, which is a compression format used across the Wii and other Nintendo consoles.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.archive.u8 Module # libWiiPy.archive.u8 Module
## Description
The `libWiiPy.archive.u8` module provides support for handling U8 archives, which are a non-compressed archive format used extensively on the Wii to join multiple files into one. The `libWiiPy.archive.u8` module provides support for handling U8 archives, which are a non-compressed archive format used extensively on the Wii to join multiple files into one.
This module exposes functions for both packing and unpacking U8 archives, as well as code to parse IMET headers. IMET headers are a header format used specifically for U8 archives containing the banner of a channel, as they store the localized name of the channel along with other banner metadata. This module exposes functions for both packing and unpacking U8 archives, as well as code to parse IMET headers. IMET headers are a header format used specifically for U8 archives containing the banner of a channel, as they store the localized name of the channel along with other banner metadata.

View File

@@ -31,7 +31,8 @@ html_logo = "banner.png"
html_title = "libWiiPy API Docs" html_title = "libWiiPy API Docs"
html_theme_options = { html_theme_options = {
"repository_url": "https://github.com/NinjaCheetah/libWiiPy", "repository_url": "https://github.com/NinjaCheetah/libWiiPy",
"use_repository_button": True "use_repository_button": True,
"show_toc_level": 3
} }
# MyST Configuration # MyST Configuration

View File

@@ -1,5 +1,7 @@
# libWiiPy.media.banner Module # libWiiPy.media.banner Module
## Description
The `libWiiPy.media.banner` module is essentially a stub at this point in time. It only provides one dataclass that is likely to become a traditional class when fully implemented. It is not recommended to use this module for anything yet. The `libWiiPy.media.banner` module is essentially a stub at this point in time. It only provides one dataclass that is likely to become a traditional class when fully implemented. It is not recommended to use this module for anything yet.
## Module Contents ## Module Contents

View File

@@ -1,13 +1,16 @@
# libWiiPy.media Package # libWiiPy.media Package
## Modules ## Description
The `libWiiPy.media` package contains modules used for parsing and editing media formats used by the Wii. This currently only includes limited support for parsing channel banners. The `libWiiPy.media` package contains modules used for parsing and editing media formats used by the Wii. This currently only includes limited support for parsing channel banners.
## Modules
| Module | Description | | Module | Description |
|----------------------------------------|---------------------------------------------------| |----------------------------------------|---------------------------------------------------|
| [libWiiPy.media.banner](/media/banner) | Provides support for basic channel banner parsing | | [libWiiPy.media.banner](/media/banner) | Provides support for basic channel banner parsing |
### libWiiPy.media Package Contents ## Full Package Contents
```{toctree} ```{toctree}
:maxdepth: 4 :maxdepth: 4

View File

@@ -1,5 +1,7 @@
# libWiiPy.nand.emunand Module # libWiiPy.nand.emunand Module
## Description
The `libWiiPy.nand.emunand` module provides support for creating and managing Wii EmuNANDs. At present, you cannot create an EmuNAND compatible with something like NEEK on a real Wii with the features provided by this library, but you can create an EmuNAND compatible with Dolphin. The `libWiiPy.nand.emunand` module provides support for creating and managing Wii EmuNANDs. At present, you cannot create an EmuNAND compatible with something like NEEK on a real Wii with the features provided by this library, but you can create an EmuNAND compatible with Dolphin.
## Module Contents ## Module Contents

View File

@@ -1,15 +1,18 @@
# libWiiPy.nand Package # libWiiPy.nand Package
## Modules ## Description
The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNANDs as well as modules for parsing and editing core system files found on the Wii's NAND. The `libWiiPy.nand` package contains modules for parsing and manipulating EmuNANDs as well as modules for parsing and editing core system files found on the Wii's NAND.
## Modules
| Module | Description | | Module | Description |
|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| |----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
| [libWiiPy.nand.emunand](/nand/emunand) | Provides support for parsing, creating, and editing EmuNANDs | | [libWiiPy.nand.emunand](/nand/emunand) | Provides support for parsing, creating, and editing EmuNANDs |
| [libWiiPy.nand.setting](/nand/setting) | Provides support for parsing, creating, and editing `setting.txt`, which is used to store the console's region and serial number | | [libWiiPy.nand.setting](/nand/setting) | Provides support for parsing, creating, and editing `setting.txt`, which is used to store the console's region and serial number |
| [libWiiPy.nand.sys](/nand/sys) | Provides support for parsing, creating, and editing `uid.sys`, which is used to store a log of all titles run on a console | | [libWiiPy.nand.sys](/nand/sys) | Provides support for parsing, creating, and editing `uid.sys`, which is used to store a log of all titles run on a console |
### libWiiPy.nand Package Contents ## Full Package Contents
```{toctree} ```{toctree}
:maxdepth: 4 :maxdepth: 4

View File

@@ -1,5 +1,7 @@
# libWiiPy.nand.setting Module # libWiiPy.nand.setting Module
## Description
The `libWiiPy.nand.setting` module provides support for handling the Wii's `setting.txt` file. This file is stored as part of the Wii Menu's save data (stored in `/title/00000001/00000002/data/`) and is an encrypted text file that's primarily used to store your console's serial number and region information. The `libWiiPy.nand.setting` module provides support for handling the Wii's `setting.txt` file. This file is stored as part of the Wii Menu's save data (stored in `/title/00000001/00000002/data/`) and is an encrypted text file that's primarily used to store your console's serial number and region information.
This module allows you to encrypt or decrypt this file, and exposes the keys stored in it for editing. This module allows you to encrypt or decrypt this file, and exposes the keys stored in it for editing.

View File

@@ -1,5 +1,7 @@
# libWiiPy.nand.sys Module # libWiiPy.nand.sys Module
## Description
The `libWiiPy.nand.sys` module provides support for editing system files used on the Wii. Currently, it only offers support for `uid.sys`, which keeps a record of the Title IDs of every title launched on the console, assigning each one a unique ID. The `libWiiPy.nand.sys` module provides support for editing system files used on the Wii. Currently, it only offers support for `uid.sys`, which keeps a record of the Title IDs of every title launched on the console, assigning each one a unique ID.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.cert Module # libWiiPy.title.cert Module
## Description
The `libWiiPy.title.cert` module provides support for parsing the various signing certificates used by the Wii for content validation. The `libWiiPy.title.cert` module provides support for parsing the various signing certificates used by the Wii for content validation.
This module allows you to write your own code for validating the authenticity of a TMD or Ticket by providing the certificates from the Wii's certificate chain. Both retail and development certificate chains are supported. This module allows you to write your own code for validating the authenticity of a TMD or Ticket by providing the certificates from the Wii's certificate chain. Both retail and development certificate chains are supported.

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.commonkeys Module # libWiiPy.title.commonkeys Module
## Description
The `libWiiPy.title.commonkeys` module simply provides easy access to the Wii's common encryption keys. The `libWiiPy.title.commonkeys` module simply provides easy access to the Wii's common encryption keys.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.content Module # libWiiPy.title.content Module
## Description
The `libWiiPy.title.content` module provides support for parsing, adding, removing, and editing content files from a digital Wii title. The `libWiiPy.title.content` module provides support for parsing, adding, removing, and editing content files from a digital Wii title.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.crypto Module # libWiiPy.title.crypto Module
## Description
The `libWiiPy.title.crypto` module provides low-level cryptography functions required for handling digital Wii titles. It does not expose many functions that are likely to be required during typical use, and instead acts more as a dependency for other modules. The `libWiiPy.title.crypto` module provides low-level cryptography functions required for handling digital Wii titles. It does not expose many functions that are likely to be required during typical use, and instead acts more as a dependency for other modules.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.iospatcher Module # libWiiPy.title.iospatcher Module
## Description
The `libWiiPy.title.iospatcher` module provides support for applying various binary patches to IOS' ES module. These patches and what they do can be found attached to the methods used to apply them. The `libWiiPy.title.iospatcher` module provides support for applying various binary patches to IOS' ES module. These patches and what they do can be found attached to the methods used to apply them.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.nus Module # libWiiPy.title.nus Module
## Description
The `libWiiPy.title.nus` module provides support for downloading digital Wii titles from the Nintendo Update Servers. This module provides easy methods for downloading TMDs, common Tickets (when present), encrypted content, and the certificate chain. The `libWiiPy.title.nus` module provides support for downloading digital Wii titles from the Nintendo Update Servers. This module provides easy methods for downloading TMDs, common Tickets (when present), encrypted content, and the certificate chain.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.ticket Module # libWiiPy.title.ticket Module
## Description
The `libWiiPy.title.ticket` module provides support for handling Tickets, which are the license files used to decrypt the content of digital titles during installation. This module allows for easy parsing and editing of Tickets. The `libWiiPy.title.ticket` module provides support for handling Tickets, which are the license files used to decrypt the content of digital titles during installation. This module allows for easy parsing and editing of Tickets.
## Module Contents ## Module Contents

View File

@@ -1,7 +1,9 @@
# libWiiPy.title Package # libWiiPy.title Package
## Description
The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, as it offers the functionality one would be most likely to need. As a result, it gets the most attention during development and should be the most reliable.
## Modules ## Modules
The `libWiiPy.title` package contains modules for interacting with Wii titles. This is the most complete package in libWiiPy, and therefore offers the most functionality.
| Module | Description | | Module | Description |
|------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| |------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
@@ -17,7 +19,7 @@ The `libWiiPy.title` package contains modules for interacting with Wii titles. T
| [libWiiPy.title.util](/title/util) | Provides some simple utility functions relating to titles | | [libWiiPy.title.util](/title/util) | Provides some simple utility functions relating to titles |
| [libWiiPy.title.wad](/title/wad) | Provides support for parsing and editing WAD files, allowing you to load each component into the other available classes | | [libWiiPy.title.wad](/title/wad) | Provides support for parsing and editing WAD files, allowing you to load each component into the other available classes |
### libWiiPy.title Package Contents ## Full Package Contents
```{toctree} ```{toctree}
:maxdepth: 4 :maxdepth: 4

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.title Module # libWiiPy.title.title Module
## Description
The `libWiiPy.title.title` module provides a high-level interface for handling all the components of a digital Wii title through one class. It allows for directly importing a WAD, and will automatically extract the various components and load them into their appropriate classes. Additionally, it provides duplicates of some methods found in those classes that require fewer arguments, as it has the context of the other components and is able to retrieve additional data automatically. The `libWiiPy.title.title` module provides a high-level interface for handling all the components of a digital Wii title through one class. It allows for directly importing a WAD, and will automatically extract the various components and load them into their appropriate classes. Additionally, it provides duplicates of some methods found in those classes that require fewer arguments, as it has the context of the other components and is able to retrieve additional data automatically.
An example of that idea can be seen with the method `get_content_by_index()`. In its original definition, which can be seen at <project:#libWiiPy.title.content.ContentRegion.get_content_by_index>, you are required to supply the Title Key for the title that the content is sourced from. In contrast, when using <project:#libWiiPy.title.title.Title.get_content_by_index>, you do not need to supply a Title Key, as the Title object already has the context of the Ticket and can retrieve the Title Key from it automatically. In a similar vein, this module provides the easiest route for verifying that a title is legitimately signed by Nintendo. The method <project:#libWiiPy.title.title.Title.get_is_signed> is able to access the entire certificate chain, the TMD, and the Ticket, and is therefore able to verify all components of the title by itself. An example of that idea can be seen with the method `get_content_by_index()`. In its original definition, which can be seen at <project:#libWiiPy.title.content.ContentRegion.get_content_by_index>, you are required to supply the Title Key for the title that the content is sourced from. In contrast, when using <project:#libWiiPy.title.title.Title.get_content_by_index>, you do not need to supply a Title Key, as the Title object already has the context of the Ticket and can retrieve the Title Key from it automatically. In a similar vein, this module provides the easiest route for verifying that a title is legitimately signed by Nintendo. The method <project:#libWiiPy.title.title.Title.get_is_signed> is able to access the entire certificate chain, the TMD, and the Ticket, and is therefore able to verify all components of the title by itself.
@@ -12,5 +14,4 @@ Because using <project:#libWiiPy.title.title.Title> allows many operations to be
.. automodule:: libWiiPy.title.title .. automodule:: libWiiPy.title.title
:members: :members:
:undoc-members: :undoc-members:
:show-inheritance:
``` ```

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.tmd Module # libWiiPy.title.tmd Module
## Description
The `libWiiPy.title.tmd` module provides support for handling TMD (Title Metadata) files, which contain the metadata of both digital and physical Wii titles. This module allows for easy parsing and editing of TMDs. The `libWiiPy.title.tmd` module provides support for handling TMD (Title Metadata) files, which contain the metadata of both digital and physical Wii titles. This module allows for easy parsing and editing of TMDs.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.util Module # libWiiPy.title.util Module
## Description
The `libWiiPy.title.util` module provides common utility functions internally. It is not designed to be used directly. The `libWiiPy.title.util` module provides common utility functions internally. It is not designed to be used directly.
## Module Contents ## Module Contents

View File

@@ -1,5 +1,7 @@
# libWiiPy.title.wad Module # libWiiPy.title.wad Module
## Description
The `libWiiPy.title.wad` module provides support for handling WAD (Wii Archive Data) files, which is the format used to deliver digital Wii titles. This module allows for extracting the various components for a WAD, as well as properly padding and writing out that data when it has been edited using other modules. The `libWiiPy.title.wad` module provides support for handling WAD (Wii Archive Data) files, which is the format used to deliver digital Wii titles. This module allows for extracting the various components for a WAD, as well as properly padding and writing out that data when it has been edited using other modules.
## Module Contents ## Module Contents

View File

@@ -4,6 +4,235 @@
# See https://wiibrew.org/wiki/LZ77 for details about the LZ77 compression format. # See https://wiibrew.org/wiki/LZ77 for details about the LZ77 compression format.
import io import io
from dataclasses import dataclass as _dataclass
from typing import List as _List
_LZ_MIN_DISTANCE = 0x01 # Minimum distance for each reference.
_LZ_MAX_DISTANCE = 0x1000 # Maximum distance for each reference.
_LZ_MIN_LENGTH = 0x03 # Minimum length for each reference.
_LZ_MAX_LENGTH = 0x12 # Maximum length for each reference.
@_dataclass
class _LZNode:
dist: int = 0
len: int = 0
weight: int = 0
def _compress_compare_bytes(buffer: _List[int], offset1: int, offset2: int, abs_len_max: int) -> int:
# Compare bytes up to the maximum length we can match. Start by comparing the first 3 bytes, since that's the
# minimum match length and this allows for a more optimized early exit.
num_matched = 0
while num_matched < abs_len_max:
if buffer[offset1 + num_matched] != buffer[offset2 + num_matched]:
break
num_matched += 1
return num_matched
def _compress_search_matches_optimized(buffer: _List[int], pos: int) -> (int, int):
bytes_left = len(buffer) - pos
global _LZ_MAX_DISTANCE, _LZ_MIN_LENGTH, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
# Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
# only look as far back as we've gone.
max_dist = min(_LZ_MAX_DISTANCE, pos)
# Default to only matching up to 18 bytes, unless fewer than 18 bytes remain, in which case we can only match
# up to that many bytes.
max_len = min(_LZ_MAX_LENGTH, bytes_left)
# Log the longest match we found and its offset.
biggest_match, biggest_match_pos = 0, 0
# Search for matches.
for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
num_matched = _compress_compare_bytes(buffer, pos - i, pos, max_len)
if num_matched > biggest_match:
biggest_match = num_matched
biggest_match_pos = i
if biggest_match == max_len:
break
return biggest_match, biggest_match_pos
def _compress_search_matches_greedy(buffer: _List[int], pos: int) -> (int, int):
# Finds and returns the first valid match, rather that finding the best one.
bytes_left = len(buffer) - pos
global _LZ_MAX_DISTANCE, _LZ_MAX_LENGTH, _LZ_MIN_DISTANCE
# Default to only looking back 4096 bytes, unless we've moved fewer than 4096 bytes, in which case we should
# only look as far back as we've gone.
max_dist = min(_LZ_MAX_DISTANCE, pos)
# Default to only matching up to 18 bytes, unless fewer than 18 bytes remain, in which case we can only match
# up to that many bytes.
max_len = min(_LZ_MAX_LENGTH, bytes_left)
match, match_pos = 0, 0
for i in range(_LZ_MIN_DISTANCE, max_dist + 1):
match = _compress_compare_bytes(buffer, pos - i, pos, max_len)
match_pos = i
if match >= _LZ_MIN_LENGTH or match == max_len:
break
return match, match_pos
def _compress_node_is_ref(node: _LZNode) -> bool:
return node.len >= _LZ_MIN_LENGTH
def _compress_get_node_cost(length: int) -> int:
if length >= _LZ_MIN_LENGTH:
num_bytes = 2
else:
num_bytes = 1
return 1 + (num_bytes * 8)
def _compress_lz77_optimized(data: bytes) -> bytes:
# Optimized compressor based around a node graph that finds optimal string matches. Originally the default
# implementation, but unfortunately it's very slow.
nodes = [_LZNode() for _ in range(len(data))]
# Iterate over the uncompressed data, starting from the end.
pos = len(data)
global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
data_list = list(data)
while pos:
pos -= 1
node = nodes[pos]
# Limit the maximum search length when we're near the end of the file.
max_search_len = min(_LZ_MAX_LENGTH, len(data_list) - pos)
if max_search_len < _LZ_MIN_DISTANCE:
max_search_len = 1
# Initialize as 1 for each, since that's all we could use if we weren't compressing.
length, dist = 1, 1
if max_search_len >= _LZ_MIN_LENGTH:
length, dist = _compress_search_matches_optimized(data_list, pos)
# Treat as direct bytes if it's too short to copy.
if length == 0 or length < _LZ_MIN_LENGTH:
length = 1
# If the node goes to the end of the file, the weight is the cost of the node.
if (pos + length) == len(data_list):
node.len = length
node.dist = dist
node.weight = _compress_get_node_cost(length)
# Otherwise, search for possible matches and determine the one with the best cost.
else:
weight_best = 0xFFFFFFFF # This was originally UINT_MAX, but that isn't a thing here so 32-bit it is!
len_best = 1
while length:
weight_next = nodes[pos + length].weight
weight = _compress_get_node_cost(length) + weight_next
if weight < weight_best:
len_best = length
weight_best = weight
length -= 1
if length != 0 and length < _LZ_MIN_LENGTH:
length = 1
node.len = len_best
node.dist = dist
node.weight = weight_best
# Write the compressed data.
with io.BytesIO() as buffer:
# Write the header data.
buffer.write(b'LZ77\x10') # The LZ type on the Wii is *always* 0x10.
buffer.write(len(data).to_bytes(3, 'little'))
src_pos = 0
while src_pos < len(data):
head = 0
head_pos = buffer.tell()
buffer.write(b'\x00') # Reserve a byte for the chunk head.
i = 0
while i < 8 and src_pos < len(data):
current_node = nodes[src_pos]
length = current_node.len
dist = current_node.dist
# This is a reference node.
if _compress_node_is_ref(current_node):
encoded = (((length - _LZ_MIN_LENGTH) & 0xF) << 12) | ((dist - _LZ_MIN_DISTANCE) & 0xFFF)
buffer.write(encoded.to_bytes(2))
head = (head | (1 << (7 - i))) & 0xFF
# This is a direct copy node.
else:
buffer.write(data[src_pos:src_pos + 1])
src_pos += length
i += 1
pos = buffer.tell()
buffer.seek(head_pos)
buffer.write(head.to_bytes(1))
buffer.seek(pos)
buffer.seek(0)
out_data = buffer.read()
return out_data
def _compress_lz77_greedy(data: bytes) -> bytes:
# Greedy compressor that processes the file start to end and saves the first matches found. Faster than the
# optimized implementation, but creates larger files.
global _LZ_MAX_LENGTH, _LZ_MIN_LENGTH, _LZ_MIN_DISTANCE
with io.BytesIO() as buffer:
# Write the header data.
buffer.write(b'LZ77\x10') # The LZ type on the Wii is *always* 0x10.
buffer.write(len(data).to_bytes(3, 'little'))
src_pos = 0
data_list = list(data)
while src_pos < len(data):
head = 0
head_pos = buffer.tell()
buffer.write(b'\x00') # Reserve a byte for the chunk head.
i = 0
while i < 8 and src_pos < len(data):
length, dist = _compress_search_matches_greedy(data_list, src_pos)
# This is a reference node.
if length >= _LZ_MIN_LENGTH:
encoded = (((length - _LZ_MIN_LENGTH) & 0xF) << 12) | ((dist - _LZ_MIN_DISTANCE) & 0xFFF)
buffer.write(encoded.to_bytes(2))
head = (head | (1 << (7 - i))) & 0xFF
src_pos += length
# This is a direct copy node.
else:
buffer.write(data[src_pos:src_pos + 1])
src_pos += 1
i += 1
pos = buffer.tell()
buffer.seek(head_pos)
buffer.write(head.to_bytes(1))
buffer.seek(pos)
buffer.seek(0)
out_data = buffer.read()
return out_data
def compress_lz77(data: bytes, compression_level: int = 1) -> bytes:
"""
Compresses data using the Wii's LZ77 compression algorithm and returns the compressed result. Supports two
different levels of compression, one based around a "greedy" LZ compression algorithm and the other based around
an optimized LZ compression algorithm. The greedy compressor, level 1, will produce a larger compressed file but
will run noticeably faster than the optimized compressor, which is level 2, especially for larger data.
Parameters
----------
data: bytes
The data to compress.
compression_level: int
The compression level to use, either 1 and 2. Default value is 1.
Returns
-------
bytes
The LZ77-compressed data.
"""
if compression_level == 1:
out_data = _compress_lz77_greedy(data)
elif compression_level == 2:
out_data = _compress_lz77_optimized(data)
else:
raise ValueError(f"Invalid compression level \"{compression_level}\"!\"")
return out_data
def decompress_lz77(lz77_data: bytes) -> bytes: def decompress_lz77(lz77_data: bytes) -> bytes:

View File

@@ -4,9 +4,9 @@
# See https://wiibrew.org/wiki/NUS for details about the NUS # See https://wiibrew.org/wiki/NUS for details about the NUS
import requests import requests
import hashlib #import hashlib
from typing import List from typing import List
from urllib.parse import urlparse as _urlparse #from urllib.parse import urlparse as _urlparse
from .title import Title from .title import Title
from .tmd import TMD from .tmd import TMD
from .ticket import Ticket from .ticket import Ticket
@@ -315,9 +315,10 @@ def _validate_endpoint(endpoint: str) -> str:
The validated NUS endpoint with the proper path. The validated NUS endpoint with the proper path.
""" """
# Find the root of the URL and then assemble the correct URL based on that. # Find the root of the URL and then assemble the correct URL based on that.
new_url = _urlparse(endpoint) # TODO: Rewrite in a way that makes more sense and un-stub
if new_url.netloc == "": #new_url = _urlparse(endpoint)
endpoint_url = "http://" + new_url.path + "/ccs/download/" #if new_url.netloc == "":
else: # endpoint_url = "http://" + new_url.path + "/ccs/download/"
endpoint_url = "http://" + new_url.netloc + "/ccs/download/" #else:
return endpoint_url # endpoint_url = "http://" + new_url.netloc + "/ccs/download/"
return endpoint

View File

@@ -128,7 +128,7 @@ class Ticket:
self.console_id = int.from_bytes(ticket_data.read(4)) self.console_id = int.from_bytes(ticket_data.read(4))
# Title ID. # Title ID.
ticket_data.seek(0x1DC) ticket_data.seek(0x1DC)
self.title_id = binascii.hexlify(ticket_data.read(8)) self.title_id = ticket_data.read(8)
# Unknown data 1. # Unknown data 1.
ticket_data.seek(0x1E4) ticket_data.seek(0x1E4)
self.unknown1 = ticket_data.read(2) self.unknown1 = ticket_data.read(2)
@@ -202,7 +202,7 @@ class Ticket:
# Console ID. # Console ID.
ticket_data += int.to_bytes(self.console_id, 4) ticket_data += int.to_bytes(self.console_id, 4)
# Title ID. # Title ID.
ticket_data += binascii.unhexlify(self.title_id) ticket_data += self.title_id
# Unknown data 1. # Unknown data 1.
ticket_data += self.unknown1 ticket_data += self.unknown1
# Title version. # Title version.
@@ -318,6 +318,8 @@ class Ticket:
return "Korean" return "Korean"
case 2: case 2:
return "vWii" return "vWii"
case _:
return "Unknown"
def get_title_key(self) -> bytes: def get_title_key(self) -> bytes:
""" """
@@ -343,7 +345,7 @@ class Ticket:
""" """
if len(title_id) != 16: if len(title_id) != 16:
raise ValueError("Invalid Title ID! Title IDs must be 8 bytes long.") raise ValueError("Invalid Title ID! Title IDs must be 8 bytes long.")
self.title_id = title_id.encode() self.title_id = binascii.unhexlify(title_id.encode())
def set_title_version(self, new_version: str | int) -> None: def set_title_version(self, new_version: str | int) -> None:
""" """