Strings With Source Tracking

This module provides a string type SourceStr() which remembers the source it originates from.

The correspondence between the contents of a SourceStr and the source file is stored in a SourceMap which stores a collection of SourceMapSpans, each mapping a contiguous range of characters.

Obtaining and Manipulating Source Strings

The easiest way to obtain a SourceStr from a file is to use the read_file function:

yosys_mau.source_str.read_file(path, *, store_content=None, relative_to=None)

Read a file into a SourceStr that track’s its source.

Parameters:

  • path (PathLike[Any] | str) – The path to the file to read.

  • store_content (bool | None) – Whether to store the file’s content in memory even when only parts of it are still referenced. By default only files below 1MiB are stored, but this option can force or prevent storing the content.

  • relative_to (PathLike[Any] | str | None) – The path to resolve relative paths against. If not given, the current working directory is used.

Returns:

A SourceStr that tracks the file’s source.

Return type:

str

The SourceStr class overrides some of the standard str methods to keep track of their contents. For some standard python string operations this isn’t possible though, and in that case alternative methods may be present.

There is also a yosys_mau.source_str.re module which provides the same API as the standard library’s re module, but with support for tracking sources when matching within SourceStr objects.

class yosys_mau.source_str.SourceStr(value, source_map=None)

Bases: str

String type which remembers the source it originates from.

A SourceStr inherits from str but has an additional attribute source_map to track the string’s origin. When a SourceStr operation returns a new string, e.g. when you slice or concatenate strings, the resulting string will often be a SourceStr with the source_map attribute set accordingly.

Parameters:

  • source_map (SourceMap | None) – The source map to associate with the string. If not given, this actually returns a plain str.

  • value (str)

Return type:

str

property source_map: SourceMap

The source map associated with this string.

Prefer using source_map() to access this, unless you know for certain that your target string is a SourceStr.

__getitem__(key)

Source tracking slicing.

Not all slicing operations are source tracking, but [start:stop], [start:], [:stop] and [:] are.

Parameters:

key (SupportsIndex | slice)

Return type:

str

__add__(other)

Source tracking concatenation.

Parameters:

other (str)

Return type:

str

splitlines(keepends=False)

Source tracking implementation of str.splitlines().

Parameters:

keepends (bool)

Return type:

list[str]

split(sep=None, maxsplit=-1)

Source tracking implementation of str.split().

Parameters:

  • sep (str | None)

  • maxsplit (SupportsIndex)

Return type:

list[str]

strip(chars=None)

Source tracking implementation of str.strip().

Parameters:

chars (str | None)

Return type:

str

rstrip(chars=None)

Source tracking implementation of str.rstrip().

Parameters:

chars (str | None)

Return type:

str

lstrip(chars=None)

Source tracking implementation of str.lstrip().

Parameters:

chars (str | None)

Return type:

str

replace(old, new, count=-1)

Source tracking implementation of str.replace().

Parameters:

  • old (str)

  • new (str)

  • count (SupportsIndex)

Return type:

str

join(iterable)

Source tracking implementation of str.join().

Note

When using join, self often isn’t a SourceStr even when the arguments are. Since we can only override methods according to the receiver self, to ensure source tracking, this has to be invoked as SourceStr.join(self, iterable).

Parameters:
Return type:

str

as_plain_str()

Return a copy of the string without source tracking information.

Return type:

str

Inspecting String Sources

The correspondence between the contents of a SourceStr and the source file is stored in a SourceMap which stores a collection of SourceMapSpans. The SourceMap class inherits from SourceSpans and SourceMapSpan from SourceSpan, where either super-class only stores the source spans without associating them to any specific string. They are mostly used when generating diagnostic messages from source strings.

To obtain a SourceMap from either SourceStr or a plain str, use the source_map function, which will return an empty mapping for a plain str:

yosys_mau.source_str.source_map(string)

The source map of a string.

Parameters:

string (str)

Return type:

SourceMap

class yosys_mau.source_str.SourceMap(spans, len)

Bases: SourceSpans

Maps string contents to their source files.

It can map different spans of the string to different source files or different parts of the same source file.

The SourceMapSpans are stored in string order, to allow for binary searches.

Parameters:
len: int

The length of the string.

spans: tuple[SourceMapSpan, ...]

Each span maps a contiguous part of the string to a part of a source file.

They are stored in order, to allow for binary searches.

property simple_span: bool

Whether the source map is a single span that maps the entire string.

detached()

Returns the source spans of this source map, detached from their position in the string.

Return type:

SourceSpans

class yosys_mau.source_str.SourceSpans(spans)

A collection of source spans.

Parameters:

spans (tuple[SourceSpan, ...])

spans: tuple[SourceSpan, ...]

Each span covers a contiguous part of a source file.

close_gaps(max_gap=3, line_mode=False)

Sorts contained spans and merges almost adjacent spans.

Parameters:

  • max_gap (int) – The maximum gap between two spans that will be merged.

  • line_mode (bool)

Returns:

A new SourceSpans with sorted and merged spans.

Return type:

SourceSpans

class yosys_mau.source_str.SourceMapSpan(len, file_start, file, str_start)

Bases: SourceSpan

Maps a span of a string to a corresponding span of a source file.

All offsets use Python’s native string indexing, i.e. they count unicode codepoints.

Parameters:

  • len (int)

  • file_start (int)

  • file (SourceFile)

  • str_start (int)

str_start: int

Start of the span in the string.

property str_end: int

End of the span in the string.

class yosys_mau.source_str.SourceSpan(len, file_start, file)

A contiguous span within a source file.

All offsets use Python’s native string indexing, i.e. they count unicode codepoints.

Parameters:

  • len (int)

  • file_start (int)

  • file (SourceFile)

len: int

Length of the span.

file_start: int

Start of the span in the source file.

file: SourceFile

The source file that contains the span.

property file_end: int

End of the span in the source file.

Generating Diagnostics

Diagnostic reporting using source tracking strings.

Warning

The API for error reporting hasn’t been fully designed yet and is likely to change.

exception yosys_mau.source_str.report.InputError(where, message)

An error in user input.

Note

The API for error reporting hasn’t been fully designed yet and is likely to change. Something simple to raise one-off errors like this will certainly stay though.

Parameters:

  • where (str | None)

  • message (str)

Return type:

None

class yosys_mau.source_str.report.Report(spans, message)

Collects diagnostic information associated to specific source locations.

Warning

The API for error reporting hasn’t been fully designed yet and is likely to change. This class in particular will undergo backwards incompatible API changes and should be considered internal to this library for now.

Parameters:

Regular Expression Support

The yosys_mau.source_str.re module provides the same API as the standard re module, but with support for tracking sources when matching within SourceStr objects.

Apart from the module contents listed her, this also re-exports the flags constants defined by the standard re module.

yosys_mau.source_str.re.compile(pattern, flags=0)

Source tracking wrapper for re.compile().

Parameters:
Return type:

Pattern

yosys_mau.source_str.re.search(pattern, string, flags=0)

Source tracking implementation of re.search().

Parameters:
Return type:

Match | None

yosys_mau.source_str.re.match(pattern, string, flags=0)

Source tracking implementation of re.match().

Parameters:
Return type:

Match | None

yosys_mau.source_str.re.fullmatch(pattern, string, flags=0)

Source tracking implementation of re.fullmatch().

Parameters:
Return type:

Match | None

yosys_mau.source_str.re.split(pattern, string, maxsplit=0, flags=0)

Source tracking implementation of re.split().

Parameters:
Return type:

list[str]

yosys_mau.source_str.re.findall(pattern, string, flags=0)

Source tracking implementation of re.findall().

Parameters:
Return type:

list[str]

yosys_mau.source_str.re.finditer(pattern, string, flags=0)

Source tracking implementation of re.finditer().

Parameters:
Return type:

Iterator[Match]

class yosys_mau.source_str.re.Pattern(wrapped)

Source tracking wrapper for re Pattern objects.

Parameters:

wrapped (Pattern[str])

wrapped: Pattern[str]

The wrapped plain re Pattern object.

search(string, pos=0, endpos=9223372036854775807)

Source tracking wrapper for re.Pattern.search().

Parameters:
Return type:

Match | None

match(string, pos=0, endpos=9223372036854775807)

Source tracking wrapper for re.Pattern.match().

Parameters:
Return type:

Match | None

fullmatch(string, pos=0, endpos=9223372036854775807)

Source tracking wrapper for re.Pattern.fullmatch().

Parameters:
Return type:

Match | None

split(string, maxsplit=0)

Source tracking implementation of re.Pattern.split().

Parameters:
Return type:

list[str]

findall(string, pos=0, endpos=9223372036854775807)

Source tracking implementation of re.Pattern.findall().

Parameters:
Return type:

list[str]

finditer(string, pos=0, endpos=9223372036854775807)

Source tracking wrapper for re.Pattern.finditer().

Parameters:
Return type:

Iterator[Match]

sub(repl, string, count=0)

Source tracking implementation of re.Pattern.sub().

Parameters:
Return type:

str

subn(repl, string, count=0)

Source tracking implementation of re.Pattern.subn().

Parameters:
Return type:

tuple[str, int]

property flags: int

Forwards to re.Pattern.flags.

property groups: int

Forwards to re.Pattern.groups.

property groupindex: Mapping[str, int]

Forwards to re.Pattern.groupindex.

property pattern: str

Forwards to re.Pattern.pattern.

class yosys_mau.source_str.re.Match(wrapped)

Source tracking wrapper for re Match objects.

Parameters:

wrapped (Match[str])

wrapped: Match[str]

The wrapped plain re Match object.

property pos: int

Forwards to re.Match.pos.

property endpos: int

Forwards to re.Match.endpos.

property string: str

Forwards to re.Match.string.

Even though this forwards to the wrapped object, this will return a <project:#SourceStr> when used as match target.

property lastindex: int | None

Forwards to re.Match.lastindex.

property lastgroup: str | None

Forwards to re.Match.lastgroup.

property re: Pattern

Source tracking wrapper for re.Match.re.

span(group=0)

Forwards to re.Match.span().

Parameters:

group (int | str)

Return type:

tuple[int, int]

start(group=0)

Forwards to re.Match.start().

Parameters:

group (int | str)

Return type:

int

end(group=0)

Forwards to re.Match.end().

Parameters:

group (int | str)

Return type:

int

group(group: Literal[0] = 0, /) → str
group(group: str | int, /) → str | None
group(group1: str | int, group2: str | int, /, *groups: str | int) → tuple[str | None, ...]

Source tracking wrapper for re.Match.group().

__getitem__(group: Literal[0]) → str
__getitem__(group: int | str) → str | None

Source tracking wrapper for re.Match.__getitem__().

groups() → tuple[str | None, ...]
groups(default: _T) → tuple[str | _T, ...]

Source tracking wrapper for re.Match.groups().

groupdict() → dict[str, str | None]
groupdict(default: _T) → dict[str, str | _T]

Source tracking wrapper for re.Match.groupdict().

expand(template)

Source tracking implementation of re.Match.expand().

Parameters:

template (str)

Return type:

str