Struct Tokens

pub struct Tokens<L = ()>
where
    L: Lang,
{ /* private fields */ }

A stream of tokens.

Structural Guarantees

This stream of tokens provides the following structural guarantees.

• Only one space occurs in sequence and indicates spacing between tokens.
• Only one push occurs in sequence and indicates that the next token should be spaced onto a new line.
• A line is never by a push since it would have no effect. A line ensures an empty line between two tokens.

use genco::Tokens;
use genco::tokens::Item;

let mut tokens = Tokens::<()>::new();

// The first push token is "overriden" by a line.
tokens.space();
tokens.space();

assert_eq!(vec![Item::Space::<()>], tokens);

let mut tokens = Tokens::<()>::new();

tokens.space();
tokens.push();
tokens.push();

assert_eq!(vec![Item::Push::<()>], tokens);

let mut tokens = Tokens::<()>::new();

// The first space and push tokens are "overriden" by a line.
tokens.space();
tokens.push();
tokens.line();

assert_eq!(vec![Item::Line::<()>], tokens);

Implementations

impl<L> Tokens<L>

where L: Lang,

pub fn new() -> Self

Create a new empty stream of tokens.

Examples

use genco::prelude::*;

let tokens = Tokens::<()>::new();

assert!(tokens.is_empty());

pub fn with_capacity(cap: usize) -> Self

Create a new empty stream of tokens with the specified capacity.

Examples

use genco::prelude::*;

let tokens = Tokens::<()>::with_capacity(10);

assert!(tokens.is_empty());

pub fn iter(&self) -> Iter<'_, L>

Construct an iterator over the token stream.

Examples

use genco::prelude::*;
use genco::tokens::{ItemStr, Item};

let tokens: Tokens<()> = quote!(foo bar baz);
let mut it = tokens.iter();

assert_eq!(Some(&Item::Literal(ItemStr::Static("foo"))), it.next());
assert_eq!(Some(&Item::Space), it.next());
assert_eq!(Some(&Item::Literal(ItemStr::Static("bar"))), it.next());
assert_eq!(Some(&Item::Space), it.next());
assert_eq!(Some(&Item::Literal(ItemStr::Static("baz"))), it.next());
assert_eq!(None, it.next());

pub fn append<T>(&mut self, tokens: T)

where T: FormatInto<L>,

Append the given tokens.

This append function takes anything implementing FormatInto making the argument’s behavior customizable. Most primitive types have built-in implementations of FormatInto treating them as raw tokens.

Most notabley, things implementing FormatInto can be used as arguments for interpolation in the quote! macro.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();
tokens.append(4u32);

assert_eq!(quote!($(4u32)), tokens);

pub fn extend<I>(&mut self, it: I)

where I: IntoIterator<Item = Item<L>>,

Extend with another stream of tokens.

This respects the structural requirements of adding one element at a time, like you would get by calling space, push, or line.

Examples

use genco::prelude::*;
use genco::tokens::{Item, ItemStr};

let mut tokens: Tokens<()> = quote!(foo bar);
tokens.extend::<Tokens<()>>(quote!($[' ']baz));

assert_eq!(tokens, quote!(foo bar baz));

pub fn walk_imports(&self) -> WalkImports<'_, L>

Walk over all imports.

The order in which the imports are returned is not defined. So if you need them in some particular order you need to sort them.

Examples

use genco::prelude::*;

let debug = rust::import("std::fmt", "Debug");
let ty = rust::import("std::collections", "HashMap");

let tokens = quote!(foo $ty<u32, dyn $debug> baz);

for import in tokens.walk_imports() {
    println!("{:?}", import);
}

pub fn register<T>(&mut self, tokens: T)

where T: Register<L>,

Add an registered custom element that is not rendered.

Registration can be used to generate imports that do not render a visible result.

Examples

use genco::prelude::*;

let write_bytes_ext = rust::import("byteorder", "WriteBytesExt").with_alias("_");

let tokens = quote!($(register(write_bytes_ext)));

assert_eq!("use byteorder::WriteBytesExt as _;\n", tokens.to_file_string()?);

pub fn is_empty(&self) -> bool

Check if tokens contain no items.

use genco::prelude::*;

let tokens: Tokens<()> = quote!();

assert!(tokens.is_empty());

pub fn space(&mut self)

Add a single spacing to the token stream.

Note that due to structural guarantees two consequent spaces may not follow each other in the same token stream.

A space operation has no effect unless it’s followed by a non-whitespace token.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();

tokens.space();
tokens.append("hello");
tokens.space();
tokens.space(); // Note: ignored
tokens.append("world");
tokens.space();

assert_eq!(
    vec![
        " hello world",
    ],
    tokens.to_file_vec()?
);

pub fn push(&mut self)

Add a single push operation.

Push operations ensure that any following tokens are added to their own line.

A push has no effect unless it’s preceeded or followed by non-whitespace tokens.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();

tokens.push();
tokens.append("hello");
tokens.push();
tokens.append("world");
tokens.push();

assert_eq!(
    vec![
        "hello",
        "world"
    ],
    tokens.to_file_vec()?
);

pub fn line(&mut self)

Add a single line operation.

A line ensures that any following tokens have one line of separation between them and the preceeding tokens.

A line has no effect unless it’s preceeded and followed by non-whitespace tokens.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();

tokens.line();
tokens.append("hello");
tokens.line();
tokens.append("world");
tokens.line();

assert_eq!(
    vec![
        "hello",
        "",
        "world"
    ],
    tokens.to_file_vec()?
);

pub fn indent(&mut self)

Increase the indentation of the token stream.

An indentation is a language-specific operation which adds whitespace to the beginning of a line preceeding any non-whitespace tokens.

An indentation has no effect unless it’s followed by non-whitespace tokens. It also acts like a push, in that it will shift any tokens to a new line.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();

tokens.indent();
tokens.append("hello");
tokens.indent();
tokens.append("world");
tokens.indent();
tokens.append("😀");

assert_eq!(
    vec![
        "    hello",
        "        world",
        "            😀",
    ],
    tokens.to_file_vec()?
);

pub fn unindent(&mut self)

Decrease the indentation of the token stream.

An indentation is a language-specific operation which adds whitespace to the beginning of a line preceeding any non-whitespace tokens.

An indentation has no effect unless it’s followed by non-whitespace tokens. It also acts like a push, in that it will shift any tokens to a new line.

Indentation can never go below zero, and will just be ignored if that were to happen. However, negative indentation is stored in the token stream, so any negative indentation in place will have to be countered before indentation starts again.

Examples

use genco::prelude::*;

let mut tokens = Tokens::<()>::new();

tokens.indent();
tokens.append("hello");
tokens.unindent();
tokens.append("world");
tokens.unindent();
tokens.append("😀");
tokens.indent();
tokens.append("😁");
tokens.indent();
tokens.append("😂");

assert_eq!(
    vec![
        "    hello",
        "world",
        "😀",
        "😁",
        "    😂",
    ],
    tokens.to_file_vec()?
);

pub fn format( &self, out: &mut Formatter<'_>, config: &L::Config, format: &L::Format, ) -> Result

Formatting function for token streams that gives full control over the formatting environment.

The configurations and format arguments will be provided to all registered language items as well, and can be used to customize formatting through LangItem::format().

The format argument is primarily used internally by Lang::format_file to provide intermediate state that can be affect how language items are formatter. So formatting something as a file might yield different results than using this raw formatting function.

Available formatters:

• fmt::VecWriter - To write result into a vector.
• fmt::FmtWriter - To write the result into something implementing fmt::Write.
• fmt::IoWriter- To write the result into something implementing io::Write.

Examples

use genco::prelude::*;
use genco::fmt;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

let stdout = std::io::stdout();
let mut w = fmt::IoWriter::new(stdout.lock());

let fmt = fmt::Config::from_lang::<Rust>()
    .with_indentation(fmt::Indentation::Space(2));
let mut formatter = w.as_formatter(&fmt);
let config = rust::Config::default();

// Default format state for Rust.
let format = rust::Format::default();

tokens.format(&mut formatter, &config, &format)?;

pub fn format_file(&self, out: &mut Formatter<'_>, config: &L::Config) -> Result

File formatting function for token streams that gives full control over the formatting environment.

File formatting will render preambles like namespace declarations and imports.

Available formatters:

• fmt::VecWriter - To write result into a vector.
• fmt::FmtWriter - To write the result into something implementing fmt::Write.
• fmt::IoWriter- To write the result into something implementing io::Write.

Examples

use genco::prelude::*;
use genco::fmt;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

let stdout = std::io::stdout();
let mut w = fmt::IoWriter::new(stdout.lock());

let fmt = fmt::Config::from_lang::<Rust>()
    .with_indentation(fmt::Indentation::Space(2));
let mut formatter = w.as_formatter(&fmt);
let config = rust::Config::default();

tokens.format_file(&mut formatter, &config)?;

impl<L> Tokens<L>

where L: Lang, L::Config: Default,

pub fn to_file_string(&self) -> Result<String>

Format the token stream as a file for the given target language to a string using the default configuration.

This is a shorthand to using FmtWriter directly in combination with format.

This function will render imports.

Examples

use genco::prelude::*;
use genco::fmt;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

assert_eq!(
    "use std::collections::HashMap;\n\nlet mut m = HashMap::new();\nm.insert(1u32, 2u32);\n",
    tokens.to_file_string()?
);

pub fn to_string(&self) -> Result<String>

Format only the current token stream as a string using the default configuration.

This is a shorthand to using FmtWriter directly in combination with format.

This function will not render imports.

Examples

use genco::prelude::*;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

assert_eq!(
    "let mut m = HashMap::new();\nm.insert(1u32, 2u32);",
    tokens.to_string()?
);

pub fn to_file_vec(&self) -> Result<Vec<String>>

Format tokens into a vector, where each entry equals a line in the resulting file using the default configuration.

This is a shorthand to using VecWriter directly in combination with format.

This function will render imports.

Examples

use genco::prelude::*;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

assert_eq!(
    vec![
        "use std::collections::HashMap;",
        "",
        "let mut m = HashMap::new();",
        "m.insert(1u32, 2u32);"
    ],
    tokens.to_file_vec()?
);

Example with Python indentation

use genco::prelude::*;

let tokens: python::Tokens = quote! {
    def foo():
        pass

    def bar():
        pass
};

assert_eq!(
    vec![
        "def foo():",
        "    pass",
        "",
        "def bar():",
        "    pass",
    ],
    tokens.to_file_vec()?
);

pub fn to_vec(&self) -> Result<Vec<String>>

Helper function to format tokens into a vector, where each entry equals a line using the default configuration.

This is a shorthand to using VecWriter directly in combination with format.

This function will not render imports.

Examples

use genco::prelude::*;

let map = rust::import("std::collections", "HashMap");

let tokens: rust::Tokens = quote! {
    let mut m = $map::new();
    m.insert(1u32, 2u32);
};

assert_eq!(
    vec![
        "let mut m = HashMap::new();",
        "m.insert(1u32, 2u32);"
    ],
    tokens.to_vec()?
);

Trait Implementations

impl<L> Clone for Tokens<L>

where L: Lang + Clone,

fn clone(&self) -> Tokens<L>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<L> Debug for Tokens<L>

where L: Lang + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<L> Default for Tokens<L>

where L: Lang,

fn default() -> Self

Returns the “default value” for a type.

impl<L> FormatInto<L> for &Tokens<L>

where L: Lang,

Formatting a reference to a token stream is exactly the same as extending the token stream with a copy of the stream being formatted.

Examples

use genco::prelude::*;

let a: &Tokens = &quote!(foo bar);

let result = quote!($a baz);

assert_eq!("foo bar baz", result.to_string()?);

fn format_into(self, tokens: &mut Tokens<L>)

Convert the type into tokens in-place.

impl<L> FormatInto<L> for Tokens<L>

where L: Lang,

fn format_into(self, tokens: &mut Self)

Convert the type into tokens in-place.

impl<'a, L> FromIterator<&'a Item<L>> for Tokens<L>

where L: Lang,

fn from_iter<I: IntoIterator<Item = &'a Item<L>>>(iter: I) -> Self

Creates a value from an iterator.

impl<L> FromIterator<Item<L>> for Tokens<L>

where L: Lang,

fn from_iter<I: IntoIterator<Item = Item<L>>>(iter: I) -> Self

Creates a value from an iterator.

impl<L> Hash for Tokens<L>

where L: Lang + Hash,

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl<'a, L> IntoIterator for &'a Tokens<L>

where L: Lang,

type Item = &'a Item<L>

The type of the elements being iterated over.

type IntoIter = Iter<'a, L>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<L> IntoIterator for Tokens<L>

where L: Lang,

Construct an owned iterator over the token stream.

Examples

use genco::prelude::*;
use genco::tokens::{ItemStr, Item};

let tokens: Tokens<()> = quote!(foo bar baz);
let mut it = tokens.into_iter();

assert_eq!(Some(Item::Literal(ItemStr::Static("foo"))), it.next());
assert_eq!(Some(Item::Space), it.next());
assert_eq!(Some(Item::Literal(ItemStr::Static("bar"))), it.next());
assert_eq!(Some(Item::Space), it.next());
assert_eq!(Some(Item::Literal(ItemStr::Static("baz"))), it.next());
assert_eq!(None, it.next());

type Item = Item<L>

The type of the elements being iterated over.

type IntoIter = IntoIter<L>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<L> Ord for Tokens<L>

where L: Lang + Ord,

fn cmp(&self, other: &Tokens<L>) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl<L> PartialEq<[Item<L>]> for Tokens<L>

where L: Lang,

fn eq(&self, other: &[Item<L>]) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<L> PartialEq<Tokens<L>> for [Item<L>]

where L: Lang,

fn eq(&self, other: &Tokens<L>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<L> PartialEq<Tokens<L>> for Vec<Item<L>>

where L: Lang,

fn eq(&self, other: &Tokens<L>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<L> PartialEq<Vec<Item<L>>> for Tokens<L>

where L: Lang,

fn eq(&self, other: &Vec<Item<L>>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<L> PartialEq for Tokens<L>

where L: Lang + PartialEq,

fn eq(&self, other: &Tokens<L>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<L> PartialOrd for Tokens<L>

where L: Lang + PartialOrd,

fn partial_cmp(&self, other: &Tokens<L>) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl<L> Eq for Tokens<L>

where L: Lang + Eq,

impl<L> StructuralPartialEq for Tokens<L>

where L: Lang,