Struct ParseBuffer

source

pub struct ParseBuffer<'a> { /* private fields */ }

Expand description

Cursor position within a buffered token stream.

This type is more commonly used through the type alias ParseStream which is an alias for &ParseBuffer.

ParseStream is the input type for all parser functions in Syn. They have the signature fn(ParseStream) -> Result<T>.

§Calling a parser function

There is no public way to construct a ParseBuffer. Instead, if you are looking to invoke a parser function that requires ParseStream as input, you will need to go through one of the public parsing entry points.

The parse_macro_input! macro if parsing input of a procedural macro;
One of the syn::parse* functions; or
A method of the Parser trait.

Implementations§

source §

impl<'a> ParseBuffer<'a>

source

pub fn parse<T: Parse>(&self) -> Result<T>

Parses a syntax tree node of type T, advancing the position of our parse stream past it.

source

pub fn call<T>( &self, function: fn(_: ParseStream<'_>) -> Result<T>, ) -> Result<T>

Calls the given parser function to parse a syntax tree node of type T from this stream.

§Example

The parser below invokes Attribute::parse_outer to parse a vector of zero or more outer attributes.

use syn::{Attribute, Ident, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses a unit struct with attributes.
//
//     #[path = "s.tmpl"]
//     struct S;
struct UnitStruct {
    attrs: Vec<Attribute>,
    struct_token: Token![struct],
    name: Ident,
    semi_token: Token![;],
}

impl Parse for UnitStruct {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(UnitStruct {
            attrs: input.call(Attribute::parse_outer)?,
            struct_token: input.parse()?,
            name: input.parse()?,
            semi_token: input.parse()?,
        })
    }
}

source

pub fn peek<T: Peek>(&self, token: T) -> bool

Looks at the next token in the parse stream to determine whether it matches the requested type of token.

Does not advance the position of the parse stream.

§Syntax

Note that this method does not use turbofish syntax. Pass the peek type inside of parentheses.

input.peek(Token![struct])
input.peek(Token![==])
input.peek(syn::Ident) (does not accept keywords)
input.peek(syn::Ident::peek_any)
input.peek(Lifetime)
input.peek(token::Brace)

§Example

In this example we finish parsing the list of supertraits when the next token in the input is either where or an opening curly brace.

use syn::{braced, token, Generics, Ident, Result, Token, TypeParamBound};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

// Parses a trait definition containing no associated items.
//
//     trait Marker<'de, T>: A + B<'de> where Box<T>: Clone {}
struct MarkerTrait {
    trait_token: Token![trait],
    ident: Ident,
    generics: Generics,
    colon_token: Option<Token![:]>,
    supertraits: Punctuated<TypeParamBound, Token![+]>,
    brace_token: token::Brace,
}

impl Parse for MarkerTrait {
    fn parse(input: ParseStream) -> Result<Self> {
        let trait_token: Token![trait] = input.parse()?;
        let ident: Ident = input.parse()?;
        let mut generics: Generics = input.parse()?;
        let colon_token: Option<Token![:]> = input.parse()?;

        let mut supertraits = Punctuated::new();
        if colon_token.is_some() {
            loop {
                supertraits.push_value(input.parse()?);
                if input.peek(Token![where]) || input.peek(token::Brace) {
                    break;
                }
                supertraits.push_punct(input.parse()?);
            }
        }

        generics.where_clause = input.parse()?;
        let content;
        let empty_brace_token = braced!(content in input);

        Ok(MarkerTrait {
            trait_token,
            ident,
            generics,
            colon_token,
            supertraits,
            brace_token: empty_brace_token,
        })
    }
}

source

pub fn peek2<T: Peek>(&self, token: T) -> bool

Looks at the second-next token in the parse stream.

This is commonly useful as a way to implement contextual keywords.

§Example

This example needs to use peek2 because the symbol union is not a keyword in Rust. We can’t use just peek and decide to parse a union if the very next token is union, because someone is free to write a mod union and a macro invocation that looks like union::some_macro! { ... }. In other words union is a contextual keyword.

use syn::{Ident, ItemUnion, Macro, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses either a union or a macro invocation.
enum UnionOrMacro {
    // union MaybeUninit<T> { uninit: (), value: T }
    Union(ItemUnion),
    // lazy_static! { ... }
    Macro(Macro),
}

impl Parse for UnionOrMacro {
    fn parse(input: ParseStream) -> Result<Self> {
        if input.peek(Token![union]) && input.peek2(Ident) {
            input.parse().map(UnionOrMacro::Union)
        } else {
            input.parse().map(UnionOrMacro::Macro)
        }
    }
}

source

pub fn peek3<T: Peek>(&self, token: T) -> bool

Looks at the third-next token in the parse stream.

source

pub fn parse_terminated<T, P>( &self, parser: fn(_: ParseStream<'_>) -> Result<T>, separator: P, ) -> Result<Punctuated<T, P::Token>>
where P: Peek, P::Token: Parse,

Parses zero or more occurrences of T separated by punctuation of type P, with optional trailing punctuation.

Parsing continues until the end of this parse stream. The entire content of this parse stream must consist of T and P.

§Example

use syn::{parenthesized, token, Ident, Result, Token, Type};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

// Parse a simplified tuple struct syntax like:
//
//     struct S(A, B);
struct TupleStruct {
    struct_token: Token![struct],
    ident: Ident,
    paren_token: token::Paren,
    fields: Punctuated<Type, Token![,]>,
    semi_token: Token![;],
}

impl Parse for TupleStruct {
    fn parse(input: ParseStream) -> Result<Self> {
        let content;
        Ok(TupleStruct {
            struct_token: input.parse()?,
            ident: input.parse()?,
            paren_token: parenthesized!(content in input),
            fields: content.parse_terminated(Type::parse, Token![,])?,
            semi_token: input.parse()?,
        })
    }
}

§See also

If your separator is anything more complicated than an invocation of the Token! macro, this method won’t be applicable and you can instead directly use Punctuated’s parser functions: parse_terminated, parse_separated_nonempty etc.

use syn::{custom_keyword, Expr, Result, Token};
use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;

mod kw {
    syn::custom_keyword!(fin);
}

struct Fin(kw::fin, Token![;]);

impl Parse for Fin {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(Self(input.parse()?, input.parse()?))
    }
}

struct Thing {
    steps: Punctuated<Expr, Fin>,
}

impl Parse for Thing {
    fn parse(input: ParseStream) -> Result<Self> {
        Ok(Thing {
            steps: Punctuated::parse_terminated(input)?,
        })
        // or equivalently, this means the same thing:
            steps: input.call(Punctuated::parse_terminated)?,
    }
}

source

pub fn is_empty(&self) -> bool

Returns whether there are tokens remaining in this stream.

This method returns true at the end of the content of a set of delimiters, as well as at the very end of the complete macro input.

§Example

use syn::{braced, token, Ident, Item, Result, Token};
use syn::parse::{Parse, ParseStream};

// Parses a Rust `mod m { ... }` containing zero or more items.
struct Mod {
    mod_token: Token![mod],
    name: Ident,
    brace_token: token::Brace,
    items: Vec<Item>,
}

impl Parse for Mod {
    fn parse(input: ParseStream) -> Result<Self> {
        let content;
        Ok(Mod {
            mod_token: input.parse()?,
            name: input.parse()?,
            brace_token: braced!(content in input),
            items: {
                let mut items = Vec::new();
                while !content.is_empty() {
                    items.push(content.parse()?);
                }
                items
            },
        })
    }
}

source

pub fn lookahead1(&self) -> Lookahead1<'a>

Constructs a helper for peeking at the next token in this stream and building an error message if it is not one of a set of expected tokens.

§Example

use syn::{ConstParam, Ident, Lifetime, LifetimeParam, Result, Token, TypeParam};
use syn::parse::{Parse, ParseStream};

// A generic parameter, a single one of the comma-separated elements inside
// angle brackets in:
//
//     fn f<T: Clone, 'a, 'b: 'a, const N: usize>() { ... }
//
// On invalid input, lookahead gives us a reasonable error message.
//
//     error: expected one of: identifier, lifetime, `const`
//       |
//     5 |     fn f<!Sized>() {}
//       |          ^
enum GenericParam {
    Type(TypeParam),
    Lifetime(LifetimeParam),
    Const(ConstParam),
}

impl Parse for GenericParam {
    fn parse(input: ParseStream) -> Result<Self> {
        let lookahead = input.lookahead1();
        if lookahead.peek(Ident) {
            input.parse().map(GenericParam::Type)
        } else if lookahead.peek(Lifetime) {
            input.parse().map(GenericParam::Lifetime)
        } else if lookahead.peek(Token![const]) {
            input.parse().map(GenericParam::Const)
        } else {
            Err(lookahead.error())
        }
    }
}

source

pub fn fork(&self) -> Self

Forks a parse stream so that parsing tokens out of either the original or the fork does not advance the position of the other.

§Performance

Forking a parse stream is a cheap fixed amount of work and does not involve copying token buffers. Where you might hit performance problems is if your macro ends up parsing a large amount of content more than once.

// Do not do this.
if input.fork().parse::<Expr>().is_ok() {
    return input.parse::<Expr>();
}

As a rule, avoid parsing an unbounded amount of tokens out of a forked parse stream. Only use a fork when the amount of work performed against the fork is small and bounded.

When complex speculative parsing against the forked stream is unavoidable, use parse::discouraged::Speculative to advance the original stream once the fork’s parse is determined to have been successful.

For a lower level way to perform speculative parsing at the token level, consider using ParseStream::step instead.

§Example

The parse implementation shown here parses possibly restricted pub visibilities.

pub
pub(crate)
pub(self)
pub(super)
pub(in some::path)

To handle the case of visibilities inside of tuple structs, the parser needs to distinguish parentheses that specify visibility restrictions from parentheses that form part of a tuple type.

struct S(pub(crate) A, pub (B, C));

In this example input the first tuple struct element of S has pub(crate) visibility while the second tuple struct element has pub visibility; the parentheses around (B, C) are part of the type rather than part of a visibility restriction.

The parser uses a forked parse stream to check the first token inside of parentheses after the pub keyword. This is a small bounded amount of work performed against the forked parse stream.

use syn::{parenthesized, token, Ident, Path, Result, Token};
use syn::ext::IdentExt;
use syn::parse::{Parse, ParseStream};

struct PubVisibility {
    pub_token: Token![pub],
    restricted: Option<Restricted>,
}

struct Restricted {
    paren_token: token::Paren,
    in_token: Option<Token![in]>,
    path: Path,
}

impl Parse for PubVisibility {
    fn parse(input: ParseStream) -> Result<Self> {
        let pub_token: Token![pub] = input.parse()?;

        if input.peek(token::Paren) {
            let ahead = input.fork();
            let mut content;
            parenthesized!(content in ahead);

            if content.peek(Token![crate])
                || content.peek(Token![self])
                || content.peek(Token![super])
            {
                return Ok(PubVisibility {
                    pub_token,
                    restricted: Some(Restricted {
                        paren_token: parenthesized!(content in input),
                        in_token: None,
                        path: Path::from(content.call(Ident::parse_any)?),
                    }),
                });
            } else if content.peek(Token![in]) {
                return Ok(PubVisibility {
                    pub_token,
                    restricted: Some(Restricted {
                        paren_token: parenthesized!(content in input),
                        in_token: Some(content.parse()?),
                        path: content.call(Path::parse_mod_style)?,
                    }),
                });
            }
        }

        Ok(PubVisibility {
            pub_token,
            restricted: None,
        })
    }
}

source

pub fn error<T: Display>(&self, message: T) -> Error

Triggers an error at the current position of the parse stream.

§Example

use syn::{Expr, Result, Token};
use syn::parse::{Parse, ParseStream};

// Some kind of loop: `while` or `for` or `loop`.
struct Loop {
    expr: Expr,
}

impl Parse for Loop {
    fn parse(input: ParseStream) -> Result<Self> {
        if input.peek(Token![while])
            || input.peek(Token![for])
            || input.peek(Token![loop])
        {
            Ok(Loop {
                expr: input.parse()?,
            })
        } else {
            Err(input.error("expected some kind of loop"))
        }
    }
}

source

pub fn step<F, R>(&self, function: F) -> Result<R>
where F: for<'c> FnOnce(StepCursor<'c, 'a>) -> Result<(R, Cursor<'c>)>,

Speculatively parses tokens from this parse stream, advancing the position of this stream only if parsing succeeds.

This is a powerful low-level API used for defining the Parse impls of the basic built-in token types. It is not something that will be used widely outside of the Syn codebase.

§Example

use proc_macro2::TokenTree;
use syn::Result;
use syn::parse::ParseStream;

// This function advances the stream past the next occurrence of `@`. If
// no `@` is present in the stream, the stream position is unchanged and
// an error is returned.
fn skip_past_next_at(input: ParseStream) -> Result<()> {
    input.step(|cursor| {
        let mut rest = *cursor;
        while let Some((tt, next)) = rest.token_tree() {
            match &tt {
                TokenTree::Punct(punct) if punct.as_char() == '@' => {
                    return Ok(((), next));
                }
                _ => rest = next,
            }
        }
        Err(cursor.error("no `@` was found after this point"))
    })
}

source

pub fn span(&self) -> Span

Returns the Span of the next token in the parse stream, or Span::call_site() if this parse stream has completely exhausted its input TokenStream.

source

pub fn cursor(&self) -> Cursor<'a>

Provides low-level access to the token representation underlying this parse stream.

Cursors are immutable so no operations you perform against the cursor will affect the state of this parse stream.

§Example

use proc_macro2::TokenStream;
use syn::buffer::Cursor;
use syn::parse::{ParseStream, Result};

// Run a parser that returns T, but get its output as TokenStream instead of T.
// This works without T needing to implement ToTokens.
fn recognize_token_stream<T>(
    recognizer: fn(ParseStream) -> Result<T>,
) -> impl Fn(ParseStream) -> Result<TokenStream> {
    move |input| {
        let begin = input.cursor();
        recognizer(input)?;
        let end = input.cursor();
        Ok(tokens_between(begin, end))
    }
}

// Collect tokens between two cursors as a TokenStream.
fn tokens_between(begin: Cursor, end: Cursor) -> TokenStream {
    assert!(begin <= end);

    let mut cursor = begin;
    let mut tokens = TokenStream::new();
    while cursor < end {
        let (token, next) = cursor.token_tree().unwrap();
        tokens.extend(std::iter::once(token));
        cursor = next;
    }
    tokens
}

fn main() {
    use quote::quote;
    use syn::parse::{Parse, Parser};
    use syn::Token;

    // Parse syn::Type as a TokenStream, surrounded by angle brackets.
    fn example(input: ParseStream) -> Result<TokenStream> {
        let _langle: Token![<] = input.parse()?;
        let ty = recognize_token_stream(syn::Type::parse)(input)?;
        let _rangle: Token![>] = input.parse()?;
        Ok(ty)
    }

    let tokens = quote! { <fn() -> u8> };
    println!("{}", example.parse2(tokens).unwrap());
}