Explain token, lexemes and pattern with example

Explain token, lexemes and  pattern with example.

Token :

• Token is a sequence of characters that can be treated as a unit in the grammar of the programming languages called token.
• A token is a pair consisting of a token name and an optional attribute value token like: identifiers,keywords,operators,special symbols,constants.

Pattern : 

• A set of string in the input for which the same token is produced as output. This set of strings is described by a rule called a pattern associated with the token.

Lexeme : 

• The sequence of character in a source program matched with a pattern for a token is called lexeme.

Token	Lexeme	Pattern
Const	Const	Const
If	If	If
Relation	<,<=,= ,< >,>=,>	< or <= or = or < > or >= or >
Id	Pi, count, n, I	letter followed by letters and digits.
Number	3.14159, 0, 6.02e23	Any numeric constant
Literal	"Darshan Institute"	Any character between “ and “ except “

Table. Examples of Tokens

Example:

total = sum + 12.5

Tokens are: total (id), = (relation), Sum (id), + (operator), 12.5 (num)

Lexemes are: total, =, sum, +, 12.5