Lexer, tokenizer, expander, and quote handling prototypes. More...

#include "structs.h"

Include dependency graph for token.h:

This graph shows which files directly or indirectly include this file:

Detailed Description

Lexer, tokenizer, expander, and quote handling prototypes.

Definition in file token.h.

Functions
t_token *	create_token (t_token_type type, char *value)
	Allocate and initialize a new token. More...

void	free_token (t_token *token)
	Free a single token and its value str. More...

void	free_token_list (t_token *head)
	Free the entire token linked list. More...

void	add_token (t_token *head, t_token new_token)
	Add a token to the end of a linked list. More...

int	is_operator (char c)
	Check if a character is a shell operator. More...

int	is_whitespace (char c)
	Check if a character is whitespace. More...

int	is_word_end (char c)
	Check if a character ends a word token. More...

t_token_type	get_operator_token_type (char str, int len)
	Determine operator token type and its length in characters. More...

int	process_word_token (char s, t_token *head)
	Process a word token composed of quoted and unquoted chunks. More...

int	process_operator_token (char input, t_token *head)
	Process an operator token. More...

t_token *	tokenize (char *input)
	Tokenize the input string into a linked list of tokens. More...

int	is_dollar_quote (t_token *tok)
	Check if a token is a `$` connected to a quoted token. More...

int	is_var_char (char c)
	Check if a character is valid in a variable name. More...

char *	extract_var_name (char str, int len)
	Extract a variable name from after the `$` More...

char *	append_char (char *str, char c)
	Append a single character to an allocated string. More...

char *	append_str (char s1, char s2)
	Concatenates 2 strings and frees the first. More...

char *	expand_variables (char str, t_env env_list, t_quote_type quote_type, int exit_status)
	Expand all $VARIABLE in a string. More...

int	expand_all_tokens (t_token tokens, t_shell shell)
	Expand all token values before parsing. More...

char *	expand_full (char str, t_env env, t_quote_type qt, int exit_status)
	Handles variable expansion and tilde expansion. More...

Function Documentation

◆ create_token()

t_token* create_token	(	t_token_type	type,
		char *	value
	)

Allocate and initialize a new token.

The new token starts with quote_type = QUOTE_NONE, connected = 0 and next = NULL

Parameters

type	Token type
value	Text content

Returns: Newly allocated token or NULL on failure

Definition at line 26 of file token.c.

References t_token::connected, t_token::next, QUOTE_NONE, t_token::quote_type, t_token::type, and t_token::value.

Referenced by process_operator_token(), and process_word_token().

Here is the caller graph for this function:

◆ free_token()

void free_token ( t_token * token )

Free a single token and its value str.

Parameters

token Token to free

Definition at line 51 of file token.c.

References t_token::value.

Referenced by free_token_list().

Here is the caller graph for this function:

◆ free_token_list()

void free_token_list ( t_token * head )

Free the entire token linked list.

Parameters

head	Head of list to free

Definition at line 64 of file token.c.

References free_token(), and t_token::next.

Referenced by process_input(), process_ui_input(), and tokenize().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ add_token()

void add_token	(	t_token **	head,
		t_token *	new_token
	)

Add a token to the end of a linked list.

Parameters

head	Pointer to head pointer
new_token	Token to add

Definition at line 84 of file token.c.

References t_token::next.

Referenced by process_operator_token(), and process_word_token().

Here is the caller graph for this function:

◆ is_operator()

int is_operator ( char c )

Check if a character is a shell operator.

Parameters

c	Character to test

Returns: 1 if operator and 0 otherwise

Definition at line 21 of file tokenizer_utils.c.

Referenced by extract_unquoted(), is_word_end(), process_word_token(), and tokenize().

Here is the caller graph for this function:

◆ is_whitespace()

int is_whitespace ( char c )

Check if a character is whitespace.

Parameters

c	Character to test

Returns: 1 if whitespace and 0 otherwise

Definition at line 32 of file tokenizer_utils.c.

Referenced by extract_unquoted(), is_word_end(), process_word_token(), and tokenize().

Here is the caller graph for this function:

◆ is_word_end()

int is_word_end ( char c )

Check if a character ends a word token.

Parameters

c	Character to test

Returns: 1 if word ending and 0 otherwise

Definition at line 43 of file tokenizer_utils.c.

References is_operator(), and is_whitespace().

Here is the call graph for this function:

◆ get_operator_token_type()

t_token_type get_operator_token_type	(	char *	str,
		int *	len
	)

Determine operator token type and its length in characters.

Parameters

str	Input at the operator position
len	Operator length (1 or 2)

Returns: Type of token for the operator

Definition at line 63 of file tokenizer_utils.c.

References TOKEN_APPEND, TOKEN_HEREDOC, TOKEN_PIPE, TOKEN_REDIR_IN, TOKEN_REDIR_OUT, and TOKEN_WORD.

Referenced by process_operator_token().

Here is the caller graph for this function:

◆ process_word_token()

int process_word_token	(	char *	s,
		t_token **	head
	)

Process a word token composed of quoted and unquoted chunks.

Parses adjacent chunks (for example hello"world"'!' -> 3 connected tokens)

Each chunk becomes a separate token with its own quote_type and token->connected = 1 if another chunk follows right after

Parameters

s	Input string at word position
head	Pointer to token lsit head

Returns: Characters consumed or -1 on error (unclosed quotes or alloc failure)

Definition at line 80 of file tokenizer.c.

References add_token(), t_token::connected, create_token(), extract_quoted(), extract_unquoted(), is_operator(), is_whitespace(), t_token::quote_type, and TOKEN_WORD.

Referenced by tokenize().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ process_operator_token()

int process_operator_token	(	char *	input,
		t_token **	head
	)

Process an operator token.

Determines the operator type and length, creates a token and appends it

Parameters

input	Input at the operator character
head	Pointer to token list head

Returns: Number of chars consumed (1 or 2) or -1 on allocation failure

Definition at line 118 of file tokenizer.c.

References add_token(), create_token(), and get_operator_token_type().

Referenced by tokenize().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize()

t_token* tokenize ( char * input )

Tokenize the input string into a linked list of tokens.

The tokenizer entry point: skips whitespaces and dispatches each segment to process_operator_token() or process_word_token()

Parameters

input Raw input string

Returns: Head of token list or NULL on error

Note: On error it frees all tokens before returning NULL

Warning: The caller has to free via free_token_list()

Definition at line 144 of file tokenizer.c.

References free_token_list(), is_operator(), is_whitespace(), process_operator_token(), and process_word_token().

Referenced by process_input(), and process_ui_input().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ is_dollar_quote()

int is_dollar_quote ( t_token * token )

Check if a token is a $ connected to a quoted token.

Detects the pattern: $"TOKEN_WORD" or $'TOKEN_WORD' + connected + next has quotes

Parameters

token Token to inspect

Returns: 1 if the pattern is valid and 0 if not

Definition at line 25 of file expander_utils.c.

References t_token::connected, t_token::next, QUOTE_NONE, and t_token::value.

Referenced by expand_all_tokens().

Here is the caller graph for this function:

◆ is_var_char()

int is_var_char ( char c )

Check if a character is valid in a variable name.

Parameters

c	Character to check

Returns: 1 if alphanumeric or _ and 0 otherwise

Definition at line 40 of file expander_utils.c.

Referenced by extract_var_name(), and handle_dollar_sign().

Here is the caller graph for this function:

◆ extract_var_name()

char* extract_var_name	(	char *	str,
		int *	len
	)

Extract a variable name from after the $

If the first char is a digit, it returns the only digit (matching BASH). Otherwise it scans for valid variable characters (alphanumeric and _)

Parameters

str	String starting after `$`
len	Number of characters consumed

Returns: Newly allocated variable name

Definition at line 55 of file expander_utils.c.

References is_var_char().

Referenced by process_dollar().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ append_char()

char* append_char	(	char *	str,
		char	c
	)

Append a single character to an allocated string.

Parameters

str	Original string
c	Character to append

Returns: New string with c appended or NULL on failure

Warning: original str is freed

Definition at line 79 of file expander_utils.c.

Referenced by expand_variables(), handle_dollar_sign(), and process_dollar().

Here is the caller graph for this function:

◆ append_str()

char* append_str	(	char *	s1,
		char *	s2
	)

Concatenates 2 strings and frees the first.

If s2 is NULL it returns s1 unchanged

Parameters

s1	First string
s2	Second string

Returns: Concatenated string or NULL on failure

Definition at line 113 of file expander_utils.c.

Referenced by get_append_value(), handle_special_var(), and process_dollar().

Here is the caller graph for this function:

◆ expand_variables()

char* expand_variables	(	char *	str,
		t_env *	env_list,
		t_quote_type	quote_type,
		int	exit_status
	)

Expand all $VARIABLE in a string.

Parameters

str	Input string
env_list	Environment list for the variables lookups
quote_type	Quote context (QUOTE_SINGLE = no expansion)
exit_status	Last exit status for $? expansion

Returns: Newly allocated expanded string or NULL

Definition at line 111 of file expander.c.

References append_char(), handle_dollar_sign(), and QUOTE_SINGLE.

Referenced by expand_full(), and write_heredoc_line().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ expand_all_tokens()

int expand_all_tokens	(	t_token *	tokens,
		t_shell *	shell
	)

Expand all token values before parsing.

Walks the token list and expands each TOKEN_WORD
Single quoted tokens and heredoc delimiters (token after <<) are not exp
A bare $ connected to a quoted token is replaced with an empty string

Parameters

tokens	Token list head
shell	Shell state

Returns: 0 on success and -1 on allocation failure

Warning: On -1 the token list could be partially expanded

Definition at line 150 of file expander.c.

References t_shell::env, t_shell::exit_status, expand_full(), is_dollar_quote(), t_token::next, QUOTE_SINGLE, t_token::quote_type, TOKEN_HEREDOC, TOKEN_WORD, t_token::type, and t_token::value.

Referenced by process_input(), and process_ui_input().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ expand_full()

char* expand_full	(	char *	str,
		t_env *	env,
		t_quote_type	qt,
		int	exit_status
	)

Handles variable expansion and tilde expansion.

Tilde expansions run first to conform with BASH and correctly handling:

export TEST="~"
echo $TEST

Parameters

str	Input string
env	Environment list
qt	Quote type
exit_status	Exit status for $?

Returns: Fully expanded string or NULL on failure

Definition at line 60 of file tilde_expand.c.

References expand_tilde(), and expand_variables().

Referenced by expand_all_tokens().

Here is the call graph for this function:

Here is the caller graph for this function:

Go to the source code of this file.

Detailed Description

Functions

Function Documentation

◆ create_token()

◆ free_token()

◆ free_token_list()

◆ add_token()

◆ is_operator()

◆ is_whitespace()

◆ is_word_end()

◆ get_operator_token_type()

◆ process_word_token()

◆ process_operator_token()

◆ tokenize()

◆ is_dollar_quote()

◆ is_var_char()

◆ extract_var_name()

◆ append_char()

◆ append_str()

◆ expand_variables()

◆ expand_all_tokens()

◆ expand_full()