Sequential decision making under uncertainty involves selecting a sequence of actions in the presence of noise to maximize an agent's expected utility. In multiagent settings, agents are uncertain not only about their actions' outcomes, their observations, and states, but also about actions of other agents sharing the environment. Therefore, an agent's behavior must be strategic and consider these uncertainties. One recognized framework relevant for decision making in multiagent settings is the interactive partially observable Markov decision process (I-POMDP). This research focuses on strategic behavior of humans and normative agents in multiagent settings.First, I study the behavior of humans in two classes of games and propose several new models of behavioral data collected when humans engaged in these games. The first class is a modified Centipede game for testing human recursive thinking. Recent experiments show that humans predominantly reason at lower levels; however, they display a higher level of reasoning if games are made simpler and more competitive. I model the data using the finitely-nested I-POMDP, appropriately simplified and augmented with models simulating human learning and choice. Results suggest that this process-oriented behavioral modeling provides a good fit of the data. My modeling further showed that humans attribute the same errors that they themselves make to others. The second class pertains to sequential bargaining where humans are widely observed as deviating from game-theoretic predictions. I construct a suite of new and existing computational process models that integrate different choice models with utility functions. Fairness and limited backward induction, both of which may possibly explain the behavioral deviations, are incorporated. My comparative analyses reveal that limited backward induction plays a crucial role in longer-round games while in shorter-round games, fairness remains the key consideration.Second, I present new methods for computing the strategic behavior of normative agents in the context of I-POMDPs. A new technique provides the first formalization of planning in finitely-nested I-POMDPs as a probabilistic inference problem. My comprehensive experimental results demonstrate that we may obtain solutions represented as compact finite state controllers whose quality is significantly better than previous policy iteration techniques though convergence may take more time.