NASA Responsible GenAI HACK Week · CryoCloud

Landscape of
Coding Agents

An orientation to the technology underneath AI coding tools

Anthony Arendt · eScience Institute Anshul Tambay · Don Setiawan · Scientific Software Engineering Center Jason Gilman · Element84
August 24, 2026

Goal for this week: a shared mental model of fundamental concepts.

Large Language Models

A prediction machine,
not a knowledge base.

A large language model is a statistical model that predicts the next chunk of text given the text that came before it. At each step the model picks a likely next token (roughly three-quarters of a word) given everything that came before.

Diagram showing tokenized input feeding into an LLM network, which outputs a probability distribution over possible next tokens, with the most likely token selected

Credit: Jason Gilman, Element 84

How models learn to be useful

Useful behaviors come from a second training phase — post-training.

Supervised fine-tuning

Human contractors write thousands of examples of ideal conversations. The model learns the shape of a helpful interaction. This is why the model follows your instructions.

Reinforcement learning from human feedback

Contractors judge which of two model responses is better. The model is tuned to produce responses that score well. This is why the model stops and decides it is done, instead of looping forever.

Tool-use fine-tuning

The model is trained on examples of conversations that include tool calls. It learns both the format and the harder meta-skill of knowing when a tool call is needed versus when it can answer from memory.

Convergent post-training

All major AI labs do post-training in roughly similar ways.

Anthropic, OpenAI, Google, and Meta all use supervised fine-tuning, reinforcement learning from human feedback, and tool-use training with different data but similar pipelines. The resulting behaviors converge: all modern models can follow instructions, use tools, and stop when done.

The practical implication

You can swap Claude for GPT for Gemini in most coding agents and things still mostly work. Your investment in learning how to work effectively with these tools transfers across products.

Chat vs Agent: What is a Harness?

The harness is the software layer that gives an LLM access to tools, memory, and your environment.

Chat

Answers questions.

Agent

Reads your files, runs your code, edits your project.

The harness is what has changed most in the last two years. Saying "I'm using Claude" is under-specified: browser Claude, Claude Code, and Claude in VS Code share the same model but have completely different harnesses.

What the model knows vs. what you tell it.

Built-in knowledge

  • Python and common libraries — the model has seen millions of lines of code and understands standard patterns, idioms, and APIs.
  • General best practices — it knows how to structure a function, write a test, or refactor a loop without being told.
  • When to stop — it has a sense of when a task is complete or when it needs more information before acting.

You don't control this — it is fixed at training time and has a knowledge cutoff date.

Context you provide

  • Your data's quirks — the model has never seen your CSV headers, your variable naming conventions, or your domain vocabulary.
  • Your lab's conventions — style guides, preferred libraries, and project structure exist nowhere in its training data.
  • Your specific goal — what "done" looks like for this task is something only you can define.

You control this!

Most of what people call "context engineering" is really just learning how to fill the right side of this diagram.

The context window

The model sees exactly what the harness puts into its context, and nothing else.

Understanding this single concept explains most of the confusing behavior researchers encounter. The context window is the total amount of text the model can consider at once, shared across every source of input.

  • Modern context windows range from roughly 100,000 tokens to 1 million tokens. Every turn of a conversation, the entire history is re-sent to the model.
  • Cost scales with conversation length. A twenty-turn conversation costs more per turn than turn one did.
  • Models have no memory between conversations. What looks like memory is the harness reloading context. Close the window, and the model has no idea who you are.
  • When the context window fills up, models get confused, forget instructions, and fail in unexpected ways.
Diagram showing a context window as a finite container with stacked colored blocks representing system instructions, user messages, agent thoughts, tool results, and agent responses, with an arrow pointing to an LLM

Credit: Jason Gilman, Element 84

A shared mental model of working with agents

We regard an agent as an eager assistant
that needs our supervision.

Every output the agent produces requires human judgment. The agent can be confidently wrong, and it will not tell you when it is. Checking what it gives you is not merely a quality control step; it is the core practice.

Failure modes

When agents go wrong, they tend to go wrong in predictable ways.

Fabrication (hallucination)

The model produces confident, plausible-sounding output that is factually wrong. This is especially dangerous in scientific contexts because the output often looks like it could be correct.

Sycophancy

The model agrees with your framing rather than pushing back, even when your framing is wrong. This is a systematic side-effect of reinforcement learning training that rewards responses users rate as helpful.

Scope creep

The model does more than you asked — adds type hints, refactors adjacent code, restructures your project. Trained to be maximally helpful, it over-delivers by default.

Context exhaustion

As the conversation grows long and the context window fills up, the model loses track of earlier instructions and makes increasingly confused decisions.

Diagnosing agent problems

Does the agent seem unaware of something specific to your work? YES NO CONTEXT PROBLEM What context is it missing? Provide Additional Context TRAINING PROBLEM Can I restructure the task? YES NO Restructure Switch models

Shaping Agent Behavior

Always loaded Guaranteed present, but costs context budget
Summoned on demand Efficient, but only loaded when triggered

AGENTS.md

Project-level context and rules — facts, conventions, and constraints the agent has in hand across every session.

Agent Skills

User or model-invoked procedures for tasks the agent is not familiar with.

...

Interactive pause — 3 minutes

Where Does Your Work Sit?

QR code linking to the Where Does Your Work Sit? audience survey

Coding agents

AI Coding Agents

Tool Cost model Notable trait
GitHub Copilot
 Free tier (usage limits apply); $10/mo Works inside your existing editor with no additional setup
Claude Code
 Anthropic API tokens Terminal-native; best for long autonomous runs and multi-step workflows
OpenAI Codex
 OpenAI API tokens OpenAI-native CLI; useful if your institution already has OpenAI credits
OpenCode
 Bring your own key Open source; connect any provider or a local model — no subscription required
Cline
 Bring your own key Open source IDE extension with the same bring-your-own-key flexibility as OpenCode

Development environments

IDEs for Coding Agents

Tool Cost model Notable trait
VS Code
 Free Universal editor; connects to GitHub Copilot, Cline, and Claude Code out of the box
Cursor Desktop
 Free tier; paid from $20/mo AI-native fork of VS Code with built-in agent capabilities and multi-file editing
JupyterLab + Jupyter AI
 Bring your own key AI chat, code completion, and model-assisted analysis inside your notebooks

Choosing a tool

There is no single best tool. The right choice depends on the constraints of the specific task and the data involved.

01

Cost

Flat subscription, metered by token (pay for every API call), or self-hosted (pay for the compute yourself).

02

Capability

Single-file edits vs. multi-file refactors vs. autonomous multi-hour tasks. More capable tools tend to cost more, but not always.

03

Integration

How deeply the tool connects to your actual work environment — IDE-native, terminal-based, or web-based.

04

Model hosting

Where the model runs: vendor cloud (fastest, most capable), institution's cloud (institutional control), or your own hardware (maximum control).

05

Privacy

What data can leave your machine? Some research data, code, or context can never go to a vendor. This constraint often dominates the other four.

Data privacy — a preview

When you use a cloud-hosted AI tool, your data leaves your machine.

Where it goes and what happens next depends on your access tier. This week's sandboxing and security tutorial on Thursday covers this in depth.

Consumer tier

Free or personal-paid accounts

Prompts are often logged and may be used for model training.

Enterprise & API

Paid API access, team accounts

Contractual guarantees that prompts are not logged or used for training.

Institutional gateway

Proxy your institution runs

Adds authentication, budget controls, and audit logging on top of a vendor API.

Local models

Runs on your machine

Nothing leaves your machine. Capability is lower, but data sensitivity constraints are satisfied by default.

The practical default: match model hosting to data sensitivity. When in doubt, do not send it. A schema description or synthetic sample is often enough.

Reproducibility

Know what to keep:
the science lives in the artifact stack.

THE CONVERSATION ephemeral · stochastic ≠ THE ARTIFACT STACK Research Plan versioned spec doc — design intent before code is written Context Files AGENTS.md, skills — conventions that shaped agent behavior Environment Spec pixi.toml, lock file — exact pinned dependencies Code & Notebooks git-versioned, runnable — the analysis itself Data Provenance dataset DOI, version identifier, access date, checksum

Git Workflows

Git workflows: essential when working with coding agents.

YOU Commit: establish a clean baseline before the agent starts, so you have a safe rollback point AGENT Agent runs large changes, fast, confident — and potentially wrong YOU Review every change read the diff — do not accept output you have not checked YOU Commit: your message is the attestation refactor snow depth aggregation to use xarray DataArray ↻ repeat for each agent task

The week ahead

Each tutorial this week adds a layer of capability, and each layer requires the one below it.

Monday AM — this tutorial

Landscape of Coding Agents

The shared mental model: LLMs, agent anatomy, context, failure modes.

Monday PM

Context Engineering

Writing AGENTS.md for a real scientific codebase. Building skills that encode your procedures.

Tuesday

MCP Servers & Workflow Patterns

Giving agents structured access to real systems. Separating deliberation from execution so design choices become reviewable artifacts before code is written.

Wednesday

GenAI-ready Data

Connecting agents to NASA Earthdata sources. Making data discoverable and queryable by language models.

Thursday

Sandboxing, Security, Validation & Metrics

Constraining what capable agents can do. Knowing that the output is correct.

Three principles for the week

Robust

Agents will confidently produce wrong output. Context engineering and the diagnostic habits you learned today are how you catch and prevent that.

Reproducible

The process is stochastic but the artifact is versioned. Separate the scaffolding from the science.

Secure

Your data has constraints that do not bend for convenience. Match model hosting to data sensitivity. Thursday makes this practical.

1 / 0