Works with
Packaged from the source skill for Claude, Codex and 4 more.
About this skill
Spinning Up in Deep RL is a structured study-note package derived from OpenAI’s educational documentation, primarily developed by Joshua Achiam. Alireza Rezvani supplies the packaged learning structure. Chapter summaries, a glossary and comparison notes help an agent retrieve the relevant material before answering a focused reinforcement-learning question.
Use this skill when
- You are learning the distinctions between policy optimisation and value-learning approaches.
- An educational experiment needs a clearer plan for measurements, random seeds and comparisons.
- You want a chapter-based explanation of an algorithm or a debugging question within the original documentation’s scope.
What it can do
The notes cover foundational concepts and the VPG, TRPO, PPO, DDPG, TD3 and SAC families discussed by the source. Topic and chapter indexes support targeted retrieval, while experiment and logging notes help organise a learning exercise. The package preserves the OpenAI copyright notice and MIT terms alongside the maintainer’s scaffold licence.
This is a notes package, not a bundled reinforcement-learning runtime or a record of experiments performed by the catalogue. Running the original educational implementations requires separate software and environment setup. The source coverage stops around the January 2020 PyTorch update; it should not be presented as a survey of later algorithms or current performance benchmarks.
Typical workflows
- Resolve a focused concept. Name the question and existing knowledge. Read the mapped chapter note and explain the idea with its assumptions, referring back to the original source where detail matters.
- Compare learning approaches. Supply the environment and constraints. Compare the relevant algorithm families conceptually without inventing measured sample efficiency or a winning benchmark.
- Prepare an experiment review. Bring a proposed configuration, measurements and seed plan. Check whether the comparison can answer its stated question and list likely silent implementation errors to investigate.
What is inside the skill
Section headings from the original SKILL.md (about 1,495 words of instructions). Read the full text in the source repository before installing.
- How to Use This Skill
- Core Frameworks & Mental Models
- The RL problem
- The two branching questions
- Policy optimization vs Q-learning — the central trade-off
- The policy gradient template
- The policy-gradient loss is not a loss function
- Broken RL code almost always fails silently
- Learn by doing
- Rigor: four standards
- The safe-step family (on-policy)
- The overestimation family (off-policy)
- Chapter Index
- Topic Index
- Supporting Files
- Scope & Limits
Bundled files
chapters/ch01-introduction.mdchapters/ch02-installation.mdchapters/ch03-algorithm-lineup.mdchapters/ch04-running-experiments.mdchapters/ch05-experiment-outputs.mdchapters/ch06-plotting-results.mdchapters/ch07-key-concepts-in-rl.mdchapters/ch08-kinds-of-rl-algorithms.mdchapters/ch09-intro-to-policy-optimization.mdchapters/ch10-spinning-up-as-a-researcher.mdchapters/ch11-key-papers-in-deep-rl.mdchapters/ch12-exercises.mdchapters/ch13-benchmarks.mdchapters/ch14-vpg.mdchapters/ch15-trpo.mdchapters/ch16-ppo.mdchapters/ch17-ddpg.mdchapters/ch18-td3.mdchapters/ch19-sac.mdchapters/ch20-logger-and-utilities.mdcheatsheet.mdglossary.mdpatterns.mdLICENSE
Example prompts
- Explain the relationship between a policy gradient, a value baseline and an advantage estimate using the supplied Spinning Up notes. State the assumptions and identify the original source sections I should read next.
- Compare PPO, TD3 and SAC for this educational continuous-control exercise. Separate conceptual trade-offs from performance claims, and list the environment and implementation details needed before choosing an experiment.
- Review this proposed reinforcement-learning experiment plan for fair baselines, seed variation and logging. Identify missing measurements and silent shape or data-distribution errors worth checking without claiming the implementation has been tested.
How to install Spinning Up Deep Rl
Claude Packaged
- Download the Claude package from the skill page. The ZIP contains one folder named after the skill, with SKILL.md and any bundled reference files.
- For Claude Code, unpack the folder into ~/.claude/skills/ to use it in every project, or into .claude/skills/ inside a repository to share it with your team.
- For Claude apps, upload the ZIP in the Skills section of the capabilities settings.
- Start a new session and describe the task in your own words. Claude picks the skill when the request matches its description.
Codex Packaged
- Download the Codex package. The ZIP contains one folder named after the skill, with SKILL.md and bundled resources.
- Unpack the complete folder into .agents/skills/ inside your repository, or into ~/.agents/skills/ for personal use. Preserve its relative reference-file paths.
- Start a new Codex session and invoke the skill by name or with $skill-name, or describe a matching task.
- AGENTS.md is separate project guidance. The ZIP does not generate or modify that file.
GPT Packaged
- Download the GPT package from the skill page.
- Create or edit a custom GPT (or a ChatGPT project) and paste the contents of SKILL.md into the instructions field.
- Upload the bundled reference files as knowledge so the GPT can consult them.
- Test with one of the example prompts from the skill page and adjust the instructions if your use case differs.
Gemini Packaged
- Download the Gemini package from the skill page.
- For a project, unpack the complete folder into .gemini/skills/ or .agents/skills/. For personal use, use ~/.gemini/skills/ or ~/.agents/skills/.
- Run /skills reload in Gemini CLI, then /skills list to check discovery. There is no need to paste the skill into GEMINI.md.
- Test a matching task and verify the referenced tools and files are available.
Cursor Packaged
- Download the Cursor package from the skill page.
- Unpack the complete skill folder into .cursor/skills/ or .agents/skills/ in your project. For personal local use, ~/.cursor/skills/ and ~/.agents/skills/ are also supported.
- Start a new Cursor session. Type / in Agent chat and select the skill name, or describe a matching task.
- Commit the skill folder and resources for team use. Personal local skills are not automatically available to remote agents or automation runs.
OpenCode Packaged
- Download the OpenCode package from the skill page.
- Unpack the complete folder into .opencode/skills/ in your project, or ~/.config/opencode/skills/ for personal use. OpenCode also supports .agents/skills/ and .claude/skills/ compatibility locations.
- Keep supporting files at their original relative paths. AGENTS.md is separate project-wide guidance and is not generated by this ZIP.
- Start OpenCode in the project and test a matching task with the model and permissions you use.
Permissions & security
Inspect the source, license and declared permissions before use. Read the scripts before running them. Review method
Spinning Up Deep Rl FAQ
What does the Spinning Up Deep Rl skill do?
Navigate structured study notes derived from OpenAI’s Spinning Up documentation, with algorithm comparisons, experiment-design questions and links back to the original educational material.
Which AI agents can use Spinning Up Deep Rl?
Packages for Claude, Codex and 4 more are built from the same source instructions.
How do I install Spinning Up Deep Rl in Claude?
Download the Claude package from the skill page. The ZIP contains one folder named after the skill, with SKILL.md and any bundled reference files. For Claude Code, unpack the folder into ~/.claude/skills/ to use it in every project, or into .claude/skills/ inside a repository to share it with your team.
Who made Spinning Up Deep Rl and what is the license?
Spinning Up Deep Rl is published by Alireza Rezvani under the MIT license. The source is https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl. The catalog keeps the original attribution.
What permissions does Spinning Up Deep Rl need?
Declared permissions: read project files, write generated skill files. Risk level: low. Check the scripts in the source before running them.