- by x32x01 ||
If you defend the use of AI to solve CTF challenges, there is a question worth asking: what exactly is the competition measuring when AI can solve the challenge for you?
I do not usually solve challenges on Hack The Box, but I decided to try one today using AI.
The result was surprisingly fast.
In less than two minutes, the AI analyzed the source code, tried several payloads, and eventually retrieved the flag. I did not develop a new exploitation technique, discover an unusual vulnerability, or spend hours analyzing the application.
I wrote a short prompt.
That raises an important question: where is the actual skill being tested?
Using AI as an assistant can certainly be useful. It can help explain unfamiliar code, suggest possible attack vectors, or point out mistakes in an exploit.
But there is a major difference between using AI to improve your own reasoning and asking AI to perform most of the reasoning for you.
If the process becomes:
A ranking is supposed to provide some indication of a competitor's ability relative to other participants. If one person solves challenges manually while another delegates a large portion of the work to an AI system, comparing their scores as if they demonstrated the same skills becomes questionable.
The issue is not simply whether AI is powerful.
The real question is whether AI-assisted solving is allowed under the rules of the competition, and if it is allowed, what the resulting ranking actually represents.
There is nothing inherently wrong with using AI when the rules explicitly permit it. But calling the resulting performance equivalent to solving the challenge independently can be misleading.
For example, a competitor might solve a challenge independently and use AI to:
The situation is very different when the AI receives the challenge and independently performs reconnaissance, analyzes the vulnerability, generates payloads, tests them, and retrieves the flag.
At that point, the competitor may be acting more like an operator supervising an automated solver.
If a CTF is specifically designed to measure human cybersecurity skills, organizers may need clear rules around AI assistance.
On the other hand, competitions could deliberately create an AI-assisted security category where participants are expected to use modern AI tools.
That could actually make the competition more interesting.
Instead of pretending that AI does not exist, organizers could explicitly test how effectively participants can combine human expertise with AI systems.
The important thing is transparency.
A score achieved with significant AI assistance should not necessarily be presented as if it represents the same skill set as a completely manual solution.
AI is becoming an important part of real-world security work, and learning how to use it effectively is a legitimate skill.
The problem appears when we use an automated system to perform the core intellectual work of a challenge and then treat the final result as a pure measurement of our own cybersecurity ability.
If an AI reads the source code, discovers the vulnerability, creates the payloads, tests multiple approaches, and retrieves the flag in under two minutes, we should at least be honest about what we accomplished.
Maybe the skill was cybersecurity.
Maybe it was prompt engineering.
Maybe it was simply knowing how to delegate the task to an AI model.
Those are different skills, and CTF rankings should make that distinction clear.
I do not usually solve challenges on Hack The Box, but I decided to try one today using AI.
The result was surprisingly fast.
In less than two minutes, the AI analyzed the source code, tried several payloads, and eventually retrieved the flag. I did not develop a new exploitation technique, discover an unusual vulnerability, or spend hours analyzing the application.
I wrote a short prompt.
That raises an important question: where is the actual skill being tested?
AI Can Turn a CTF Challenge Into a Prompting Exercise
CTF competitions are designed to test skills such as vulnerability analysis, exploitation, reverse engineering, scripting, cryptography, and problem solving.Using AI as an assistant can certainly be useful. It can help explain unfamiliar code, suggest possible attack vectors, or point out mistakes in an exploit.
But there is a major difference between using AI to improve your own reasoning and asking AI to perform most of the reasoning for you.
If the process becomes:
- Give the challenge to an AI model.
- Let it analyze the source code.
- Let it generate payloads.
- Let it test different approaches.
- Take the flag when it succeeds.
The Problem With Ranking
This becomes even more important when competition rankings are involved.A ranking is supposed to provide some indication of a competitor's ability relative to other participants. If one person solves challenges manually while another delegates a large portion of the work to an AI system, comparing their scores as if they demonstrated the same skills becomes questionable.
The issue is not simply whether AI is powerful.
The real question is whether AI-assisted solving is allowed under the rules of the competition, and if it is allowed, what the resulting ranking actually represents.
There is nothing inherently wrong with using AI when the rules explicitly permit it. But calling the resulting performance equivalent to solving the challenge independently can be misleading.
AI Assistance Is Not the Same as AI Doing the Work
There is a reasonable middle ground.For example, a competitor might solve a challenge independently and use AI to:
- Explain a confusing function.
- Review an exploit.
- Suggest alternative approaches.
- Help debug a script.
- Explain an unfamiliar protocol.
- Review code for mistakes.
The situation is very different when the AI receives the challenge and independently performs reconnaissance, analyzes the vulnerability, generates payloads, tests them, and retrieves the flag.
At that point, the competitor may be acting more like an operator supervising an automated solver.
What Should CTF Competitions Measure?
There is no single answer for every competition.If a CTF is specifically designed to measure human cybersecurity skills, organizers may need clear rules around AI assistance.
On the other hand, competitions could deliberately create an AI-assisted security category where participants are expected to use modern AI tools.
That could actually make the competition more interesting.
Instead of pretending that AI does not exist, organizers could explicitly test how effectively participants can combine human expertise with AI systems.
The important thing is transparency.
A score achieved with significant AI assistance should not necessarily be presented as if it represents the same skill set as a completely manual solution.
The Bigger Question
The point is not that AI should never be used in cybersecurity competitions.AI is becoming an important part of real-world security work, and learning how to use it effectively is a legitimate skill.
The problem appears when we use an automated system to perform the core intellectual work of a challenge and then treat the final result as a pure measurement of our own cybersecurity ability.
If an AI reads the source code, discovers the vulnerability, creates the payloads, tests multiple approaches, and retrieves the flag in under two minutes, we should at least be honest about what we accomplished.
Maybe the skill was cybersecurity.
Maybe it was prompt engineering.
Maybe it was simply knowing how to delegate the task to an AI model.
Those are different skills, and CTF rankings should make that distinction clear.