How to specify and evaluate a system
A specification built around site requirements rather than product features produces bids that can be compared and an award that holds up under scrutiny.
Key facts
- A specification should describe what has to be detected at what range and under what conditions, rather than listing the features of a preferred product.
- Detection range figures are only meaningful alongside the target they were measured against, because the same system reports very different ranges for a large fixed-wing aircraft and a small quadcopter.
- Under the SAFER SKIES framework, systems acquired by certified state, local, tribal and territorial agencies must appear on an Authorized Systems List, which is maintained through a law enforcement portal rather than published openly.
- A procurement written around one vendor feature set is more likely to be challenged, and a sole-source position needs a documented basis whether or not the choice is sound.
- Evaluation should include how the system behaves in false alarm conditions, because an operator who stops trusting alerts has effectively lost the capability.
- Integration, installation and support belong in the specification, since these are frequently the items that decide whether a system works in service.
Start from the site, not the product
The most common weakness in a counter-drone specification is that it describes a product. That usually happens because a supplier demonstration came first and the document was written afterwards.
The alternative is to describe the requirement. What approaches matter at this site. What has to be detected, at what distance, and how early. What conditions the system has to work in, including darkness, weather and the local radio environment. What has to happen when something is detected, and who acts.
A specification written that way allows different technical approaches to compete, which is how a buyer finds out whether radar, RF or a combination actually suits the site.
The reference target
This single item improves a procurement more than any other.
Detection range depends on what is being detected. A system will report a much longer range against a large fixed-wing aircraft than against a small consumer quadcopter, and both figures can be accurate. If the specification does not state the reference target, competing bids will quote ranges measured against different aircraft in different conditions, and the numbers cannot be compared.
State the aircraft type and size the site is concerned with, and require each bidder to report performance against it.
Include the parts that get left out
Integration, installation, commissioning, training and support are usually where a system succeeds or fails in service, and they are usually thinner in a bid than the sensor specification.
Ask where the alert arrives, in which system, and what the operator sees. Ask what installation involves at your site, including mounting, power and any area classification requirements. Ask what support covers, what it costs annually, and how detection performance will be verified over time.
Evaluation
Score against the specification rather than against the bids relative to one another. The criteria and their weighting should be set before responses are opened, and recorded.
False alarm behaviour deserves explicit weight. A system that alerts frequently and wrongly will be switched off or ignored, and a capability that nobody trusts has been lost regardless of what the datasheet says.
Where the buyer is a certified agency acquiring mitigation capability, confirm the approved-systems position before award. The list is maintained through a law enforcement portal, so the agency holds the access, and an adviser works through it rather than around it.
Related work
FAQ
▸Should we specify a technology or a requirement?
A requirement, in almost all cases. Specifying radar, or RF, or a particular product, transfers the design decision to whoever wrote the specification and narrows the field before the bids arrive. Describing what must be detected, at what range and in what conditions, lets suppliers propose an approach and lets you compare them on the outcome.
▸How do we compare detection range claims?
By insisting on the target. A range is only meaningful with the aircraft size and the conditions it was measured against, so the specification should state the reference target and ask each bidder to report range against it. Without that, the figures in competing bids are not describing the same thing.
▸What should we ask about false alarms?
The rate, the conditions that produce them, and what the operator is expected to do with an uncertain alert. A system that raises frequent false alarms tends to be ignored within weeks, and that outcome should be treated as a failure of the capability rather than an inconvenience.
▸Do we need to check the Authorized Systems List?
Where the buyer is a certified state, local, tribal or territorial agency acquiring mitigation capability, the applicable list governs what may be acquired. It is maintained through a law enforcement portal, so the agency rather than an adviser holds the access. Confirming status before award is preferable to discovering it afterwards.