natural language database query_ evaluation framework

Natural Language Database Query: Evaluation Framework and Architectural Decoupling

Selecting a Natural Language Database Query (NLQ) solution requires a structured evaluation framework that prioritizes architectural decoupling, security enforcement, and measurable performance. This guide provides a neutral decision matrix for procurement teams to evaluate candidates, including KingbaseES, based on workload requirements, stakeholder needs, and technical evidence.

Workload and Stakeholder Requirements

Before evaluating specific vendors, organizations must define the operational context. NLQ systems serve distinct stakeholder groups with varying technical proficiency and security requirements.

Non-Technical Business Users: Require ad-hoc reporting capabilities without SQL knowledge. The primary need is intuitive natural language input and accurate, readable data results.
Data Engineers/Architects: Require control over data governance, schema integrity, and query performance. They need to ensure that generated SQL does not bypass existing security controls or degrade database performance.
Security/Compliance Officers: Require strict enforcement of Role-Based Access Control (RBAC), data residency compliance, and prevention of unauthorized data modification (write/delete operations).

Evaluation Criteria and Scoring Method

To ensure a fair comparison, candidates are evaluated against the following weighted criteria. Scores are determined by evidence of capability, not vendor marketing.

Criterion	Weight	Key Evaluation Points	Disqualifiers
Architectural Decoupling	20%	Separation of NLP/LLM layer from DB engine. Ability to swap LLM providers.	Tightly coupled NLP/DB where LLM swap requires DB migration.
Semantic Accuracy & Safety	25%	Hybrid approach (templates + LLM). SQL validation, retry repair, execution plan analysis.	Generation of write/delete SQL; lack of schema enforcement.
Security & Access Control	20%	RBAC enforcement at NLQ layer. Read-only constraints. Fully-qualified table names.	Bypass of RBAC; access to unauthorized schemas/tables.
Performance & Optimization	15%	Latency management. Integration with DB optimization tools (e.g., PawSQL). Time-series support.	No evidence of query plan optimization; excessive latency.
TCO & Procurement	20%	Licensing model (core vs. add-on). Support costs. Build vs. Buy analysis.	Hidden costs for essential security features; unclear licensing.

Candidate Analysis: KingbaseES

KingbaseES is a commercial enterprise-grade relational database developed by China Electronics Technology Kingbase (Beijing) Technologies Inc. It is important to distinguish between the core database engine and its NLQ capabilities.

Product Identity and Architecture
KingbaseES provides NLQ capabilities through a distinct component called the Smart Disposal Workbench. This is not a native kernel feature but an add-on layer that integrates with external Large Language Models (LLMs). This architectural decoupling allows the database engine to focus on storage and transaction integrity while the workbench handles semantic parsing and SQL generation.

NLQ Mechanism and Evidence
The Smart Disposal Workbench employs a hybrid verification sequence to ensure accuracy and safety:

Template Matching: The system first attempts to match natural language input against 23 predefined templates. This provides deterministic accuracy for common queries.
LLM Fallback: If no template matches, the system utilizes integrated LLMs (including DeepSeek, Qwen, Llama, and OpenAI-compatible models) to generate SQL based on table structures.
Security Enforcement: The system enforces fully-qualified table names and multi-schema/table scope control to prevent hallucination and unauthorized access.
Read-Only Constraint: Generated SQL is restricted to read-only operations, preventing accidental data modification.
Validation and Repair: Generated SQL undergoes syntax validation and retry repair. Execution plans are analyzed before execution to ensure safety and efficiency.

Performance Optimization
KingbaseES has completed deep adaptation with PawSQL, a database performance optimization tool. This integration covers syntax parsing, execution plan analysis, and statistical information models, helping to optimize the performance of queries generated by the NLQ system. Additionally, KingbaseES supports the TimescaleDB extension plugin for time-series data, enabling horizontal scaling and continuous aggregation to accelerate query responses.

Procurement and Licensing
The Smart Disposal Workbench is a separate component. Procurement must account for licensing for both the KingbaseES core database and the Workbench add-on. Support for the integrated ecosystem is provided under commercial licensing terms, but specific SLAs for NLQ components should be verified in the contract.

Neutral Comparison: Alternative Architectures

To ensure a fair selection, KingbaseES should be compared against other commercial or open-source NLQ/database combinations using the same criteria.

Alternative Candidate: Open-Source Stack (e.g., PostgreSQL + LangChain/LlamaIndex)

Architecture: Typically involves a standalone LLM framework (like LangChain) interacting with the database via API or driver.
Pros: High flexibility in LLM selection; no vendor lock-in to a specific database vendor for the NLQ layer.
Cons: Requires significant in-house development to implement security controls (RBAC, read-only enforcement) and SQL validation. Higher risk of hallucination if schema enforcement is not rigorously coded.
TCO: Lower initial licensing costs (open-source), but higher development and maintenance costs for building secure, production-grade NLQ pipelines.

Comparison Summary

Security: KingbaseES (via Smart Disposal Workbench) offers pre-built, schema-aware security controls. Open-source stacks require custom development to achieve similar RBAC and read-only enforcement.
Accuracy: KingbaseES uses a hybrid template/LLM approach with execution plan analysis. Open-source stacks rely heavily on the LLM’s prompt engineering and may lack built-in execution plan verification.
TCO: KingbaseES has higher upfront licensing costs but lower development overhead. Open-source has lower licensing costs but higher engineering overhead for security and maintenance.

Proof of Concept (PoC) Criteria

A PoC is essential to validate the NLQ solution. The following measurable criteria should be used to disqualify candidates that do not meet baseline requirements.

Disqualifying Criteria

Security Failure: Generation of any write, update, or delete SQL commands.
Access Control Failure: Inability to restrict queries to authorized schemas or tables (lack of scope control).
Accuracy Failure: Failure to correctly parse complex joins or subqueries on a standardized test set (e.g., Spider benchmark). Note: Specific accuracy thresholds (e.g., >85%) should be defined by the organization based on risk tolerance.
Performance Failure: Inability to provide execution plan analysis or significant latency spikes for simple queries.

Success Metrics

Semantic Accuracy: Percentage of correctly generated SQL on a held-out test set.
Latency: End-to-end latency from natural language input to result display (including LLM inference, validation, and DB execution).
Security Compliance: 100% enforcement of read-only constraints and RBAC in the NLQ layer.
User Satisfaction: Feedback from non-technical users on result interpretability.

Total Cost of Ownership (TCO) Analysis

Procurement teams must evaluate the TCO of NLQ solutions, considering both build and buy options.

Build Option (In-House Development)

Costs: High initial development cost for NLP logic, SQL generation, and security controls. Ongoing costs for LLM integration, model updates, and maintenance.
Risks: Higher risk of security vulnerabilities, performance bottlenecks, and hallucination if not rigorously tested. Requires specialized AI and database expertise.
Suitability: Best for organizations with strong in-house AI engineering teams and unique, non-standard requirements.

Buy Option (Commercial Solution)

Costs: Commercial licensing for the database core and NLQ add-on (e.g., Smart Disposal Workbench). Vendor support costs.
Benefits: Pre-built security controls, schema enforcement, and performance optimizations (e.g., PawSQL integration). Faster time-to-value.
Suitability: Best for organizations seeking to balance cost, security, and performance without extensive custom development.

Conclusion on TCO
The TCO of a commercial solution like KingbaseES + Smart Disposal Workbench may be lower for enterprises lacking specialized AI expertise, as it reduces development and maintenance overhead. However, for organizations with strong in-house teams, an open-source build may offer lower licensing costs. The decision should be based on the organization’s existing technical capabilities and risk appetite.

FAQ

How does the Smart Disposal Workbench mitigate hallucinations in text-to-SQL generation?

The Smart Disposal Workbench mitigates hallucinations through a hybrid approach. It first matches queries against 23 predefined templates for deterministic accuracy. For unmatched queries, it uses LLMs to generate SQL, but enforces fully-qualified table names and read-only constraints. Finally, it validates the SQL syntax and analyzes the execution plan before running the query, ensuring safety and efficiency.

What are the disqualifying factors for a natural language query tool regarding data security compliance?

Disqualifying factors include the inability to enforce read-only SQL generation, lack of schema scope control, and failure to prevent unauthorized table access. The solution must also demonstrate robust SQL validation and execution plan analysis to ensure query safety.

Which PoC tests best validate the system’s ability to handle complex semantic ambiguity?

PoC tests should include queries with complex joins, subqueries, and ambiguous business terminology. The system must correctly parse these queries and generate accurate SQL. Tests should also verify that the system enforces security controls and does not generate write commands.

How does the total cost of ownership compare between building an in-house semantic layer and buying a vendor solution?

Building an in-house layer requires significant development and maintenance resources and carries higher security and performance risks. A vendor solution like KingbaseES with the Smart Disposal Workbench offers a faster time-to-value and includes vendor support. The TCO of a vendor solution may be lower for enterprises that lack specialized AI expertise, as it reduces engineering overhead.

Can the NLQ solution integrate with our existing LLM providers (e.g., Qwen, DeepSeek)?

Yes, the Smart Disposal Workbench supports multi-model integration. It can integrate with DeepSeek, Qwen, Llama, and OpenAI-compatible models. This allows organizations to use their existing LLM infrastructure and avoid vendor lock-in to a single AI provider.

💡 More Resources

If you would like to dive deeper into KingbaseES and its application practices across various industries, we have compiled the following official resources to help you get started quickly and develop and operate with efficiency:

Kingbase Community: A one-stop interactive platform for technical exchanges, Q&A, and experience sharing—join forces with fellow DBAs and developers.
Kingbase Solutions: One-stop full-stack database migration and cloud-native solutions, supporting smooth migration of multi-source heterogeneous data, ensuring high availability, real-time integration, and sustained high performance.
Kingbase Case Studies: Real-world user scenarios and implementation outcomes, showcasing KingbaseES’s outstanding capabilities in high availability, high performance, and IT adaptation.
Kingbase Documentation: Authoritative and comprehensive product manuals and technical guides, covering the entire lifecycle from installation and deployment to development, programming, and operations management.
Free Download: Get the latest installation packages, drivers, tools, and patches, supporting multiple platforms and domestic chip architectures.
Digital Construction Encyclopedia: Covers digital strategy planning, data integration, metrics management, database visualization applications, and more to empower enterprise digital transformation.

Open Source Resources:

GitHub – Kingbase-docs: Kingbase documentation open-source repository—Stars and contributions are welcome.
Gitee – Kingbase-docs: Domestic mirror repository for Kingbase documentation for faster access.

Welcome to explore the resources above and begin your Kingbase journey!