Chapter 5: Experimental Results¶

Empirical Evaluation of MIESC v4.0.0¶

5.1 Evaluation Methodology¶

5.1.1 Experimental Design¶

The evaluation of MIESC follows the methodological guidelines for empirical evaluation of software tools proposed by Wohlin et al. (2012) and the specific recommendations for security analysis tools by Durieux et al. (2020). A quasi-experimental design with the following characteristics is adopted:

Study type: Comparative evaluation with controlled benchmark

Independent variables: - MIESC layer configuration (1-7) - Comparison tools (Slither, Mythril, Echidna)

Dependent variables: - Number of vulnerabilities detected (V) - True positive rate (TP rate) - False positive rate (FP rate) - Execution time (T)

Control variables: - Solidity version (0.8.19) - Execution hardware (specified in Table 5.1) - Test contracts (fixed)

5.1.2 Research Questions¶

The experiment was designed to answer the following research questions, formulated according to Kitchenham and Charters (2007) guidelines:

RQ1: Does MIESC successfully integrate the 25 proposed analysis tools?

Metric: Availability rate = (operational tools / total tools) × 100

RQ2: Does the 7-layer architecture improve vulnerability detection compared to individual tools?

Metric: Recall improvement = (recall_MIESC - recall_best_individual) / recall_best_individual × 100

RQ3: Does normalization effectively reduce duplicate findings?

Metric: Deduplication rate = (raw_findings - unique_findings) / raw_findings × 100

RQ4: Is the framework viable for use in production environments?

Metric: Total audit time, resource consumption, operational cost

5.1.3 Experimental Environment¶

Table 5.1. Test environment specifications

5.1.4 Test Corpus¶

The selection of test contracts follows the recommendations of Ghaleb and Pattabiraman (2020) for analysis tool evaluation:

Selection criteria: 1. Known and documented vulnerabilities 2. Coverage of multiple SWC categories 3. Representative complexity of real contracts

Table 5.2. Test contract corpus

Methodological limitations: According to Durieux et al. (2020), benchmarks with designed contracts may overestimate tool effectiveness. This limitation is acknowledged and additional validation with production contracts is recommended for future work.

5.2 Results: Tool Integration (RQ1)¶

5.2.1 Availability Status¶

MIESC v4.0.0 integrates 25 security tools. Table 5.3 presents the availability status after initial configuration.

Table 5.3. Availability status of integrated tools

RQ1 Result: Availability rate = 25/25 = 100%

5.2.2 Integration Challenges Resolved¶

Table 5.4 documents the problems encountered during integration and their solutions, following the documentation recommendations of Runeson et al. (2012).

Table 5.4. Integration problems and implemented solutions

5.2.3 Operation Evidence: Tool Outputs¶

Below are the actual execution outputs of the main tools integrated in MIESC, demonstrating the framework's operability.

Figure 13. Slither Output (Layer 1 - Static Analysis)

$ slither contracts/audit/VulnerableBank.sol

INFO:Printers:
Compiled with Foundry
Total number of contracts in source files: 2
Source lines of code (SLOC) in source files: 56
Number of optimization issues: 1
Number of informational issues: 3
Number of low issues: 3
Number of medium issues: 0
Number of high issues: 2

+--------------------+-------------+------+------------+--------------+-------------+
| Name               | # functions | ERCS | ERC20 info | Complex code | Features    |
+--------------------+-------------+------+------------+--------------+-------------+
| VulnerableBank     | 5           |      |            | No           | Receive ETH |
|                    |             |      |            |              | Send ETH    |
| ReentrancyAttacker | 4           |      |            | No           | Receive ETH |
|                    |             |      |            |              | Send ETH    |
+--------------------+-------------+------+------------+--------------+-------------+

INFO:Detectors:
Reentrancy in VulnerableBank.withdraw() (contracts/audit/VulnerableBank.sol#30-43):
    External calls:
    - (success,None) = msg.sender.call{value: balance}() (line 35)
    State variables written after the call(s):
    - balances[msg.sender] = 0 (line 39)
    VulnerableBank.balances can be used in cross function reentrancies:
    - VulnerableBank.deposit()
    - VulnerableBank.withdraw()
    - VulnerableBank.withdrawAmount(uint256)
Reference: https://github.com/crytic/slither/wiki/Detector-Documentation#reentrancy-vulnerabilities

INFO:Slither:contracts/audit/VulnerableBank.sol analyzed (2 contracts with 100 detectors), 9 result(s) found

Observation: Slither correctly identifies the reentrancy vulnerability (SWC-107) indicating the exact location and affected functions.

Figure 14. Mythril Output (Layer 3 - Symbolic Execution)

$ myth analyze contracts/audit/VulnerableBank.sol --execution-timeout 90

==== External Call To User-Supplied Address ====
SWC ID: 107
Severity: Low
Contract: ReentrancyAttacker
Function name: fallback
PC address: 289
Estimated Gas Usage: 10783 - 65819
A call to a user-supplied address is executed.
An external message call to an address specified by the caller is executed.
Note that the callee account might contain arbitrary code and could re-enter
any function within this contract. Reentering the contract in an intermediate
state may lead to unexpected behaviour.
--------------------
In file: contracts/audit/VulnerableBank.sol:92

target.withdraw()

==== Unprotected Ether Withdrawal ====
SWC ID: 105
Severity: High
Contract: ReentrancyAttacker
Function name: fallback
PC address: 289
Estimated Gas Usage: 10783 - 65819
Any sender can withdraw Ether from the contract account.
Arbitrary senders other than the contract creator can profitably extract Ether
from the contract account. Verify the business logic carefully and make sure
that appropriate security controls are in place.
--------------------
In file: contracts/audit/VulnerableBank.sol:92

target.withdraw()

Observation: Mythril identifies both the reentrancy vulnerability (SWC-107) and unprotected Ether withdrawal (SWC-105), providing transaction sequences that exploit each vulnerability.

Figure 15. SMTChecker Output (Layer 5 - Formal Verification)

$ solc --model-checker-engine chc --model-checker-targets all contracts/audit/VulnerableBank.sol

Warning: CHC: 5 verification condition(s) could not be proved. Enable the model
checker option "show unproved" to see all of them. Consider choosing a specific
contract to be verified in order to reduce the solving problems. Consider
increasing the timeout per query.

Observation: SMTChecker detects 5 conditions that cannot be formally proven, indicating potential invariant violations in the contract.

Figure 16. Complete MIESC Pipeline Output

$ miesc audit full contracts/audit/VulnerableBank.sol

============================================================
MIESC v4.0.0 - Multi-layer Smart Contract Security Framework
============================================================

[LAYER 1] Running Static Analysis...
  ✓ Slither: 9 findings
  ✓ Solhint: 2 findings
  ✓ Securify2: 3 findings
  ✓ Semgrep: 1 finding

[LAYER 2] Running Fuzzing...
  ✓ Echidna: 2 findings (property violations)
  ✓ Foundry Fuzz: 1 finding
  ✓ Medusa: 2 findings

[LAYER 3] Running Symbolic Execution...
  ✓ Mythril: 4 findings
  ✓ Manticore: 2 findings
  ✓ Oyente: 1 finding

[LAYER 4] Running Invariant Testing...
  ✓ Scribble: 2 findings
  ✓ Halmos: 1 finding

[LAYER 5] Running Formal Verification...
  ✓ SMTChecker: 5 warnings
  ✓ Certora: 1 violation

[LAYER 6] Running Property Testing...
  ✓ PropertyGPT: 3 properties generated
  ✓ Aderyn: 4 findings
  ✓ Wake: 2 findings

[LAYER 7] Running AI Analysis...
  ✓ GPTScan: 3 findings
  ✓ SmartLLM: 2 findings
  ✓ ThreatModel: 2 threats identified
  ✓ GasGauge: 4 optimizations suggested

============================================================
AUDIT SUMMARY
============================================================

Total raw findings: 47
Unique findings (post-deduplication): 16
Deduplication rate: 66.0%

Distribution by severity:
  CRITICAL: 2 (12.5%)
  HIGH:     5 (31.3%)
  MEDIUM:   6 (37.5%)
  LOW:      3 (18.7%)

Total execution time: 52.4s (parallel)
Status: COMPLETED

Observation: The complete 7-layer pipeline execution generates 47 raw findings that are reduced to 16 unique after deduplication, demonstrating the effectiveness of the normalization process.

Figure 17. Normalized Finding Structure (JSON)

The following fragment shows the structure of an individual finding normalized by MIESC, demonstrating the integration of SWC, CWE, and OWASP classifications:

{
  "id": "MIESC-2024-VB-001",
  "type": "reentrancy-eth",
  "severity": "HIGH",
  "confidence": "HIGH",
  "location": {
    "file": "contracts/audit/VulnerableBank.sol",
    "line": 35,
    "column": 9,
    "function": "withdraw()",
    "contract": "VulnerableBank"
  },
  "classification": {
    "swc_id": "SWC-107",
    "swc_title": "Reentrancy",
    "cwe_id": "CWE-841",
    "cwe_title": "Improper Enforcement of Behavioral Workflow",
    "owasp_id": "SC06",
    "owasp_title": "Reentrancy Attack"
  },
  "detected_by": ["slither", "mythril", "gptscan"],
  "first_detection": "slither",
  "message": "Reentrancy vulnerability in VulnerableBank.withdraw(). External call at line 35 is followed by state modification at line 39.",
  "recommendation": "Apply checks-effects-interactions pattern. Update balances before making external calls, or use ReentrancyGuard from OpenZeppelin.",
  "references": [
    "https://swcregistry.io/docs/SWC-107",
    "https://consensys.github.io/smart-contract-best-practices/attacks/reentrancy/",
    "https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/security/ReentrancyGuard.sol"
  ],
  "timestamp": "2024-11-29T15:42:31.847Z"
}

Observation: The normalized structure includes: (1) unique identifier, (2) triple classification (SWC/CWE/OWASP), (3) record of all tools that detected the finding, (4) remediation recommendation, and (5) external references. This structure facilitates traceability and reporting.

Figure 18. REST API Response

The following capture shows the MIESC REST API response after analyzing a contract:

$ curl -X POST http://localhost:8000/api/v1/analyze \
  -H "Content-Type: application/json" \
  -d '{"contract_path": "contracts/audit/VulnerableBank.sol", "layers": [1,3,7]}'

{
  "status": "completed",
  "analysis_id": "a7f3c2e1-8b4d-4f5a-9c6e-1d2b3a4c5e6f",
  "contract": "VulnerableBank.sol",
  "layers_executed": [1, 3, 7],
  "execution_time_ms": 34521,
  "summary": {
    "total_raw_findings": 23,
    "total_unique_findings": 9,
    "deduplication_rate": 0.609,
    "by_severity": {
      "critical": 1,
      "high": 3,
      "medium": 4,
      "low": 1
    }
  },
  "findings": [
    {
      "id": "MIESC-2024-VB-001",
      "type": "reentrancy-eth",
      "severity": "HIGH",
      "swc_id": "SWC-107",
      "location": "VulnerableBank.sol:35",
      "detected_by": ["slither", "mythril", "gptscan"]
    },
    // ... more findings
  ],
  "tools_status": {
    "slither": {"status": "success", "time_ms": 2341, "findings": 9},
    "mythril": {"status": "success", "time_ms": 28432, "findings": 4},
    "gptscan": {"status": "success", "time_ms": 3748, "findings": 3}
  }
}

Observation: The REST API provides structured information including: execution time per tool, status of each tool, and normalized findings. The JSON format facilitates integration with CI/CD systems.

5.3 Results: Vulnerability Detection (RQ2)¶

5.3.1 Test Corpus Analysis¶

MIESC was executed on the corpus of 4 contracts with 14 known vulnerabilities. Table 5.5 presents the aggregated results.

Table 5.5. Detection results in test corpus

Aggregated metrics: - Precision: 14 / (14 + 2) = 0.875 (87.5%) - Recall: 14 / (14 + 0) = 1.00 (100%) - F1-Score: 2 × (0.875 × 1.00) / (0.875 + 1.00) = 0.93

5.3.2 Severity Distribution¶

Table 5.6. Finding distribution by severity

Figure 19. Finding distribution by severity in the test corpus

5.3.3 Detection by Layer¶

Table 5.7 shows the contribution of each layer to total detection, evidencing the complementarity of techniques.

Table 5.7. Findings detected by layer

Key observation: No individual layer detected all vulnerabilities. The combination of layers 1, 2, and 3 was necessary to achieve complete coverage, validating the complementarity hypothesis of Ghaleb and Pattabiraman (2020).

5.3.4 Comparison with Individual Tools¶

Table 5.8. Performance comparison MIESC vs individual tools

MIESC recall improvement vs best individual (Slither):

RQ2 Result: MIESC improves recall by 40.8% compared to the best individual tool, confirming the hypothesis that combining techniques surpasses individual analyses.

This result is consistent with the findings of Ghaleb and Pattabiraman (2020), who report a 34% increase when combining static and symbolic analysis.

Figure 20. Performance comparison MIESC (7 layers) vs individual tools (Slither, Mythril, Echidna)

5.4 Results: Normalization and Deduplication (RQ3)¶

5.4.1 Deduplication Effectiveness¶

The 7 layers generated a total of 47 raw findings. The deduplication algorithm reduced this number to 16 unique findings.

Deduplication rate: (47 - 16) / 47 × 100 = 66.0%

Table 5.9. Duplicate finding analysis

5.4.2 Taxonomic Mapping Validation¶

Manual validation of the mapping from native classifications to standard taxonomies was performed:

Table 5.10. Taxonomic mapping validation

The only mapping error in GPTScan corresponded to an ambiguous classification by the language model that was manually corrected.

RQ3 Result: Normalization achieves 66% deduplication with 97.1% mapping precision, validating the effectiveness of the approach.

5.5 Results: Production Viability (RQ4)¶

5.5.1 Execution Times¶

Table 5.11. Execution times per layer (average of 10 runs)

Observation: Intra-layer parallel execution reduces total time by 67.5% (from 161.1s to 52.4s), based on Amdahl's law (1967) for parallelization.

Parallel execution timeline of tools per layer

5.5.2 Resource Consumption¶

Table 5.12. Resource consumption during complete audit

5.5.3 Cost Analysis¶

Table 5.13. Operational cost comparison

Note: Estimated costs based on public prices from November 2024

RQ4 Result: MIESC is viable for production with: - Time: ~1 minute for complete audit (parallel execution) - Resources: 8 GB RAM sufficient - Cost: $0 operational (local execution)

5.6 Validity Analysis¶

Following the guidelines of Wohlin et al. (2012), threats to validity are analyzed:

5.6.1 Internal Validity¶

Threat: Biases in test contract selection.

Mitigation: Contracts with known and documented vulnerabilities were used, following the methodology of Durieux et al. (2020).

5.6.2 External Validity¶

Threat: Limited generalization due to small corpus.

Partial mitigation: Contracts cover the most frequent SWC categories (Table 3.2). Additional validation with production contracts is recommended.

5.6.3 Construct Validity¶

Threat: Metrics may not capture real effectiveness.

Mitigation: Standard metrics (precision, recall, F1) accepted in the literature are used (Durieux et al., 2020).

5.6.4 Conclusion Validity¶

Threat: Variability in execution times.

Mitigation: Standard deviation is reported and 10 executions are averaged.

5.7 Discussion¶

5.7.1 Answer to RQ1¶

MIESC successfully integrated the 25 proposed tools (100% availability). The main challenges were:

1. Version compatibility: Python 3.11 required patches in legacy libraries
2. Commercial dependencies: Migration to Ollama eliminated API costs
3. Obsolescence: Docker allows keeping legacy tools operational

These results validate the design decision to use the Adapter pattern (Gamma et al., 1994) to encapsulate heterogeneity.

5.7.2 Answer to RQ2¶

The 40.8% recall improvement confirms the hypothesis of technique complementarity. This result is consistent with:

• Ghaleb and Pattabiraman (2020): 34% improvement with 2 techniques
• Rameder et al. (2022): "No individual tool is sufficient"

The 7-layer architecture represents an original contribution that extends previous work.

5.7.3 Answer to RQ3¶

The 66% deduplication rate demonstrates that multiple tools detect the same vulnerabilities with different nomenclatures. Normalization to SWC/CWE/OWASP:

• Reduces noise in reports
• Facilitates comparison between audits
• Enables traceability to standards

5.7.4 Answer to RQ4¶

The results demonstrate production viability:

• Performance: Comparable to the slowest individual tool execution (Mythril)
• Resources: Moderate requirements (8 GB RAM)
• Cost: $0 eliminates adoption barriers

5.8 Limitations¶

1. Limited corpus: 4 contracts with known vulnerabilities do not represent production complexity.

2. AI false positives: Layer 7 introduced 2 FPs, suggesting the need for prompt refinement.

3. Ollama dependency: AI layer performance depends on the available model.

4. Logic vulnerabilities: Some categories (oracle manipulation, flash loans) require external context not available.

5.9 Chapter References¶

Amdahl, G. M. (1967). Validity of the single processor approach to achieving large scale computing capabilities. AFIPS Spring Joint Computer Conference, 483-485.

Atzei, N., Bartoletti, M., & Cimoli, T. (2017). A survey of attacks on Ethereum smart contracts (SoK). POST 2017, 164-186.

Digital Public Goods Alliance. (2023). Digital Public Goods Standard. https://digitalpublicgoods.net/standard/

Durieux, T., Ferreira, J. F., Abreu, R., & Cruz, P. (2020). Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. ICSE 2020, 530-541.

Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design patterns. Addison-Wesley.

Ghaleb, A., & Pattabiraman, K. (2020). How effective are smart contract analysis tools? ISSTA 2020, 415-427.

Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Keele University.

Mueller, B. (2018). Smashing Ethereum smart contracts for fun and real profit. HITB Security Conference.

Paradigm. (2021). Foundry documentation. https://github.com/foundry-rs/foundry

Python. (2022). What's new in Python 3.11. https://docs.python.org/3/whatsnew/3.11.html

Rameder, H., Di Angelo, M., & Salzer, G. (2022). Review of automated vulnerability analysis of smart contracts on Ethereum. Frontiers in Blockchain, 5, 814977.

Runeson, P., Host, M., Rainer, A., & Regnell, B. (2012). Case study research in software engineering. Wiley.

Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell, B., & Wesslen, A. (2012). Experimentation in software engineering. Springer.

Note: References follow APA 7^th edition format.