Skip to content

Add first-phase Android guest analyzer (APK install / launch / process lifetime) - #3210

Open
nyrivera wants to merge 8 commits into
kevoreilly:masterfrom
nyrivera:android-analyzer-phase1
Open

nyrivera wants to merge 8 commits into
kevoreilly:masterfrom
nyrivera:android-analyzer-phase1

Conversation

@nyrivera

@nyrivera nyrivera commented Sep 5, 2026

Copy link
Copy Markdown

What this is

A first-phase Android guest analyzer for CAPE.

It:

  • installs a submitted APK on an Android-x86 guest;
  • launches the application;
  • watches process lifetime for the configured analysis timeout;
  • completes through the existing guest agent + GuestManager /execpy pipeline;
  • routes a bare .apk (or --package apk) to platform=android so find_machine_to_service_task() does not pick a free Windows VM.

It does not include Frida or any other dynamic instrumentation. Reports from this phase will not contain behavioral events, signatures, or malscore. That absence is intentional. Instrumentation is a follow-up.

Host routing

  • File.get_type() containing Android package (APK) wins over sflock_identify(), which returns jar for real APKs (only sflock.unpack().package says apk).
  • "apk" is in sandbox_packages.
  • File.get_platform() has an android branch.
  • demux.py treats apk like jar (whitelist, not blacklist).
  • demux_sample_and_add_to_db() sets platform=android when package == "apk" and no platform was given — including explicit --package apk, and again after the extracted_files unpack.
python utils/submit.py /path/to/sample.apk
python utils/submit.py --package apk /path/to/sample.apk

both store task.package=apk, task.platform=android.

Guest paths

Analyzer and sample land under /data/local/tmp (determine_system_drive() / determine_temp_path()). Windows and Linux path behavior is unchanged. Clock set uses toybox date @UNIXTIME, not GNU date -s.

Known limitations (not treated as defects)

  • tasks.status stays reported for both guest success and guest failure. That is CAPE-wide (GuestManager maps agent complete/failed to DB complete). Look at the guest log (Analysis failed vs completed successfully) and report.json.
  • Analyzer loopback POST to http://127.0.0.1:8000/status fails on guests where agent.py cannot bind 0.0.0.0 and binds only the guest IP. Host completion uses /execpy child exit via GET /status. Do not retarget the callback to the guest’s routed IP; agent pinning would drop it. The bind must be fixed later. When the POST does succeed, a crashed run now sends failed.
  • upload_scripts() still forces \ in the guest path (pre-existing). Unused unless pre/during scripts are configured.
  • A file whose magic is only Zip archive data (no Android package (APK)) still goes through sflock_identify() → jar.
  • Process.terminate() exists on the Android Process class (Linux’s class is missing it even though the Linux analyzer shutdown loop calls it). Intentional.

Validation

  • Task #40: F-Droid APK, full timeout, report.json with correct hashes, no behavioral telemetry. Early run had empty platform (fixed later).
  • Task #41: corrupt APK → host logs Analysis failed, timeout=false, pm install error in debug.log. sys.exit(1) path.
  • Task #43/#44: bare submit, both VMs free → package=apk, platform=android, machine=android1.
  • Task #45: submit.py --package apk --timeout 30 with no --platform, cuckoo1 unlocked → package=apk, platform=android, machine=android1.

🤖 Generated with Claude Code

nyrivera and others added 6 commits September 4, 2026 22:29
First-phase Android guest analyzer, ported from analyzer/linux's
lib/ scaffolding (procfs semantics are identical -- Android runs a
real Linux kernel). No dynamic instrumentation (Frida) yet -- this
covers install + launch + process-tree monitoring + clean shutdown
only.

New:
- lib/core/packages.py: Android-specific Package base (the Linux
  one is tightly coupled to strace, which doesn't apply here)
- modules/packages/apk.py: installs the submitted APK, resolves its
  package name via pm list packages diffing (no aapt dependency for
  the common case), launches via monkey, resolves the resulting PID
  via pidof with a /proc fallback
- lib/api/process.py: adds a working Process.terminate() -- upstream
  analyzer/linux's Process is missing this entirely, so its own
  terminate_processes cleanup path silently no-ops today
- lib/common/constants.py: falls back to TMPDIR=/data/local/tmp
  since stock Android has no writable /tmp

Validated end-to-end on a live Android-x86 9.0 guest: installed a
real third-party APK, launched it (confirmed on screen), tracked its
PID through a full monitored run, and shut down cleanly. Along the
way, found and fixed a real bug: /system/bin/monkey ships with no
shebang line on this build, so subprocess exec's it directly with
ENOEXEC where an interactive shell would silently retry through sh.

Known follow-ups, not fixed here:
- analyzer.py's status report-back hardcodes 127.0.0.1:8000; agent.py
  can't bind 0.0.0.0 on this Android/bionic build (socket.gaierror),
  so it binds the concrete guest IP instead, and loopback then
  refuses the connection
- lib/cuckoo/core/guest.py hardcodes /tmp as the non-Windows temp
  path for placing the submitted sample; needs an android case (or a
  guest-side /tmp symlink) to line up with constants.py's override
Two real bugs found and fixed by actually submitting tasks through a
live CAPE instance against a real android1 machine (not just manual
testing):

- analyzer.py: Android's /system/bin/date is toybox, not GNU
  coreutils -- it has no -s flag at all (date: Unknown option s),
  so every task crashed immediately in prepare() when a clock value
  was set. toybox's SET syntax takes a bare @unixtime positional
  argument instead, which sidesteps needing to match its other SET
  format (MMDDhhmm[[CC]YY][.ss]).

- guest.py: determine_system_drive() returned / for android, same
  as the generic non-Windows fallback, but stock Android mounts /
  read-only (tmpfs). upload_analyzer()'s mkdtemp(dir=/) would have
  failed outright. Added an android case returning /data/local/tmp/,
  the writable partition analyzer/android's own constants.py already
  standardizes on. determine_temp_path() gets the matching case for
  where the submitted sample gets placed.

Validated against a live android1 KVM machine (kvm.conf, not
committed -- local deployment config) added via git-log-visible
submit.py runs: task correctly selected android1, uploaded the
analyzer, installed and launched a real third-party APK, ran for the
full configured timeout, and produced a complete report.json with
platform=android, package=apk. Confirmed the live-snapshot revert
model works cleanly end-to-end (agent.py comes back up immediately
on revert, no reprovisioning needed per task).
Addresses all four confirmed findings from a full review pass, each
re-validated live through the real submission pipeline (task IDs
below), not just re-read as code:

- analyzer.py never called sys.exit(), so its process always exited 0
  regardless of whether the analysis crashed. agent.py's
  get_subprocess_status() (polled by the host via /execpy) maps
  exitcode to Status.COMPLETE/FAILED, so this made guest-level
  success/failure indistinguishable to anything that depends on it
  (the "Analysis failed" vs "Analysis completed successfully" log
  line in guest.py, and the accuracy of the report's "timeout" field).
  Confirmed via task kevoreilly#41 (deliberately corrupt "APK"): guest.py now
  correctly logs "Analysis failed", and report.json shows
  timeout=false plus the real pm-install error text in debug.log,
  where before it would've read identically to a genuine success.
  Note the top-level tasks.status column still normalizes to
  "reported" for both outcomes -- that's a pipeline-completion flag
  CAPE uses for every platform, not a defect; the real signal lives
  one level down.

- Stop-task-early was broken on both sides. Guest-side: analyzer.py's
  marker path used PATHS["root"] (a random per-run directory) instead
  of the shared TMPDIR path guest.py actually creates the marker in --
  a regression against analyzer/linux, which this was ported from.
  Host-side: web/apiv2/views.py had no android branch for dest_folder,
  which would've raised UnboundLocalError. Both fixed together since
  neither half works without the other.

- No host-side routing to Android existed at all. Traced and
  empirically confirmed three independent gaps: File.get_platform()
  had no android branch (defaulted to "windows"); "apk" wasn't in
  sandbox_packages, so _identify_aux_func() never promoted it out of
  sflock's own generic classification; and even that classification
  was wrong -- sflock_identify() returns "jar" for a real APK (only
  sflock.unpack()'s separate .package attribute correctly says "apk"),
  so a bare .apk with no flags would've been silently treated as a
  Java jar, a package analyzer/android doesn't implement. Fixed by
  checking our own libmagic-based File.get_type() first (it already
  distinguishes them correctly: "Android package (APK)"), adding the
  android platform branch, and moving "apk" from blacklist_extensions
  to whitelist_extensions in demux.py (matching "jar", another
  zip-based format already handled that way) as defense in depth for
  the fallback path.

  Package detection alone isn't sufficient for correct machine
  selection: find_machine_to_service_task() filters on task.platform
  independently of task.package, so _identify_aux_func() resolving
  "apk" doesn't by itself keep a bare submission off a Windows
  machine. demux_sample_and_add_to_db() now also sets platform =
  "android" when package resolves to "apk" and no platform was given.

  Confirmed via task kevoreilly#43 (both android1 and cuckoo1 free at submit
  time): `submit.py --timeout 30 file.apk` with no other flags
  produces task.package=apk, task.platform=android, machine=android1,
  and completes normally.

- No unit tests existed for the new package. Added
  analyzer/android/tests/modules/packages/test_apk.py covering
  _install()'s package-name resolution (including the ambiguous
  reinstall/aapt-fallback/multiple-new-packages branches), _launch()'s
  exact argv construction (asserting package_name only ever reaches
  the shell as a positional parameter, never string-interpolated into
  the script body), and the empty-PID failure path in start(). Matches
  analyzer/linux/tests' existing structure and pytest.ini convention.
demux_sample_and_add_to_db only wrote platform=android after auto-identify,
so submit.py --package apk left task.platform empty and
find_machine_to_service_task() could pick a free Windows VM.

Move the assignment out of `if not package` so it also applies before
demux_sample (which echoes the incoming platform for a pinned package).
Re-apply after the extracted_files unpack, which shadows the outer
platform name, so archive-extracted APKs identified in the loop get
the same pin.

Also send status=failed on the analyzer loopback POST when the run
errored, matching the sys.exit(1) path.
### What was changed:
1. **Multi-Process Sibling Monitoring (`analyzer/android/analyzer.py`)**:
   * Added the `get_package_pids(package_name)` helper function to dynamically scan `/proc` for running sibling processes whose command line matches the target `<package_name>` or its subcomponents (e.g., `<package_name>:remote`).
   * Updated `monitor_new_processes` to combine both procfs-based direct child process tracking (for native execution/forks) and Zygote-forked sibling tracking.
   * Passed the target APK’s resolved `package_name` down to the monitoring thread.

2. **Robust Local Routing Fallback (`analyzer/android/analyzer.py`)**:
   * Added the `get_local_ip_via_routing(host_ip)` helper function which uses a lightweight UDP socket routing check to determine the exact local IP used to reach the CAPE host.
   * Updated the loopback status reporting logic to attempt a `/status` POST to loopback (`127.0.0.1:8000`) first, falling back gracefully to the concrete local IP address if the local agent bound there due to Android/Bionic `0.0.0.0` socket errors.

3. **Backward Compatibility in `pm install` (`analyzer/android/modules/packages/apk.py`)**:
   * Modified `_install()` to catch unrecognized option errors for `-g` (e.g. `Unknown option: -g` on Android API < 23) and fall back gracefully to a standard `pm install -r` without crashing the entire analysis package.

4. **Case-Insensitive Host-Side magic File Detection (`lib/cuckoo/core/data/tasking.py`)**:
   * Updated the host's `libmagic` type classification check to be case-insensitive (`"android package" in file_type.lower()`) to ensure reliable APK identification across different distribution builds of host-side `libmagic`.
@doomedraven

Copy link
Copy Markdown
Collaborator

thank you, how do you create android analysis vm? do you install android studio inside of the linux vm?

@dsecuma

dsecuma commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

thanks for your initial perspective @nyrivera .

I have a few thoughts and questions regarding the architecture.

  • ARM architecture & avd: What about ARM apk and architecture? Have you tested this against any virtual devices (like avds or similar emulators) like @doomedraven asked? Moving towards an environment that closely resembles a real phone is probably the best approach long-term, especially since x86 distributions of android are largely obsolete now.

  • eBPF & Detection Bypass: Have you considered event extraction and testing device detection bypass using ebpf for android?

  • User Interaction: What is the interaction plan for the guest? are we going to expose the vnc interface so users can interact with the ui?

  • Rootbeer Test: Could you try to analyze scottyab/rootbeer using your current setup and share the output? It would be very interesting to see how the sandbox behaves against those checks.

Thanks again for driving this!

@nyrivera

nyrivera commented Sep 5, 2026

Copy link
Copy Markdown
Author

@doomedraven Not Android Studio, and not a Linux VM with an emulator nested inside.

This PR was validated on a KVM domain whose guest OS is Android-x86 9.0 (x86_64), registered in CAPE as platform=android, with a libvirt snapshot named clean. agent.py is in the snapshot and comes back on revert. Analyzer + sample land under /data/local/tmp (stock Android has no writable /tmp). kvm.conf and the qcow are local deploy artifacts and are not in this PR.

I can write a short guest-image note (ISO → KVM → agent.py → snapshot → machines table) if you want that in-tree. Happy to treat ARM/AVD as a later machine type; this slice is the x86_64 KVM path only.

@nyrivera

nyrivera commented Sep 5, 2026

Copy link
Copy Markdown
Author

@dsecuma Thanks — those are the right long-term questions. This PR is deliberately install/launch/process-lifetime + host routing only.

  • ARM / AVD: not tested. Guest here is Android-x86 9.0 x86_64 KVM. ARM and phone-like AVDs should be additional machine types, not a rewrite of this analyzer.
  • eBPF: not in this PR. Linux CAPE already has a tracee-style capture path we can learn from later; Android eBPF/detection-bypass is a later slice.
  • VNC / user interaction: not exposed. Current launch is monkey on the LAUNCHER activity.
  • Rootbeer: good next live sample. I can run scottyab/rootbeer on this guest and paste report.json + guest log. Expect no behavioral telemetry from this phase (no Frida/signatures yet).

Documents the ISO → KVM → agent.py → snapshot → machines table
path used to validate analyzer/android. x86_64 KVM only; ARM/AVD
is a later machine type.
Answer the PR review questions in-tree: ARM/AVD is a later machine
type, eBPF/bypass and CAPE-facing VNC are later slices, Rootbeer is
a live sample on this x86_64 KVM guest (rooted userdebug, so detectors
should fire).
@nyrivera

nyrivera commented Sep 5, 2026

Copy link
Copy Markdown
Author

@dsecuma Follow-up on your architecture questions. The in-tree write-up is now in docs/book/src/installation/guest/android.rst (section Architecture follow-ups). Short version:

ARM / AVD. Not tested. This PR’s guest is Android-x86 9.0 x86_64 KVM so it sits next to the Windows domain: libvirt snapshot revert, agent.py on :8000, platform=android in the machines table. That is a poor stand-in for a modern phone; x86 Android is obsolete as a long-term target. The right shape is additional machine types (ARM64 AVD or a phone-like image), not a second analyzer. analyzer/android can stay. Play APKs that ship only armeabi-v7a / arm64-v8a .so files will miss those libs here; Java still runs.

eBPF / detection bypass. Not in this PR. Linux CAPE already has a first-cut capture path (Tracee → results["tracee"], no signatures). Android eBPF on this 4.19 Android-x86 kernel is a different stack, and Magisk-style hide is image hardening. Dynamic events on Android are a parallel Frida-capture track, not eBPF in this slice.

VNC / user interaction. This slice launches with monkey on the LAUNCHER activity. There is no CAPE web VNC. Operators can already attach virt-manager / spice / VNC to the libvirt domain for debugging. Exposing that as a user-facing analysis feature (Guacamole-style) is a later product decision.

RootBeer. The image is userdebug with su; the agent runs as uid=0. Root detectors are expected to report rooted — that is honest for this guest, not a failed task. GitHub releases for scottyab/rootbeer do not attach a sample APK (Play listing is gone; F-Droid has no com.scottyab.rootbeer.sample). I will run the sample once we have a build from that repo and the guest is free, and paste report.json + the guest log here. Native ARM-only checks may no-op on x86_64; the Java checks (su on PATH, test-keys, dangerous packages) should still fire.

@nyrivera

nyrivera commented Sep 5, 2026

Copy link
Copy Markdown
Author

@dsecuma RootBeer baseline on the existing x86_64 guest (control sample for later work).

Sample. Built from source at scottyab/rootbeer 0.1.2 (:app:assembleDebug). Package com.scottyab.rootbeer.sample.debug. Fat APK with libtoolChecker.so for armeabi-v7a, arm64-v8a, x86, and x86_64 — native checks are not ABI-inapplicable on this guest.

sha256 7f3184097c11c0730529e195a49bd98ce6a04c0cdc6ac3050413287a6ba92baa
type: Android package (APK), with gradle app-metadata.properties, with APK Signing Block

CAPE task #48. package=apk, machine=android1 (platform=android), timeout 90s, info.timeout=true (full timeout, not a crash). Guest log:

pm install output: Success
Added new process to list with pid: 9164
Analysis timeout hit, terminating analysis
Analysis completed

report.json: malscore 0, no signatures, behavior.processes empty. That is this PR’s contract (install/launch/lifetime only). The report does not contain per-check RootBeer booleans — those live in the sample UI / logcat / a later Frida slice.

Why isRooted() is expected true (guest facts, not UI scrape). Image is android_x86_64-userdebug 9 / eng.lh.20200325.112926 test-keys, agent runs as uid=0, su is present. Mapping to RootBeer 0.1.2 isRooted():

Check Expected on this guest Why
detectTestKeys true Build.TAGS contains test-keys
checkForSuBinary / checkSuExists (which su) true su on the userdebug image
checkForDangerousProps true userdebug (ro.debuggable / ro.secure)
checkForRootNative applicable lib/x86_64/libtoolChecker.so is in the APK
checkForMagiskBinary likely false no Magisk install in clean
detectRootManagementApps / dangerous / cloak packages likely false none of those package IDs on the image
checkForBusyBoxBinary not in default isRooted() only isRootedWithBusyBoxCheck()

So the important result is not merely “rooted.” It is: this sandbox is a rooted userdebug x86_64 KVM, RootBeer’s Java + native paths can load, and CAPE still installs/launches/times out cleanly. That is the control for ARM / Frida / hardening later.

Caveat. The live analyzer tree on the host already has an in-progress Frida auxiliary (not in this PR). Task 48’s guest log shows Frida injected into pid 9164, but report.json has no results["frida"]. Do not treat #48 as the Frida A/B. A follow-up branch will replay this same SHA with structured events.

No further feature commits planned on this PR.

@nyrivera

nyrivera commented Sep 5, 2026 •

Copy link
Copy Markdown
Author

Sprint lock / #3210 freeze

This PR is frozen after the x86_64 RootBeer control sample (task #48). Further work stays off this branch.

Follow-up order is now locked as:

Baseline → Observability → Architecture → Realism → Additional instrumentation → Human interaction

Concrete sequence:

  1. RootBeer baseline on the existing x86_64 guest — done. The important result is not merely isRooted == true. The guest is a userdebug test-keys image with su; that is the control sample for every later architectural change.
  2. Frida capture v1 — isolated branch. Small scope: prove CAPE can collect structured runtime events into something like results["frida"] on a deterministic app (Activity.onResume, Runtime.exec, one native call). No eBPF until this is reliable.
  3. RootBeer + Frida — replay the same APK as the baseline. That is the A/B: sandbox behavior vs sandbox behavior + runtime observability (which Java checks run, whether it searches for su, whether it shells out, which native checks are reached).
  4. ARM64 feasibility spike — before bypass or eBPF. Constrained: ARM64 guest boots → agent reachable → snapshot/revert → APK install → analyzer launch → results return. Then the same RootBeer APK. No full ARM product yet.
  5. Frida on ARM — same experiment, then RootBeer+Frida on ARM. At that point the core architecture is proven.
  6. Detection-bypass experiment — keep this image as the intentionally detectable reference. Clone a separate hardened image and reduce obvious indicators experimentally. Goal is not “fool RootBeer completely.” The question is how phone-like the guest can become without making instrumentation brittle, and whether CAPE still functions when those indicators disappear.
  7. eBPF feasibility spike — eBPF is another observation plane, not a bypass. First question only: can this kernel load one BPF program, attach to one useful event, and return one structured event to CAPE? If kernel config / sepolicy / BTF / verifier / age says no, document it and stop. Adopt only if it provides telemetry Frida cannot provide reliably.
  8. Interactive UI / VNC — two different questions. Operator virt-manager/SPICE/VNC is infrastructure validation and can be checked whenever the VM is idle. Exposing interactive Android sessions through CAPE’s web UI is a later product feature (session lifecycle, auth, task ownership, concurrency, input capture, timeouts, and analyst interaction changing malware behavior).

@nyrivera

nyrivera commented Sep 6, 2026

Copy link
Copy Markdown
Author

@doomedraven Hey — whenever you have a spare look, this slice is frozen and ready for review.

It is just APK install / launch / process lifetime plus host routing, and the short Android-x86 guest note. RootBeer control is task #48 in the thread. No Frida or extra feature commits on this branch unless you want changes.

Thanks for the VM-build question earlier — happy to follow up on anything that is still unclear.

@nyrivera

Copy link
Copy Markdown
Author

Hey @doomedraven — checking back in on this. It's been about 8 days since I froze this slice (APK install / launch / process lifetime, RootBeer control sample included) and pinged for review. No pressure, just want to make sure it didn't fall through the cracks. Happy to split it further or clarify anything that'd make review easier.

@doomedraven

doomedraven commented Sep 15, 2026 via email

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants