If you’re running Apache Spark inside Docker using Bitnami’s images and suddenly encounter an Ivy error that says:

Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ?/.ivy2/local

You’re not alone. This blog post walks you through the root cause and offers two solid solutions, including how to pin your Docker image to a known working version from mid-2024.

---

Understanding the Error: The error is thrown by Apache Ivy, which Spark uses to resolve external dependencies. The message essentially means that it’s trying to access a directory like ~/.ivy2/local, but the environment doesn’t have a proper HOME path set, so it resolves to an invalid location like ?/.ivy2/local.

This issue surfaced more frequently with updates to the Bitnami Spark Docker image, particularly in versions after mid-2024, where the HOME variable is no longer implicitly set for the root user.

---

Solution 1: Set the HOME Environment Variable You can explicitly set the HOME variable in your docker-compose.yml to avoid this issue. Here is the snippet you can add to your spark-master service:

environment:
  - SPARK_MODE=master
  - HOME=/opt/bitnami/spark

This ensures Ivy has a valid absolute path for its local repository.

---

Solution 2: Pin the Image Version to Mid-2024 Instead of relying on latest, which may introduce breaking changes, pin the image to a known working version like 3.3.2-debian-11-r10.

Update your docker-compose.yml like so:

image: bitnami/spark:3.3.2-debian-11-r10

Apply this to all your Spark services (master and workers). Then run:

docker-compose down --volumes --remove-orphans
docker-compose pull
docker-compose up -d

This should give you a stable environment without the Ivy error.

---

Conclusion: This error is a good reminder that relying on :latest in Docker can introduce surprises. Whether you choose to set the HOME variable or pin your image version, both solutions are valid. For long-term stability, we recommend pinning to a specific version and only upgrading with testing.