All In One Custom PostgreSQL For Database Developer Docker Image

$ docker load < bigdata.tgz
$ docker load < etl.tgz
$ docker load < docker_samza.tar.gz
$ docker load < postgresql.tgz
$ docker image lsREPOSITORY                    TAG       IMAGE ID       CREATED        SIZE
jentekllc/postgresql latest 1557485c708b 22 hours ago 1.43GB
jentekllc/etl latest c3b44db4c5d6 8 days ago 6.61GB
jentekllc/bigdata latest c7d3fb6d8221 8 days ago 5.53GB
jentekllc/samza latest e9086c0ddaab 10 days ago 12.3GB
$ tar -xzf additional_files_20211125.tar.gz
$ nohup docker-compose -p j up --scale spark-worker=3 &
$ docker psea6ccaad5924   nginx:latest                  "/docker-entrypoint.…"   9 minutes ago   Up 9 minutes   80/tcp, 0.0.0.0:5000->5000/tcp                                                                                                                            nginx-lb
0ae10942def2 jentekllc/bigdata:latest "/run_sshd_worker.sh" 9 minutes ago Up 9 minutes 22/tcp, 0.0.0.0:50000->38080/tcp j_spark-worker_3
9075364e60f3 jentekllc/bigdata:latest "/run_sshd_worker.sh" 9 minutes ago Up 9 minutes 22/tcp, 0.0.0.0:50001->38080/tcp j_spark-worker_2
c92b4e131c1d jentekllc/bigdata:latest "/run_sshd_worker.sh" 9 minutes ago Up 9 minutes 22/tcp, 0.0.0.0:50002->38080/tcp j_spark-worker_1
0ed5d7ed0d22 jentekllc/samza:latest "/home/hadoop/start_…" 9 minutes ago Up 9 minutes 0.0.0.0:50022->22/tcp, 0.0.0.0:58088->8088/tcp, 0.0.0.0:58888->8888/tcp, 0.0.0.0:58889->8889/tcp samza-server
0178dc3df8b1 jentekllc/bigdata:latest "/run_sshd_hive.sh" 9 minutes ago Up 9 minutes 0.0.0.0:30022->22/tcp, 0.0.0.0:38088->8088/tcp, 0.0.0.0:39000->9000/tcp, 0.0.0.0:39083->9083/tcp hadoop-hive
e11a0f1c4dd9 dpage/pgadmin4:latest "/entrypoint.sh" 9 minutes ago Up 9 minutes 443/tcp, 0.0.0.0:20080->80/tcp pgadmin4-container
4effecf897e6 jentekllc/etl:latest "/start_etl.sh" 9 minutes ago Up 9 minutes 0.0.0.0:40022->22/tcp, 0.0.0.0:48080->8080/tcp, 0.0.0.0:48888->8888/tcp, 0.0.0.0:48889->8889/tcp, 0.0.0.0:49000->9000/tcp, 0.0.0.0:40000->30000/tcp etl-server
7b25d8dcd063 jentekllc/postgresql:latest "/start_postgres_ssh…" 9 minutes ago Up 9 minutes 5050/tcp, 0.0.0.0:10022->22/tcp, 0.0.0.0:15432->5432/tcp, 0.0.0.0:18888->8888/tcp, 0.0.0.0:18889->8889/tcp postgresql-container
af0f519a43ce jentekllc/bigdata:latest "/run_sshd_master.sh" 9 minutes ago Up 9 minutes 0.0.0.0:4040->4040/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8088->8088/tcp, 0.0.0.0:8888-8889->8888-8889/tcp, 0.0.0.0:20022->22/tcp spark-master
$ docker-compose -p j down
http://localhost:8080
http://localhost:5000
http://localhost:8888
http://localhost:48888
http://localhost:48080
http://localhost:49000
http://localhost:58088
http://localhost:58888
#From host
$ ssh -p 20022 hadoop@localhost
#Inside container
$ cd $SPARK_HOME
$ bin/spark-submit /spark/examples/src/main/python/pi.py
$ spark-sql2021-11-06 23:26:42,842 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java class es where applicable
Setting default log level to "WARN".
Spark master: local[*], Application Id: local-1636241205417
spark-sql>
$ ssh -p 40022 hadoop@localhost
$ ssh -p 50022 hadoop@localhost
$ jps144 ResourceManager
1233 ClusterBasedJobCoordinatorRunner
1729 LocalContainerRunner
1848 LocalContainerRunner
1689 LocalContainerRunner
586 Kafka
1946 Jps
426 NodeManager
76 QuorumPeerMain
1324 ClusterBasedJobCoordinatorRunner
1421 ClusterBasedJobCoordinatorRunner
$ file * | grep directoryhello-samza:        directory
samza: directory
http://localhost:20080
http://localhost:18888
#On the host, password for user postgres is 123456$ ssh -p 10022 postgres@localhost#Inside the container$ psqlpsql (14.1 (Debian 14.1-1.pgdg110+1))
Type "help" for help.
postgres=# \dn List of schemas
Name | Owner
--------+----------
public | postgres
(1 row)

--

--

--

I am founder of Jen Tek LLC, a startup company in East Bay California developing AI powered, cloud based documentation/publishing software as a service.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Why from therapy to coding

How to Set Custom 404 Errors with WP2Static and Netlify

What Does Mortal Kombat And Software Development Have In Common?

Hello, Planet?

The Microservice Problem

From MySQL to MySQL(RDS) | Using AWS DMS to migrate your databases to AWS via Terraform

Host Unity game on the web

Find Out How Your Kubernetes Workloads Compare in Our New Benchmark Report

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
George Jen

George Jen

I am founder of Jen Tek LLC, a startup company in East Bay California developing AI powered, cloud based documentation/publishing software as a service.

More from Medium

Creating Dynamically DAG’s Apache Airflow with Various and Dependencies Task

Apache Airflow in Nutshell

Testing of Apache Airflow’s DAGs with docker compose and pytest

Data pipeline automation using Airflow DAGs in Docker containers