Add kinesis integration testing by qhool · Pull Request #75 · AdRoll/erlmld

qhool · 2023-07-07T14:54:40Z

docker compose.yml creates a testing environment: * Use kinesalite and dynalite (github.com/mhart) to provide kinesis and dynamodb work-alikes * Create ca-certificate and server cert signed with this ca, so that kinesalite and dynalite can pose as the real deals * Insert the ca-cert into trusted certificats for the test container * Create dns aliases for standard aws endpoints
New consume_SUITE.erl and load_from_dir script enable testing against real AWS or using the docker compose env
Add Makefile to hold the required commands to start tests under docker

* docker compose.yml creates a testing environment: * Use kinesalite and dynalite (github.com/mhart) to provide kinesis and dynamodb work-alikes * Create ca-certificate and server cert signed with this ca, so that kinesalite and dynalite can pose as the real deals * Insert the ca-cert into trusted certificats for the test container * Create dns aliases for standard aws endpoints * New consume_SUITE.erl and load_from_dir script enable testing against real AWS or using the docker compose env * Add Makefile to hold the required commands to start tests under docker

raz-adroll

I'm on a running it in my mac and I get:

error: failed switching to "work_user": unable to find user work_user: no matching entries in passwd file

I highly recommend documenting the setup in the README.

raz-adroll · 2023-07-07T15:41:47Z

test/compose/ca/cacert.sh

+openssl genrsa -des3 -passout pass:foobar -out ca/ca.key $KEY_SIZE || exit 1
+
+openssl req -new -x509 -passin pass:foobar \
+        -subj "/C=US/ST=CA/O=ACME/CN=acme.fake/emailAddress=wcoyote@acme.fake" \


I got that reference

test/compose/dynamo/Dockerfile

test/compose/test_runner/Dockerfile

test/bin/dump_to_dir

qhool · 2023-07-07T16:09:05Z

I'm on a running it in my mac and I get:

error: failed switching to "work_user": unable to find user work_user: no matching entries in passwd file

What command did you use exactly?

I highly recommend documenting the setup in the README.

I'll add something once I understand what's going on.. for me the setup was apt install docker-ce then make test; obviously that lacks .. something?

Docker desktop uses a utility vm to host the containers. In addition to making things slower, this has other advantages: for instance, by running as root and mapping file ownership to root, it can break workflows that assume the docker user is unpriviledged.

raz-adroll · 2023-07-07T21:10:29Z

Almost working, but I still got this error while I was running the last commit.

test-tests-1    | ===> Running Common Test suites...
test-tests-1    | %%% consume_SUITE:
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.262740 ===
test-tests-1    | erlmld_sup_simple_consume_test listening on 40593
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.265542 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1136)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:02.608564 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1136,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2132.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.609100 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1157)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:02.966878 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1157,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2135.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:02.967439 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1178)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:03.307353 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1178,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2138.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:03.307840 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1199)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:03.652304 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1199,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2141.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:03.652876 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1220)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.028469 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1220,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2144.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.029812 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1241)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.417206 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1241,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2147.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.417784 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1262)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:04.764537 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1262,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2150.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:04.765446 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1283)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.152821 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1283,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2153.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.153767 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1304)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.495754 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1304,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2156.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.496381 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1325)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:05.838265 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1325,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2159.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =INFO REPORT==== 7-Jul-2023::20:59:05.838710 ===
test-tests-1    | erlmld_runner_simple_consume_test launched "/home/work/copy/_build/test/lib/erlmld/priv/run_kinesis.sh" (pid 1346)
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:06.211647 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: child_terminated
test-tests-1    |     reason: {child_exited,1346,{exit_status,256}}
test-tests-1    |     offender: [{pid,<0.2162.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    | =SUPERVISOR REPORT==== 7-Jul-2023::20:59:06.211692 ===
test-tests-1    |     supervisor: {local,erlmld_sup_simple_consume_test}
test-tests-1    |     errorContext: shutdown
test-tests-1    |     reason: reached_max_restart_intensity
test-tests-1    |     offender: [{pid,<0.2162.0>},
test-tests-1    |                {id,mld_runner},
test-tests-1    |                {mfargs,{erlmld_runner,start_link,
test-tests-1    |                                       [erlmld_runner_simple_consume_test,
test-tests-1    |                                        "/tmp/erlmld/erlmld.simple_consume_test.properties",
test-tests-1    |                                        kinesis]}},
test-tests-1    |                {restart_type,permanent},
test-tests-1    |                {significant,false},
test-tests-1    |                {shutdown,brutal_kill},
test-tests-1    |                {child_type,worker}]
test-tests-1    |
test-tests-1    |
test-tests-1    | %%% consume_SUITE ==> simple_consume_test: FAILED
test-tests-1    | %%% consume_SUITE ==> {'EXIT',shutdown}
test-tests-1    |
test-tests-1    | EXPERIMENTAL: Writing retry specification at /home/work/copy/_build/test/logs/retry.spec
test-tests-1    |               call rebar3 ct with '--retry' to re-run failing cases.
test-tests-1    | Failed 1 tests. Passed 0 tests.
test-tests-1    | Results written to "/home/work/copy/_build/test/logs/index.html".
test-tests-1    | ===> Failures occurred running tests: 1

pbrudnick · 2023-07-10T19:45:57Z

test/compose.yml

+    depends_on:
+      ca:
+        condition: service_healthy
+    domainname: kinesis.us-east-1.amazaonaws.com


typo here

Suggested change

domainname: kinesis.us-east-1.amazaonaws.com

domainname: kinesis.us-east-1.amazonaws.com

tak30 · 2023-07-11T14:06:06Z

Please do not merge this PR before it has been tested in mac arm and mac x86. As soon as we get it working on x86 I'll test it in arm. Thaaanks!

raz-adroll reviewed Jul 7, 2023

View reviewed changes

qhool added 3 commits July 7, 2023 12:22

Remove dead code and appease hank

4a25c42

Fix format

016a7d2

Work around Docker desktop

5ba2891

Docker desktop uses a utility vm to host the containers. In addition to making things slower, this has other advantages: for instance, by running as root and mapping file ownership to root, it can break workflows that assume the docker user is unpriviledged.

Downgrade node for kinesalite, to avoid docker issuea

e1ac126

pbrudnick reviewed Jul 10, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kinesis integration testing#75

Add kinesis integration testing#75
qhool wants to merge 5 commits intoAdRoll:mainfrom
qhool:RTI-14728_kinesis_test_env

qhool commented Jul 7, 2023

Uh oh!

raz-adroll left a comment

Uh oh!

raz-adroll Jul 7, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qhool commented Jul 7, 2023

Uh oh!

raz-adroll commented Jul 7, 2023

Uh oh!

pbrudnick Jul 10, 2023

Uh oh!

tak30 commented Jul 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	domainname: kinesis.us-east-1.amazaonaws.com
	domainname: kinesis.us-east-1.amazonaws.com

Conversation

qhool commented Jul 7, 2023

Uh oh!

raz-adroll left a comment

Choose a reason for hiding this comment

Uh oh!

raz-adroll Jul 7, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qhool commented Jul 7, 2023

Uh oh!

raz-adroll commented Jul 7, 2023

Uh oh!

pbrudnick Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

tak30 commented Jul 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants