By using AWS re:Post, you agree to the Terms of Use

Analytics

AWS provides the broadest selection of analytics services that fit all your data analytics needs and enables organizations of all sizes and industries reinvent their business with data. From data movement, data storage, data lakes, big data analytics, machine learning, and anything in between, AWS offers purpose-built services that provide the best price-performance, scalability, and lowest cost.

Recent questions

see all
1/18

bootstrap failure due to requirement of arm64 version of numpy required for r6 instances?

Was trying to upgrade to the latest r6 instances from r5s and ran into an issue with installing numpy in our bootstrap script via pip. Found[ this post](https://repost.aws/questions/QUdF4dL0k9RTeAZaUFiPDJCw/emr-bootstrap-script-with-pip-numpy-installation-fails-on-r-6-instances) that is related to my issue. Was anyone able to resolve this without building your own wheel file of the arm64 version of numpy? EC2/EMR Cluster Config ``` Release label: emr-6.5.0 Instance Type: r6gd.8xlarge ``` Snippet of the bootstrap ``` #!/bin/bash # python version pyv="$(python3 -V 2>&1)" echo "Python version: $pyv" # misc code to link up the requirements.txt echo "`date -u` install python dependencies" #Install Python deps sudo python3 -m pip install wheel sudo python3 -m pip install -r requirements.txt ``` requirements.txt ``` boto3==1.18.46 Cython==0.29.24 pandas==1.3.3 numpy==1.21.2 ``` Log Output ``` + echo 'Python version: Python 3.7.10' ... + echo 'Thu Sep 29 22:16:46 UTC 2022 install python dependencies' + sudo python3 -m pip install wheel WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead. WARNING: The script wheel is installed in '/usr/local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. + sudo python3 -m pip install -r job-requirements.txt WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead. ERROR: Command errored out with exit status 1: command: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel cwd: None Complete output (866 lines): WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead. Ignoring numpy: markers 'python_version < "3.7"' don't match your environment Collecting cython>=0.29 Using cached Cython-0.29.32-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB) Collecting numpy==1.16.0 Downloading numpy-1.16.0.zip (5.1 MB) Collecting setuptools Downloading setuptools-65.4.0-py3-none-any.whl (1.2 MB) Collecting setuptools_scm Downloading setuptools_scm-7.0.5-py3-none-any.whl (42 kB) Collecting wheel Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting packaging>=20.0 Downloading packaging-21.3-py3-none-any.whl (40 kB) Collecting tomli>=1.0.0 Downloading tomli-2.0.1-py3-none-any.whl (12 kB) Collecting typing-extensions Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB) Collecting importlib-metadata; python_version < "3.8" Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting zipp>=0.5 Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB) ... _configtest.c:1:10: fatal error: Python.h: No such file or directory #include <Python.h> ^~~~~~~~~~ compilation terminated. failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "<string>", line 1, in <module> File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 415, in <module> setup_package() File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 407, in setup_package setup(**metadata) File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/core.py", line 171, in setup return old_setup(**new_attr) File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 165, in setup return distutils.core.setup(**attrs) File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 62, in run r = self.setuptools_run() File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run return distutils_install.run(self) File "/usr/lib64/python3.7/distutils/command/install.py", line 556, in run self.run_command('build') File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build.py", line 47, in run old_build.run(self) File "/usr/lib64/python3.7/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 148, in run self.build_sources() File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 165, in build_sources self.build_extension_sources(ext) File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 322, in build_extension_sources sources = self.generate_sources(sources, ext) File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 375, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 423, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 47, in check_types out = check_types(*a, **kw) File "numpy/core/setup.py", line 281, in check_types "install {0}-dev|{0}-devel.".format(python)) SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel. ---------------------------------------- ERROR: Command errored out with exit status 1: /bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"'; __file__='"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /mnt/tmp/pip-record-paofd9vx/install-record.txt --single-version-externally-managed --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --compile --install-headers /mnt/tmp/pip-build-env-yy928eo_/overlay/include/python3.7m/numpy Check the logs for full command output. ---------------------------------------- ERROR: Command errored out with exit status 1: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output. ```
0
answers
0
votes
6
views
asked 3 days ago

How to containerize kafka-kinesis-connector?

I have an on-prem data pipeline with MQTT + Kafka, each containerized locally. Now, I want to enable the upstream connection to the Cloud/Internet with AWS Kinesis, but I need a Kafka/Kinesis connector. ``` version: '3' services: nodered: container_name: nodered image: nodered/node-red ports: - "1880:1880" volumes: - ./nodered:/data depends_on: - mosquitto environment: - TZ=America/Toronto - NODE_RED_ENABLE_PROJECTS=true restart: always mosquitto: image: eclipse-mosquitto container_name: mqtt restart: always ports: - "1883:1883" volumes: - "./mosquitto/config:/mosquitto/config" - "./mosquitto/data:/mosquitto/data" - "./mosquitto/log:/mosquitto/log" environment: - TZ=America/Toronto user: "${PUID}:${PGID}" portainer: ports: - "9000:9000" container_name: portainer restart: always volumes: - "/var/run/docker.sock:/var/run/docker.sock" - "./portainer/portainer_data:/data" image: portainer/portainer-ce zookeeper: image: zookeeper:3.4 container_name: zookeeper ports: - "2181:2181" volumes: - "zookeeper_data:/data" kafka: image: wurstmeister/kafka:1.0.0 container_name: kafka ports: - "9092:9092" - "9093:9093" volumes: - "kafka_data:/data" environment: - KAFKA_ZOOKEEPER_CONNECT=10.0.0.129:2181 - KAFKA_ADVERTISED_HOST_NAME=10.0.0.129 - JMX_PORT=9093 - KAFKA_ADVERTISED_PORT=9092 - KAFKA_LOG_RETENTION_HOURS=1 - KAFKA_MESSAGE_MAX_BYTES=10000000 - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000 - KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000 - KAFKA_NUM_PARTITIONS=2 - KAFKA_DELETE_RETENTION_MS=1000 depends_on: - zookeeper restart: on-failure cmak: image: hlebalbau/kafka-manager:1.3.3.16 container_name: kafka-manager restart: always depends_on: - kafka - zookeeper ports: - "9080:9080" environment: - ZK_HOSTS=10.0.0.129 - APPLICATION_SECRET=letmein command: -Dconfig.file=/kafka-manager/conf/application.conf -Dapplication.home=/kafkamanager -Dhttp.port=9080 volumes: zookeeper_data: driver: local kafka_data: driver: local ``` I found this one from your labs: https://github.com/awslabs/kinesis-kafka-connector Again, I run everything from a docker-compose and that works, but now I'm not sure if there's either an existing image or documentation that can help me figure out how to containerize this connector. Will I have to create my own custom image via a DockerFile? Any examples? Thank you.
1
answers
0
votes
12
views
asked 3 days ago

Recent articles

see all
1/3

Popular users

see all
1/18

Learn AWS faster by following popular topics

1/3