Questions tagged with Database
Content language: English
Sort by most recent
Hello,
Since a Sunday maintenance window, we began to receive
```
ERROR: unexpected pageaddr
```
for all logical replication apps (AWS DMS and Debezium on Kafka Connect).
This error is somehow recoverable and periodic:
Otherwise DMS shows no error on this but Debezium kind of fails and recovered by connecting Postgres again.
We are wondering if this could be related to Aurora DB minor version update? Or more like we are wrongly using some DB parameters (reboot actually activate the changes).
Thanks a lot in advance :).
Best regards,
David
We have a Django app running in lambda. It connects to the RDS Database through RDS PROXY Using IAM Auth. When we are doing load testing after certain load we start getting error saying Too many requests to the IAM AUTH service. When we tried creating a new AWS RDS PROXY without IAM AUTH configuration our load tests performed much better. But we wanted to check what is the best and scalable architecture. Should we remove the IAM AUTH and keep the communication direct BW the application and the IAM AUTH or there is a better way to do this.
Can someone Please Help me in Fetching the AWS RDS Snapshots which are older than 1 month using "AWS CLI" Command.
Regards,
kalyan varma
Below is a sample javascript sdk v3 Athena query that uses a prepared statement and parameters that are passed to the query
```
const { AthenaClient } = require("@aws-sdk/client-athena");
const REGION = 'us-east-1';
const athenaClient = new AthenaClient({region: REGION});
module.exports = {athenaClient};
```
```
const tableName = 'employees';
const sqlString = "SELECT firstname, lastname, state FROM " + tableName + " WHERE " +
"zipcode = ? AND " +
"companyname = ?";
const queryExecutionInput = {
QueryString: sqlString,
QueryExecutionContext: {
Database: 'sample-employee',
Catalog: 'awscatalogname'
},
ResultConfiguration: {
OutputLocation: 's3://athena-query-bucket'
},
WorkGroup: 'primary',
ExecutionParameters: ["12345", "Test 1"]
}
const queryExecutionId = await athenaClient.send(new StartQueryExecutionCommand(queryExecutionInput));
const command = new GetQueryExecutionCommand(queryExecutionId);
const response = await athenaClient.send(command);
const state = response.QueryExecution?.Status?.State;
if(state === QueryExecutionState.QUEUED || state === QueryExecutionState.RUNNING) {
await setTimeout(this.config.pollInterval); //wait for pollInterval before calling again.
return this.waitForQueryExecution(queryExecutionId);
} else if(state === QueryExecutionState.SUCCEEDED) {
const resultParams = { QueryExecutionId: response.QueryExecution.QueryExecutionId, MaxResults: this.config.maxResults};
const getQueryResultsCommand:any = new GetQueryResultsCommand(resultParams);
const resp = await athenaClient.send(getQueryResultsCommand);
console.log("GetQueryResultsCommand : ", resp.ResultSet.ResultSetMetadata.ColumnInfo);
console.log("GetQueryResultsCommand : ", resp.ResultSet.Rows);
} else if(state === QueryExecutionState.FAILED) {
throw new Error(`Query failed: ${response.QueryExecution?.Status?.StateChangeReason}`);
} else if(state === QueryExecutionState.CANCELLED) {
throw new Error("Query was cancelled");
}
```
This table has about 50 records that match this query. When the query is run this is what is returned for all 50 records.
```
{
"ResultSetMetadata":
{
"Rows":
[
{
"Data":
[
{
"VarCharValue": "firstname"
},
{
"VarCharValue": "lastname"
},
{
"VarCharValue": "state"
}
]
}
]
}
}
```
Only the column names are listed but no data from these columns.
I see the exact same issue when I try it using the CLI as well
```
aws athena start-query-execution --query-string "SELECT firstname, lastname, state FROM employees WHERE zipcode = CAST(? as varchar) AND companyname = CAST(? as varchar)"
--query-execution-context "Database"="sample-employee"
--result-configuration "OutputLocation"="s3://athena-query-bucket/"
--execution-parameters "12345" "Test 1"
aws athena get-query-execution --query-execution-id "<query-execution-id>"
aws athena get-query-results --query-execution-id "<query-execution-id>"
```
FYI ColumnInfo in the ResultSetMetadata object has been removed to keep the json simple
```
{
"ResultSetMetadata":
{
"Rows":
[
{
"Data":
[
{
"VarCharValue": "firstname"
},
{
"VarCharValue": "lastname"
},
{
"VarCharValue": "state"
}
]
}
]
}
}
```
So, not exactly sure what I might be doing wrong. Any help/pointers on this would be great. We are currently running Athena engine version 2.
I am trying to find a way to create Athena queries that handle information from AWS Security Hub, such as the 'Findings' displayed within it. Athena's input data comes from S3. Is there a way to specify a location in S3 that will receive the findings from AWS Security Hub, or is there already a location I should try looking into? Is there any other way to feed Security Hub information into Athena?
How do I connect to localhost from MySQL Workbench?
Cluster parameters:
1. Engine: Aurora (MySQL 5.7) 2.10.3
2. VPC security group: default and allow-prod-corp-port-81928200
3. Database authentication: password authentication
MySQL Workbench parameters:
1. Connection method: Standard (TCP/IP)
2. Hostname: using correct host address
3. Port: 8192
4. Username: using correct master user name
5. Password: using correct master password
Results:
1. From test connection: Failed to connect to MySQL at <database address>:8192 with user <master user name> . Unable to connect to localhost.
2. From open connection: Cannot connect to database server. Your connection attempt failed for <username> at <database address>:8192. Unable to connect to localhost.
I have enabled performance insight for my RDS.deadlock error occured in my database.I need to know about the querry which got deadlock.
My team have had issues using StackExchange.Redis's client for C# with MemoryDB:
https://stackoverflow.com/questions/72545009/connection-to-aws-memorydb-cluster-sometimes-fails
We've also written to StackExchange.Redis' github here:
https://github.com/StackExchange/StackExchange.Redis/issues/2013
In short, the issue is that when we manually failover MemoryDB while having 2 or 3 nodes (1 primary, 1 or 2 replicas, 1 shard), we're unable to reach the MemoryDB cluster via the cluster endpoint unless we restart the ECS task (or reconnect using ConnectionMultiplexer).
We've found that if we do:
```
foreach (var endpoint in connectionMultiplexer.GetEndPoints(false))
{
Console.WriteLine(endpoint.ToString());
}
```
We only get back the cluster endpoint. But our understanding is that it should be retunring all discovered nodes.
Is there anything we can do to further debug this issue?
I am unable to modify the RDS aurora cluster, located in Ireland region, due to unexpected error as in the screenshot attached. I tried to clear caching and cookies, using different browsers and devices and even different users but no luck.
Hello
1. What is difference between Amazon Athena and Amazon Redshift?
2. Which one is the right choice?
3. If we are using RDS, Amazon Athena and Amazon Redshift together in architecture. How does it works together?
4. what is Amazon QuickSight and how does it works ?
Thanks,
Monica
Login request was received, the username and password were correctly extracted from the request body, and a user with ID 1 was found in the database. The form still 504 fails eventually.
my index.js, db.js, users.js, and login.html all seem fine.
I'm on Lightsail so unfortunately I've had to use SQL Workbench this whole time.
Not sure if there's an issue with the Lightsail to DB communication? It's been a pain to try to figure out Lightsail with the 'module' stuff like databases.
users.js :
```
const connection = require('./db');
const bcrypt = require('bcrypt');
const saltRounds = 10;
class User {
constructor(id, username, password, email, createdAt, updatedAt) {
this.id = id;
this.username = username;
this.password = password;
this.email = email;
this.createdAt = createdAt;
this.updatedAt = updatedAt;
}
static create(username, password, email) {
const now = new Date().toISOString();
const sql = `INSERT INTO loginserver (username, password, email, created_at, updated_at) VALUES (?, ?, ?, ?, ?)`;
bcrypt.hash(password, saltRounds, (err, hash) => {
if (err) {
console.error('Error hashing password:', err);
return;
}
const values = [username, hash, email, now, now];
connection.query(sql, values, (err, result) => {
if (err) {
console.error('Error creating user:', err);
return;
}
console.log('User created with ID', result.insertId);
const user = new User(result.insertId, username, hash, email, now, now);
return user;
});
});
}
static getByUsername(username) {
const sql = `SELECT * FROM loginserver WHERE username = ?`;
connection.query(sql, [username], (err, results) => {
if (err) {
console.error('Error getting user by username:', err);
return;
}
if (results.length === 0) {
console.log('User not found');
return null;
}
const { id, username, password, email, created_at, updated_at } = results[0];
console.log('User found with ID', id);
const user = new User(id, username, password, email, created_at, updated_at);
return user;
});
}
checkPassword(password) {
return new Promise((resolve, reject) => {
bcrypt.compare(password, this.password, (err, isMatch) => {
if (err) {
console.error('Error checking password:', err);
reject(err);
} else {
resolve(isMatch);
}
});
});
}
update() {
const now = new Date().toISOString();
const sql = `UPDATE loginserver SET username = ?, password = ?, email = ?, updated_at = ? WHERE id = ?`;
const values = [this.username, this.password, this.email, now, this.id];
connection.query(sql, values, (err) => {
if (err) {
console.error('Error updating user:', err);
return;
}
console.log('User updated with ID', this.id);
this.updatedAt = now;
return this;
});
}
delete() {
const sql = `DELETE FROM loginserver WHERE id = ?`;
connection.query(sql, [this.id], (err) => {
if (err) {
console.error('Error deleting user:', err);
return;
}
console.log('User deleted with ID', this.id);
return;
});
}
}
module.exports = User;
```
index.js :
```
const express = require('express');
const https = require('https');
const socketIO = require('socket.io');
const path = require('path');
const fs = require('fs');
const mysql = require('mysql');
const User = require('./server/users');
const bodyParser = require('body-parser');
const app = express();
const server = https.createServer({
key: fs.readFileSync('/etc/letsencrypt/live/ispeedrun.tv/privkey.pem'),
cert: fs.readFileSync('/etc/letsencrypt/live/ispeedrun.tv/fullchain.pem')
}, app);
const io = socketIO(server);
// Add this before the routes
app.use((req, res, next) => {
console.log('Request received');
next();
});
app.use(express.static(path.join(__dirname, 'views', 'public')));
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.get('/', (req, res) => {
res.sendFile(path.join(__dirname, 'views', 'index.html'));
});
app.get('/live', (req, res) => {
res.sendFile(path.join(__dirname, 'views', 'live.html'));
});
const connection = mysql.createConnection({
host: 'ls-7f5846c26112d5a110aa9ce17f20838297ce7c51.cdnunzehdfq0.us-east-2.rds.amazonaws.com',
port: '3306',
user: 'dbmasteruser',
password: '',
database: ''
});
connection.connect((err) => {
if (err) {
console.error('Failed to connect to MySQL:', err);
return;
}
console.log('Connected to MySQL database');
});
io.on('connection', (socket) => {
console.log('WebSocket connection established');
socket.on('message', (msg) => {
console.log('message: ' + msg);
io.emit('message', msg);
});
socket.on('disconnect', () => {
console.log('WebSocket connection closed');
});
});
// add this route to handle form submission
app.post('/login', (req, res) => {
console.log('Received login request');
console.log('Login request received:', req.body); // Log the received request
const { username, password } = req.body;
User.getByUsername(username, (err, user) => {
if (err) {
console.error('Error getting user:', err);
res.status(500).send('Internal server error');
return;
}
if (!user) {
res.status(401).send('Invalid username or password');
return;
}
user.checkPassword(password, (err, isMatch) => {
if (err) {
console.error('Error checking password:', err);
res.status(500).send('Internal server error');
return;
}
if (!isMatch) {
res.status(401).send('Invalid username or password');
return;
}
res.status(200).send(); // Send a 200 status code to indicate a successful login
});
});
});
// Add this after the routes
app.use((req, res, next) => {
console.log('Response sent');
next();
});
const PORT = process.env.PORT || 6611;
server.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
});
```
login.html :
```
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>iSpeedrun.TV - Login</title>
<link rel="stylesheet" href="styles.css">
<style>
/* Keep the same styles as index.html */
.main-container {
display: flex;
flex-direction: row;
}
.video-container {
width: 1280px;
height: 720px;
margin-right: 20px;
}
.video-container iframe {
width: 100%;
height: 100%;
}
.sidebar {
width: 300px;
height: 720px;
display: flex;
flex-direction: column;
justify-content: space-between;
}
.sidebar-item {
display: flex;
align-items: center;
padding: 10px;
background-color: #222;
color: #fff;
font-size: 14px;
}
.sidebar-item img {
width: 60px;
height: 60px;
margin-right: 10px;
}
header {
display: flex;
justify-content: space-between;
align-items: center;
background-color: #222;
color: #fff;
padding: 10px;
}
nav ul {
display: flex;
list-style: none;
padding: 0;
margin: 0;
}
nav li {
margin-right: 20px;
}
nav a {
color: #fff;
text-decoration: none;
font-weight: bold;
font-size: 16px;
text-transform: uppercase;
}
nav a:hover {
color: #ff0000;
}
.login-container {
background-color: #fff;
padding: 40px;
border-radius: 10px;
width: 70%;
margin: 20px auto;
box-shadow: 0 0 20px rgba(0, 0, 0, 0.5);
}
.login-container label {
font-size: 20px;
margin-bottom: 20px;
}
.login-container input[type="text"],
.login-container input[type="password"] {
width: 100%;
height: 40px;
margin-bottom: 30px;
padding: 10px;
font-size: 16px;
border-radius: 5px;
border: none;
box-shadow: 1px 1px 5px rgba(0, 0, 0, 0.3);
}
.login-container button[type="submit"] {
display: block;
width: 100%;
height: 50px;
background-color: #e74c3c;
color: #fff;
border: none;
border-radius: 5px;
font-size: 18px;
cursor: pointer;
transition: background-color 0.2s;
}
.login-container button[type="submit"]:hover {
background-color: #c0392b;
}
#message {
font-size: 18px;
color: red;
margin-bottom: 15px;
}
</style>
</head>
<body>
<header>
<h1>iSpeedrun.TV - Login</h1>
<nav>
<ul>
<li><a href="index.html">Home</a></li>
<li><a href="livestream.html">Live Streams</a></li>
<li><a href="about.html">About Us</a></li>
<li><a href="contact.html">Contact</a></li>
<li><a href="login.html">Login</a></li>
</ul>
</nav>
</header>
<main class="main-container">
<div class="sidebar">
<div class="sidebar-item">
<img src="https://via.placeholder.com/60x60.png?text=User+1" alt="User 1">
<p>User 1</p>
</div>
<div class="sidebar-item">
<img src="https://via.placeholder.com/60x60.png?text=User+2" alt="User 2">
<p>User 2</p>
</div>
<div class="sidebar-item">
<img src="https://via.placeholder.com/60x60.png?text=User+3" alt="User 3">
<p>User 3</p>
</div>
<div class="sidebar-item">
<img src="https://via.placeholder.com/60x60.png?text=User+4" alt="User 4">
<p>User 4</p>
</div>
</div>
<div class="video-container">
<form class="login-container" action="/login" method="post" id="login-form">
<label for="username">Username:</label>
<input type="text" id="username" name="username">
<label for="password">Password:</label>
<input type="password" id="password" name="password">
<div id="message"></div>
<button type="submit">Login</button>
</form>
</div>
</main>
<script>
const form = document.getElementById('login-form');
const message = document.getElementById('message');
form.addEventListener('submit', async function(event) {
console.log('Form submitted');
event.preventDefault(); // Prevent the form from submitting normally
const username = document.getElementById('username').value;
const password = document.getElementById('password').value;
try {
console.log('Sending request to server');
const response = await fetch('/login', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ username, password }),
});
console.log('Server responded with status:', response.status);
if (response.status === 200) {
localStorage.setItem('loggedIn', 'true');
window.location.href = 'index.html';
} else {
const error = await response.json();
message.textContent = error.message;
}
} catch (error) {
console.error('Error:', error);
message.textContent = 'An error occurred. Please try again.';
}
});
</script>
</body>
</html>
```
We have a encrypted dynamodb table and recently noticed latency on ddb calls, looking further it was actually kms calls that was taking time. So we looked into caching solutions and found that CachingMostRecentProvider as the general suggestion. But would like to know why does CachingMostRecentProvider/MetaStore need a DDB table? Why cannot it store the required data in memory and on expiry can it not fetch using DirectKMS?