Minimal latency for EFS OPEN(new file), WRITE(1 byte), RENAME and REMOVE?

0

Minimal latency for EFS OPEN(new file), WRITE(1 byte), RENAME and REMOVE?

Thanks in advance for any help with this. I am evaluating EFS for use with an existing proprietary technology stack. Within the system there are many shards that each correspond to a database. When these databases are first opened there are (currently) several small files created, renamed and removed. The requirements are for each shard to be opened quickly, so ideally in under 50ms 95% of the time. I have noticed high latency with such operations when testing on EFS and am now wondering how to obtain minimal latency?

I am testing with m4.10xlarge instance type in us-east-1d (using EFS DNS to mount in the same availability zone). I am in a VPC, could the VPC be adding latency?

Model       vCPU* Mem (GiB) Storage  Dedicated EBS Bandwidth (Mbps) Network Performance
m4.10xlarge 40    160       EBS-only 4,000                          10 Gigabit

Running amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2 (ami-0080e4c5bc078760e). I started with a RHEL7.6 AMI but switched.
I have tested EFS throughput modes provisioned 1024 MiB/s and Bursting, performance mode Max I/O and General Purpose (I read that MaxIO can have higher latency and I have observed this). All with 1.2TB of files on the filesystem and in the case of Bursting, plenty of Burst Credit Balance. Testing without encrytion at rest. Mount options are the default from the "Amazon EC2 mount instructions (from local VPC)", NFS client, so:

    mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-XXXX.efs.us-east-1.amazonaws.com:/ /efs
Testing without ssl.  

What (NFS RTT) latency figures to expect?

So far, after 10 runs of the command below, I am seeing the 1 byte write (to a new file) client NFS RTT is around 10 millisenconds, with the open, rename and remove all being between 5ms to 8ms:

This is on a 1024 MiB/s PIOPS General Purpose EFS.

 mountstats /efs | egrep -B 3 'RTT: (([1-9][0-9])|([3-9]))' --no-group-separator

WRITE:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 328      avg bytes received per op: 176
        backlog wait: 0.012500  RTT: 10.775000  total execute time: 10.803125 (milliseconds)
OPEN:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 404      avg bytes received per op: 460
        backlog wait: 0.009375  RTT: 7.390625   total execute time: 7.456250 (milliseconds)
REMOVE:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 288      avg bytes received per op: 116
        backlog wait: 0.003125  RTT: 6.390625   total execute time: 6.431250 (milliseconds)
RENAME:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 440      avg bytes received per op: 152
        backlog wait: 0.009375  RTT: 5.750000   total execute time: 5.771875 (milliseconds)

This is on a 1024 MiB/s PIOPS MaxIO EFS.

  mountstats /efs | egrep -B 3 'RTT: (([1-9][0-9])|([6-9]))' --no-group-separator
WRITE:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 328      avg bytes received per op: 176
        backlog wait: 0.012500  RTT: 13.746875  total execute time: 13.775000 (milliseconds)
OPEN:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 404      avg bytes received per op: 460
        backlog wait: 0.009375  RTT: 27.175000  total execute time: 27.196875 (milliseconds)
REMOVE:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 288      avg bytes received per op: 116
        backlog wait: 0.003125  RTT: 19.465625  total execute time: 19.515625 (milliseconds)
RENAME:
        320 ops (12%)   0 retrans (0%)  0 major timeouts
        avg bytes sent per op: 440      avg bytes received per op: 152
        backlog wait: 0.012500  RTT: 19.046875  total execute time: 19.068750 (milliseconds)

Testing with this command:

export DIR=/efs ; rm $DIR/*.tmp ; ( time bash -c 'seq 1 32 | xargs -I {} bash -c "time ( dd if=/dev/zero of=$DIR/{}.tmp bs=1 count=1 conv=fdatasync ; mv $DIR/{}.tmp $DIR/mv{}.tmp ; rm $DIR/mv{}.tmp )" ' ) 2>&1 | grep real

Is this around what to expect from EFS? Can anything be done to lower this latency?

I have read https://docs.aws.amazon.com/efs/latest/ug/performance.html

"The distributed nature of Amazon EFS enables high levels of availability, durability, and scalability. This distributed architecture results in a small latency overhead for each file operation. Due to this per-operation latency, overall throughput generally increases as the average I/O size increases, because the overhead is amortized over a larger amount of data. Amazon EFS supports highly parallelized workloads (for example, using concurrent operations from multiple threads and multiple Amazon EC2 instances), which enables high levels of aggregate throughput and operations per second."


Can also be seen with this mainprog:

// Compile with: g++ -Wall -std=c++11 test.cc
#include "stdio.h"
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sstream>
#include <vector>
#include <iostream>
#include <chrono>

int main(int argc, char* argv[]){

  std::vector<std::string> args(argv, argv + argc);
  if (args.size() != 2){
    std::cout << "Usage: " << args[0] << " dir_path" << std::endl;
    return 1;
  }

  for(int i=1;i<32;i++){
    std::ostringstream oss_file;
    std::ostringstream oss_file_rename;
    oss_file        << args[1] << "/test_" << i << ".tmp";
    oss_file_rename << args[1] << "/test_" << i << "_rename.tmp";
    FILE *fptr;
    auto start = std::chrono::system_clock::now();
    auto start_for_total = start;
    fptr = fopen(oss_file.str().c_str(), "w");
    auto stop = std::chrono::system_clock::now();
    if( NULL == fptr ){
      printf("Could not open file '%s': %s\n", oss_file.str().c_str(), strerror(errno));
    }
    printf("time in ms for fopen = %3ld ",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count());
    start = std::chrono::system_clock::now();
    if( write( fileno(fptr), "X",1 ) <= 0 ){
      printf("Could not write to file '%s': %s\n", oss_file.str().c_str(), strerror(errno));
    }
    stop = std::chrono::system_clock::now();
    printf("write = %3ld ",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count());
    start = std::chrono::system_clock::now();
    if( 0 != fdatasync( fileno(fptr) )){
      printf("Could not fdatasync file '%s': %s\n", oss_file.str().c_str(), strerror(errno));
    }
    stop = std::chrono::system_clock::now();
    printf("fdatasync = %3ld ",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count());
    start = std::chrono::system_clock::now();
    if( 0 != fclose(fptr)){
      printf("Could not fclose file '%s': %s\n", oss_file.str().c_str(), strerror(errno));
    }
    stop = std::chrono::system_clock::now();
    printf("fclose = %3ld ",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count());
    start = std::chrono::system_clock::now();
    if( 0 != rename(oss_file.str().c_str(), oss_file_rename.str().c_str())){
      printf("Could not rename file '%s' to file '%s': %s\n", oss_file.str().c_str(), oss_file_rename.str().c_str(), strerror(errno));
    }
    stop = std::chrono::system_clock::now();
    printf("rename = %3ld ",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count());
    start = std::chrono::system_clock::now();
    if(unlink(oss_file_rename.str().c_str())!=0){
      printf("Could not unlink file '%s': %s\n", oss_file.str().c_str(), strerror(errno));
    }
    stop = std::chrono::system_clock::now();
    printf("unlink = %3ld total = %3ld\n",std::chrono::duration_cast<std::chrono::milliseconds>(stop - start).count(), std::chrono::duration_cast<std::chrono::milliseconds>(stop - start_for_total).count());
  }
}

On the EBS SSD /tmp filesystem:

> time ./a.out /tmp
time in ms for fopen =   0 write =   0 fdatasync =   3 fclose =   0 rename =   0 unlink =   0 total =   3
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   3 fclose =   0 rename =   0 unlink =   0 total =   3
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
time in ms for fopen =   0 write =   0 fdatasync =   3 fclose =   0 rename =   0 unlink =   0 total =   3
time in ms for fopen =   0 write =   0 fdatasync =   2 fclose =   0 rename =   0 unlink =   0 total =   2
real    0m0.086s

On EFS General Purpose with 1024 MiB/s provisioned IOPS:

> time ./a.out /efs_gp_1024piops
time in ms for fopen =  12 write =   0 fdatasync =  10 fclose =   0 rename =   7 unlink =   5 total =  37
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   0 rename =   7 unlink =   5 total =  32
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   1 rename =  13 unlink =   9 total =  42
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   1 rename =   9 unlink =   7 total =  37
time in ms for fopen =   7 write =   0 fdatasync =  12 fclose =   2 rename =  11 unlink =   6 total =  40
time in ms for fopen =  10 write =   0 fdatasync =  13 fclose =   4 rename =  11 unlink =   5 total =  46
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   0 rename =  20 unlink =   5 total =  44
time in ms for fopen =   8 write =   0 fdatasync =  15 fclose =   6 rename =  14 unlink =   7 total =  52
time in ms for fopen =  11 write =   0 fdatasync =  11 fclose =   3 rename =  15 unlink =   6 total =  48
time in ms for fopen =   8 write =   0 fdatasync =  17 fclose =   1 rename =  11 unlink =   6 total =  44
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   1 rename =   8 unlink =   5 total =  32
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   1 rename =   8 unlink =   6 total =  34
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   0 rename =   8 unlink =   6 total =  34
time in ms for fopen =   8 write =   0 fdatasync =  10 fclose =   0 rename =   8 unlink =   5 total =  33
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   0 rename =   7 unlink =   5 total =  33
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   1 rename =   7 unlink =   5 total =  32
time in ms for fopen =   8 write =   0 fdatasync =  11 fclose =   0 rename =   7 unlink =   6 total =  34
time in ms for fopen =   7 write =   0 fdatasync =   9 fclose =   0 rename =   7 unlink =   5 total =  31
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   1 rename =   8 unlink =   5 total =  32
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   0 rename =   8 unlink =   6 total =  35
time in ms for fopen =   8 write =   0 fdatasync =  10 fclose =   1 rename =   7 unlink =   5 total =  32
time in ms for fopen =   8 write =   0 fdatasync =  10 fclose =   1 rename =   8 unlink =   5 total =  33
time in ms for fopen =  28 write =   0 fdatasync =  10 fclose =   0 rename =   7 unlink =   5 total =  54
time in ms for fopen =   8 write =   0 fdatasync =  10 fclose =   1 rename =   7 unlink =   6 total =  35
time in ms for fopen =   7 write =   0 fdatasync =  12 fclose =   1 rename =  11 unlink =   6 total =  39
time in ms for fopen =   8 write =   0 fdatasync =   9 fclose =   0 rename =   7 unlink =   6 total =  33
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   1 rename =   8 unlink =   5 total =  35
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   1 rename =  11 unlink =   5 total =  35
time in ms for fopen =   7 write =   0 fdatasync =  11 fclose =   0 rename =   8 unlink =   6 total =  35
time in ms for fopen =   7 write =   0 fdatasync =  10 fclose =   1 rename =   8 unlink =   5 total =  33
time in ms for fopen =   8 write =   0 fdatasync =  10 fclose =   1 rename =   7 unlink =   5 total =  33
real    0m1.167s

On EFS MaxIO with 1024 MiB/s provisioned IOPS:

> time ./a.out /efs_maxio_1024piops
time in ms for fopen =  35 write =   0 fdatasync =  13 fclose =   0 rename =  22 unlink =  19 total =  91
time in ms for fopen =  26 write =   0 fdatasync =  12 fclose =   1 rename =  23 unlink =  19 total =  82
time in ms for fopen =  29 write =   0 fdatasync =  12 fclose =   1 rename =  31 unlink =  20 total =  95
time in ms for fopen =  27 write =   0 fdatasync =  13 fclose =   1 rename =  28 unlink =  19 total =  90
time in ms for fopen =  25 write =   0 fdatasync =  14 fclose =   1 rename =  24 unlink =  18 total =  84
time in ms for fopen =  28 write =   0 fdatasync =  11 fclose =   1 rename =  24 unlink =  22 total =  88
time in ms for fopen =  24 write =   0 fdatasync =  13 fclose =   1 rename =  32 unlink =  18 total =  90
time in ms for fopen =  27 write =   0 fdatasync =  11 fclose =   1 rename =  24 unlink =  19 total =  84
time in ms for fopen =  24 write =   0 fdatasync =  14 fclose =   1 rename =  22 unlink =  17 total =  80
time in ms for fopen =  27 write =   0 fdatasync =  12 fclose =   0 rename =  24 unlink =  21 total =  86
time in ms for fopen =  26 write =   0 fdatasync =  13 fclose =   1 rename =  26 unlink =  18 total =  85
time in ms for fopen =  26 write =   0 fdatasync =  13 fclose =   1 rename =  24 unlink =  17 total =  83
time in ms for fopen =  26 write =   0 fdatasync =  13 fclose =   1 rename =  23 unlink =  19 total =  84
time in ms for fopen =  27 write =   0 fdatasync =  12 fclose =   1 rename =  23 unlink =  18 total =  82
time in ms for fopen =  28 write =   0 fdatasync =  16 fclose =   0 rename =  23 unlink =  18 total =  87
time in ms for fopen =  28 write =   0 fdatasync =  13 fclose =   0 rename =  25 unlink =  19 total =  87
time in ms for fopen =  24 write =   0 fdatasync =  10 fclose =   0 rename =  23 unlink =  18 total =  77
time in ms for fopen =  28 write =   0 fdatasync =  15 fclose =   0 rename =  23 unlink =  19 total =  88
time in ms for fopen =  26 write =   0 fdatasync =  13 fclose =   1 rename =  21 unlink =  18 total =  81
time in ms for fopen =  25 write =   0 fdatasync =  13 fclose =   1 rename =  21 unlink =  16 total =  78
time in ms for fopen =  24 write =   0 fdatasync =  14 fclose =   1 rename =  26 unlink =  17 total =  83
time in ms for fopen =  26 write =   0 fdatasync =  14 fclose =   1 rename =  27 unlink =  20 total =  90
time in ms for fopen =  27 write =   0 fdatasync =  11 fclose =   1 rename =  25 unlink =  21 total =  86
time in ms for fopen =  24 write =   0 fdatasync =  11 fclose =   0 rename =  21 unlink =  17 total =  75
time in ms for fopen =  29 write =   0 fdatasync =  16 fclose =   1 rename =  24 unlink =  17 total =  88
time in ms for fopen =  27 write =   0 fdatasync =  13 fclose =   0 rename =  23 unlink =  31 total =  96
time in ms for fopen =  25 write =   0 fdatasync =  14 fclose =   1 rename =  23 unlink =  17 total =  83
time in ms for fopen =  27 write =   0 fdatasync =  13 fclose =   1 rename =  21 unlink =  17 total =  81
time in ms for fopen =  28 write =   0 fdatasync =  14 fclose =   1 rename =  22 unlink =  17 total =  84
time in ms for fopen =  24 write =   0 fdatasync =  13 fclose =   1 rename =  23 unlink =  18 total =  81
time in ms for fopen =  26 write =   0 fdatasync =  12 fclose =   0 rename =  23 unlink =  18 total =  81
real    0m2.649s

On EFS General Purpose in Bursting config:

> time ./a.out /efs_burst
time in ms for fopen =   7 write =   0 fdatasync =  30 fclose =   0 rename =  25 unlink =   4 total =  68
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   4 total =  23
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   4 total =  23
time in ms for fopen =   5 write =   0 fdatasync =   8 fclose =   0 rename =   5 unlink =   4 total =  24
time in ms for fopen =   6 write =   0 fdatasync =   8 fclose =   0 rename =   5 unlink =   4 total =  25
time in ms for fopen =   4 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   6 total =  25
time in ms for fopen =   6 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   3 total =  25
time in ms for fopen =   5 write =   0 fdatasync =   8 fclose =   0 rename =   5 unlink =   4 total =  24
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   7 total =  28
time in ms for fopen =   4 write =   0 fdatasync =   8 fclose =   0 rename =   5 unlink =   4 total =  23
time in ms for fopen =   5 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  25
time in ms for fopen =   5 write =   0 fdatasync =   9 fclose =   0 rename =   7 unlink =   5 total =  28
time in ms for fopen =   6 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  26
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   4 total =  24
time in ms for fopen =   5 write =   0 fdatasync =   9 fclose =   0 rename =   6 unlink =   4 total =  26
time in ms for fopen =   6 write =   0 fdatasync =   9 fclose =   0 rename =   6 unlink =   4 total =  27
time in ms for fopen =   6 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  26
time in ms for fopen =   6 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   3 total =  25
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   4 total =  23
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   6 unlink =   4 total =  24
time in ms for fopen =   7 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  26
time in ms for fopen =   5 write =   0 fdatasync =  10 fclose =   0 rename =   6 unlink =   4 total =  28
time in ms for fopen =   5 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  25
time in ms for fopen =   5 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  25
time in ms for fopen =   5 write =   0 fdatasync =  11 fclose =   0 rename =   5 unlink =   4 total =  27
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   3 total =  23
time in ms for fopen =   6 write =   0 fdatasync =  16 fclose =   0 rename =   6 unlink =   4 total =  33
time in ms for fopen =   7 write =   0 fdatasync =   8 fclose =   0 rename =   6 unlink =   4 total =  26
time in ms for fopen =   5 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   4 total =  23
time in ms for fopen =   4 write =   0 fdatasync =   7 fclose =   0 rename =   5 unlink =   4 total =  23
time in ms for fopen =   4 write =   0 fdatasync =   7 fclose =   0 rename =   7 unlink =   3 total =  24
real    0m0.845s

Thanks again for any input.

Edited by: Indiana on Apr 25, 2019 12:28 PM

Edited by: Indiana on Apr 25, 2019 12:28 PM

Edited by: Indiana on Apr 26, 2019 2:40 AM

Edited by: Indiana on Apr 26, 2019 2:54 AM

Indiana
asked 5 years ago455 views
1 Answer
0

Hi Indiana,

We will be sending you a private message shortly regarding this post.

Thanks,
Lilian
Amazon EFS Team

answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions