Amazon Web Services - Tutorials by Qwiklabs

01 Apr 2016

Amazon Web Services (AWS)

EC2

• all instances have a private + public IP (NAT-ed)
• Load Balancers
  • use the security group assigned when the instance was created
  • thresholds can be easily set/configured, as well as the “ping-path”
  • tags are supported
  • all metrics are automatically reported to cloudwatch
• Elastic IPs
  • can be assigned to running instances: they will be static IPs and can be assigned to one instance, work across availability zones
• instances tags
• in SecurityGroups of EC2 you can set IPs as instance ID
• EC2 security guide - http://aws.amazon.com/articles/Amazon-EC2/9001172542712674
• EC2 root device guide - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html
• instances can have termination protection in order to prevent accidental deletion
• Spot Instances for EC2
  • best suited for distributed fault-tolerant takss
  • name your price for EC2 instances
  • price varies in real time, by the hour, last hour is not charged if instance is terminated
  • instances may not start immediately
  • you need to place a spot instance request - valid dates / type one-time or persistent / launch group / AZ group
  • if you bid .25 and the price will be .23 for the period, you will pay the lower amount, if it goes over your instance stops
  • there is a Spot Instance Price History for the past 90 days
  • one-time request remains active until instance terminates, request expires, or request is cancelled
    • persistent requests remain active until expired or cancelled
  • in the last step of the request you can see current prices depending on the AZ selected
• windows host + iis 169.254.169.254/latest/meta-data/mac
• Security Groups
  • if your Security Groups are not in a VPC, you’re using EC2 classic
  • “all things” are accessible through the API, e.g. get security groups, VPC id, instances where applied, all rules IN/OUT for a SecGroup
  • for example, with the SDK for various programming languages we can create a simple dashboard to monitor all the VPCs and SecGroups
• Docker can be installed and is compatible w/ all Amazon Linux instances
  • docker and git need to be installed
  • docker pull jboss/wildfly; docker images - to see available ones; docker run -it –rm jboss/wildfly # run interactive remove at end
  • docker build –tag=name_assigned; we can upload files to a “location” which we later can re-build and package in the container

CloudFront

• content delivery web service, accelerated delivery of content through Amazon edge locations
  • no minimum usage restrictions
  • dynamic, static, streaming content
  • default allows users to access objects through HTTP and HTTPS
  • cache for location for 24 hours
  • after creating a distribution you might need to wait up to 15 minutes while its status will be DEPLOYED
• as everything is cached, if you need to update various files, one idea would be to have the files versioned (e.g. _v2)
  • if a file is changed CloudFront will still serve the cached version up to 1 day
  • to invalidate the cache, in the Distribution Settings we can create a new Invalidations and specify the files we’ve (we’re) updated(-ing)
• CNAME can be added, up to 10 CNAMEs per distribution so you can use domain names e.g. www.mysite.com instead of …..amazonaws.com/
• default root (e.g. index.html) is the main path if none is specified
• you can select the region limits where your content is accessible

CodeDeploy

• automates code deploy to EC2 instances
• launch, control, and monitor instances for each instance running in a DTAP environment
• no additional charge for CodeDeploy
  • for the sample application the CloudFormation tool is used

AWS Trusted Advisor

• Best practices of security are in-built: VPC, IAM, network ACLs, … and T.A. offers over 30 checks to monitor and improve security
  • Cost optimization
  • Security
  • Fault tolerance
  • Performance improvement
• 4 of the best practices are free to all customers, to have access to full 37 Business or Enterprise support is needed
  • service limits for resources that are 80% utilized
  • access unrestricted (e.g. 0.0.0.0/0) to specific hosts
  • MFA (Multi Factor Authentication) on Root account
• remove rules which allow access unrestricted to hosts and/or ports (e.g. port 21 FTP)
  • link will take you directly to Security Groups
• analysis (refresh) can be performed every 5 minutes - hovering will give you time to next refresh
• trusted weekly notification over email can be configured in Preferences of Trusted Advisor
• AWS Security Best Practices - http://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf

Amazon Elastic Transcoder

• media transcoding in the cloud
• HLS (HTTP Live Streaming) streaming can adapt to the network speed
• pipeline to transcode a file
• a master playlist can be created which will contain all streams, depending on the bitrate the best will be selected
• depending on the S3 bucket location (upload/download) and the cloudfront location we can obtain the full path to our files (used for testing mainly)

CloudWatch

• monitoring (ELB) dashboard
• details about zones, and request types
• we can create an Alarm in CloudWatch that triggers an event in the auto-scale group

S3 (Simple Storage System)

• secure, durable, highly scalable object storage
• charged is done based on storage and requests of data (in and out), not on bucket(s) creation
• name of bucket must be unique across all S3 buckets (really?!?)
• backup and storage, media and app hosting, high traffic hosting
• data is represented as Objects in Buckets
  • object = file + metadata
  • one or more buckets in the system
  • access control to buckets, access logs and regions can be configured/set
• Permissions can be set as JSON (not very convenient, but there are some examples)
  • by default all objects/buckets are private
• Static Website Hosting (endpoint - Endpoint: webapplication-site-9988.s3-website-us-east-1.amazonaws.com)
• Enable CORS (Cross-Origin Resource Sharing)
  • for JS, “wide array of configrations options”
• to be readable by everyone you will need to have Everyone - List as Add more permissions

EBS (Elastic Block Store)

• persistent network attached storage, independent from the instance itself
• automatically replicated in the same Availability Zone (AZ)
• snapshotting is possible, with the snapshots saved in S3
• IOPS
  • can be provisioned
  • bursty for non-provisioned ones (default)
• for pre-existing images for AMIs a volume is created from a snapshot of the OS image
• on AMIs termination they can be saved or discarded (default behavior)
• volumes w/ sizes between 1GB to 1TB can be created
  • create, attach (to instances)
  • volumes can have tags
• Snapshots
  • create from a running instance
  • for DB we should use a read replica, for which writes are temporarily suspended to improve snapshotting speeds
  • new volumes can be created directly from existing snapshots

EMR (Elastic Map-Reduce) - intro youtube - https://www.youtube.com/watch?v=Hhj3fOdt7zo

• process large amounts of data w/o managing instances
• launch & scale Hadoop applications using EC2 & S3
  • Apache Hive, Pig, Pig Latin support, most known Hadoop apps are directly available
• EC2 images are preconfigured
• Master the node which distributes jobs to other nodes
  • worker nodes
• Metrics about jobs can be viewed in CloudWatch
• cluster can be terminated on job completion
• wordSplitter.py for MR AWS file
• input and output (including logs) must be saved to S3, or be provisioned from S3
  • worker jobs can be as well from S3

ELB (Elastic Load Balancing)

• distributes traffic against multiple instances and multiple zones, automatically switching between all options
• works with VPC, you can create an internal LB environment for your internal network within the VPC
• integrated certificate management for SSL

EBS (Elastic Bean Stalk)

• deploy quickly web applications w/o knowledge of pieces required for cloud
  • monitoring, auto-scaling, load balancing all handled by EBS
  • no charge for EBS, you pay the resources that you’re using
• various options to select the extras needed
  • web server - preconfigured options for most major platforms (Ruby, python, node.js, …)
  • database server (RDS DB) - engine selection from MySQL, PostgreSQL, SQL Server, Oracle
    • instance type and database size
• next step is that all necessary AWS services are started and configured to support your environment

IAM

• Security Groups
  • virtual firewall fo in&out traffic
    • you can specify allow rules but not deny rules
    • by default no inbound traffic is allowed
  • Bastion server
    • single server which has access, with all other resources and applications removed for extra security
• in the VPC you can use both
  • SecurityGroups as well as - on the VPC portal
  • ACL (Access Control Lists) - Network ACLs
• Groups
• Users
  • Permissions
• users login - https://.signin.aws.amazon.com

CloudTrail

• web service that records AWS API calls
  • source, destination, parameters, also made via Web, SDK, CLI, etc
  • you can easily get the history of all calls for security analysis, in order to track changes to AWS resources, or for a compliance trail
  • you can see the trail in S3 buckets

DynamoDB

• NoSQL, single-digit millisecond latency, document or key-value store

SQS (Simple Queue Service)

• fully managed message queing service
• any level of data, at any throughput w/o losing anything

Route 53

• Highly available and scalable DNS
  • CNAME or Canonical Name is a resource record used to specify that a domain is an alias for another domain, the canonical domain
  • having a DNS failover means that traffic always is routed to the live application, also notifications can be sent when issues occur
• monitors health and performance of your web application and other resources
• Route 53 can route internet traffic between multiple services performing same functions only to the healthy ones (email servers, web servers)
• automatically creates a NS (Name Server) resource when creating a new Hosted Zone (HZ)
  • HZ = collection a resource records sets hosted by Route 53
  • NS + SOA required for the HZ to work correctly
• if the HZ is created, you can add (Create Record Set) pointing to an Elastic IP from an existing EC2 instance
• in the Health Check we can create notifications sent to SNS
• we can create a WWW Failover alias which takes content from pre-populated S3 messages (already available, no creation needed)
  • main A alias must be set as Failover primary

SNS (Simple Notification Service)

• fully managed push notification service
  • push to devices, Windows, mobile, Baidu cloud, sms text, email, http to any endpoint
• create topic / email / create subscription - on ARN
• Topic ARN is necessary to manage it from the CLI tools
• to see all notifications as-describe-notification-configurations
• to update notifications for this LB group as-put-notificaation-configuration
  • we will need the identifier -> arn:aws:sns:us-east-1:22549:lab-as-topic
  • as-put-notification-configuration lab-as-group –topic-arn arn:aws:sns:us-east-1:22549:lab-as-topic –notification-types autoscaling:EC2_INSTANCE_LAUNCH, autoscaling:EC2_INSTANCE_TERMINATE

Device Farm

• app testing service for iOS, Android and FireOS including PhoneGap, Xamarin, Titanium, Unity
  • Android support: Appium, Calabash, Instrumentation, UI Automator, Explorer
  • iOS: Appium, Calabash, UI Automation, XCTest (including KIF)
  • built-in fuzz test, random events
• Integrated w/ CloudTrail, a service that captures API calls made to AWS and delivers logs to S3 buckets
  • radios settings can be defined (WiFi, BT, NFC) and location + “other data” can be set

ECS (EC2 Container Service)

• shared state, optimistic concurrency
• An Amazon ECS cluster is a regional grouping of one or more container instances on which you can run task requests.
  • Each account receives a default cluster the first time you use the Amazon ECS service.
  • Clusters may contain more than one Amazon EC2 instance type.
• Task definitions specify the container information for your application
  • such as how many containers are part of your task
  • what resources they will use
  • how they are linked together
  • which host ports they will use
• the LB is an ELB instance from EC2 dashboard

CloudFormation

• create a stack of applications, from a JSON file that we can store in S3
  • can be easily managed in source control
  • collection of related AWS resources (stack), provisioning and updating them
  • QueueWatcherCount can be set to >0, so we can monitor the actions done by CloudFormation
  • only mandatory element is the Resources object, which must declare at least one resource
  • Events lower tab we can see the progress of the current actions/operations
• Resources tab in the lower part shows the queues it uses/publishes to and policies
• many resources have meaningful default values, so only if we need specific changes we should update them in the JSON file
• Some examples of possible actions from the template file
  • install yum packages
  • create files, whilst specifying the contents inside the JSON template file
    • set file permissions, owner, groups
  • starting and enabling services
  • specify IAM user profiles and access
  • any property that supports updating, can be updated on subsequent runs
  • some values can be defined as parameters (e.g. instance type t2/m1/…) and can be chosen during the stack creation/update
  • we can add a WaitCondition in order to pause the stack creation until some actions are taken/time has passed
• for Mutable changes, AWS will update the underlying resource, for Immutable changes new resources will be created and then the old ones will be deleted
• we can also add properties that were not initially specified, such as SSH key-pair, and open the SecGroup to allow for SSH connections
  • for example we can update from a single LAMP server to a auto-scaling, load balanced multi host site just by updating the CloudFormation template
• in case something goes wrong, or not all configurations are correct, the changes will be automatically rolled back
• Sample templates depending on AZ - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-sample-templates.html https://us-east-1-aws-training.s3.amazonaws.com/self-paced-lab-10/CloudFormation-WebApp-v2.template

Amazon ElasiCache

• In-memory caching web service, supports Memcached and Redis
• configure the EC2 so that is allows the cache cluster to connect to it
  • add it to the SecGroup that your instances are running

Amazon OpsWorks

• select the type of application stack you wish to use Chef 11 or Chef 12 are available
• Chef based stack deployments
• Add Stack -> Add 1 or more Layers -> Add 1 or more Instances (EC2 w/ IAM as chef-ec2) -> Add 1 or more Apps (depending on Layers)

Amazon RDS (Relational Data Service)

• setup a relational db (MySQL, Oracle, PostgreSQL)

Amazon Redshift

• peta-byte scale datawarehouse
  • working with a cluster of nodes, parallel processing and columnar storage
  • 4 node types, 2 small and 2 large, SSD or spinning disks, max storage between 5TB to 2PB

SQS (Simple Queue Service)

• in actions of a particular queue we can see or delete messages (View/Delete messages), but while we’re doing this no other applications have access to this queue
• various services (can) push messages to SQS/SNS so we can use them to register or track actions in AWS
• push notifications from SNS to SQS to be stored, later retrieved or analyzed

VPC (Virtual Private Cloud)

• can create logically isolated AWS networking section w/ servers w/ or w/o internet access, own set of IP addresses and ranges
• when using the VPC wizard to create a VPC, we can use a NAT host for NAT-ing
• when launching EC2 instances we can assign them automatically to internal IP range and VPC
• best practice - mirror environment across 2 zones and add ELB to balance between them
• permits inbound and outbound rules for filtering traffic
  • Network ACLs are very expensive from a performance point of view
  • rules can be added to restrict also security groups
• creation of route tables and subnets can be performed from the left hand menus
  • by default the subnets don’t have internet acces, safety first principle
• in SecurityGroups of EC2 you can set IPs as instance ID
• Bastion servers sit between the public internet and the private VPC, act as entrance point to the VPC or private network
  • m1.small should serve all but the largest networks, also no SSH-ing into this instance, on error just replace
• JSON
  • Mappings section; allows you to re-use the same template in all AZ, selecting only the required server types
  • VPC section: define the VPC address
    • here you need to specify the InternetGateway which will allow internet connectivity, AttachGateway
    • next specify the subnets (Type: AWS::EC2::Subnet), by referring (Ref) to the VPC we defined previously
    • routing tables (AWS::EC2::RouteTable): one for the public and one for the private networks, there can be up to 1 per subnet
      • we will specify the NAT server as router for the private section
      • next, we associate the subnets w/ the routing tables
    • next create Network ACLs (::PublicSubnetAcl and ::NetworkAcl) and then add the rules for ACLs
      • one pair ingress + egress, at least one pair needed
    • associate ACLs w/ subnets
  • create a NAT server (is a virtual EC2 appliance, we specified before)
    • to describe it we can specify REFerences to the Mappings we described in the JSON, e.g. Fn::FindInMap
    • NAT needs public (Elastic IP) to communicate w/ the Internet
    • define a NAT security group in order to specify which ports are opened or closed
    • also a private security group for communication inside the private Network
  • add an Output section
• to delete the stack we need to DISABLE termination protection in the EC2 for the NAT instance

CodeCommit

• create secure repositories for sharing code in the cloud :)
• Create triggers that will send an email or connect to a webhook through Amazon Simple Notification Service (SNS)
  • or run your custom integrations through AWS Lambda
• using AWS IAM, storage is S3 and DynamoDB
• unlimited repos and files

Advanced topics

Auto Scaling

• the policy consits of
  • a launch configuration (AMI, instance type, security group) and
  • an auto scaling group (min/max numbers, AZ, ELB)
• the triggers can be done by CloudWatch, manually or on a schedule
• all costs are per hour, so one should pull up/down instances quickly
• exact params for as-launch are:
  • AMI image ID
  • Instance type
  • Key pair generated
  • Name of Security Group
  • launch config file
  • ec2-metadata -z (zone)
    • lots of other informations can be gathered using this tool
  • extra credentials are needed for authentication of the load balancer commands
  • create AS groups w/ as-create-auto-scaling-group
  • create launch config w/ s-create-launch-config
• we can also create the same config from the AWS CLI
  • aws create-launch-configuration ....
    • if we start spot instances, we need to specify the needed spot price aws autoscaling create-launch-configuration ... --spot-price 0.03
    • –min-size A –max-size B or we can specify the number of always active instances w/ –desired-capacity
  • check if creation was successful w/ aws autoscaling describe-launch-configurations
  • auto-scaling groups can be queried w/ aws autoscaling describe-auto-scaling-groups
    • metrics can be aggregated on group level and sent to CloudWatch that way
• auto-scaling instances are launched without names
  • can be tagged with as-create-or-update-tags
• AS will be integrated with ELB, and you can get notifications from SNS (Simple Notification Service) on changes
  • see details in SNS below
• we can create a scale up/down policy
  • as-put-scaling-policy lab-scale-up-policy –auto-scaling-group lab-as-group –adjustment=1 –type ChangeInCapacity –cooldown 300
  • as-put-scaling-policy lab-scale-down-policy –auto-scaling-group lab-as-group “–adjustment=-1” –type ChangeInCapacity –cooldown 300
  • to stop to AS group, we can set the capacity to 0 (zero)
  • we can set a CloudWatch alarm in which the sum of all instances in instance hours surpasses a threshold, the auto-scaled instances are terminated
    • also an email is sent to a list of addresses :)
• we can create an Alarm in CloudWatch that triggers an event in the auto-scale group
• all scaling can be analyzed by using as-describe-scaling-activities --show-long
• all autoscaling processes can be suspended/resumed w/ as-supend/resume-...
• quick reference card for Auto Scaling - http://awsdocs.s3.amazonaws.com/AutoScaling/latest/as-qrc.pdf

Dynamic Registration

• Elastic environment - is highly utilized all the time, by using just in time provisioning
  • add instances as some metric increases (e.g. CPU), remove them when decreasing
• Scalable architecture - on-demand elastic growth w/o changing of architecture
  • Horizontal scaling: add/remove same size instances
• Bootstraping: automatically configure the instance on boot
  • by accessing already available and accessible meta-data
  • one such tool is CloudInit from Canonical/Ubuntu
  • CloudFormation: create and delete AWS (instances, AS groups, DynamoDB, etc) called stacks by using a JSON template
    • stacks can be used to create/update existing stacks