You can start your own registry using the official Docker registry s as gitlab s in a private environment. All of the above rmendations are just a few but are a short list of my go-to and I have used all 3. All are great ways of setting up private Docker registries that require authentication in order to allowmit access via docker push < after using a docker login
What are some tips, tricks and gotchas for writing AWS Lambda functions in Python?
Some things to keep in mind Remember that the setup is stateless . Between one invocation to another there is no shared state. You just process the current event. Do not assume any implicit ordering of events invocations - Since it is a distributed setup events can be processed in any order. For instance if you use records in an S3 bucket to trigger the input eventspute using Lambda and store corresponding results in a different S3 bucket the order of results won't correspond to order of input events. Beware of limits Invocation timeout - Each invocation of your function can only run for a certain time. Configure this with care. The Con object provides you the remaining time available at any given moment but if your module needs to really use this to ensurepletion then something may be wrong (at least be prepared to lose out on some events in this case). 1 Checkout the list of AWS Lambda Limits and be aware of them. 1 Throttling of concurrent events - If your application is massive beware of the number of concurrent events being processed by AWS Lambda. Some things to note here 1 The limit is not determined by just by #events per second but also by processing time per event. For instance if you have 1 events per second and a processing time of 1 seconds per event your application may have already hit the throttling limit. 2 The limit is 'account-wide'. If you have multiple functions actively deployed you must factor that in. 2 Minimize startup time in your function - This repeats with every invocation and you may save valuable time in doing so. Remember the idea is to aplish 'large number of parallel lightweight statelessputes' Authorization - If you are building a large architecture involving several cloud services set up your IAM role accounts wisely ensuring segregation of role function and access to data Error handling - Do not assume that events are just processed if sent. Handle exceptions and if you are directly invoking Lambda (instead of through S3 events) then you may need to retry. If using S3 beware of retry limits. Logging - Use logging wisely. This is what is going to help you debug in production. Versioning - Remember to version your functions (and perhaps even responses if you persist any). If you have a function that is evolving semantically and is handling an evolving data input versioning can be of real help. Optimization - Use CloudWatch to monitor your run times and identify areas to optimize. Cleanup - Clean up unused code. Yes there are code size limits too. Python specific - Know the supported version (It is Python 2.7 as of time of writing this - 13 Nov 215). Use libraries accordingly. Also checkout the Best Practices for Working with AWS Lambda Functions nAll the best.
Is the Docker Swarm mode ready for production?
We recently Dockerized the main part of our event processing pipeline using the release. It been awesome using Docker in Swarm mode and weve been really impressed with the ease of setup and use of Swarm mode. Our event pipeline processes around 22 million application errors per day approximately 15k per minute and performs a massive variety of processing tasks it a critical piece of our infrastructure. The new version of Docker Engine introduces native clustering capabilities built directly into the daemon (as opposed to running a separate set of processes). This in our opinion positions Docker as a seriouspetitor in the container orchestration space (alongside giants like Kubernetes and Mesos) and will make building and operating large distributed applications easier for everyone. Docker Swarm mode A quick look at Bugsnag error processing architecture As you probably know Bugsnag is an error monitoring platform. As part of this we have to accept process store and display a large amount of events in near-real-time containing errors for more than 4 different platforms that are written in over 16 different programming languages. That is no small feat. The event processing pipeline is a fairly large application that pulls items (events that contain errors) off of a queue performs various s of work on each item and stores it in our databases. Why we chose Docker Swarm mode We decided to modernize our setup when we saw the need to upgrade to a newer LTS version. After quickly realizing we were reinventing the wheel with a homegrown solution we decided to revisit the space of container management. While ecosystems that were already well-established (Mesos and Kubernetes) seemed like good candidates upon evaluating them it became obvious they were overlyplex for our use case to run a polling app on a fleet of nodes and would have been an overkill to deploy and operate. So we decided to try out Docker then-newly-released engine for our needs. Setting up a Swarm mode cluster came down to simply bootstrapping the manager and adding workers which is as easy as 3mands docker swarm init (on 1 manager node) docker swarm join --token manager_token (on the rest of the manager nodes) docker swarm join --token worker_token (on the workers) Personally it took me no more then 2 minutes to bring up a Swarm mode cluster from reading the docs to having it ready for work. Compare that to my Mesos experience back in the day which took about 2-3 hours of thoroughly reading through documentation for multipleplexponents (ZooKeeper the Mesos master and the Mesos slaves). After that it took me another two hours to fully bring up the cluster and get it ready to accept work. The time savings from setting up and operating a Swarm mode clusterbined with its spot-on UX and interface sets Docker up as a seriouspetitor in the space and truly enables container management at scale for the masses. Tips for running Docker in Swarm mode Weve been running the event processing pipeline in production in Swarm mode for over a month now and along the way we ran into issues that helped us uncover important lessons for working with this system. Here a list of key takeaways that are valuable to know before starting they took some time to figure out but have a significant impact. Beware of issues in the network stack Weve coded our application to make it easy to deploy instead of servicing inbound requests it pulls from a set of queues. Outbound connections in Swarm mode work fine. However when we started looking into adding services that dealt with requests on inbound connections (listening on ports) we quickly found the routing mesh feature of Swarm mode to be unreliable. The idea behind the routing mesh is to intelligently route service requests to machines that are running the service container even if the host receiving the request isn running the container. This is achieved through a gossip layer between the workers in the pool that allows them to talk to each other directing requests to the appropriate container. However when we tested it out the intelligent routing was very spotty sometimes hanging during a request and sometimes not even accepting inbound connections at all. As it stands at the writing of this post there are several open GitHub Issues related to this problem (we are tracking #25325) and themunity is working hard to fix the underlying cause. Have a solid labeling system By labels were referring to Docker native labeling system both for containers and the Docker engine itself. While the use case for container labels may be obvious labeling the Docker daemon itself has some benefits as well. For example we have automation in place that reads all tags of an s More Docker Tutorials for Beginners Introduction to Docker Container - tetratutorials s Docker Tutorial for Beginners What is Docker Container? - Part 1 s Docker Tutorial for Beginners How to Install Docker Engine Part 3 s ordered-list
How can I scale up a 10m user social network website with AWS, currently running on local Linux servers and local CDN?
What impacts needed architecture more is concurrent (simultaneous active) users and amount of interactions per user per day (data storage needs). First define the API endpoints youll need to realize your use cases. Example login password edit profile settings post connections postment browse Youll want to back your service with a database and likely a write-through cache to handle millions of concurrent users. Your back end can be microservices API (Dockerized API manages by Kubernetes) and Lambda backend processors or jus API Gateway proxy to Lambda functions (now with custom runtimes). I suggest you keep your schema simple and even if not using a GraphDB model after graph using bi-directional relations. Example object (id label owner data status sort published) relation (source_id s target_id t relation meta status published) event (id actor obj target published) media_item (id mime size label owner path filename status sort published) Use a CQRS-ES architecture separating reads from writes. You can still use an RDBMS and be ACID and write-through distributed cache for near real time updates to users. Writes leverage Kinesis or SQS as trigger for write functions to populate cache and databases. Every write logs an event likely persisted in Dynamo DB or RDS. Every event logs two relation entries (LIKES IS_LIKED_BY) and the associated objects respectively. File uploads (photos aka media_item) are persisted to S3. Reads your API layer can serve data from cache if available otherwise from db. With write-through it should have a high hit rate and you can ore-load on restarts. View logic pushed to client either in app or web using React Vue or Angular. You may consider a PWA for optimal performance. I don use AWS much anymore so there may be other rmended tools but that should get you started.