Amazon is a real Ops first/hard-core technology company. DevOps is a popular key word in IT world today, but only a few who can put it into best practice and drive core business growth.
- CI/CD tools is first step. Must have a good pipeline from development stage to production stage. That pipeline should integrate with security check, integration test etc.
- Code review and bar raiser.
- One main branch Git.
- Canary Deployment tools in order to release the new feature to a subset of users progressively.
- Alarms and Monitor tools (infra, exceptions, errors…)
- Dashboard Metrics tools (gathering users info after launch )
- Developers take life cycle of code (dev, test, production bugs fix etc.)
- High skills coworkers ( Managers, Product Managers, Developers)
- Good communication channel (Instant message tools, ticket tracking tools, knowledge sharing tools)
No real DevOps can achieve if missing any above step, in my option.
This blog is inspired and based on the Alicloud WeChat article https://mp.weixin.qq.com/s/X1MtLk71LLZnsYZBVv-hHA
This year 2019-11.11, Alibaba Tmall double 11 sales event accomplished Turnover of 268.4 billion Chinese Yuan! The peak value of orders reached 544,000 units/second, and the data processing capacity per day reached 970PB! And all the system is based on Alibaba Cloud.
According to CTO Alibaba said in this blog, there were four technology secrets behind this.
- 3rd Generation of X-Dragon Architecture. An AWS nitro similar technology.
- OceanBase and PolarDB. Those are Alibaba’s self-made Databases.
- Calculation and storage are separated. The storage is on remote and could be easily for expansion
- RDMA(RemoteDirect Memory Access) in order to access the remote storage data quickly.
We could see in order to support and boost such large data requesting case, we should improve on the physical machine side and also database sides. Reading retrieving data quickly is the key.
Verify if Bucket already exists
Create Subdirectory (folder): using
PutObjectRequest request = new PutObjectRequest(bucketName, "folder/", new File(fileName));
Remove all the files under folder:
key like :
Using Presigned Url to share object (public url)
Often we need upload a file (image) to S3 and get a tempory public URL of this object. (Be careful, you should always set private-access to this object)
We could use Presigned-url with an expiration period.